Files
PaperClipAI/scripts/typecheck-test-programs.mjs
T
DottaandClaude Opus 5.5 64ec273845 perf(ci): run the typecheck lane's checks in parallel and drop its duplicate server type check (#21404)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the trusted PR workflow, and the `Typecheck
+ Release Registry` job is one of its required checks
> - That job takes about 9.2 minutes on the 4 vCPU / 16 GB fleet runner
(run 43344818387), against a target of 7 minutes or less for every job
> - The job runs six steps one after another, and the first step
type-checks the whole server program a second time, at about 13.5 GiB,
which makes the runner swap
> - Typecheck results are the check that authors most want back quickly,
so this job is on the critical path of most pull requests
> - This pull request removes the duplicate server type check and runs
the independent checks in parallel through one small script, inside the
same memory budget
> - The benefit is about 39% less wall time for the job's checks on the
same machine, with peak memory down from about 14 GiB (plus swap) to
about 8.5 GiB

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The `Typecheck + Release Registry` job in
`.github/workflows/pr-trusted.yml`. It takes about 9.2 minutes. The
target for every job in the workflow is 7 minutes or less.

**Subsystem affected**

CI (`.github/workflows/pr-trusted.yml`) and the typecheck scripts under
`scripts/` and `server/scripts/`.

**Current behavior**

After about 75 s of setup, the job runs six steps in sequence on a 4
vCPU runner, about 470 s in all (run 43344818387):
`typecheck:build-gaps` 121 s, test programs 85 s, server layers 128 s,
test projects 38 s, CLI guidance test 5 s, release registry tests 95 s.
Most of these steps use one thread. The `typecheck:build-gaps` step
builds the server with a full type check of the whole server program.
The server layer step then checks the same files again, and the `Build`
job checks them a third time. That full program peaks at about 13.5 GiB,
so the job peaks at about 14 GiB of 16 GiB and uses swap.

**Proposed behavior**

The job runs `node scripts/ci-typecheck-lane.mjs`. The script builds the
server without type checks (`PAPERCLIP_TSC_EMIT_ONLY=1`, as the Docker
image does) and still checks the runtime assets. It runs the steps that
write shared files first, one at a time. Then it runs the remaining
checks in parallel, one per CPU, with the server declaration layers
first, because the test layers and test projects read them. The checks
that are not on that path run at a lower CPU priority.

**Reason and benefit**

On the same runner, the job's checks take 61% of the time and about 60%
of the memory (see Verification). The job should take about 6 minutes on
the fleet instead of 9.2.

**Breaking changes**

None. The same files are type-checked with the same options. Local
commands do not change. `pnpm run typecheck:build-gaps` still works as
before.

## What Changed

- `scripts/ci-typecheck-lane.mjs` (new): runs the job's checks. It
writes shared outputs first, in sequence: links, plugin SDK, server
build without type checks, the runtime asset check, the sandbox provider
installs, and the Runner typecheck (it rebuilds the eval kernel, which
other checks read). Then it runs these checks in parallel: the server
declaration layers, each server test layer, each test project, the test
programs, the other build-gap workspaces, the CLI guidance test and the
release registry tests. A passing check prints its output in a collapsed
log group. A failing check prints its full output between `=== FAILED`
and `=== end of` lines, with an error annotation. The script ends with a
table of each check's start, duration and result, and exits non-zero if
any check fails.
- `server/scripts/typecheck-layers.mjs`: new `--layer <name>` option
that checks one layer against the configs and declarations of an earlier
`--declarations-only` run, and an exported `listLayers`.
- `scripts/typecheck-test-programs.mjs`: new `--prepare-only` and
`--skip-prepare` options, so the installs happen before the parallel
checks start.
- `scripts/run-typecheck-build-gaps.mjs`: new `--list-packages` option,
so the script still finds the build-gap workspaces and still fails when
one has no `typecheck` script.
- `.github/workflows/pr-trusted.yml`: the `typecheck_release_registry`
job runs the script instead of the six steps. No other job changes.
- Tests: new `scripts/__tests__/ci-typecheck-lane.test.mjs` (added to
`test:release-registry`), new cases in
`server/scripts/typecheck-layers.test.mjs` and
`scripts/typecheck-test-programs.test.mjs`, and the workflow wiring
assertions in `.github/scripts/tests/ci-plan.test.mjs` and
`scripts/__tests__/runner-release-crate-cache.test.mjs`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: repinned, because
`package.json` changed.
- `doc/DEVELOPING.md`: new "Typecheck lane in CI" section, and the new
options.

## Verification

This PR's own CI ran the old job definition, because PR runs use the
workflow from master. So I measured the commands on a stand-in machine:
a temporary workflow (`.github/workflows/typecheck-lane-bench-temp.yml`,
now deleted) on GitHub-hosted `ubuntu-latest` (4 vCPU Intel Xeon
6975P-C, 15.7 GiB, no secrets, `contents: read`). It ran in runs
[43444356017](https://github.com/paperclipai/paperclip/actions/runs/43444356017),
[43445393958](https://github.com/paperclipai/paperclip/actions/runs/43445393958),
[43446786105](https://github.com/paperclipai/paperclip/actions/runs/43446786105),
[43448707837](https://github.com/paperclipai/paperclip/actions/runs/43448707837)
and
[43450786493](https://github.com/paperclipai/paperclip/actions/runs/43450786493).
Each step ran under `/usr/bin/time -v`, and a sampler read
`/proc/meminfo` every 0.25 s.

**Old steps, one at a time (3 runs, run 43445393958):**

| Step | Wall s | CPU s | Peak RSS of one process | System memory peak |
| --- | --- | --- | --- | --- |
| `typecheck:build-gaps` | 128–155 | 376–400 | 13.2–13.6 GiB | 14.2–14.7
GiB |
| ...of which the full server `tsc` | 40–51 | 146–164 | 12.8–13.8 GiB |
13.7–14.7 GiB, swap up to 738 MiB |
| ...replaced by the emit-only server build | 6–10 | 20–22 | 2.9 GiB |
3.9 GiB |
| Test programs | 67–87 | 147–170 | 1.9 GiB | 2.9 GiB |
| Server layers | 102–127 | 150–172 | 4.0–4.2 GiB | 5.4–5.6 GiB |
| Test projects | 31–37 | 49–59 | 2.1–2.2 GiB | 3.1–3.2 GiB |
| CLI guidance test | 2–4 | 2–3 | 0.2 GiB | 1.6 GiB |
| Release registry tests | 82–95 | 159–183 | 1.7 GiB | 3.0 GiB |

CPU seconds are about twice the wall seconds for the `tsc` steps even on
one checker thread. So each step uses about two of the four vCPUs.

**Old sequence and new lane on the same runner (4 runs, run
43450786493).** Each job copied one installed checkout, then ran both,
in alternating order:

| Run | Old steps, wall s | New lane, wall s | Ratio | Old memory peak |
New memory peak |
| --- | --- | --- | --- | --- | --- |
| 1 | 503 | 314 | 0.62 | 13.98 GiB | 8.33 GiB |
| 2 | 647 | 388 | 0.60 | 14.04 GiB, swap 637 MiB | 8.13 GiB |
| 3 | 627 | 385 | 0.61 | 13.98 GiB | 8.63 GiB |
| 4 | 382 | 237 | 0.62 | 14.10 GiB, swap 1.1 GiB | 7.99 GiB |

Hosted runners differ in speed by up to 1.7 times, so only the
same-runner ratio is meaningful. Both arms include a cold compile of the
Runner release crate (45–80 s) that the fleet restores from cache. The
fleet ran the old steps in 470 s; at the measured ratio, the new lane
should take about 290 s, and the job about 6 minutes with setup. The new
lane peaks at about 55% of memory, with no swap.

Local checks (on a loaded Mac, for correctness only):

- `node --test scripts/__tests__/ci-typecheck-lane.test.mjs` — 8 pass.
- `node --test .github/scripts/tests/*.test.mjs` — 807 pass.
- `node --test scripts/__tests__/e2e-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/runner-release-crate-cache.test.mjs
scripts/__tests__/run-typecheck-build-gaps.test.mjs` — pass.
- `pnpm exec vitest run scripts/typecheck-layers.test.mjs` in `server/`
— the new cases pass.

## Risks

- This changes a required job that every PR depends on, and its own CI
cannot prove it. Please watch the first `Typecheck + Release Registry`
runs on master after merge: the wall time (expect about 6 minutes), the
summary table at the end of the step, and any `terminated by` message.
- A type error now shows in that check's `=== FAILED` block, not in a
step of its own. Passing checks are folded, so the failure is the only
expanded block.
- When the server declaration layers fail, the test layers and test
projects are reported as skipped. Before, the test layers still ran
after a source error. Fix the source errors first; the next run checks
the tests.
- The Honeycomb CI time board and any step-name queries see one step,
`Typecheck, CLI guidance safety and release registry tests`, instead of
six. The script prints the duration of each check.
- The release registry tests run beside the type checks at normal
priority. Their slowest test took 46–78 s here, against a 180 s timeout,
and 114 s on the slowest stand-in runner.
- Low risk for memory: the new lane peaked at 8.0–8.6 GiB of 15.7 GiB,
far below the old 14 GiB.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool
use (shell, GitHub CLI, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:06:01 -06:00

268 lines
11 KiB
JavaScript

#!/usr/bin/env node
// Typechecks the test files that no build or package typecheck reads.
//
// Most package tsconfigs exclude tests so that `tsc` does not emit them, and
// Vitest strips types, so a type error in those tests fails nothing. Each program
// in PROGRAMS reads them without emitting. Sandbox providers are outside the
// workspace, so the root install has none of their dependencies. They install from
// typecheck-sandbox-providers/pnpm-lock.yaml here, because their SDK types are part
// of what a test checks. A provider with its own lockfile (LOCKED_PROVIDERS) installs
// from that lockfile instead, so each lockfile has one owner and one Dependabot entry.
//
// The script then fails if a tracked test file is in no program. tsc decides that,
// not a glob: `include` silently drops foo.test.tsx when foo.test.ts exists.
//
// Usage: node scripts/typecheck-test-programs.mjs [--low-memory] [tsconfig ...]
// Naming a tsconfig runs only that program and skips the coverage check.
import { spawnSync } from "node:child_process";
import { existsSync } from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { linkExcludedPlugins } from "./link-plugin-dev-sdk.mjs";
import { LOW_MEMORY_VARIABLE, runTsc } from "./run-tsc.mjs";
import { PROJECTS } from "./typecheck-test-projects.mjs";
export const PROVIDER_WORKSPACE = "scripts/typecheck-sandbox-providers";
export const SANDBOX_PROVIDERS = [
"packages/plugins/sandbox-providers/boxd",
"packages/plugins/sandbox-providers/cloudflare",
"packages/plugins/sandbox-providers/cloudflare/bridge-template",
"packages/plugins/sandbox-providers/createos",
"packages/plugins/sandbox-providers/daytona",
"packages/plugins/sandbox-providers/e2b",
"packages/plugins/sandbox-providers/exe-dev",
"packages/plugins/sandbox-providers/kubernetes",
"packages/plugins/sandbox-providers/modal",
"packages/plugins/sandbox-providers/novita",
];
// Providers that commit their own pnpm-lock.yaml. The release build and the Daytona
// end-to-end job install it frozen, so the shared workspace leaves them out.
export const LOCKED_PROVIDERS = ["packages/plugins/sandbox-providers/daytona"];
export const PROGRAMS = [
"packages/db/tsconfig.test.json",
"packages/kv-demo-mcp-server/tsconfig.test.json",
"packages/paperclip-runner/tsconfig.browser.test.json",
"packages/paperclip-runner/tsconfig.surfaces.test.json",
"packages/paperclip-runner/tsconfig.test.json",
"packages/plugins/examples/plugin-orchestration-smoke-example/tsconfig.test.json",
"packages/plugins/plugin-workspace-diff/tsconfig.test.json",
"packages/plugins/sdk/tsconfig.test.json",
...SANDBOX_PROVIDERS.map((directory) => `${directory}/tsconfig.test.json`),
"tests/runner-acceptance/tsconfig.json",
"tests/tsconfig.playwright.json",
];
// server/scripts/typecheck-layers.mjs reads this tree. Everything else must be in a program.
export const CHECKED_ELSEWHERE = ["server/src/"];
const PREPARE_STEPS = [
["--filter", "@paperclipai/plugin-sdk", "ensure-build-deps"],
["--filter", "@paperclipai/paperclip-runner", "run", "ensure:eval-build-deps"],
];
// Replacing a provider's standalone node_modules would otherwise ask for confirmation.
const INSTALL_ARGS = ["install", "--frozen-lockfile", "--ignore-scripts", "--prefer-offline", "--config.confirmModulesPurge=false"];
const LOCKED_INSTALL_ARGS = ["install", "--ignore-workspace", ...INSTALL_ARGS.slice(1)];
// Contributors do not commit provider lockfiles: the lockfile refresh workflow regenerates them after
// merge, as it does the root pnpm-lock.yaml. Until then a changed manifest leaves the lockfile stale,
// so resolve it in place, like the root install in CI.
export const RESOLVE_ARGS = ["install", "--resolution-only", "--ignore-scripts", "--no-frozen-lockfile"];
const LOCKED_RESOLVE_ARGS = ["install", "--ignore-workspace", ...RESOLVE_ARGS.slice(1)];
export function findUncheckedTests({ trackedFiles, checkedFiles, checkedElsewhere = CHECKED_ELSEWHERE }) {
const checked = new Set(checkedFiles);
return trackedFiles.filter(
(file) =>
/\.(test|spec)\.tsx?$/.test(file) &&
!checked.has(file) &&
!checkedElsewhere.some((prefix) => file.startsWith(prefix)),
);
}
function runPnpm(args, { cwd, env }) {
const result = spawnSync("pnpm", args, {
cwd,
env,
stdio: "inherit",
// A bare `pnpm` on Windows is pnpm.cmd, which needs the shell to resolve.
shell: process.platform === "win32",
});
if (result.error) throw result.error;
return result.status ?? 1;
}
function listProgramFiles(config, { cwd, env }) {
const tsc = path.join(cwd, "node_modules", "typescript", "bin", "tsc");
const result = spawnSync(process.execPath, [tsc, "-p", config, "--listFilesOnly"], {
cwd,
env,
encoding: "utf8",
maxBuffer: 1 << 28,
});
if (result.error) throw result.error;
return {
status: result.status ?? 1,
output: result.stdout,
files: result.stdout
.split("\n")
.filter(Boolean)
.map((file) => path.relative(cwd, file).split(path.sep).join("/")),
};
}
function listWorkspaceConfigs(root) {
const result = spawnSync("pnpm", ["ls", "-r", "--depth", "-1", "--json"], {
cwd: root,
encoding: "utf8",
maxBuffer: 1 << 26,
shell: process.platform === "win32",
});
if (result.error) throw result.error;
if (result.status !== 0) throw new Error("Unable to list pnpm workspace packages.");
return JSON.parse(result.stdout)
.map((workspace) => path.relative(root, path.join(workspace.path, "tsconfig.json")).split(path.sep).join("/"))
.filter((config) => config !== "server/tsconfig.json" && existsSync(path.join(root, config)));
}
function listTrackedFiles(root) {
const result = spawnSync("git", ["ls-files", "-z"], { cwd: root, encoding: "utf8", maxBuffer: 1 << 28 });
if (result.error) throw result.error;
if (result.status !== 0) throw new Error("Unable to list tracked files.");
return result.stdout.split("\0").filter(Boolean);
}
/** Returns the test files in no program, or null if a program could not be listed. */
export function checkCoverage({
root,
env = process.env,
programs = PROGRAMS,
workspaceConfigs = listWorkspaceConfigs,
trackedFiles = listTrackedFiles,
listFiles = listProgramFiles,
logError = console.error,
}) {
const checkedFiles = [];
// `pnpm build` and `typecheck:build-gaps` check each workspace's tsconfig.json.
// scripts/typecheck-test-projects.mjs checks these same files through tsconfig.layers.json.
const projectConfigs = PROJECTS.map((project) => `${project}/tsconfig.json`);
for (const config of [...workspaceConfigs(root), ...projectConfigs, ...programs]) {
const listing = listFiles(config, { cwd: root, env });
if (listing.status !== 0) {
logError(`typecheck:test-programs: could not list the files of ${config}\n${listing.output ?? ""}`);
return null;
}
checkedFiles.push(...listing.files);
}
return findUncheckedTests({ trackedFiles: trackedFiles(root), checkedFiles });
}
export function prepareTestPrograms({
root,
env = process.env,
pnpm = runPnpm,
linkPlugins = linkExcludedPlugins,
log = console.log,
logError = console.error,
}) {
for (const args of PREPARE_STEPS) {
const status = pnpm(args, { cwd: root, env });
if (status !== 0) return status;
}
const install = (directory, frozenArgs, resolveArgs) => {
const cwd = path.join(root, directory);
let status = pnpm(frozenArgs, { cwd, env });
if (status !== 0) {
log(
`typecheck:test-programs: ${directory}/pnpm-lock.yaml does not match its manifests. ` +
`Resolving it in place. Do not commit it: the lockfile refresh workflow updates it after merge.`,
);
status = pnpm(resolveArgs, { cwd, env });
if (status === 0) status = pnpm(frozenArgs, { cwd, env });
}
if (status !== 0) logError(`typecheck:test-programs: could not install the sandbox providers in ${directory}`);
return status;
};
const installed = install(PROVIDER_WORKSPACE, INSTALL_ARGS, RESOLVE_ARGS);
if (installed !== 0) return installed;
for (const provider of LOCKED_PROVIDERS) {
const status = install(provider, LOCKED_INSTALL_ARGS, LOCKED_RESOLVE_ARGS);
if (status !== 0) return status;
}
// The installs remove the SDK link from each provider's node_modules.
linkPlugins();
return 0;
}
export function typecheckTestPrograms({
root,
env = process.env,
programs = PROGRAMS,
coverage = true,
prepare = true,
pnpm = runPnpm,
linkPlugins = linkExcludedPlugins,
run = runTsc,
checkFiles = checkCoverage,
log = console.log,
logError = console.error,
}) {
if (prepare) {
const prepared = prepareTestPrograms({ root, env, pnpm, linkPlugins, log, logError });
if (prepared !== 0) return prepared;
}
let exitCode = 0;
for (const config of programs) {
log(`typecheck:test-programs: ${config}`);
const { status, signal } = run(["-p", config], { cwd: root, env });
if (signal) return status || 1;
if (status !== 0) exitCode = status;
}
if (!coverage) return exitCode;
const unchecked = checkFiles({ root, env, logError });
if (unchecked === null) return exitCode || 1;
if (unchecked.length > 0) {
logError(
`typecheck:test-programs: ${unchecked.length} test files are in no typechecked program. ` +
`Add them to a program in scripts/typecheck-test-programs.mjs. A foo.test.tsx beside a foo.test.ts needs a new name.\n` +
unchecked.map((file) => ` ${file}`).join("\n"),
);
return exitCode || 1;
}
return exitCode;
}
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
const args = process.argv.slice(2);
const flags = args.filter((arg) => arg.startsWith("--"));
const programs = args.filter((arg) => !arg.startsWith("--"));
const unknownFlags = flags.filter((flag) => !["--low-memory", "--prepare-only", "--skip-prepare"].includes(flag));
const unknown = programs.filter((program) => !PROGRAMS.includes(program));
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
const env = flags.includes("--low-memory") ? { ...process.env, [LOW_MEMORY_VARIABLE]: "1" } : process.env;
if (unknownFlags.length > 0) {
console.error(`Unknown option: ${unknownFlags.join(", ")}`);
process.exitCode = 2;
} else if (flags.includes("--prepare-only") && (flags.includes("--skip-prepare") || programs.length > 0)) {
console.error("--prepare-only takes no programs and cannot be combined with --skip-prepare");
process.exitCode = 2;
} else if (unknown.length > 0) {
console.error(`Unknown program: ${unknown.join(", ")}. Known programs:\n ${PROGRAMS.join("\n ")}`);
process.exitCode = 2;
} else if (flags.includes("--prepare-only")) {
process.exitCode = prepareTestPrograms({ root, env });
} else {
process.exitCode = typecheckTestPrograms({
root,
env,
programs: programs.length > 0 ? programs : PROGRAMS,
coverage: programs.length === 0,
prepare: !flags.includes("--skip-prepare"),
});
}
}