mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:48:15 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the trusted PR workflow, and the `Typecheck + Release Registry` job is one of its required checks > - That job takes about 9.2 minutes on the 4 vCPU / 16 GB fleet runner (run 43344818387), against a target of 7 minutes or less for every job > - The job runs six steps one after another, and the first step type-checks the whole server program a second time, at about 13.5 GiB, which makes the runner swap > - Typecheck results are the check that authors most want back quickly, so this job is on the critical path of most pull requests > - This pull request removes the duplicate server type check and runs the independent checks in parallel through one small script, inside the same memory budget > - The benefit is about 39% less wall time for the job's checks on the same machine, with peak memory down from about 14 GiB (plus swap) to about 8.5 GiB ## Linked Issues or Issue Description **What existing behavior does this improve?** The `Typecheck + Release Registry` job in `.github/workflows/pr-trusted.yml`. It takes about 9.2 minutes. The target for every job in the workflow is 7 minutes or less. **Subsystem affected** CI (`.github/workflows/pr-trusted.yml`) and the typecheck scripts under `scripts/` and `server/scripts/`. **Current behavior** After about 75 s of setup, the job runs six steps in sequence on a 4 vCPU runner, about 470 s in all (run 43344818387): `typecheck:build-gaps` 121 s, test programs 85 s, server layers 128 s, test projects 38 s, CLI guidance test 5 s, release registry tests 95 s. Most of these steps use one thread. The `typecheck:build-gaps` step builds the server with a full type check of the whole server program. The server layer step then checks the same files again, and the `Build` job checks them a third time. That full program peaks at about 13.5 GiB, so the job peaks at about 14 GiB of 16 GiB and uses swap. **Proposed behavior** The job runs `node scripts/ci-typecheck-lane.mjs`. The script builds the server without type checks (`PAPERCLIP_TSC_EMIT_ONLY=1`, as the Docker image does) and still checks the runtime assets. It runs the steps that write shared files first, one at a time. Then it runs the remaining checks in parallel, one per CPU, with the server declaration layers first, because the test layers and test projects read them. The checks that are not on that path run at a lower CPU priority. **Reason and benefit** On the same runner, the job's checks take 61% of the time and about 60% of the memory (see Verification). The job should take about 6 minutes on the fleet instead of 9.2. **Breaking changes** None. The same files are type-checked with the same options. Local commands do not change. `pnpm run typecheck:build-gaps` still works as before. ## What Changed - `scripts/ci-typecheck-lane.mjs` (new): runs the job's checks. It writes shared outputs first, in sequence: links, plugin SDK, server build without type checks, the runtime asset check, the sandbox provider installs, and the Runner typecheck (it rebuilds the eval kernel, which other checks read). Then it runs these checks in parallel: the server declaration layers, each server test layer, each test project, the test programs, the other build-gap workspaces, the CLI guidance test and the release registry tests. A passing check prints its output in a collapsed log group. A failing check prints its full output between `=== FAILED` and `=== end of` lines, with an error annotation. The script ends with a table of each check's start, duration and result, and exits non-zero if any check fails. - `server/scripts/typecheck-layers.mjs`: new `--layer <name>` option that checks one layer against the configs and declarations of an earlier `--declarations-only` run, and an exported `listLayers`. - `scripts/typecheck-test-programs.mjs`: new `--prepare-only` and `--skip-prepare` options, so the installs happen before the parallel checks start. - `scripts/run-typecheck-build-gaps.mjs`: new `--list-packages` option, so the script still finds the build-gap workspaces and still fails when one has no `typecheck` script. - `.github/workflows/pr-trusted.yml`: the `typecheck_release_registry` job runs the script instead of the six steps. No other job changes. - Tests: new `scripts/__tests__/ci-typecheck-lane.test.mjs` (added to `test:release-registry`), new cases in `server/scripts/typecheck-layers.test.mjs` and `scripts/typecheck-test-programs.test.mjs`, and the workflow wiring assertions in `.github/scripts/tests/ci-plan.test.mjs` and `scripts/__tests__/runner-release-crate-cache.test.mjs`. - `tests/runner-e2e/source-sdk-public-inputs.json`: repinned, because `package.json` changed. - `doc/DEVELOPING.md`: new "Typecheck lane in CI" section, and the new options. ## Verification This PR's own CI ran the old job definition, because PR runs use the workflow from master. So I measured the commands on a stand-in machine: a temporary workflow (`.github/workflows/typecheck-lane-bench-temp.yml`, now deleted) on GitHub-hosted `ubuntu-latest` (4 vCPU Intel Xeon 6975P-C, 15.7 GiB, no secrets, `contents: read`). It ran in runs [43444356017](https://github.com/paperclipai/paperclip/actions/runs/43444356017), [43445393958](https://github.com/paperclipai/paperclip/actions/runs/43445393958), [43446786105](https://github.com/paperclipai/paperclip/actions/runs/43446786105), [43448707837](https://github.com/paperclipai/paperclip/actions/runs/43448707837) and [43450786493](https://github.com/paperclipai/paperclip/actions/runs/43450786493). Each step ran under `/usr/bin/time -v`, and a sampler read `/proc/meminfo` every 0.25 s. **Old steps, one at a time (3 runs, run 43445393958):** | Step | Wall s | CPU s | Peak RSS of one process | System memory peak | | --- | --- | --- | --- | --- | | `typecheck:build-gaps` | 128–155 | 376–400 | 13.2–13.6 GiB | 14.2–14.7 GiB | | ...of which the full server `tsc` | 40–51 | 146–164 | 12.8–13.8 GiB | 13.7–14.7 GiB, swap up to 738 MiB | | ...replaced by the emit-only server build | 6–10 | 20–22 | 2.9 GiB | 3.9 GiB | | Test programs | 67–87 | 147–170 | 1.9 GiB | 2.9 GiB | | Server layers | 102–127 | 150–172 | 4.0–4.2 GiB | 5.4–5.6 GiB | | Test projects | 31–37 | 49–59 | 2.1–2.2 GiB | 3.1–3.2 GiB | | CLI guidance test | 2–4 | 2–3 | 0.2 GiB | 1.6 GiB | | Release registry tests | 82–95 | 159–183 | 1.7 GiB | 3.0 GiB | CPU seconds are about twice the wall seconds for the `tsc` steps even on one checker thread. So each step uses about two of the four vCPUs. **Old sequence and new lane on the same runner (4 runs, run 43450786493).** Each job copied one installed checkout, then ran both, in alternating order: | Run | Old steps, wall s | New lane, wall s | Ratio | Old memory peak | New memory peak | | --- | --- | --- | --- | --- | --- | | 1 | 503 | 314 | 0.62 | 13.98 GiB | 8.33 GiB | | 2 | 647 | 388 | 0.60 | 14.04 GiB, swap 637 MiB | 8.13 GiB | | 3 | 627 | 385 | 0.61 | 13.98 GiB | 8.63 GiB | | 4 | 382 | 237 | 0.62 | 14.10 GiB, swap 1.1 GiB | 7.99 GiB | Hosted runners differ in speed by up to 1.7 times, so only the same-runner ratio is meaningful. Both arms include a cold compile of the Runner release crate (45–80 s) that the fleet restores from cache. The fleet ran the old steps in 470 s; at the measured ratio, the new lane should take about 290 s, and the job about 6 minutes with setup. The new lane peaks at about 55% of memory, with no swap. Local checks (on a loaded Mac, for correctness only): - `node --test scripts/__tests__/ci-typecheck-lane.test.mjs` — 8 pass. - `node --test .github/scripts/tests/*.test.mjs` — 807 pass. - `node --test scripts/__tests__/e2e-shard.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/run-typecheck-build-gaps.test.mjs` — pass. - `pnpm exec vitest run scripts/typecheck-layers.test.mjs` in `server/` — the new cases pass. ## Risks - This changes a required job that every PR depends on, and its own CI cannot prove it. Please watch the first `Typecheck + Release Registry` runs on master after merge: the wall time (expect about 6 minutes), the summary table at the end of the step, and any `terminated by` message. - A type error now shows in that check's `=== FAILED` block, not in a step of its own. Passing checks are folded, so the failure is the only expanded block. - When the server declaration layers fail, the test layers and test projects are reported as skipped. Before, the test layers still ran after a source error. Fix the source errors first; the next run checks the tests. - The Honeycomb CI time board and any step-name queries see one step, `Typecheck, CLI guidance safety and release registry tests`, instead of six. The script prints the duration of each check. - The release registry tests run beside the type checks at normal priority. Their slowest test took 46–78 s here, against a 180 s timeout, and 114 s on the slowest stand-in runner. - Low risk for memory: the new lane peaked at 8.0–8.6 GiB of 15.7 GiB, far below the old 14 GiB. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use (shell, GitHub CLI, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
268 lines
11 KiB
JavaScript
268 lines
11 KiB
JavaScript
#!/usr/bin/env node
|
|
// Typechecks the test files that no build or package typecheck reads.
|
|
//
|
|
// Most package tsconfigs exclude tests so that `tsc` does not emit them, and
|
|
// Vitest strips types, so a type error in those tests fails nothing. Each program
|
|
// in PROGRAMS reads them without emitting. Sandbox providers are outside the
|
|
// workspace, so the root install has none of their dependencies. They install from
|
|
// typecheck-sandbox-providers/pnpm-lock.yaml here, because their SDK types are part
|
|
// of what a test checks. A provider with its own lockfile (LOCKED_PROVIDERS) installs
|
|
// from that lockfile instead, so each lockfile has one owner and one Dependabot entry.
|
|
//
|
|
// The script then fails if a tracked test file is in no program. tsc decides that,
|
|
// not a glob: `include` silently drops foo.test.tsx when foo.test.ts exists.
|
|
//
|
|
// Usage: node scripts/typecheck-test-programs.mjs [--low-memory] [tsconfig ...]
|
|
// Naming a tsconfig runs only that program and skips the coverage check.
|
|
|
|
import { spawnSync } from "node:child_process";
|
|
import { existsSync } from "node:fs";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
import { linkExcludedPlugins } from "./link-plugin-dev-sdk.mjs";
|
|
import { LOW_MEMORY_VARIABLE, runTsc } from "./run-tsc.mjs";
|
|
import { PROJECTS } from "./typecheck-test-projects.mjs";
|
|
|
|
export const PROVIDER_WORKSPACE = "scripts/typecheck-sandbox-providers";
|
|
|
|
export const SANDBOX_PROVIDERS = [
|
|
"packages/plugins/sandbox-providers/boxd",
|
|
"packages/plugins/sandbox-providers/cloudflare",
|
|
"packages/plugins/sandbox-providers/cloudflare/bridge-template",
|
|
"packages/plugins/sandbox-providers/createos",
|
|
"packages/plugins/sandbox-providers/daytona",
|
|
"packages/plugins/sandbox-providers/e2b",
|
|
"packages/plugins/sandbox-providers/exe-dev",
|
|
"packages/plugins/sandbox-providers/kubernetes",
|
|
"packages/plugins/sandbox-providers/modal",
|
|
"packages/plugins/sandbox-providers/novita",
|
|
];
|
|
|
|
// Providers that commit their own pnpm-lock.yaml. The release build and the Daytona
|
|
// end-to-end job install it frozen, so the shared workspace leaves them out.
|
|
export const LOCKED_PROVIDERS = ["packages/plugins/sandbox-providers/daytona"];
|
|
|
|
export const PROGRAMS = [
|
|
"packages/db/tsconfig.test.json",
|
|
"packages/kv-demo-mcp-server/tsconfig.test.json",
|
|
"packages/paperclip-runner/tsconfig.browser.test.json",
|
|
"packages/paperclip-runner/tsconfig.surfaces.test.json",
|
|
"packages/paperclip-runner/tsconfig.test.json",
|
|
"packages/plugins/examples/plugin-orchestration-smoke-example/tsconfig.test.json",
|
|
"packages/plugins/plugin-workspace-diff/tsconfig.test.json",
|
|
"packages/plugins/sdk/tsconfig.test.json",
|
|
...SANDBOX_PROVIDERS.map((directory) => `${directory}/tsconfig.test.json`),
|
|
"tests/runner-acceptance/tsconfig.json",
|
|
"tests/tsconfig.playwright.json",
|
|
];
|
|
|
|
// server/scripts/typecheck-layers.mjs reads this tree. Everything else must be in a program.
|
|
export const CHECKED_ELSEWHERE = ["server/src/"];
|
|
|
|
const PREPARE_STEPS = [
|
|
["--filter", "@paperclipai/plugin-sdk", "ensure-build-deps"],
|
|
["--filter", "@paperclipai/paperclip-runner", "run", "ensure:eval-build-deps"],
|
|
];
|
|
// Replacing a provider's standalone node_modules would otherwise ask for confirmation.
|
|
const INSTALL_ARGS = ["install", "--frozen-lockfile", "--ignore-scripts", "--prefer-offline", "--config.confirmModulesPurge=false"];
|
|
const LOCKED_INSTALL_ARGS = ["install", "--ignore-workspace", ...INSTALL_ARGS.slice(1)];
|
|
// Contributors do not commit provider lockfiles: the lockfile refresh workflow regenerates them after
|
|
// merge, as it does the root pnpm-lock.yaml. Until then a changed manifest leaves the lockfile stale,
|
|
// so resolve it in place, like the root install in CI.
|
|
export const RESOLVE_ARGS = ["install", "--resolution-only", "--ignore-scripts", "--no-frozen-lockfile"];
|
|
const LOCKED_RESOLVE_ARGS = ["install", "--ignore-workspace", ...RESOLVE_ARGS.slice(1)];
|
|
|
|
export function findUncheckedTests({ trackedFiles, checkedFiles, checkedElsewhere = CHECKED_ELSEWHERE }) {
|
|
const checked = new Set(checkedFiles);
|
|
return trackedFiles.filter(
|
|
(file) =>
|
|
/\.(test|spec)\.tsx?$/.test(file) &&
|
|
!checked.has(file) &&
|
|
!checkedElsewhere.some((prefix) => file.startsWith(prefix)),
|
|
);
|
|
}
|
|
|
|
function runPnpm(args, { cwd, env }) {
|
|
const result = spawnSync("pnpm", args, {
|
|
cwd,
|
|
env,
|
|
stdio: "inherit",
|
|
// A bare `pnpm` on Windows is pnpm.cmd, which needs the shell to resolve.
|
|
shell: process.platform === "win32",
|
|
});
|
|
if (result.error) throw result.error;
|
|
return result.status ?? 1;
|
|
}
|
|
|
|
function listProgramFiles(config, { cwd, env }) {
|
|
const tsc = path.join(cwd, "node_modules", "typescript", "bin", "tsc");
|
|
const result = spawnSync(process.execPath, [tsc, "-p", config, "--listFilesOnly"], {
|
|
cwd,
|
|
env,
|
|
encoding: "utf8",
|
|
maxBuffer: 1 << 28,
|
|
});
|
|
if (result.error) throw result.error;
|
|
return {
|
|
status: result.status ?? 1,
|
|
output: result.stdout,
|
|
files: result.stdout
|
|
.split("\n")
|
|
.filter(Boolean)
|
|
.map((file) => path.relative(cwd, file).split(path.sep).join("/")),
|
|
};
|
|
}
|
|
|
|
function listWorkspaceConfigs(root) {
|
|
const result = spawnSync("pnpm", ["ls", "-r", "--depth", "-1", "--json"], {
|
|
cwd: root,
|
|
encoding: "utf8",
|
|
maxBuffer: 1 << 26,
|
|
shell: process.platform === "win32",
|
|
});
|
|
if (result.error) throw result.error;
|
|
if (result.status !== 0) throw new Error("Unable to list pnpm workspace packages.");
|
|
return JSON.parse(result.stdout)
|
|
.map((workspace) => path.relative(root, path.join(workspace.path, "tsconfig.json")).split(path.sep).join("/"))
|
|
.filter((config) => config !== "server/tsconfig.json" && existsSync(path.join(root, config)));
|
|
}
|
|
|
|
function listTrackedFiles(root) {
|
|
const result = spawnSync("git", ["ls-files", "-z"], { cwd: root, encoding: "utf8", maxBuffer: 1 << 28 });
|
|
if (result.error) throw result.error;
|
|
if (result.status !== 0) throw new Error("Unable to list tracked files.");
|
|
return result.stdout.split("\0").filter(Boolean);
|
|
}
|
|
|
|
/** Returns the test files in no program, or null if a program could not be listed. */
|
|
export function checkCoverage({
|
|
root,
|
|
env = process.env,
|
|
programs = PROGRAMS,
|
|
workspaceConfigs = listWorkspaceConfigs,
|
|
trackedFiles = listTrackedFiles,
|
|
listFiles = listProgramFiles,
|
|
logError = console.error,
|
|
}) {
|
|
const checkedFiles = [];
|
|
// `pnpm build` and `typecheck:build-gaps` check each workspace's tsconfig.json.
|
|
// scripts/typecheck-test-projects.mjs checks these same files through tsconfig.layers.json.
|
|
const projectConfigs = PROJECTS.map((project) => `${project}/tsconfig.json`);
|
|
for (const config of [...workspaceConfigs(root), ...projectConfigs, ...programs]) {
|
|
const listing = listFiles(config, { cwd: root, env });
|
|
if (listing.status !== 0) {
|
|
logError(`typecheck:test-programs: could not list the files of ${config}\n${listing.output ?? ""}`);
|
|
return null;
|
|
}
|
|
checkedFiles.push(...listing.files);
|
|
}
|
|
return findUncheckedTests({ trackedFiles: trackedFiles(root), checkedFiles });
|
|
}
|
|
|
|
export function prepareTestPrograms({
|
|
root,
|
|
env = process.env,
|
|
pnpm = runPnpm,
|
|
linkPlugins = linkExcludedPlugins,
|
|
log = console.log,
|
|
logError = console.error,
|
|
}) {
|
|
for (const args of PREPARE_STEPS) {
|
|
const status = pnpm(args, { cwd: root, env });
|
|
if (status !== 0) return status;
|
|
}
|
|
const install = (directory, frozenArgs, resolveArgs) => {
|
|
const cwd = path.join(root, directory);
|
|
let status = pnpm(frozenArgs, { cwd, env });
|
|
if (status !== 0) {
|
|
log(
|
|
`typecheck:test-programs: ${directory}/pnpm-lock.yaml does not match its manifests. ` +
|
|
`Resolving it in place. Do not commit it: the lockfile refresh workflow updates it after merge.`,
|
|
);
|
|
status = pnpm(resolveArgs, { cwd, env });
|
|
if (status === 0) status = pnpm(frozenArgs, { cwd, env });
|
|
}
|
|
if (status !== 0) logError(`typecheck:test-programs: could not install the sandbox providers in ${directory}`);
|
|
return status;
|
|
};
|
|
const installed = install(PROVIDER_WORKSPACE, INSTALL_ARGS, RESOLVE_ARGS);
|
|
if (installed !== 0) return installed;
|
|
for (const provider of LOCKED_PROVIDERS) {
|
|
const status = install(provider, LOCKED_INSTALL_ARGS, LOCKED_RESOLVE_ARGS);
|
|
if (status !== 0) return status;
|
|
}
|
|
// The installs remove the SDK link from each provider's node_modules.
|
|
linkPlugins();
|
|
return 0;
|
|
}
|
|
|
|
export function typecheckTestPrograms({
|
|
root,
|
|
env = process.env,
|
|
programs = PROGRAMS,
|
|
coverage = true,
|
|
prepare = true,
|
|
pnpm = runPnpm,
|
|
linkPlugins = linkExcludedPlugins,
|
|
run = runTsc,
|
|
checkFiles = checkCoverage,
|
|
log = console.log,
|
|
logError = console.error,
|
|
}) {
|
|
if (prepare) {
|
|
const prepared = prepareTestPrograms({ root, env, pnpm, linkPlugins, log, logError });
|
|
if (prepared !== 0) return prepared;
|
|
}
|
|
|
|
let exitCode = 0;
|
|
for (const config of programs) {
|
|
log(`typecheck:test-programs: ${config}`);
|
|
const { status, signal } = run(["-p", config], { cwd: root, env });
|
|
if (signal) return status || 1;
|
|
if (status !== 0) exitCode = status;
|
|
}
|
|
if (!coverage) return exitCode;
|
|
|
|
const unchecked = checkFiles({ root, env, logError });
|
|
if (unchecked === null) return exitCode || 1;
|
|
if (unchecked.length > 0) {
|
|
logError(
|
|
`typecheck:test-programs: ${unchecked.length} test files are in no typechecked program. ` +
|
|
`Add them to a program in scripts/typecheck-test-programs.mjs. A foo.test.tsx beside a foo.test.ts needs a new name.\n` +
|
|
unchecked.map((file) => ` ${file}`).join("\n"),
|
|
);
|
|
return exitCode || 1;
|
|
}
|
|
return exitCode;
|
|
}
|
|
|
|
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
|
const args = process.argv.slice(2);
|
|
const flags = args.filter((arg) => arg.startsWith("--"));
|
|
const programs = args.filter((arg) => !arg.startsWith("--"));
|
|
const unknownFlags = flags.filter((flag) => !["--low-memory", "--prepare-only", "--skip-prepare"].includes(flag));
|
|
const unknown = programs.filter((program) => !PROGRAMS.includes(program));
|
|
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
|
const env = flags.includes("--low-memory") ? { ...process.env, [LOW_MEMORY_VARIABLE]: "1" } : process.env;
|
|
if (unknownFlags.length > 0) {
|
|
console.error(`Unknown option: ${unknownFlags.join(", ")}`);
|
|
process.exitCode = 2;
|
|
} else if (flags.includes("--prepare-only") && (flags.includes("--skip-prepare") || programs.length > 0)) {
|
|
console.error("--prepare-only takes no programs and cannot be combined with --skip-prepare");
|
|
process.exitCode = 2;
|
|
} else if (unknown.length > 0) {
|
|
console.error(`Unknown program: ${unknown.join(", ")}. Known programs:\n ${PROGRAMS.join("\n ")}`);
|
|
process.exitCode = 2;
|
|
} else if (flags.includes("--prepare-only")) {
|
|
process.exitCode = prepareTestPrograms({ root, env });
|
|
} else {
|
|
process.exitCode = typecheckTestPrograms({
|
|
root,
|
|
env,
|
|
programs: programs.length > 0 ? programs : PROGRAMS,
|
|
coverage: programs.length === 0,
|
|
prepare: !flags.includes("--skip-prepare"),
|
|
});
|
|
}
|
|
}
|