Files
PaperClipAI/scripts
DottaandClaude Opus 5.5 64ec273845 perf(ci): run the typecheck lane's checks in parallel and drop its duplicate server type check (#21404)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the trusted PR workflow, and the `Typecheck
+ Release Registry` job is one of its required checks
> - That job takes about 9.2 minutes on the 4 vCPU / 16 GB fleet runner
(run 43344818387), against a target of 7 minutes or less for every job
> - The job runs six steps one after another, and the first step
type-checks the whole server program a second time, at about 13.5 GiB,
which makes the runner swap
> - Typecheck results are the check that authors most want back quickly,
so this job is on the critical path of most pull requests
> - This pull request removes the duplicate server type check and runs
the independent checks in parallel through one small script, inside the
same memory budget
> - The benefit is about 39% less wall time for the job's checks on the
same machine, with peak memory down from about 14 GiB (plus swap) to
about 8.5 GiB

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The `Typecheck + Release Registry` job in
`.github/workflows/pr-trusted.yml`. It takes about 9.2 minutes. The
target for every job in the workflow is 7 minutes or less.

**Subsystem affected**

CI (`.github/workflows/pr-trusted.yml`) and the typecheck scripts under
`scripts/` and `server/scripts/`.

**Current behavior**

After about 75 s of setup, the job runs six steps in sequence on a 4
vCPU runner, about 470 s in all (run 43344818387):
`typecheck:build-gaps` 121 s, test programs 85 s, server layers 128 s,
test projects 38 s, CLI guidance test 5 s, release registry tests 95 s.
Most of these steps use one thread. The `typecheck:build-gaps` step
builds the server with a full type check of the whole server program.
The server layer step then checks the same files again, and the `Build`
job checks them a third time. That full program peaks at about 13.5 GiB,
so the job peaks at about 14 GiB of 16 GiB and uses swap.

**Proposed behavior**

The job runs `node scripts/ci-typecheck-lane.mjs`. The script builds the
server without type checks (`PAPERCLIP_TSC_EMIT_ONLY=1`, as the Docker
image does) and still checks the runtime assets. It runs the steps that
write shared files first, one at a time. Then it runs the remaining
checks in parallel, one per CPU, with the server declaration layers
first, because the test layers and test projects read them. The checks
that are not on that path run at a lower CPU priority.

**Reason and benefit**

On the same runner, the job's checks take 61% of the time and about 60%
of the memory (see Verification). The job should take about 6 minutes on
the fleet instead of 9.2.

**Breaking changes**

None. The same files are type-checked with the same options. Local
commands do not change. `pnpm run typecheck:build-gaps` still works as
before.

## What Changed

- `scripts/ci-typecheck-lane.mjs` (new): runs the job's checks. It
writes shared outputs first, in sequence: links, plugin SDK, server
build without type checks, the runtime asset check, the sandbox provider
installs, and the Runner typecheck (it rebuilds the eval kernel, which
other checks read). Then it runs these checks in parallel: the server
declaration layers, each server test layer, each test project, the test
programs, the other build-gap workspaces, the CLI guidance test and the
release registry tests. A passing check prints its output in a collapsed
log group. A failing check prints its full output between `=== FAILED`
and `=== end of` lines, with an error annotation. The script ends with a
table of each check's start, duration and result, and exits non-zero if
any check fails.
- `server/scripts/typecheck-layers.mjs`: new `--layer <name>` option
that checks one layer against the configs and declarations of an earlier
`--declarations-only` run, and an exported `listLayers`.
- `scripts/typecheck-test-programs.mjs`: new `--prepare-only` and
`--skip-prepare` options, so the installs happen before the parallel
checks start.
- `scripts/run-typecheck-build-gaps.mjs`: new `--list-packages` option,
so the script still finds the build-gap workspaces and still fails when
one has no `typecheck` script.
- `.github/workflows/pr-trusted.yml`: the `typecheck_release_registry`
job runs the script instead of the six steps. No other job changes.
- Tests: new `scripts/__tests__/ci-typecheck-lane.test.mjs` (added to
`test:release-registry`), new cases in
`server/scripts/typecheck-layers.test.mjs` and
`scripts/typecheck-test-programs.test.mjs`, and the workflow wiring
assertions in `.github/scripts/tests/ci-plan.test.mjs` and
`scripts/__tests__/runner-release-crate-cache.test.mjs`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: repinned, because
`package.json` changed.
- `doc/DEVELOPING.md`: new "Typecheck lane in CI" section, and the new
options.

## Verification

This PR's own CI ran the old job definition, because PR runs use the
workflow from master. So I measured the commands on a stand-in machine:
a temporary workflow (`.github/workflows/typecheck-lane-bench-temp.yml`,
now deleted) on GitHub-hosted `ubuntu-latest` (4 vCPU Intel Xeon
6975P-C, 15.7 GiB, no secrets, `contents: read`). It ran in runs
[43444356017](https://github.com/paperclipai/paperclip/actions/runs/43444356017),
[43445393958](https://github.com/paperclipai/paperclip/actions/runs/43445393958),
[43446786105](https://github.com/paperclipai/paperclip/actions/runs/43446786105),
[43448707837](https://github.com/paperclipai/paperclip/actions/runs/43448707837)
and
[43450786493](https://github.com/paperclipai/paperclip/actions/runs/43450786493).
Each step ran under `/usr/bin/time -v`, and a sampler read
`/proc/meminfo` every 0.25 s.

**Old steps, one at a time (3 runs, run 43445393958):**

| Step | Wall s | CPU s | Peak RSS of one process | System memory peak |
| --- | --- | --- | --- | --- |
| `typecheck:build-gaps` | 128–155 | 376–400 | 13.2–13.6 GiB | 14.2–14.7
GiB |
| ...of which the full server `tsc` | 40–51 | 146–164 | 12.8–13.8 GiB |
13.7–14.7 GiB, swap up to 738 MiB |
| ...replaced by the emit-only server build | 6–10 | 20–22 | 2.9 GiB |
3.9 GiB |
| Test programs | 67–87 | 147–170 | 1.9 GiB | 2.9 GiB |
| Server layers | 102–127 | 150–172 | 4.0–4.2 GiB | 5.4–5.6 GiB |
| Test projects | 31–37 | 49–59 | 2.1–2.2 GiB | 3.1–3.2 GiB |
| CLI guidance test | 2–4 | 2–3 | 0.2 GiB | 1.6 GiB |
| Release registry tests | 82–95 | 159–183 | 1.7 GiB | 3.0 GiB |

CPU seconds are about twice the wall seconds for the `tsc` steps even on
one checker thread. So each step uses about two of the four vCPUs.

**Old sequence and new lane on the same runner (4 runs, run
43450786493).** Each job copied one installed checkout, then ran both,
in alternating order:

| Run | Old steps, wall s | New lane, wall s | Ratio | Old memory peak |
New memory peak |
| --- | --- | --- | --- | --- | --- |
| 1 | 503 | 314 | 0.62 | 13.98 GiB | 8.33 GiB |
| 2 | 647 | 388 | 0.60 | 14.04 GiB, swap 637 MiB | 8.13 GiB |
| 3 | 627 | 385 | 0.61 | 13.98 GiB | 8.63 GiB |
| 4 | 382 | 237 | 0.62 | 14.10 GiB, swap 1.1 GiB | 7.99 GiB |

Hosted runners differ in speed by up to 1.7 times, so only the
same-runner ratio is meaningful. Both arms include a cold compile of the
Runner release crate (45–80 s) that the fleet restores from cache. The
fleet ran the old steps in 470 s; at the measured ratio, the new lane
should take about 290 s, and the job about 6 minutes with setup. The new
lane peaks at about 55% of memory, with no swap.

Local checks (on a loaded Mac, for correctness only):

- `node --test scripts/__tests__/ci-typecheck-lane.test.mjs` — 8 pass.
- `node --test .github/scripts/tests/*.test.mjs` — 807 pass.
- `node --test scripts/__tests__/e2e-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/runner-release-crate-cache.test.mjs
scripts/__tests__/run-typecheck-build-gaps.test.mjs` — pass.
- `pnpm exec vitest run scripts/typecheck-layers.test.mjs` in `server/`
— the new cases pass.

## Risks

- This changes a required job that every PR depends on, and its own CI
cannot prove it. Please watch the first `Typecheck + Release Registry`
runs on master after merge: the wall time (expect about 6 minutes), the
summary table at the end of the step, and any `terminated by` message.
- A type error now shows in that check's `=== FAILED` block, not in a
step of its own. Passing checks are folded, so the failure is the only
expanded block.
- When the server declaration layers fail, the test layers and test
projects are reported as skipped. Before, the test layers still ran
after a source error. Fix the source errors first; the next run checks
the tests.
- The Honeycomb CI time board and any step-name queries see one step,
`Typecheck, CLI guidance safety and release registry tests`, instead of
six. The script prints the duration of each check.
- The release registry tests run beside the type checks at normal
priority. Their slowest test took 46–78 s here, against a 180 s timeout,
and 114 s on the slowest stand-in runner.
- Low risk for memory: the new lane peaked at 8.0–8.6 GiB of 15.7 GiB,
far below the old 14 GiB.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool
use (shell, GitHub CLI, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:06:01 -06:00
..
2026-04-09 06:14:12 -05:00