2 Commits

Include renames

Author	SHA1	Message	Date
Nicky LeachandClaude Opus 5.5	8ab89f41e4	fix(build): typecheck test projects in 4 GiB (#18738) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors and agents typecheck this repository, often on machines with limited memory > - paperclipai/paperclip#18737 made `pnpm -r typecheck` fit in 4 GiB because the server now checks in layers > - The test projects outside `pnpm -r` still import server source through relative paths, so each one checks most of the server as one program > - These projects peaked between 6.9 GiB and 14.9 GiB, so they fail on a 4 GiB machine > - This pull request makes these projects read the server layers as declarations, and adds a CI step that keeps them that way > - The benefit is that every TypeScript project in the repository can now be checked in 4 GiB, and a later change cannot silently undo this ## Linked Issues or Issue Description Refs paperclipai/paperclip#18737, which is merged. This pull request builds on the server layers that it added. No issue exists for this gap. **What happened?** `tsc -p tests/runner-e2e/tsconfig.json` peaked at 14.9 GiB at default settings. `tests/speko-native` peaked at 12.2 GiB, `server/src/__tests__/helpers/maintenance` at 9.6 GiB, and `tests/lifecycle-baseline` at 6.9 GiB. On one thread, `tests/runner-e2e` still peaked at 7.7 GiB. **Expected behavior** Each project passes with 4 GiB of memory. **Steps to reproduce** Run `tsc -p tests/runner-e2e/tsconfig.json` in a Linux container with `--memory=4g --memory-swap=4g`. The kernel stops the process. ## What Changed - Added `tsconfig.layers.json` to the four projects. Each file extends the project's `tsconfig.json` and references the generated server declarations. - Added `scripts/typecheck-test-projects.mjs`. It builds the server declarations and then checks the named projects, or all four. It takes `--low-memory`. - Changed `test:e2e:runner:typecheck` and `test:lifecycle-baseline:typecheck` to call that script. `docker-runner-check.yml` calls the first script by name, so that workflow needs no edit. - Added `--declarations-only` to `server/scripts/typecheck-layers.mjs`. The script now also generates `tsconfig.declarations.json`, so hand-written configs do not name individual layers. - Exported `runPnpm` from `scripts/typecheck-low-memory.mjs`, so the new script can use it. - Added a step to the `Typecheck + Release Registry` job. It runs straight after the server typecheck and reuses its declarations through `--reuse-declarations`. - The script now fails if a project loads server source, even when the types pass. Test support files, declaration files and JSON files do not count. This check does not enforce a memory limit. - Added a unit test that fails if a new project under `tests/` is neither in `PROJECTS` nor named as a single program. `tests/runner-acceptance` is the one named project. - Added `server/src/types/express.d.ts` to `tests/lifecycle-baseline/tsconfig.json`. That plain config failed alone with 11 TS2339 errors, and it now passes. - Exported `isTestFile` from `server/scripts/typecheck-layers.mjs`, so the source check uses the same rule as the layers. - Turned off `declaration` in the layered configs. These projects emit nothing, and declaration checks gave a false TS2883 error when `tests/runner-e2e` read the server as declarations. - Added tests, one of which runs the real compiler. Updated `doc/DEVELOPING.md` and `tests/lifecycle-baseline/README.md`. The references are not in each project's `tsconfig.json` because Vitest reads that file. A test showed that Vitest fails when a referenced config does not exist yet. The `tsconfig.json` files do not change, so editors and Vitest see the same configs as before. No new `package.json` scripts were added, as `.greptile/rules.md` requires. ## Verification I tested commit `9763880fab`, which is based on `1c9ca5829f`. I copied the commit into a named Docker volume and ran a fresh install. Each row ran in its own Linux arm64 container with `--memory=4g --memory-swap=4g` and 8 CPUs. The first row started with no build output. All rows used `--low-memory`. CI does not set a memory limit. I used the limit only for this measurement. | Project | Errors | Peak (cgroup) | Max process RSS | OOM kills | Time | | --- | --- | --- | --- | --- | --- | | `tests/runner-e2e` | 0 | 3.34 GiB | 3.06 GiB | 0 | 57 s | | `tests/speko-native` | 0 | 3.60 GiB | 3.27 GiB | 0 | 51 s | | `tests/lifecycle-baseline` | 0 | 3.43 GiB | 3.15 GiB | 0 | 50 s | | `server/src/__tests__/helpers/maintenance` | 0 | 3.39 GiB | 3.13 GiB | 0 | 52 s | `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm test:e2e:runner:typecheck` and `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm test:lifecycle-baseline:typecheck` also passed in the same limit, with peaks of 3.55 GiB and 3.52 GiB. I also ran the CI sequence in one container with the same limit: `pnpm --filter @paperclipai/server typecheck`, then `node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations`. Both passed, with a peak of 3.46 GiB. The new step took 16 s. The server layer build sets the peak in each row. Against the declarations, the projects peaked between 0.9 GiB and 2.2 GiB at default settings on macOS. Server source files that each project loads, without test support, declaration or JSON files: | Project | Plain `tsconfig.json` | `tsconfig.layers.json` | | --- | --- | --- | | `tests/runner-e2e` | 1,345 | 0 | | `tests/speko-native` | 1,647 | 0 | | `tests/lifecycle-baseline` | 1,343 | 0 | | `server/src/__tests__/helpers/maintenance` | 1,716 | 0 | I counted these on `3f0f31b2f9`. The source check in the script confirmed 0 for all four projects on this commit. **Limits of this proof:** - Without `--low-memory`, the 4 GiB run does not pass. The kernel stopped the layer build for `tests/runner-e2e` on `3f0f31b2f9`, and the script exited with status 1. I did not repeat this run on the final commit. The script does not turn on low-memory mode by itself. - The Linux runs used arm64, not x64. - The new CI step has not run in CI. `pr-trusted.yml` runs from `@master`, so the step first runs after this pull request merges. Check its first run. **Other checks on this commit:** - `server/scripts/typecheck-layers.test.mjs`: 38 tests pass. One test uses the real compiler to show that the source check tells a layered config from a config with no references. - With the references removed from `tests/speko-native/tsconfig.layers.json`, the script exits with status 1 and reports 1,644 server source files. - With the declarations removed, `--reuse-declarations` exits with status 1 and reports TS6053. - A type error in a test project makes the script exit with status 1. - `tests/lifecycle-baseline/tsconfig.json` passes with 0 errors. - `tests/lifecycle-baseline` Vitest suite: 533 tests pass. - Workflow tests that read `pr-trusted.yml` and CI routing: 114 tests pass. **Earlier checks:** - `server/src/__tests__/server-package-build-script.test.ts` passed 10 tests on `3f0f31b2f9`. This test failed in `ci / General tests (server (9/12))` on an older head. It tested a build script split that the final #18737 removed. - A changed signature in `server/src/services/run-liveness.ts` failed `tests/lifecycle-baseline/authority.test.ts` through the rebuilt declarations, before the rebases. When I restored the signature, the check passed. **Not run locally:** - `pnpm test:run` and `pnpm build`. CI covers these. **Failures that this pull request does not cause:** `pnpm test:e2e:runner:unit` has 13 failures on this commit, in `cursor-cloud-replay-fault`, `first-task`, `operating-receipts`, `pi-controls-evidence` and `workflow-security`. The same 13 tests fail on `master` at `6ef37398c8`. A comparison of the test names shows no differences. PR CI does not run this suite. ## Risks - Low risk to runtime behavior. No shipped code changes. - **Slower check:** `pnpm test:e2e:runner:typecheck` now builds the server layers on each run. Before the rebase, it took 30 s on macOS at default settings, about the same as the old single program for that project. On this commit it took 57 s in low-memory mode in the 4 GiB container. The new CI step reuses the declarations, so it took 16 s. - **Narrower check:** Server source is now checked only by the server's own layers, not again under each project's compiler options. The projects also skip declaration-portability checks. These projects emit nothing, so those checks had no output to protect. - **Unchanged path:** `tests/runner-e2e/native-completion-checks.mjs` still runs `tsc -p tests/runner-e2e/tsconfig.json` as one program. It needs about 15 GiB at default settings. I did not change it because that file and its inputs are part of a fingerprinted source contract, and no workflow calls it. - **Layer growth:** The new checks do not measure memory. If the layers or the import cycle grow, the checks still pass and the peak rises. This is intentional. Crossing 4 GiB does not fail a build. - **Project list:** The CI step checks only the projects in `PROJECTS`. The unit test covers new projects under `tests/`, but not a new project in another directory. - **Reused declarations:** `--reuse-declarations` trusts the declarations on disk. CI uses it straight after the server typecheck on the same commit. If that typecheck fails, the job stops first. Stale declarations can pass locally, so `doc/DEVELOPING.md` warns against this. - **Job dependency:** A type error in a test project now fails `Typecheck + Release Registry`, and so `ci / verify`. - **Not covered:** The new step runs no tests. `node scripts/typecheck-low-memory.mjs` and the Docker image build also have no new guard. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 11:09:12 -08:00
Nicky LeachandClaude	6ef37398c8	fix(build): fit typecheck and image build in 4 GiB (#18737) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors, agents, and image builds run `tsc` to check and compile the repository. > - The repository uses TypeScript 7, which is a native binary. Node heap flags do not limit it. > - The server is one large program, so one `tsc` process needs about 10 GiB. > - The operating system stops `tsc` under a 4 GiB memory limit. > - This pull request checks the server in layers, removes the server from the CLI program, and removes declarations and checks from the image emit. > - The benefit is that the full typecheck and the image `tsc` step now complete in 4 GiB. ## Linked Issues or Issue Description I found no duplicate pull request or issue. A search for `tsc`, memory, typecheck, and OOM found only runtime memory work, which is a different subject. **What happened?** `tsc` used too much memory, and the kernel stopped it with exit code 137 in resource-constrained environments. Measured peaks before this change: | Command | Peak | | --- | --- | | Server `tsc --noEmit` | About 10 GiB | | Server `tsc` emit, as in the Docker build | About 10.7 GiB | | CLI `tsc --noEmit` | About 7 GiB | | `pnpm -r typecheck`, whole process tree | About 10.2 GiB | **Expected behavior** The typecheck and the image `tsc` step complete under a 4 GiB memory limit. **Steps to reproduce** 1. Start a Linux container with `--memory=4g --memory-swap=4g --cpus=4`. 2. Install the dependencies on the base commit. 3. Run `pnpm -r typecheck`. The kernel stops the process with exit code 137. 4. Run `pnpm --filter @paperclipai/server build`. The `tsc` step needs more than 4 GiB. **Paperclip version or commit** Base commit `09162fe893`, with TypeScript 7.0.2. **Deployment mode** Not applicable. This issue applies to development, CI, and image builds. It does not change the server at runtime. ## What Changed - Add `server/scripts/typecheck-layers.mjs`. It reads the import graph on each run and runs one `tsc` process per layer. The layers are the files below the largest import cycle, the cycle, the files above the cycle, and the tests. Each layer reads the earlier layers as declarations. - Add `scripts/run-tsc.mjs`. It adds `--singleThreaded` when `PAPERCLIP_TYPECHECK_LOW_MEMORY=1` is set. The server layers and the UI typecheck use it. - Add `scripts/typecheck-low-memory.mjs`. It checks one package at a time in this mode. It has no `package.json` entry, as `.greptile/rules.md` requires. - Change the server `typecheck` script to build only the Runner TypeScript output. It does not build the Runner release binary now. `pnpm build` and `typecheck:build-gaps` still run the full Runner build and its generated-file checks. - Change the `tsc` step of the server `build` script to `scripts/run-tsc.mjs`. The `Dockerfile` sets `PAPERCLIP_TSC_EMIT_ONLY=1` for that build, so the image emits JavaScript without checks or declarations. Without this variable, `build` still does a full check for npm packages and CI. - Use a non-literal specifier for the `@paperclipai/server` import in `cli/src/commands/run.ts`. The import result already had no useful type, and the literal specifier added the full server to the CLI program. - Add a layered server typecheck step to the `pr-trusted` typecheck job, because `typecheck:build-gaps` does not run this script. - Update `AGENTS.md` and `doc/DEVELOPING.md`. ## Verification Results on Linux use a clean copy of this branch, a new `pnpm install`, Node 24, Rust 1.97.1, 4 CPUs, and a 4 GiB memory limit with no swap. Results before the merge of `master`: | Command | Result | Peak non-reclaimable memory | | --- | --- | --- | | `node scripts/typecheck-low-memory.mjs` | Exit 0, 0 OOM kills | 3,492 MiB | | `PAPERCLIP_TSC_EMIT_ONLY=1 pnpm --filter @paperclipai/server build` | Exit 0 | 3,656 MiB, from the Rust release build | | Server single-program `tsc --noEmit --singleThreaded` | Exit 137, 1 OOM kill | Reached the limit | Results on macOS on the merged head, with peak memory for the full process tree: | Command | Before | After | | --- | --- | --- | | `pnpm -r typecheck` | 10.2 GiB, exit 0 | 5.47 GiB, exit 0 | | `node scripts/typecheck-low-memory.mjs` | Not available | 3.81 GiB, exit 0 | | CLI typecheck | 7.06 GiB | 1.68 GiB, before the merge | Other checks: - Compiler tests: new tests run the real compiler on a small fixture. They prove that the layers emit declarations and pass, that a renamed export fails the layer that imports it, and that a changed type fails the test layer. - Declaration imports: a new compiler test proves that types from `.d.mts` and JSON imports stay correct in later layers. On the real server, a wrong use of `AgentFileManifest` in the cycle layer failed with `TS2353`. - Coverage: a test on the real server source asserts that each source file is in one layer and that no import goes to a later layer. The test layer contains the remaining files. `src/__tests__` stays out of the root list, as in `server/tsconfig.json`. - Error detection: I added one type error to one file in each of the four layers. The typecheck reported all four errors and exited with 1. - Cross-layer errors: I renamed an export in the lower layer. The cycle and upper layers reported the error and the typecheck exited with 1. I reverted these temporary edits. - Emit: full `tsc` and `--noCheck --declaration false --declarationMap false` gave byte-identical JavaScript and source maps for 2,249 files on the base commit. - Runtime: `scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs` passed with a bundle from the emit-only build. - CLI: the esbuild bundle keeps the dynamic import, and `node --check` passed. - Tests on the merged head: `typecheck-layers`, `prepare-runner-vendor`, `cli-invocation-safety`, the CLI `runtime` tests, and all 503 `.github/scripts/tests` tests passed. Before the merge, `release-verify-workflow` and `runner-release-crate-cache` passed. - Full `pnpm --filter @paperclipai/server build` passed before the merge. Failures during this work: - The first low-memory run failed in the UI with `TS5023: Unknown compiler option '-b'`, because the flag came before `-b`. I corrected the argument order and added a test that runs build mode with the flag. - The first rename check did not rename the export, because BSD `sed` does not support `\b`. Its exit 0 was not evidence. I ran the check again with `perl`, and the result above is from that run. - The first version of the real-compiler fixture had a `TS2774` error in its own source. The test found this error, and I corrected the fixture. - CI failed `server-package-build-script.test.ts` on `1abc5e5a43`, because that test reads the steps of the server `build` script, and I had moved them to `build:prepare` and `build:assets`. My first search for readers of this script did not include `server/src`. I removed those scripts and `build:image`, restored one `build` script, and kept the test unchanged. - `GOMEMLIMIT` did not help enough, and I removed it from the plan. On these steps, `3GiB` saved at most about 300 MiB. With default checkers, the cycle layer was stopped with and without it, and the lower layer went from 7 seconds to 151 seconds. Not run or not proven: - I did not run the full `pnpm test:run`, a root `pnpm build`, or a full `docker build`. I expect CI to run the test and build gates. - I did not measure the Docker build steps other than the server build. - I did not run the 4 GiB checks on x86-64. - I did not measure the merged head in the 4 GiB container. The macOS result on the merged head is 3.81 GiB, and the merge added one source file to the lower layer. ## Risks - The margin is small. In the 4 GiB container, the cycle layer peaks about 600 MiB below the limit, and the Rust release build in the image build peaks about 440 MiB below it. Growth of the largest import cycle uses this margin first. - `node scripts/typecheck-low-memory.mjs` applies only to `pnpm -r` packages. These commands still check the server as one program and need more than 4 GiB: `pnpm build`, `test:e2e:runner:typecheck`, `test:lifecycle-baseline:typecheck`, and the `tests/speko-native` project. - A local flow that used the Runner binary from a previous typecheck must now run `pnpm --filter @paperclipai/paperclip-runner build`. - The image does not include server declarations now. I found no consumer in this repository. A downstream image that reads `server/dist/*.d.ts` must change. - An image build does not find type errors now. A typecheck in CI must find them before the image build. - The import scan uses a regular expression. A false edge can only make a layer larger. A missed edge causes a `TS6307` failure, so it cannot skip a file. - The new CI step adds about one minute to the typecheck job. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with reasoning, tool use, and code execution. The runtime does not show the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-03 08:53:35 -08:00