mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 11:01:16 +01:00
Nicky LeachandClaude
6ef37398c8
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors, agents, and image builds run `tsc` to check and compile the repository. > - The repository uses TypeScript 7, which is a native binary. Node heap flags do not limit it. > - The server is one large program, so one `tsc` process needs about 10 GiB. > - The operating system stops `tsc` under a 4 GiB memory limit. > - This pull request checks the server in layers, removes the server from the CLI program, and removes declarations and checks from the image emit. > - The benefit is that the full typecheck and the image `tsc` step now complete in 4 GiB. ## Linked Issues or Issue Description I found no duplicate pull request or issue. A search for `tsc`, memory, typecheck, and OOM found only runtime memory work, which is a different subject. **What happened?** `tsc` used too much memory, and the kernel stopped it with exit code 137 in resource-constrained environments. Measured peaks before this change: | Command | Peak | | --- | --- | | Server `tsc --noEmit` | About 10 GiB | | Server `tsc` emit, as in the Docker build | About 10.7 GiB | | CLI `tsc --noEmit` | About 7 GiB | | `pnpm -r typecheck`, whole process tree | About 10.2 GiB | **Expected behavior** The typecheck and the image `tsc` step complete under a 4 GiB memory limit. **Steps to reproduce** 1. Start a Linux container with `--memory=4g --memory-swap=4g --cpus=4`. 2. Install the dependencies on the base commit. 3. Run `pnpm -r typecheck`. The kernel stops the process with exit code 137. 4. Run `pnpm --filter @paperclipai/server build`. The `tsc` step needs more than 4 GiB. **Paperclip version or commit** Base commit `09162fe893`, with TypeScript 7.0.2. **Deployment mode** Not applicable. This issue applies to development, CI, and image builds. It does not change the server at runtime. ## What Changed - Add `server/scripts/typecheck-layers.mjs`. It reads the import graph on each run and runs one `tsc` process per layer. The layers are the files below the largest import cycle, the cycle, the files above the cycle, and the tests. Each layer reads the earlier layers as declarations. - Add `scripts/run-tsc.mjs`. It adds `--singleThreaded` when `PAPERCLIP_TYPECHECK_LOW_MEMORY=1` is set. The server layers and the UI typecheck use it. - Add `scripts/typecheck-low-memory.mjs`. It checks one package at a time in this mode. It has no `package.json` entry, as `.greptile/rules.md` requires. - Change the server `typecheck` script to build only the Runner TypeScript output. It does not build the Runner release binary now. `pnpm build` and `typecheck:build-gaps` still run the full Runner build and its generated-file checks. - Change the `tsc` step of the server `build` script to `scripts/run-tsc.mjs`. The `Dockerfile` sets `PAPERCLIP_TSC_EMIT_ONLY=1` for that build, so the image emits JavaScript without checks or declarations. Without this variable, `build` still does a full check for npm packages and CI. - Use a non-literal specifier for the `@paperclipai/server` import in `cli/src/commands/run.ts`. The import result already had no useful type, and the literal specifier added the full server to the CLI program. - Add a layered server typecheck step to the `pr-trusted` typecheck job, because `typecheck:build-gaps` does not run this script. - Update `AGENTS.md` and `doc/DEVELOPING.md`. ## Verification Results on Linux use a clean copy of this branch, a new `pnpm install`, Node 24, Rust 1.97.1, 4 CPUs, and a 4 GiB memory limit with no swap. Results before the merge of `master`: | Command | Result | Peak non-reclaimable memory | | --- | --- | --- | | `node scripts/typecheck-low-memory.mjs` | Exit 0, 0 OOM kills | 3,492 MiB | | `PAPERCLIP_TSC_EMIT_ONLY=1 pnpm --filter @paperclipai/server build` | Exit 0 | 3,656 MiB, from the Rust release build | | Server single-program `tsc --noEmit --singleThreaded` | Exit 137, 1 OOM kill | Reached the limit | Results on macOS on the merged head, with peak memory for the full process tree: | Command | Before | After | | --- | --- | --- | | `pnpm -r typecheck` | 10.2 GiB, exit 0 | 5.47 GiB, exit 0 | | `node scripts/typecheck-low-memory.mjs` | Not available | 3.81 GiB, exit 0 | | CLI typecheck | 7.06 GiB | 1.68 GiB, before the merge | Other checks: - Compiler tests: new tests run the real compiler on a small fixture. They prove that the layers emit declarations and pass, that a renamed export fails the layer that imports it, and that a changed type fails the test layer. - Declaration imports: a new compiler test proves that types from `.d.mts` and JSON imports stay correct in later layers. On the real server, a wrong use of `AgentFileManifest` in the cycle layer failed with `TS2353`. - Coverage: a test on the real server source asserts that each source file is in one layer and that no import goes to a later layer. The test layer contains the remaining files. `src/__tests__` stays out of the root list, as in `server/tsconfig.json`. - Error detection: I added one type error to one file in each of the four layers. The typecheck reported all four errors and exited with 1. - Cross-layer errors: I renamed an export in the lower layer. The cycle and upper layers reported the error and the typecheck exited with 1. I reverted these temporary edits. - Emit: full `tsc` and `--noCheck --declaration false --declarationMap false` gave byte-identical JavaScript and source maps for 2,249 files on the base commit. - Runtime: `scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs` passed with a bundle from the emit-only build. - CLI: the esbuild bundle keeps the dynamic import, and `node --check` passed. - Tests on the merged head: `typecheck-layers`, `prepare-runner-vendor`, `cli-invocation-safety`, the CLI `runtime` tests, and all 503 `.github/scripts/tests` tests passed. Before the merge, `release-verify-workflow` and `runner-release-crate-cache` passed. - Full `pnpm --filter @paperclipai/server build` passed before the merge. Failures during this work: - The first low-memory run failed in the UI with `TS5023: Unknown compiler option '-b'`, because the flag came before `-b`. I corrected the argument order and added a test that runs build mode with the flag. - The first rename check did not rename the export, because BSD `sed` does not support `\b`. Its exit 0 was not evidence. I ran the check again with `perl`, and the result above is from that run. - The first version of the real-compiler fixture had a `TS2774` error in its own source. The test found this error, and I corrected the fixture. - CI failed `server-package-build-script.test.ts` on `1abc5e5a43`, because that test reads the steps of the server `build` script, and I had moved them to `build:prepare` and `build:assets`. My first search for readers of this script did not include `server/src`. I removed those scripts and `build:image`, restored one `build` script, and kept the test unchanged. - `GOMEMLIMIT` did not help enough, and I removed it from the plan. On these steps, `3GiB` saved at most about 300 MiB. With default checkers, the cycle layer was stopped with and without it, and the lower layer went from 7 seconds to 151 seconds. Not run or not proven: - I did not run the full `pnpm test:run`, a root `pnpm build`, or a full `docker build`. I expect CI to run the test and build gates. - I did not measure the Docker build steps other than the server build. - I did not run the 4 GiB checks on x86-64. - I did not measure the merged head in the 4 GiB container. The macOS result on the merged head is 3.81 GiB, and the merge added one source file to the lower layer. ## Risks - The margin is small. In the 4 GiB container, the cycle layer peaks about 600 MiB below the limit, and the Rust release build in the image build peaks about 440 MiB below it. Growth of the largest import cycle uses this margin first. - `node scripts/typecheck-low-memory.mjs` applies only to `pnpm -r` packages. These commands still check the server as one program and need more than 4 GiB: `pnpm build`, `test:e2e:runner:typecheck`, `test:lifecycle-baseline:typecheck`, and the `tests/speko-native` project. - A local flow that used the Runner binary from a previous typecheck must now run `pnpm --filter @paperclipai/paperclip-runner build`. - The image does not include server declarations now. I found no consumer in this repository. A downstream image that reads `server/dist/*.d.ts` must change. - An image build does not find type errors now. A typecheck in CI must find them before the image build. - The import scan uses a regular expression. A false edge can only make a layer larger. A missed edge causes a `TS6307` failure, so it cannot skip a file. - The new CI step adds about one minute to the typecheck job. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with reasoning, tool use, and code execution. The runtime does not show the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>