mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 11:01:16 +01:00
fix(build): fit typecheck and image build in 4 GiB (#18737)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors, agents, and image builds run `tsc` to check and compile the repository. > - The repository uses TypeScript 7, which is a native binary. Node heap flags do not limit it. > - The server is one large program, so one `tsc` process needs about 10 GiB. > - The operating system stops `tsc` under a 4 GiB memory limit. > - This pull request checks the server in layers, removes the server from the CLI program, and removes declarations and checks from the image emit. > - The benefit is that the full typecheck and the image `tsc` step now complete in 4 GiB. ## Linked Issues or Issue Description I found no duplicate pull request or issue. A search for `tsc`, memory, typecheck, and OOM found only runtime memory work, which is a different subject. **What happened?** `tsc` used too much memory, and the kernel stopped it with exit code 137 in resource-constrained environments. Measured peaks before this change: | Command | Peak | | --- | --- | | Server `tsc --noEmit` | About 10 GiB | | Server `tsc` emit, as in the Docker build | About 10.7 GiB | | CLI `tsc --noEmit` | About 7 GiB | | `pnpm -r typecheck`, whole process tree | About 10.2 GiB | **Expected behavior** The typecheck and the image `tsc` step complete under a 4 GiB memory limit. **Steps to reproduce** 1. Start a Linux container with `--memory=4g --memory-swap=4g --cpus=4`. 2. Install the dependencies on the base commit. 3. Run `pnpm -r typecheck`. The kernel stops the process with exit code 137. 4. Run `pnpm --filter @paperclipai/server build`. The `tsc` step needs more than 4 GiB. **Paperclip version or commit** Base commit `09162fe893`, with TypeScript 7.0.2. **Deployment mode** Not applicable. This issue applies to development, CI, and image builds. It does not change the server at runtime. ## What Changed - Add `server/scripts/typecheck-layers.mjs`. It reads the import graph on each run and runs one `tsc` process per layer. The layers are the files below the largest import cycle, the cycle, the files above the cycle, and the tests. Each layer reads the earlier layers as declarations. - Add `scripts/run-tsc.mjs`. It adds `--singleThreaded` when `PAPERCLIP_TYPECHECK_LOW_MEMORY=1` is set. The server layers and the UI typecheck use it. - Add `scripts/typecheck-low-memory.mjs`. It checks one package at a time in this mode. It has no `package.json` entry, as `.greptile/rules.md` requires. - Change the server `typecheck` script to build only the Runner TypeScript output. It does not build the Runner release binary now. `pnpm build` and `typecheck:build-gaps` still run the full Runner build and its generated-file checks. - Change the `tsc` step of the server `build` script to `scripts/run-tsc.mjs`. The `Dockerfile` sets `PAPERCLIP_TSC_EMIT_ONLY=1` for that build, so the image emits JavaScript without checks or declarations. Without this variable, `build` still does a full check for npm packages and CI. - Use a non-literal specifier for the `@paperclipai/server` import in `cli/src/commands/run.ts`. The import result already had no useful type, and the literal specifier added the full server to the CLI program. - Add a layered server typecheck step to the `pr-trusted` typecheck job, because `typecheck:build-gaps` does not run this script. - Update `AGENTS.md` and `doc/DEVELOPING.md`. ## Verification Results on Linux use a clean copy of this branch, a new `pnpm install`, Node 24, Rust 1.97.1, 4 CPUs, and a 4 GiB memory limit with no swap. Results before the merge of `master`: | Command | Result | Peak non-reclaimable memory | | --- | --- | --- | | `node scripts/typecheck-low-memory.mjs` | Exit 0, 0 OOM kills | 3,492 MiB | | `PAPERCLIP_TSC_EMIT_ONLY=1 pnpm --filter @paperclipai/server build` | Exit 0 | 3,656 MiB, from the Rust release build | | Server single-program `tsc --noEmit --singleThreaded` | Exit 137, 1 OOM kill | Reached the limit | Results on macOS on the merged head, with peak memory for the full process tree: | Command | Before | After | | --- | --- | --- | | `pnpm -r typecheck` | 10.2 GiB, exit 0 | 5.47 GiB, exit 0 | | `node scripts/typecheck-low-memory.mjs` | Not available | 3.81 GiB, exit 0 | | CLI typecheck | 7.06 GiB | 1.68 GiB, before the merge | Other checks: - Compiler tests: new tests run the real compiler on a small fixture. They prove that the layers emit declarations and pass, that a renamed export fails the layer that imports it, and that a changed type fails the test layer. - Declaration imports: a new compiler test proves that types from `.d.mts` and JSON imports stay correct in later layers. On the real server, a wrong use of `AgentFileManifest` in the cycle layer failed with `TS2353`. - Coverage: a test on the real server source asserts that each source file is in one layer and that no import goes to a later layer. The test layer contains the remaining files. `src/__tests__` stays out of the root list, as in `server/tsconfig.json`. - Error detection: I added one type error to one file in each of the four layers. The typecheck reported all four errors and exited with 1. - Cross-layer errors: I renamed an export in the lower layer. The cycle and upper layers reported the error and the typecheck exited with 1. I reverted these temporary edits. - Emit: full `tsc` and `--noCheck --declaration false --declarationMap false` gave byte-identical JavaScript and source maps for 2,249 files on the base commit. - Runtime: `scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs` passed with a bundle from the emit-only build. - CLI: the esbuild bundle keeps the dynamic import, and `node --check` passed. - Tests on the merged head: `typecheck-layers`, `prepare-runner-vendor`, `cli-invocation-safety`, the CLI `runtime` tests, and all 503 `.github/scripts/tests` tests passed. Before the merge, `release-verify-workflow` and `runner-release-crate-cache` passed. - Full `pnpm --filter @paperclipai/server build` passed before the merge. Failures during this work: - The first low-memory run failed in the UI with `TS5023: Unknown compiler option '-b'`, because the flag came before `-b`. I corrected the argument order and added a test that runs build mode with the flag. - The first rename check did not rename the export, because BSD `sed` does not support `\b`. Its exit 0 was not evidence. I ran the check again with `perl`, and the result above is from that run. - The first version of the real-compiler fixture had a `TS2774` error in its own source. The test found this error, and I corrected the fixture. - CI failed `server-package-build-script.test.ts` on `1abc5e5a43`, because that test reads the steps of the server `build` script, and I had moved them to `build:prepare` and `build:assets`. My first search for readers of this script did not include `server/src`. I removed those scripts and `build:image`, restored one `build` script, and kept the test unchanged. - `GOMEMLIMIT` did not help enough, and I removed it from the plan. On these steps, `3GiB` saved at most about 300 MiB. With default checkers, the cycle layer was stopped with and without it, and the lower layer went from 7 seconds to 151 seconds. Not run or not proven: - I did not run the full `pnpm test:run`, a root `pnpm build`, or a full `docker build`. I expect CI to run the test and build gates. - I did not measure the Docker build steps other than the server build. - I did not run the 4 GiB checks on x86-64. - I did not measure the merged head in the 4 GiB container. The macOS result on the merged head is 3.81 GiB, and the merge added one source file to the lower layer. ## Risks - The margin is small. In the 4 GiB container, the cycle layer peaks about 600 MiB below the limit, and the Rust release build in the image build peaks about 440 MiB below it. Growth of the largest import cycle uses this margin first. - `node scripts/typecheck-low-memory.mjs` applies only to `pnpm -r` packages. These commands still check the server as one program and need more than 4 GiB: `pnpm build`, `test:e2e:runner:typecheck`, `test:lifecycle-baseline:typecheck`, and the `tests/speko-native` project. - A local flow that used the Runner binary from a previous typecheck must now run `pnpm --filter @paperclipai/paperclip-runner build`. - The image does not include server declarations now. I found no consumer in this repository. A downstream image that reads `server/dist/*.d.ts` must change. - An image build does not find type errors now. A typecheck in CI must find them before the image build. - The import scan uses a regular expression. A false edge can only make a layer larger. A missed edge causes a `TS6307` failure, so it cannot skip a file. - The new CI step adds about one minute to the typecheck job. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with reasoning, tool use, and code execution. The runtime does not show the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Nicky LeachandClaude authored and GitHub committed
1 parent
5f1b59ba2c
commit
6ef37398c8
12 files changed
+876
-7
No files matched your search
@@ -416,6 +416,11 @@ jobs:
|
||||
- name: Typecheck workspaces whose build scripts skip TypeScript
|
||||
run: pnpm run typecheck:build-gaps
|
||||
|
||||
- name: Typecheck the server in low-memory layers
|
||||
env:
|
||||
PAPERCLIP_TYPECHECK_LOW_MEMORY: "1"
|
||||
run: pnpm --filter @paperclipai/server typecheck
|
||||
|
||||
- name: Verify repository-wide CLI guidance safety
|
||||
run: pnpm exec vitest run --project @paperclipai/server server/src/__tests__/cli-invocation-safety.test.ts
|
||||
|
||||
|
||||
@@ -223,6 +223,8 @@ pnpm test:run
|
||||
pnpm build
|
||||
```
|
||||
|
||||
If memory is limited, use `node scripts/typecheck-low-memory.mjs` instead of `pnpm -r typecheck`. See `doc/DEVELOPING.md`.
|
||||
|
||||
If anything cannot be run, explicitly report what was not run and why.
|
||||
|
||||
## 8. API and Auth Expectations
|
||||
|
||||
+4
-1
@@ -146,8 +146,11 @@ RUN pnpm --filter @paperclipai/plugin-sdk build
|
||||
# next RUN as an environment variable. The production stage below declares the
|
||||
# same ARG again for the runtime fallback; an ARG goes out of scope at the
|
||||
# end of its stage. Empty for local `docker build`, which then writes no stamp.
|
||||
# NODE_OPTIONS covers the Node build steps. tsc is a native binary and ignores it.
|
||||
ENV NODE_OPTIONS=--max-old-space-size=4096
|
||||
RUN pnpm --filter @paperclipai/server build
|
||||
# The image needs no declarations, and CI type-checks this source separately.
|
||||
# Emitting without either cuts the tsc step from more than 10 GiB to about 2.5 GiB.
|
||||
RUN PAPERCLIP_TSC_EMIT_ONLY=1 pnpm --filter @paperclipai/server build
|
||||
RUN test -f server/dist/index.js || (echo "ERROR: server build output missing" && exit 1)
|
||||
RUN pnpm --filter @paperclipai/server build:runtime \
|
||||
&& test -f server/dist-runtime/index.mjs || (echo "ERROR: server runtime bundle missing" && exit 1)
|
||||
|
||||
@@ -232,8 +232,11 @@ async function importServerEntry(): Promise<StartedServer> {
|
||||
|
||||
// Production mode: import the published @paperclipai/server package
|
||||
try {
|
||||
const mod = await import("@paperclipai/server");
|
||||
return await startServerFromModule(mod, "@paperclipai/server");
|
||||
// A literal specifier would pull the whole server program into the CLI
|
||||
// typecheck. This import is untyped either way, like the dev import above.
|
||||
const serverPackage = "@paperclipai/server";
|
||||
const mod = await import(serverPackage);
|
||||
return await startServerFromModule(mod, serverPackage);
|
||||
} catch (err) {
|
||||
const missingSpecifier = getMissingModuleSpecifier(err);
|
||||
const missingServerEntrypoint = !missingSpecifier || missingSpecifier === "@paperclipai/server";
|
||||
|
||||
@@ -1132,6 +1132,46 @@ falls back to the fresh path automatically, so the cache can make a suite
|
||||
slower but never fail it or skip it: a migration that cannot apply still fails
|
||||
the suite on the fresh path.
|
||||
|
||||
#### Typecheck memory
|
||||
|
||||
TypeScript 7 is a native binary, so `NODE_OPTIONS=--max-old-space-size` does not
|
||||
limit `tsc`. Its peak memory depends on how much source one process checks and on
|
||||
the number of checkers, because each checker keeps its own copy of the types.
|
||||
|
||||
`pnpm --filter @paperclipai/server typecheck` does not check one program.
|
||||
`server/scripts/typecheck-layers.mjs` reads the import graph and runs one `tsc`
|
||||
process per layer. Each layer reads the earlier layers as declarations:
|
||||
|
||||
| Layer | Files |
|
||||
| --- | --- |
|
||||
| `below` | Files the largest import cycle depends on, and unrelated files |
|
||||
| `cycle` | The largest import cycle |
|
||||
| `above` | Files that depend on the cycle |
|
||||
| `tests` | Test files |
|
||||
|
||||
The generated configs and declarations go in
|
||||
`server/node_modules/.cache/paperclip-typecheck`. You do not maintain a layer list.
|
||||
`server/tsconfig.json` remains the single program for editors and `pnpm build`.
|
||||
The server typecheck builds only the Runner's TypeScript output. Use
|
||||
`pnpm --filter @paperclipai/paperclip-runner build` when a local check needs the
|
||||
Runner binary.
|
||||
|
||||
If the memory available to the task is limited, use this command:
|
||||
|
||||
```sh
|
||||
node scripts/typecheck-low-memory.mjs
|
||||
```
|
||||
|
||||
It peaks at about 3.8 GiB, against about 5.5 GiB for `pnpm -r typecheck` on 16
|
||||
cores. It also takes about twice as long.
|
||||
|
||||
It checks one package at a time and uses one thread for the server layers and the
|
||||
UI. Do not add `GOMEMLIMIT` to this path. It saves at most 300 MiB here, and it
|
||||
slows `tsc` by up to 20 times when the live heap goes above the limit.
|
||||
`pnpm build` still checks the whole server program and needs more than 8 GiB. The
|
||||
Docker image sets `PAPERCLIP_TSC_EMIT_ONLY=1` for that build, which emits
|
||||
JavaScript without checks or declarations.
|
||||
|
||||
#### Reused Runner build in the server build
|
||||
|
||||
The server build vendors `packages/paperclip-runner/dist`, so its
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env node
|
||||
// Runs the calling package's tsc, honoring the repository's low-memory mode.
|
||||
//
|
||||
// TypeScript 7 is a native binary, so Node heap flags do not limit it. Each
|
||||
// checker keeps its own copy of the types, which makes the checker count the
|
||||
// most useful control a package has over its peak memory.
|
||||
// scripts/typecheck-low-memory.mjs sets PAPERCLIP_TYPECHECK_LOW_MEMORY=1 so the
|
||||
// largest programs check on a single thread.
|
||||
//
|
||||
// A build that only needs JavaScript sets PAPERCLIP_TSC_EMIT_ONLY=1. That build
|
||||
// skips type checks and declarations, which use most of the memory of an emit.
|
||||
// The Docker image uses this mode.
|
||||
//
|
||||
// Usage: node ../scripts/run-tsc.mjs [tsc arguments]
|
||||
|
||||
import { spawnSync } from "node:child_process";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
export const LOW_MEMORY_VARIABLE = "PAPERCLIP_TYPECHECK_LOW_MEMORY";
|
||||
export const EMIT_ONLY_VARIABLE = "PAPERCLIP_TSC_EMIT_ONLY";
|
||||
|
||||
export function lowMemoryTscArgs(env = process.env) {
|
||||
return env[LOW_MEMORY_VARIABLE] === "1" ? ["--singleThreaded"] : [];
|
||||
}
|
||||
|
||||
export function emitOnlyTscArgs(env = process.env) {
|
||||
return env[EMIT_ONLY_VARIABLE] === "1"
|
||||
? ["--noCheck", "--declaration", "false", "--declarationMap", "false"]
|
||||
: [];
|
||||
}
|
||||
|
||||
export function runTsc(args, { cwd = process.cwd(), env = process.env } = {}) {
|
||||
const tsc = path.join(cwd, "node_modules", "typescript", "bin", "tsc");
|
||||
// tsc only accepts -b as its first argument, so the caller's arguments go first.
|
||||
const result = spawnSync(process.execPath, [tsc, ...args, ...lowMemoryTscArgs(env), ...emitOnlyTscArgs(env)], {
|
||||
cwd,
|
||||
env,
|
||||
stdio: "inherit",
|
||||
});
|
||||
if (result.error) throw result.error;
|
||||
return { status: result.status ?? 1, signal: result.signal };
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
process.exitCode = runTsc(process.argv.slice(2)).status;
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env node
|
||||
// Runs the workspace typecheck with less memory, but more slowly. It checks one
|
||||
// package at a time, and scripts/run-tsc.mjs uses one thread for the largest
|
||||
// programs. See "Typecheck memory" in doc/DEVELOPING.md.
|
||||
//
|
||||
// Usage: node scripts/typecheck-low-memory.mjs
|
||||
|
||||
import { spawnSync } from "node:child_process";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { LOW_MEMORY_VARIABLE } from "./run-tsc.mjs";
|
||||
|
||||
export const STEPS = [
|
||||
["run", "preflight:workspace-links"],
|
||||
["-r", "--workspace-concurrency=1", "typecheck"],
|
||||
];
|
||||
|
||||
function runPnpm(args, options) {
|
||||
const result = spawnSync("pnpm", args, {
|
||||
...options,
|
||||
stdio: "inherit",
|
||||
// A bare `pnpm` on Windows is pnpm.cmd, which needs the shell to resolve.
|
||||
shell: process.platform === "win32",
|
||||
});
|
||||
if (result.error) throw result.error;
|
||||
return result.status ?? 1;
|
||||
}
|
||||
|
||||
export function typecheckLowMemory({ cwd, env = process.env, run = runPnpm }) {
|
||||
for (const args of STEPS) {
|
||||
const status = run(args, { cwd, env: { ...env, [LOW_MEMORY_VARIABLE]: "1" } });
|
||||
if (status !== 0) return status;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
process.exitCode = typecheckLowMemory({
|
||||
cwd: path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."),
|
||||
});
|
||||
}
|
||||
+2
-2
@@ -36,14 +36,14 @@
|
||||
"dev": "tsx src/index.ts",
|
||||
"dev:watch": "cross-env PAPERCLIP_MIGRATION_PROMPT=never PAPERCLIP_MIGRATION_AUTO_APPLY=true tsx ./scripts/dev-watch.ts",
|
||||
"prepare:ui-dist": "bash ../scripts/prepare-server-ui-dist.sh",
|
||||
"build": "pnpm run prepare:runner-vendor && node scripts/verify-runner-vendor-dependencies.mjs && tsc && mkdir -p dist/onboarding-assets dist/built-ins dist/services/scripts dist/vendor/paperclip-runner && cp -R src/onboarding-assets/. dist/onboarding-assets/ && cp -R src/built-ins/. dist/built-ins/ && cp -R src/services/scripts/. dist/services/scripts/ && cp -Rf ../packages/paperclip-runner/dist/. dist/vendor/paperclip-runner/ && cp ../packages/paperclip-runner/scripts/install-copilot-assets.mjs ../packages/paperclip-runner/scripts/materialize-copilot-binary.mjs ../packages/paperclip-runner/scripts/copilot-inner-distribution.mjs dist/ && node dist/install-copilot-assets.mjs && node scripts/write-build-stamp.mjs",
|
||||
"build": "pnpm run prepare:runner-vendor && node scripts/verify-runner-vendor-dependencies.mjs && node ../scripts/run-tsc.mjs && mkdir -p dist/onboarding-assets dist/built-ins dist/services/scripts dist/vendor/paperclip-runner && cp -R src/onboarding-assets/. dist/onboarding-assets/ && cp -R src/built-ins/. dist/built-ins/ && cp -R src/services/scripts/. dist/services/scripts/ && cp -Rf ../packages/paperclip-runner/dist/. dist/vendor/paperclip-runner/ && cp ../packages/paperclip-runner/scripts/install-copilot-assets.mjs ../packages/paperclip-runner/scripts/materialize-copilot-binary.mjs ../packages/paperclip-runner/scripts/copilot-inner-distribution.mjs dist/ && node dist/install-copilot-assets.mjs && node scripts/write-build-stamp.mjs",
|
||||
"prepack": "pnpm run prepare:ui-dist && pnpm run build",
|
||||
"postpack": "rm -rf ui-dist",
|
||||
"clean": "rm -rf dist",
|
||||
"build:runtime": "node scripts/bundle-runtime.mjs",
|
||||
"start": "node dist/index.js",
|
||||
"prepare:runner-vendor": "node scripts/prepare-runner-vendor.mjs",
|
||||
"typecheck": "pnpm run prepare:runner-vendor && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && tsc --noEmit",
|
||||
"typecheck": "pnpm --filter @paperclipai/paperclip-runner run build:typescript && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && node scripts/typecheck-layers.mjs",
|
||||
"postinstall": "node scripts/install-copilot-assets.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
// Makes sure packages/paperclip-runner/dist is a current build before the
|
||||
// server build vendors it, rebuilding the Runner only when that is needed.
|
||||
//
|
||||
// The server build (and typecheck) used to run the Runner's full `build`
|
||||
// The server build used to run the Runner's full `build`
|
||||
// unconditionally here. That is what lets a standalone
|
||||
// `pnpm --filter @paperclipai/server build` produce a current vendored Runner
|
||||
// on its own, which the Dockerfile relies on. But inside `pnpm -r build` the
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
// Typechecks the server in layers so that no single tsc process holds the whole
|
||||
// program in memory.
|
||||
//
|
||||
// tsc keeps every type it creates until it exits, so its peak grows with the
|
||||
// source it checks. The server is about 1,700 source files and 500 test files,
|
||||
// and one program over all of them needs more than 8 GiB at any setting. Most
|
||||
// of those files are not in an import cycle, so they can be checked separately
|
||||
// and read back as declarations:
|
||||
//
|
||||
// below files the largest import cycle depends on, plus unrelated files
|
||||
// cycle the largest import cycle, reading `below` as declarations
|
||||
// above files that depend on the cycle, reading both as declarations
|
||||
// tests test files, reading all three as declarations
|
||||
//
|
||||
// The layers come from the import graph on every run, so nothing has to move
|
||||
// and no list can go stale. The import scan does not have to be perfect: if it
|
||||
// misses an edge, a file imports one outside its layer's references and tsc
|
||||
// fails with TS6307 instead of skipping the file.
|
||||
//
|
||||
// server/tsconfig.json stays the single program that editors and `pnpm build`
|
||||
// use.
|
||||
//
|
||||
// Usage: node scripts/typecheck-layers.mjs
|
||||
|
||||
import { mkdirSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { runTsc } from "../../scripts/run-tsc.mjs";
|
||||
|
||||
const IMPORT_SPECIFIER = /(?:from\s+|import\s*\(\s*|import\s+)["'](\.[^"']+)["']/g;
|
||||
const OUTPUT_DIRECTORY = path.join("node_modules", ".cache", "paperclip-typecheck");
|
||||
|
||||
const isTestFile = (file) => /\.test\.tsx?$/.test(file) || file.split("/").includes("__tests__");
|
||||
// tsconfig.json excludes this tree. A test reaches its helpers through imports.
|
||||
const isExcludedFromProgram = (file) => file.startsWith("src/__tests__/");
|
||||
|
||||
export function listSourceFiles(serverRoot) {
|
||||
const files = [];
|
||||
const walk = (directory) => {
|
||||
for (const entry of readdirSync(path.join(serverRoot, directory), { withFileTypes: true })) {
|
||||
const relative = `${directory}/${entry.name}`;
|
||||
if (entry.isDirectory()) walk(relative);
|
||||
else if (/\.tsx?$/.test(entry.name)) files.push(relative);
|
||||
}
|
||||
};
|
||||
walk("src");
|
||||
return files.sort();
|
||||
}
|
||||
|
||||
/** Resolves the relative imports of each file. `files` maps a path to its text. */
|
||||
export function readImports(files) {
|
||||
const imports = new Map();
|
||||
for (const [file, text] of files) {
|
||||
const modules = new Set();
|
||||
const json = new Set();
|
||||
for (const [, specifier] of text.matchAll(IMPORT_SPECIFIER)) {
|
||||
const target = path.posix.join(path.posix.dirname(file), specifier);
|
||||
if (target.endsWith(".json")) {
|
||||
json.add(target);
|
||||
continue;
|
||||
}
|
||||
const candidates = [
|
||||
target,
|
||||
target.replace(/\.js$/, ".ts"),
|
||||
target.replace(/\.js$/, ".tsx"),
|
||||
`${target}.ts`,
|
||||
`${target}/index.ts`,
|
||||
];
|
||||
const resolved = candidates.find((candidate) => files.has(candidate));
|
||||
if (resolved) modules.add(resolved);
|
||||
}
|
||||
imports.set(file, { modules, json });
|
||||
}
|
||||
return imports;
|
||||
}
|
||||
|
||||
/** Tarjan's algorithm, iterative because import chains can be very deep. */
|
||||
function largestCycle(graph) {
|
||||
const index = new Map();
|
||||
const low = new Map();
|
||||
const onStack = new Set();
|
||||
const stack = [];
|
||||
let next = 0;
|
||||
let largest = [];
|
||||
|
||||
for (const root of graph.keys()) {
|
||||
if (index.has(root)) continue;
|
||||
const work = [{ node: root, edges: [...graph.get(root)], position: 0 }];
|
||||
index.set(root, next);
|
||||
low.set(root, next);
|
||||
next += 1;
|
||||
stack.push(root);
|
||||
onStack.add(root);
|
||||
while (work.length > 0) {
|
||||
const frame = work[work.length - 1];
|
||||
if (frame.position < frame.edges.length) {
|
||||
const edge = frame.edges[frame.position];
|
||||
frame.position += 1;
|
||||
if (!index.has(edge)) {
|
||||
index.set(edge, next);
|
||||
low.set(edge, next);
|
||||
next += 1;
|
||||
stack.push(edge);
|
||||
onStack.add(edge);
|
||||
work.push({ node: edge, edges: [...graph.get(edge)], position: 0 });
|
||||
} else if (onStack.has(edge)) {
|
||||
low.set(frame.node, Math.min(low.get(frame.node), index.get(edge)));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
work.pop();
|
||||
const parent = work[work.length - 1];
|
||||
if (parent) low.set(parent.node, Math.min(low.get(parent.node), low.get(frame.node)));
|
||||
if (low.get(frame.node) === index.get(frame.node)) {
|
||||
const component = [];
|
||||
let member;
|
||||
do {
|
||||
member = stack.pop();
|
||||
onStack.delete(member);
|
||||
component.push(member);
|
||||
} while (member !== frame.node);
|
||||
if (component.length > largest.length) largest = component;
|
||||
}
|
||||
}
|
||||
}
|
||||
return largest.length > 1 ? new Set(largest) : new Set();
|
||||
}
|
||||
|
||||
function reachableFrom(start, graph) {
|
||||
const seen = new Set(start);
|
||||
const pending = [...start];
|
||||
while (pending.length > 0) {
|
||||
for (const edge of graph.get(pending.pop()) ?? []) {
|
||||
if (!seen.has(edge)) {
|
||||
seen.add(edge);
|
||||
pending.push(edge);
|
||||
}
|
||||
}
|
||||
}
|
||||
return seen;
|
||||
}
|
||||
|
||||
/**
|
||||
* Sorts every file into the layers, in check order. `files` maps a path to its
|
||||
* text. Declaration files are ambient, so every layer lists them.
|
||||
*/
|
||||
export function planLayers(files) {
|
||||
const declarations = [...files.keys()].filter((file) => file.endsWith(".d.ts"));
|
||||
const candidates = new Map([...files].filter(([file]) => !file.endsWith(".d.ts")));
|
||||
const imports = readImports(candidates);
|
||||
|
||||
const graph = new Map();
|
||||
const reversed = new Map();
|
||||
for (const file of candidates.keys()) {
|
||||
if (!isTestFile(file)) {
|
||||
graph.set(file, new Set());
|
||||
reversed.set(file, new Set());
|
||||
}
|
||||
}
|
||||
for (const file of graph.keys()) {
|
||||
for (const target of imports.get(file).modules) {
|
||||
if (graph.has(target)) {
|
||||
graph.get(file).add(target);
|
||||
reversed.get(target).add(file);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const cycle = largestCycle(graph);
|
||||
const dependents = reachableFrom(cycle, reversed);
|
||||
const above = [...dependents].filter((file) => !cycle.has(file));
|
||||
const below = [...graph.keys()].filter((file) => !dependents.has(file));
|
||||
const tests = [...candidates.keys()].filter((file) => isTestFile(file) && !isExcludedFromProgram(file));
|
||||
|
||||
const layer = (name, members, references) => {
|
||||
const sorted = [...members].sort();
|
||||
const json = new Set(sorted.flatMap((file) => [...imports.get(file).json]));
|
||||
return { name, sourceFiles: sorted, files: [...sorted, ...[...json].sort(), ...declarations], references };
|
||||
};
|
||||
return [
|
||||
layer("below", below, []),
|
||||
layer("cycle", cycle, ["below"]),
|
||||
layer("above", above, ["below", "cycle"]),
|
||||
layer("tests", tests, ["below", "cycle", "above"]),
|
||||
].filter((candidate) => candidate.sourceFiles.length > 0);
|
||||
}
|
||||
|
||||
/** Builds the tsconfig for one layer. Paths are relative to the output directory. */
|
||||
export function layerConfig(layer, { emptyLayers = new Set() } = {}) {
|
||||
const fromOutput = (file) => path.posix.join("../../..", file);
|
||||
const references = layer.references
|
||||
.filter((name) => !emptyLayers.has(name))
|
||||
.map((name) => ({ path: `./tsconfig.${name}.json` }));
|
||||
const output =
|
||||
layer.name === "tests"
|
||||
? { composite: false, declaration: false, declarationMap: false, sourceMap: false, noEmit: true }
|
||||
: {
|
||||
composite: true,
|
||||
declaration: true,
|
||||
declarationMap: false,
|
||||
sourceMap: false,
|
||||
emitDeclarationOnly: true,
|
||||
outDir: "./dts",
|
||||
tsBuildInfoFile: `./${layer.name}.tsbuildinfo`,
|
||||
};
|
||||
return {
|
||||
extends: fromOutput("tsconfig.json"),
|
||||
compilerOptions: { ...output, rootDir: fromOutput("src") },
|
||||
files: layer.files.map(fromOutput),
|
||||
include: [],
|
||||
references,
|
||||
};
|
||||
}
|
||||
|
||||
export function typecheckLayers({
|
||||
serverRoot,
|
||||
run = runTsc,
|
||||
log = console.log,
|
||||
}) {
|
||||
const sourceFiles = listSourceFiles(serverRoot);
|
||||
const layers = planLayers(
|
||||
new Map(sourceFiles.map((file) => [file, readFileSync(path.join(serverRoot, file), "utf8")])),
|
||||
);
|
||||
const planned = new Set(layers.map((layer) => layer.name));
|
||||
const emptyLayers = new Set(["below", "cycle", "above", "tests"].filter((name) => !planned.has(name)));
|
||||
|
||||
const outputRoot = path.join(serverRoot, OUTPUT_DIRECTORY);
|
||||
rmSync(outputRoot, { recursive: true, force: true });
|
||||
mkdirSync(outputRoot, { recursive: true });
|
||||
// Without this, the path under node_modules would make the declarations CommonJS.
|
||||
writeFileSync(path.join(outputRoot, "package.json"), `${JSON.stringify({ type: "module" })}\n`);
|
||||
for (const layer of layers) {
|
||||
writeFileSync(
|
||||
path.join(outputRoot, `tsconfig.${layer.name}.json`),
|
||||
`${JSON.stringify(layerConfig(layer, { emptyLayers }), null, 2)}\n`,
|
||||
);
|
||||
}
|
||||
|
||||
let exitCode = 0;
|
||||
for (const layer of layers) {
|
||||
log(`typecheck:layers: ${layer.name} (${layer.sourceFiles.length} files)`);
|
||||
const config = path.join(OUTPUT_DIRECTORY, `tsconfig.${layer.name}.json`);
|
||||
const { status, signal } = run(["-p", config], { cwd: serverRoot });
|
||||
// A killed process leaves no declarations, so later layers would only add noise.
|
||||
if (signal) return status || 1;
|
||||
// Later layers still run after type errors. Each file is checked in one
|
||||
// layer, so this reports everything a single tsc run would.
|
||||
if (status !== 0) exitCode = status;
|
||||
}
|
||||
return exitCode;
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
process.exitCode = typecheckLayers({
|
||||
serverRoot: path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."),
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,471 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
|
||||
import { emitOnlyTscArgs, lowMemoryTscArgs, runTsc } from "../../scripts/run-tsc.mjs";
|
||||
import { STEPS, typecheckLowMemory } from "../../scripts/typecheck-low-memory.mjs";
|
||||
import { layerConfig, listSourceFiles, planLayers, readImports, typecheckLayers } from "./typecheck-layers.mjs";
|
||||
|
||||
const serverRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const sourcesOf = (layers, name) => layers.find((layer) => layer.name === name)?.sourceFiles;
|
||||
|
||||
describe("readImports", () => {
|
||||
it("resolves every import form that TypeScript follows", () => {
|
||||
const files = new Map([
|
||||
[
|
||||
"src/a.ts",
|
||||
[
|
||||
`import { b } from "./b.js";`,
|
||||
`export * from "./c.js";`,
|
||||
`import "./d.js";`,
|
||||
`const e = await import("./e.js");`,
|
||||
`type F = typeof import("./f.js");`,
|
||||
`import g from "./g/index.js";`,
|
||||
`import h from "./h.js";`,
|
||||
`import data from "./data.json" with { type: "json" };`,
|
||||
`import { z } from "zod";`,
|
||||
].join("\n"),
|
||||
],
|
||||
["src/b.ts", ""],
|
||||
["src/c.ts", ""],
|
||||
["src/d.ts", ""],
|
||||
["src/e.ts", ""],
|
||||
["src/f.ts", ""],
|
||||
["src/g/index.ts", ""],
|
||||
["src/h.tsx", ""],
|
||||
]);
|
||||
|
||||
const { modules, json } = readImports(files).get("src/a.ts");
|
||||
|
||||
expect([...modules].sort()).toEqual([
|
||||
"src/b.ts",
|
||||
"src/c.ts",
|
||||
"src/d.ts",
|
||||
"src/e.ts",
|
||||
"src/f.ts",
|
||||
"src/g/index.ts",
|
||||
"src/h.tsx",
|
||||
]);
|
||||
expect([...json]).toEqual(["src/data.json"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("planLayers", () => {
|
||||
it("separates the largest cycle from what it depends on and what depends on it", () => {
|
||||
const layers = planLayers(
|
||||
new Map([
|
||||
["src/leaf.ts", ""],
|
||||
["src/unrelated.ts", ""],
|
||||
["src/one.ts", `import "./two.js"; import "./leaf.js";`],
|
||||
["src/two.ts", `import "./one.js";`],
|
||||
["src/route.ts", `import "./one.js";`],
|
||||
["src/route.test.ts", `import "./route.js";`],
|
||||
["src/types/ambient.d.ts", ""],
|
||||
]),
|
||||
);
|
||||
|
||||
expect(sourcesOf(layers, "below")).toEqual(["src/leaf.ts", "src/unrelated.ts"]);
|
||||
expect(sourcesOf(layers, "cycle")).toEqual(["src/one.ts", "src/two.ts"]);
|
||||
expect(sourcesOf(layers, "above")).toEqual(["src/route.ts"]);
|
||||
expect(sourcesOf(layers, "tests")).toEqual(["src/route.test.ts"]);
|
||||
for (const layer of layers) expect(layer.files).toContain("src/types/ambient.d.ts");
|
||||
});
|
||||
|
||||
it("keeps test support files in the test layer and leaves src/__tests__ to imports", () => {
|
||||
const layers = planLayers(
|
||||
new Map([
|
||||
["src/a.ts", ""],
|
||||
["src/services/__tests__/fixtures.ts", ""],
|
||||
["src/__tests__/helpers/db.ts", ""],
|
||||
["src/__tests__/big.test.ts", ""],
|
||||
]),
|
||||
);
|
||||
|
||||
expect(sourcesOf(layers, "below")).toEqual(["src/a.ts"]);
|
||||
expect(sourcesOf(layers, "tests")).toEqual(["src/services/__tests__/fixtures.ts"]);
|
||||
});
|
||||
|
||||
it("lists imported JSON in the layer that imports it", () => {
|
||||
const layers = planLayers(
|
||||
new Map([
|
||||
["src/a.ts", `import data from "./data.json";`],
|
||||
["src/one.ts", `import "./two.js"; import data from "./cycle.json";`],
|
||||
["src/two.ts", `import "./one.js";`],
|
||||
]),
|
||||
);
|
||||
|
||||
expect(layers.find((layer) => layer.name === "below").files).toContain("src/data.json");
|
||||
expect(layers.find((layer) => layer.name === "cycle").files).toContain("src/cycle.json");
|
||||
});
|
||||
|
||||
it("omits layers with no files, as when the source has no cycle", () => {
|
||||
const layers = planLayers(new Map([["src/a.ts", ""], ["src/b.ts", `import "./a.js";`]]));
|
||||
|
||||
expect(layers.map((layer) => layer.name)).toEqual(["below"]);
|
||||
expect(sourcesOf(layers, "below")).toEqual(["src/a.ts", "src/b.ts"]);
|
||||
});
|
||||
|
||||
it("covers the real server: one layer per file, and no import reaches a later layer", () => {
|
||||
const sourceFiles = listSourceFiles(serverRoot);
|
||||
const files = new Map(
|
||||
sourceFiles.map((file) => [file, readFileSync(path.join(serverRoot, file), "utf8")]),
|
||||
);
|
||||
const layers = planLayers(files);
|
||||
|
||||
const planned = layers.flatMap((layer) => layer.sourceFiles);
|
||||
expect(new Set(planned).size).toBe(planned.length);
|
||||
expect(planned.sort()).toEqual(
|
||||
sourceFiles.filter((file) => !file.endsWith(".d.ts") && !file.startsWith("src/__tests__/")).sort(),
|
||||
);
|
||||
|
||||
const position = new Map(
|
||||
layers.flatMap((layer, index) => layer.sourceFiles.map((file) => [file, index])),
|
||||
);
|
||||
const imports = readImports(new Map([...files].filter(([file]) => !file.endsWith(".d.ts"))));
|
||||
for (const layer of layers.filter((candidate) => candidate.name !== "tests")) {
|
||||
for (const file of layer.sourceFiles) {
|
||||
for (const target of imports.get(file).modules) {
|
||||
if (position.has(target)) expect(position.get(target)).toBeLessThanOrEqual(position.get(file));
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("layerConfig", () => {
|
||||
const layer = { name: "cycle", files: ["src/a.ts"], references: ["below"] };
|
||||
|
||||
it("emits declarations for source layers and references earlier ones", () => {
|
||||
const config = layerConfig(layer);
|
||||
|
||||
expect(config.compilerOptions).toMatchObject({ composite: true, emitDeclarationOnly: true, outDir: "./dts" });
|
||||
expect(config.extends).toBe("../../../tsconfig.json");
|
||||
expect(config.files).toEqual(["../../../src/a.ts"]);
|
||||
expect(config.references).toEqual([{ path: "./tsconfig.below.json" }]);
|
||||
});
|
||||
|
||||
it("emits nothing for tests and drops references to empty layers", () => {
|
||||
const config = layerConfig(
|
||||
{ name: "tests", files: [], references: ["below", "cycle"] },
|
||||
{ emptyLayers: new Set(["cycle"]) },
|
||||
);
|
||||
|
||||
expect(config.compilerOptions).toMatchObject({ composite: false, noEmit: true });
|
||||
expect(config.references).toEqual([{ path: "./tsconfig.below.json" }]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("typecheckLayers", () => {
|
||||
const roots = [];
|
||||
|
||||
afterEach(() => {
|
||||
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function serverFixture() {
|
||||
const root = mkdtempSync(path.join(tmpdir(), "typecheck-layers-"));
|
||||
roots.push(root);
|
||||
mkdirSync(path.join(root, "src"));
|
||||
writeFileSync(path.join(root, "src", "leaf.ts"), "");
|
||||
writeFileSync(path.join(root, "src", "one.ts"), `import "./two.js"; import "./leaf.js";`);
|
||||
writeFileSync(path.join(root, "src", "two.ts"), `import "./one.js";`);
|
||||
writeFileSync(path.join(root, "src", "route.ts"), `import "./one.js";`);
|
||||
writeFileSync(path.join(root, "src", "route.test.ts"), `import "./route.js";`);
|
||||
return root;
|
||||
}
|
||||
|
||||
it("runs the layers in order and marks the output directory as ESM", () => {
|
||||
const root = serverFixture();
|
||||
const configs = [];
|
||||
|
||||
const status = typecheckLayers({
|
||||
serverRoot: root,
|
||||
log: () => {},
|
||||
run: (args) => {
|
||||
configs.push(path.basename(args[1]));
|
||||
return { status: 0, signal: null };
|
||||
},
|
||||
});
|
||||
|
||||
expect(status).toBe(0);
|
||||
expect(configs).toEqual([
|
||||
"tsconfig.below.json",
|
||||
"tsconfig.cycle.json",
|
||||
"tsconfig.above.json",
|
||||
"tsconfig.tests.json",
|
||||
]);
|
||||
const output = path.join(root, "node_modules", ".cache", "paperclip-typecheck");
|
||||
expect(JSON.parse(readFileSync(path.join(output, "package.json"), "utf8"))).toEqual({ type: "module" });
|
||||
});
|
||||
|
||||
it("reports type errors from every layer", () => {
|
||||
const root = serverFixture();
|
||||
const configs = [];
|
||||
|
||||
const status = typecheckLayers({
|
||||
serverRoot: root,
|
||||
log: () => {},
|
||||
run: (args) => {
|
||||
configs.push(args[1]);
|
||||
return { status: configs.length === 1 ? 2 : 0, signal: null };
|
||||
},
|
||||
});
|
||||
|
||||
expect(status).toBe(2);
|
||||
expect(configs).toHaveLength(4);
|
||||
});
|
||||
|
||||
it("stops when a process is killed, because later layers have no declarations", () => {
|
||||
const root = serverFixture();
|
||||
const configs = [];
|
||||
|
||||
const status = typecheckLayers({
|
||||
serverRoot: root,
|
||||
log: () => {},
|
||||
run: (args) => {
|
||||
configs.push(args[1]);
|
||||
return { status: 1, signal: "SIGKILL" };
|
||||
},
|
||||
});
|
||||
|
||||
expect(status).toBe(1);
|
||||
expect(configs).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("typecheckLayers with the real compiler", () => {
|
||||
const tsc = path.join(serverRoot, "node_modules", "typescript", "bin", "tsc");
|
||||
const roots = [];
|
||||
|
||||
afterEach(() => {
|
||||
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function compilerFixture() {
|
||||
const root = mkdtempSync(path.join(tmpdir(), "typecheck-layers-real-"));
|
||||
roots.push(root);
|
||||
mkdirSync(path.join(root, "src"));
|
||||
writeFileSync(path.join(root, "package.json"), `${JSON.stringify({ type: "module" })}\n`);
|
||||
writeFileSync(
|
||||
path.join(root, "tsconfig.json"),
|
||||
`${JSON.stringify({
|
||||
compilerOptions: {
|
||||
target: "ES2023",
|
||||
module: "NodeNext",
|
||||
moduleResolution: "NodeNext",
|
||||
strict: true,
|
||||
types: [],
|
||||
outDir: "dist",
|
||||
rootDir: "src",
|
||||
resolveJsonModule: true,
|
||||
skipLibCheck: true,
|
||||
},
|
||||
include: ["src"],
|
||||
})}\n`,
|
||||
);
|
||||
const sources = {
|
||||
"leaf.ts": `export const leaf = 1;\n`,
|
||||
"one.ts": `import { two } from "./two.js";\nimport { leaf } from "./leaf.js";\nexport const one = () => leaf + two();\n`,
|
||||
"two.ts": `import { one } from "./one.js";\nexport const two = (): number => one.length;\n`,
|
||||
"route.ts": `import { one } from "./one.js";\nexport const route = () => one();\n`,
|
||||
"route.test.ts": `import { route } from "./route.js";\nexport const checked: number = route();\n`,
|
||||
};
|
||||
for (const [name, text] of Object.entries(sources)) {
|
||||
writeFileSync(path.join(root, "src", name), text);
|
||||
}
|
||||
return root;
|
||||
}
|
||||
|
||||
function check(root) {
|
||||
const output = [];
|
||||
const status = typecheckLayers({
|
||||
serverRoot: root,
|
||||
log: () => {},
|
||||
run: (args, { cwd }) => {
|
||||
const result = spawnSync(process.execPath, [tsc, ...args], { cwd, encoding: "utf8" });
|
||||
output.push(result.stdout, result.stderr);
|
||||
return { status: result.status ?? 1, signal: result.signal };
|
||||
},
|
||||
});
|
||||
return { status, output: output.join("") };
|
||||
}
|
||||
|
||||
it("emits declarations between layers and checks every layer", () => {
|
||||
const root = compilerFixture();
|
||||
|
||||
const { status, output } = check(root);
|
||||
|
||||
expect(output).toBe("");
|
||||
expect(status).toBe(0);
|
||||
const declarations = path.join(root, "node_modules", ".cache", "paperclip-typecheck", "dts");
|
||||
for (const name of ["leaf.d.ts", "one.d.ts", "two.d.ts", "route.d.ts"]) {
|
||||
expect(existsSync(path.join(declarations, name))).toBe(true);
|
||||
}
|
||||
expect(existsSync(path.join(declarations, "route.test.d.ts"))).toBe(false);
|
||||
}, 60_000);
|
||||
|
||||
it("reports a changed export in the layers that depend on it", () => {
|
||||
const root = compilerFixture();
|
||||
writeFileSync(path.join(root, "src", "leaf.ts"), `export const renamedLeaf = 1;\n`);
|
||||
|
||||
const { status, output } = check(root);
|
||||
|
||||
expect(status).not.toBe(0);
|
||||
expect(output).toMatch(/src\/one\.ts\(\d+,\d+\): error TS\d+/);
|
||||
expect(output).toContain("leaf");
|
||||
}, 60_000);
|
||||
|
||||
it("reports a type change through declarations in the test layer", () => {
|
||||
const root = compilerFixture();
|
||||
writeFileSync(
|
||||
path.join(root, "src", "route.ts"),
|
||||
`import { one } from "./one.js";\nexport const route = (): string => String(one());\n`,
|
||||
);
|
||||
|
||||
const { status, output } = check(root);
|
||||
|
||||
expect(status).not.toBe(0);
|
||||
expect(output).toMatch(/src\/route\.test\.ts\(\d+,\d+\): error TS2322/);
|
||||
}, 60_000);
|
||||
|
||||
// Emitted declarations keep imports of .d.mts and JSON files, which tsc does not
|
||||
// copy to the output directory. If those imports stopped resolving, skipLibCheck
|
||||
// would hide it and the types would become `any`, so these errors would disappear.
|
||||
it("keeps types from .d.mts and JSON imports in later layers", () => {
|
||||
const root = compilerFixture();
|
||||
mkdirSync(path.join(root, "src", "scripts"));
|
||||
writeFileSync(path.join(root, "src", "scripts", "helper.mjs"), `export {};\n`);
|
||||
writeFileSync(path.join(root, "src", "scripts", "helper.d.mts"), `export interface Manifest { version: number }\n`);
|
||||
writeFileSync(path.join(root, "src", "catalog.json"), `${JSON.stringify({ tools: [{ name: "a" }] })}\n`);
|
||||
writeFileSync(
|
||||
path.join(root, "src", "leaf.ts"),
|
||||
[
|
||||
`import catalog from "./catalog.json" with { type: "json" };`,
|
||||
`import type { Manifest } from "./scripts/helper.mjs";`,
|
||||
`export type { Manifest };`,
|
||||
`export type Tool = (typeof catalog.tools)[number];`,
|
||||
`export const leaf = 1;`,
|
||||
``,
|
||||
].join("\n"),
|
||||
);
|
||||
const misuse = [
|
||||
`import type { Manifest, Tool } from "./leaf.js";`,
|
||||
`export const manifest: Manifest = { wrong: 1 };`,
|
||||
`export const tool: Tool = { wrong: 1 };`,
|
||||
``,
|
||||
].join("\n");
|
||||
writeFileSync(path.join(root, "src", "cycle-misuse.ts"), `import { one } from "./one.js";\nexport const used = one;\n${misuse}`);
|
||||
writeFileSync(
|
||||
path.join(root, "src", "two.ts"),
|
||||
`import { one } from "./one.js";\nimport { used } from "./cycle-misuse.js";\nexport const two = (): number => one.length + used.length;\n`,
|
||||
);
|
||||
writeFileSync(path.join(root, "src", "misuse.test.ts"), misuse);
|
||||
|
||||
const { status, output } = check(root);
|
||||
|
||||
expect(status).not.toBe(0);
|
||||
for (const file of ["cycle-misuse", "misuse\\.test"]) {
|
||||
expect(output).toMatch(new RegExp(`src/${file}\\.ts\\(\\d+,\\d+\\): error TS2353.*'Manifest'`));
|
||||
expect(output).toMatch(new RegExp(`src/${file}\\.ts\\(\\d+,\\d+\\): error TS2353.*'\\{ name: string; \\}'`));
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
describe("typecheckLowMemory", () => {
|
||||
it("runs every step in low-memory mode and stops at a failure", () => {
|
||||
const calls = [];
|
||||
const run = (args, { env }) => {
|
||||
calls.push([args, env.PAPERCLIP_TYPECHECK_LOW_MEMORY]);
|
||||
return calls.length === 1 ? 0 : 3;
|
||||
};
|
||||
|
||||
expect(typecheckLowMemory({ cwd: serverRoot, env: {}, run })).toBe(3);
|
||||
expect(calls).toEqual(STEPS.map((args) => [args, "1"]));
|
||||
});
|
||||
|
||||
it("does not run the typecheck after a failed preflight", () => {
|
||||
const calls = [];
|
||||
|
||||
const status = typecheckLowMemory({
|
||||
cwd: serverRoot,
|
||||
env: {},
|
||||
run: (args) => {
|
||||
calls.push(args);
|
||||
return 1;
|
||||
},
|
||||
});
|
||||
|
||||
expect(status).toBe(1);
|
||||
expect(calls).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("runTsc", () => {
|
||||
it("emits JavaScript without checks or declarations in emit-only mode", () => {
|
||||
const root = mkdtempSync(path.join(tmpdir(), "run-tsc-emit-only-"));
|
||||
try {
|
||||
mkdirSync(path.join(root, "src"));
|
||||
mkdirSync(path.join(root, "node_modules"));
|
||||
symlinkSync(
|
||||
realpathSync(path.join(serverRoot, "node_modules", "typescript")),
|
||||
path.join(root, "node_modules", "typescript"),
|
||||
);
|
||||
writeFileSync(
|
||||
path.join(root, "tsconfig.json"),
|
||||
`${JSON.stringify({
|
||||
compilerOptions: { declaration: true, declarationMap: true, types: [], outDir: "dist", rootDir: "src" },
|
||||
include: ["src"],
|
||||
})}\n`,
|
||||
);
|
||||
writeFileSync(path.join(root, "src", "a.ts"), `export const value: number = "not a number";\n`);
|
||||
|
||||
const { status } = runTsc([], { cwd: root, env: { ...process.env, PAPERCLIP_TSC_EMIT_ONLY: "1" } });
|
||||
|
||||
expect(status).toBe(0);
|
||||
expect(existsSync(path.join(root, "dist", "a.js"))).toBe(true);
|
||||
expect(existsSync(path.join(root, "dist", "a.d.ts"))).toBe(false);
|
||||
} finally {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
it("accepts build mode with low-memory mode on", () => {
|
||||
const { status } = runTsc(["-b", "--dry"], {
|
||||
cwd: serverRoot,
|
||||
env: { ...process.env, PAPERCLIP_TYPECHECK_LOW_MEMORY: "1" },
|
||||
});
|
||||
|
||||
expect(status).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("emitOnlyTscArgs", () => {
|
||||
it("removes checks and declarations only in emit-only mode", () => {
|
||||
expect(emitOnlyTscArgs({ PAPERCLIP_TSC_EMIT_ONLY: "1" })).toEqual([
|
||||
"--noCheck",
|
||||
"--declaration",
|
||||
"false",
|
||||
"--declarationMap",
|
||||
"false",
|
||||
]);
|
||||
expect(emitOnlyTscArgs({})).toEqual([]);
|
||||
});
|
||||
|
||||
it("is what the Dockerfile selects for the server build", () => {
|
||||
const dockerfile = readFileSync(path.join(serverRoot, "..", "Dockerfile"), "utf8");
|
||||
|
||||
expect(dockerfile).toContain("RUN PAPERCLIP_TSC_EMIT_ONLY=1 pnpm --filter @paperclipai/server build\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("lowMemoryTscArgs", () => {
|
||||
it("uses one thread only when low-memory mode is on", () => {
|
||||
expect(lowMemoryTscArgs({ PAPERCLIP_TYPECHECK_LOW_MEMORY: "1" })).toEqual(["--singleThreaded"]);
|
||||
expect(lowMemoryTscArgs({ PAPERCLIP_TYPECHECK_LOW_MEMORY: "0" })).toEqual([]);
|
||||
expect(lowMemoryTscArgs({})).toEqual([]);
|
||||
});
|
||||
});
|
||||
+1
-1
@@ -20,7 +20,7 @@
|
||||
"storybook": "storybook dev -p 6006 -c storybook/.storybook",
|
||||
"build-storybook": "storybook build -c storybook/.storybook -o storybook-static",
|
||||
"preview": "vite preview",
|
||||
"typecheck": "node ../scripts/storybook-catalog.mjs && tsc -b",
|
||||
"typecheck": "node ../scripts/storybook-catalog.mjs && node ../scripts/run-tsc.mjs -b",
|
||||
"clean": "rm -rf dist dist-preview storybook-static tsconfig.tsbuildinfo",
|
||||
"prepack": "rm -f package.dev.json && cp package.json package.dev.json && node ../scripts/generate-ui-package-json.mjs",
|
||||
"postpack": "if [ -f package.dev.json ]; then mv package.dev.json package.json; fi"
|
||||
|
||||
Reference in new issue
Block a user