mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process serves every request and agent turn on one thread. The same process saves each remote workspace as one encrypted archive in object storage, and restores it before a turn > - `server/src/services/remote-workspace-snapshot-storage.ts` hashed, encrypted and decrypted every byte of those archives on that thread. `server/src/storage/s3-multipart.ts` computed the Content-MD5 of each uploaded part on that thread too > - Hashes and ciphers are synchronous in Node. A CPU profile of a busy hosted server showed this work on the main thread, which was at 93% of one core > - Each megabyte of a snapshot delays every request and turn that waits behind it > - This pull request moves those passes to a small, bounded set of worker threads. The checks, their order, their failures and the stored format do not change > - The benefit is that a 1 GiB save uses about 0.7 s of the server's thread instead of about 2.8 s, and an unrelated request during the save is answered in 1.3 ms (p99) instead of 16.7 ms ## Linked Issues or Issue Description No public issue exists. I searched open and closed pull requests for `workspace snapshot hash encrypt`, `event loop` and `archive io`. Related merged work: #19900 gave these stream passes 1 MiB buffers, #20554 removed a read-back from migration, #20722 took runner artifact hashes off the event loop, #20709 records long event-loop blocks. No open pull request covers this change. **What happened?** A 30 second CPU profile of the main thread of a hosted server (commit1d153f62b6, Node 24, main thread at 93% of one core) showed: - the two stream transforms in the snapshot save at 3.9% and 1.6% inclusive. The first hashes the archive (SHA-256). The second hashes the ciphertext (SHA-256); - native hash `update` at 5.4% self time for the whole process. About 1.0 s of it was under those two transforms and 0.25 s was under `fileRangeMd5`; - the AES-256-GCM cipher of the same pipeline, and Buffer allocation under the file read streams. Garbage collection was 25% of the thread. What the code did with each byte, on the server's thread: | Path | Pass | Digest or cipher | Purpose | | --- | --- | --- | --- | | Save | read archive | SHA-256 of plaintext | bound into the wrapped key, checked on each restore | | Save | same read | AES-256-GCM | confidentiality and authenticity | | Save | same read | SHA-256 of stored bytes | recorded on the snapshot row, checked on each restore and read-back | | Save | read each part (S3) | MD5 | `Content-MD5`: the object store rejects a part it did not receive intact | | Save, read-back callers | read the kept file | SHA-256 of plaintext | proof that the published file is the saved archive | | Restore | download | SHA-256 of stored bytes | integrity of the object against the snapshot row | | Restore | read ciphertext | AES-256-GCM | authenticity | | Restore | read plaintext | SHA-256 of plaintext | integrity against the digest in the wrapped key | | Restore | read plaintext again | none (copy) | publication beside the destination | **Expected behavior** Digest and cipher passes over a snapshot do not run on the thread that serves requests. Every check stays, with the same failure. **Steps to reproduce** 1. On the parent commit, run `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (copy the script from this branch). 2. Read `upload.mainThreadCpuMs`, `upload.mainThreadBusyMs` and `upload.unrelatedRequest`. The benchmark uses the real save and restore functions, the real S3 client, and a local object store that checks each part's `Content-MD5`. A separate process requests a trivial route from the measured server and times each reply with its own clock. **Paperclip version or commit** `master` at9e910ff9ae. The profile is from1d153f62b6. **Deployment mode** All modes that save remote workspaces. The profile is from a hosted server with S3 storage. **Node.js version** Profile: Node 24. Local benchmark and tests: Node 26.4.0 on darwin arm64. ## What Changed - `server/src/storage/file-crypto-pool.ts` (new): runs file digests and AES-256-GCM passes on worker threads. - Bounds: at most 2 jobs run at once. Each worker has two reused 1 MiB read buffers and one write in flight. A waiting job holds file names and byte ranges, not file contents. Waiting jobs are served in order. - Cancellation: a waiting job is cancelled at once. A running job stops at its next chunk, closes its files, and then the caller sees the same `AbortError` as before. - Shutdown: an idle worker does not keep the process alive. A busy worker does, until its job ends or its caller's signal aborts it. - Key handling: the pool copies the data key for the worker, moves that copy (it is not cloned), and the worker erases it. A copy that never reaches a worker is erased on this thread. The caller keeps and erases its own key as before. - The worker program is sent as source text, as `remote-workspace-snapshot-archive.ts` already does. It needs no extra file in the build or the runtime bundle. - Save (`uploadWorkspaceSnapshot`): the one read that digests the archive, encrypts it and digests the stored bytes runs in a worker. The identity check around that read (`archive_changed`) stays on this thread and is unchanged. - Part upload (`fileRangeMd5`): the `Content-MD5` of each part is computed in a worker. The first round of parts is digested before any part is sent, so the parts still leave together and a rate limit still answers one burst (the existing throttle test covers this). Every digest of that round has ended, also after a failure or a cancellation, before the upload settles. - Restore (`loadWorkspaceSnapshotArchive`): the download no longer hashes on this thread. One worker digests the downloaded object while another decrypts it into private scratch. The results are then checked in the previous order: object size and digest, header, key, GCM tag. - Restore, one pass less: the decrypted archive was read twice (digest, then copy). It is now digested while it is copied beside the destination, and linked to the destination only when the digest matches. The digest covers exactly the published bytes. - Read-back callers (`verifiedArchivePath`): the digest of the kept file runs in a worker, through the same open descriptor. The file identity checks are unchanged. - `server/scripts/benchmark-workspace-snapshot-crypto.ts` (new): the reproduction and benchmark. - Docs: `doc/run-log-events.md` (what each `storage.snapshot.*` phase now does) and `doc/DEVELOPING.md` (the worker threads and the benchmark). Module boundaries (`rules/module-boundaries.mdc`): the pool is in `server/src/storage/` because both consumers are parts of the same upload, and storage must not import from services. Its contract is file paths, byte ranges, an algorithm, and AES-GCM parameters. The snapshot format (magic bytes, additional data, key wrapping, the order of checks) stays in the snapshot service. The part layout and `Content-MD5` stay in the S3 module. Not changed, and why: - The stored format, the ledger, attempt journaling, atomic publication, and `recoverWorkspaceSnapshotUpload` / `recoverWorkspaceSnapshotArchive`. - `verifyObject` (full read-back for a store that did not verify the upload, and for a retried save) still hashes on this thread. Its bytes arrive from the network on this thread. To move it, the object must first be written to disk. - Socket writes of part bodies stay on this thread. I measured 1 MiB part-body read chunks in place of 64 KiB: main-thread time did not change and the longest event-loop delay increased, so I did not keep it. - `remote-workspace-transport.ts` also hashes an archive on this thread. It is outside the snapshot storage path. ## Verification Regression tests, red then green (`cd server && pnpm exec vitest run src/services/remote-workspace-snapshot-storage.test.ts src/storage/file-crypto-pool.test.ts`): - Parent implementation with the tests of this branch: 3 tests fail. - `keeps encrypting a save while the server thread is occupied by other work`: `expected 4194323 to be 25166144`. The test blocks the thread until the encrypted file is complete. On the parent the file stops at 4 MiB. - `answers unrelated requests while a save is being encrypted, and digests or encrypts none of it on the server thread`: `expected 8388627 to be less than 65536` (bytes hashed on the thread). - `digests and decrypts none of a save or a restore on the server thread when the store verified the upload`: `expected 33556772 to be less than 65536`. - This branch: 58 of 58 and 6 of 6 pass. - The tests use synchronization, not time. A named pipe keeps a save in the encrypt step for as long as the test feeds it. Deadlines are 30 s and matter only for a failure. - Integrity tests that pass on both the parent implementation and this branch, which shows that the failures are the same: a truncated object (`object_hash_mismatch`), an oversized object (`object_size_mismatch`), an archive that grows, shrinks or is rewritten in place after its identity was read (`archive_changed`, before any upload), a ciphertext that was changed with the recorded digest changed to match (GCM failure), a plaintext that differs from the digest in the key (`plaintext_hash_mismatch`, staged copy removed), and the order object, header, key. - Cancellation: during encryption the save rejects with `AbortError`, and leaves no scratch file, no object and no recorded attempt. During the store write it leaves no object. - Part digests (`src/__tests__/storage-s3-multipart.test.ts`): `leaves no part digest running or waiting when the source is cancelled|removed during the first round`. Before the review fix ina354242865it failed with `{ running: 2, waiting: 0 }` at multipart cleanup. - Pool tests: digests equal single-threaded digests for a file, a range and a descriptor; encrypt and decrypt round trip; each damaged ciphertext is reported; at most 2 jobs run and the others wait in order; a waiting job and a running job are both cancelled. Benchmark, same machine, 5 alternating pairs (parent implementation, then this branch) for each size. Values are medians, with the range in parentheses. Command: `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (and `--mib=200`). 1 GiB archive: | Measure | Save, before | Save, after | Restore, before | Restore, after | | --- | --- | --- | --- | --- | | Main-thread CPU time | 2,834 ms (2,717 to 3,007) | 742 ms (658 to 881) | 1,602 ms (1,378 to 1,837) | 510 ms (470 to 589) | | Main-thread busy time (event-loop utilization) | 2,938 ms | 858 ms | 1,673 ms | 597 ms | | Total process CPU time | 6,735 ms | 4,762 ms | 5,564 ms | 3,306 ms | | Wall time | 3,914 ms (3,656 to 4,253) | 2,580 ms (2,507 to 3,200) | 2,502 ms (1,756 to 2,922) | 1,799 ms (1,508 to 2,492) | | Longest event-loop delay | 12.6 ms | 6.0 ms | 10.4 ms | 5.4 ms | | Event-loop delay p99 | 9.2 ms | 2.0 ms | 2.1 ms | 1.8 ms | | Unrelated request p99 (p50 is 0.2 to 0.3 ms in all runs) | 16.7 ms | 1.3 ms | 4.7 ms | 1.3 ms | | Unrelated request, slowest | 18.5 ms | 4.5 ms | 10.0 ms | 3.5 ms | | Peak resident memory | 495 MiB | 408 MiB | 524 MiB | 476 MiB | 200 MiB archive: | Measure | Save, before | Save, after | Restore, before | Restore, after | | --- | --- | --- | --- | --- | | Main-thread CPU time | 571 ms | 181 ms | 378 ms | 141 ms | | Main-thread busy time | 602 ms | 236 ms | 395 ms | 167 ms | | Total process CPU time | 1,235 ms | 935 ms | 1,385 ms | 840 ms | | Wall time | 842 ms | 679 ms | 538 ms | 382 ms | | Longest event-loop delay | 12.4 ms | 4.0 ms | 7.3 ms | 2.9 ms | | Unrelated request p99 | 16.5 ms | 2.0 ms | 1.9 ms | 1.9 ms | | Peak resident memory | 437 MiB | 398 MiB | 510 MiB | 424 MiB | How to read these numbers: - The parent did not block the thread for seconds at one time. It worked in 1 MiB slices, so the longest single delay was about 12 ms. The cost was the sum: the thread was busy for about 75% of a 1 GiB save. That sum is what the profile showed, and it is what decreases. - A worker thread moves CPU time; it does not remove it. Total CPU time also decreased here (about 29% for a save, 41% for a restore) because the workers reuse two buffers in place of one new 1 MiB Buffer for each chunk, and a restore reads the decrypted archive once in place of twice. - Wall time decreased because the part digests no longer wait for the thread that sends the parts, and a restore digests and decrypts at the same time. - What remains on the server's thread is network and file streaming: socket writes of the parts, the download, and the database calls. - Limits: the machine was shared with other work (1-minute load average 8 to 14 on 18 cores) and timings varied between runs, which is why the pairs alternate. Earlier runs on a busier machine (load 15 to 28) showed the same CPU-time ratios with wall times up to 4 times longer for both implementations. The object store was on loopback without TLS. Other checks: - `cd server && pnpm exec vitest run src/storage src/services/remote-workspace-snapshot-storage.test.ts src/services/remote-workspace-snapshot-reuse.test.ts src/__tests__/remote-workspace-file-resources.test.ts` and the 8 `src/__tests__/storage-s3-*.test.ts` files: 249 tests in 15 files pass. - `tsc --noEmit` over the changed TypeScript files: clean. `pnpm check:module-boundaries`: pass. - The pool, bundled with the options of `server/scripts/bundle-runtime.mjs` and run with plain `node`: encrypt, decrypt and digest round trip, and the process exits. - `pnpm verify` exits 2 (broader verification required, because of the new file in `server/scripts`). Required CI supplies that coverage. - Not verified: the hosted server after deploy, Linux x86 with Node 24 outside CI, a real object store with TLS, and archives larger than 1 GiB. ## Risks - This is integrity and confidentiality code. The mitigation is that each check and its failure is pinned by a test that passes on the parent implementation and on this branch, and that the format is unchanged, so objects saved before and after this change restore with either version. - Order of work in a restore: the ciphertext is now decrypted into private scratch while the object digest is computed, not after it. No output is used before the digest, header, key and tag checks pass, and the scratch directory is removed on every failure. A damaged object is reported with the same error, but after the decrypt work. - The staged copy beside the destination (`.workspace-snapshot-*.tmp`) is now written before the plaintext digest is known, and after the GCM tag is verified. It is removed on a mismatch. The destination itself is linked only after the digest matches, as before. - Resource trade-off: up to 2 more threads, each with about 2 MiB of buffers and a small heap (64 MB limit), that stay for the life of the process once used. In exchange the server's thread does about 70% less work for each snapshot. With more than 2 snapshot passes at one time, the others wait. A part digest can wait behind a long encryption. - Run-log timing shifts: `storage.snapshot.digest` is now about 0 ms, `storage.snapshot.verify_plaintext` includes the copy, and `storage.snapshot.publish` is only the link. `doc/run-log-events.md` describes this. - A worker that stops unexpectedly fails its job with `file_crypto_worker_exited`. The save or restore then fails as for any other I/O error, and the next job starts a new worker. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with tool use (shell, file edits, local test and benchmark runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>