Files
PaperClipAI/server/scripts
DottaandClaude Fable 5.2 1af987690d perf(storage): hash and encrypt workspace snapshots off the server thread (#21138)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process serves every request and agent turn on one
thread. The same process saves each remote workspace as one encrypted
archive in object storage, and restores it before a turn
> - `server/src/services/remote-workspace-snapshot-storage.ts` hashed,
encrypted and decrypted every byte of those archives on that thread.
`server/src/storage/s3-multipart.ts` computed the Content-MD5 of each
uploaded part on that thread too
> - Hashes and ciphers are synchronous in Node. A CPU profile of a busy
hosted server showed this work on the main thread, which was at 93% of
one core
> - Each megabyte of a snapshot delays every request and turn that waits
behind it
> - This pull request moves those passes to a small, bounded set of
worker threads. The checks, their order, their failures and the stored
format do not change
> - The benefit is that a 1 GiB save uses about 0.7 s of the server's
thread instead of about 2.8 s, and an unrelated request during the save
is answered in 1.3 ms (p99) instead of 16.7 ms

## Linked Issues or Issue Description

No public issue exists. I searched open and closed pull requests for
`workspace snapshot hash encrypt`, `event loop` and `archive io`.
Related merged work: #19900 gave these stream passes 1 MiB buffers,
#20554 removed a read-back from migration, #20722 took runner artifact
hashes off the event loop, #20709 records long event-loop blocks. No
open pull request covers this change.

**What happened?**

A 30 second CPU profile of the main thread of a hosted server (commit
1d153f62b6, Node 24, main thread at 93% of one core) showed:

- the two stream transforms in the snapshot save at 3.9% and 1.6%
inclusive. The first hashes the archive (SHA-256). The second hashes the
ciphertext (SHA-256);
- native hash `update` at 5.4% self time for the whole process. About
1.0 s of it was under those two transforms and 0.25 s was under
`fileRangeMd5`;
- the AES-256-GCM cipher of the same pipeline, and Buffer allocation
under the file read streams. Garbage collection was 25% of the thread.

What the code did with each byte, on the server's thread:

| Path | Pass | Digest or cipher | Purpose |
| --- | --- | --- | --- |
| Save | read archive | SHA-256 of plaintext | bound into the wrapped
key, checked on each restore |
| Save | same read | AES-256-GCM | confidentiality and authenticity |
| Save | same read | SHA-256 of stored bytes | recorded on the snapshot
row, checked on each restore and read-back |
| Save | read each part (S3) | MD5 | `Content-MD5`: the object store
rejects a part it did not receive intact |
| Save, read-back callers | read the kept file | SHA-256 of plaintext |
proof that the published file is the saved archive |
| Restore | download | SHA-256 of stored bytes | integrity of the object
against the snapshot row |
| Restore | read ciphertext | AES-256-GCM | authenticity |
| Restore | read plaintext | SHA-256 of plaintext | integrity against
the digest in the wrapped key |
| Restore | read plaintext again | none (copy) | publication beside the
destination |

**Expected behavior**

Digest and cipher passes over a snapshot do not run on the thread that
serves requests. Every check stays, with the same failure.

**Steps to reproduce**

1. On the parent commit, run `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (copy the
script from this branch).
2. Read `upload.mainThreadCpuMs`, `upload.mainThreadBusyMs` and
`upload.unrelatedRequest`. The benchmark uses the real save and restore
functions, the real S3 client, and a local object store that checks each
part's `Content-MD5`. A separate process requests a trivial route from
the measured server and times each reply with its own clock.

**Paperclip version or commit**

`master` at 9e910ff9ae. The profile is from 1d153f62b6.

**Deployment mode**

All modes that save remote workspaces. The profile is from a hosted
server with S3 storage.

**Node.js version**

Profile: Node 24. Local benchmark and tests: Node 26.4.0 on darwin
arm64.

## What Changed

- `server/src/storage/file-crypto-pool.ts` (new): runs file digests and
AES-256-GCM passes on worker threads.
- Bounds: at most 2 jobs run at once. Each worker has two reused 1 MiB
read buffers and one write in flight. A waiting job holds file names and
byte ranges, not file contents. Waiting jobs are served in order.
- Cancellation: a waiting job is cancelled at once. A running job stops
at its next chunk, closes its files, and then the caller sees the same
`AbortError` as before.
- Shutdown: an idle worker does not keep the process alive. A busy
worker does, until its job ends or its caller's signal aborts it.
- Key handling: the pool copies the data key for the worker, moves that
copy (it is not cloned), and the worker erases it. A copy that never
reaches a worker is erased on this thread. The caller keeps and erases
its own key as before.
- The worker program is sent as source text, as
`remote-workspace-snapshot-archive.ts` already does. It needs no extra
file in the build or the runtime bundle.
- Save (`uploadWorkspaceSnapshot`): the one read that digests the
archive, encrypts it and digests the stored bytes runs in a worker. The
identity check around that read (`archive_changed`) stays on this thread
and is unchanged.
- Part upload (`fileRangeMd5`): the `Content-MD5` of each part is
computed in a worker. The first round of parts is digested before any
part is sent, so the parts still leave together and a rate limit still
answers one burst (the existing throttle test covers this). Every digest
of that round has ended, also after a failure or a cancellation, before
the upload settles.
- Restore (`loadWorkspaceSnapshotArchive`): the download no longer
hashes on this thread. One worker digests the downloaded object while
another decrypts it into private scratch. The results are then checked
in the previous order: object size and digest, header, key, GCM tag.
- Restore, one pass less: the decrypted archive was read twice (digest,
then copy). It is now digested while it is copied beside the
destination, and linked to the destination only when the digest matches.
The digest covers exactly the published bytes.
- Read-back callers (`verifiedArchivePath`): the digest of the kept file
runs in a worker, through the same open descriptor. The file identity
checks are unchanged.
- `server/scripts/benchmark-workspace-snapshot-crypto.ts` (new): the
reproduction and benchmark.
- Docs: `doc/run-log-events.md` (what each `storage.snapshot.*` phase
now does) and `doc/DEVELOPING.md` (the worker threads and the
benchmark).

Module boundaries (`rules/module-boundaries.mdc`): the pool is in
`server/src/storage/` because both consumers are parts of the same
upload, and storage must not import from services. Its contract is file
paths, byte ranges, an algorithm, and AES-GCM parameters. The snapshot
format (magic bytes, additional data, key wrapping, the order of checks)
stays in the snapshot service. The part layout and `Content-MD5` stay in
the S3 module.

Not changed, and why:

- The stored format, the ledger, attempt journaling, atomic publication,
and `recoverWorkspaceSnapshotUpload` /
`recoverWorkspaceSnapshotArchive`.
- `verifyObject` (full read-back for a store that did not verify the
upload, and for a retried save) still hashes on this thread. Its bytes
arrive from the network on this thread. To move it, the object must
first be written to disk.
- Socket writes of part bodies stay on this thread. I measured 1 MiB
part-body read chunks in place of 64 KiB: main-thread time did not
change and the longest event-loop delay increased, so I did not keep it.
- `remote-workspace-transport.ts` also hashes an archive on this thread.
It is outside the snapshot storage path.

## Verification

Regression tests, red then green (`cd server && pnpm exec vitest run
src/services/remote-workspace-snapshot-storage.test.ts
src/storage/file-crypto-pool.test.ts`):

- Parent implementation with the tests of this branch: 3 tests fail.
- `keeps encrypting a save while the server thread is occupied by other
work`: `expected 4194323 to be 25166144`. The test blocks the thread
until the encrypted file is complete. On the parent the file stops at 4
MiB.
- `answers unrelated requests while a save is being encrypted, and
digests or encrypts none of it on the server thread`: `expected 8388627
to be less than 65536` (bytes hashed on the thread).
- `digests and decrypts none of a save or a restore on the server thread
when the store verified the upload`: `expected 33556772 to be less than
65536`.
- This branch: 58 of 58 and 6 of 6 pass.
- The tests use synchronization, not time. A named pipe keeps a save in
the encrypt step for as long as the test feeds it. Deadlines are 30 s
and matter only for a failure.
- Integrity tests that pass on both the parent implementation and this
branch, which shows that the failures are the same: a truncated object
(`object_hash_mismatch`), an oversized object (`object_size_mismatch`),
an archive that grows, shrinks or is rewritten in place after its
identity was read (`archive_changed`, before any upload), a ciphertext
that was changed with the recorded digest changed to match (GCM
failure), a plaintext that differs from the digest in the key
(`plaintext_hash_mismatch`, staged copy removed), and the order object,
header, key.
- Cancellation: during encryption the save rejects with `AbortError`,
and leaves no scratch file, no object and no recorded attempt. During
the store write it leaves no object.
- Part digests (`src/__tests__/storage-s3-multipart.test.ts`): `leaves
no part digest running or waiting when the source is cancelled|removed
during the first round`. Before the review fix in a354242865 it failed
with `{ running: 2, waiting: 0 }` at multipart cleanup.
- Pool tests: digests equal single-threaded digests for a file, a range
and a descriptor; encrypt and decrypt round trip; each damaged
ciphertext is reported; at most 2 jobs run and the others wait in order;
a waiting job and a running job are both cancelled.

Benchmark, same machine, 5 alternating pairs (parent implementation,
then this branch) for each size. Values are medians, with the range in
parentheses.

Command: `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (and
`--mib=200`).

1 GiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 2,834 ms (2,717 to 3,007) | 742 ms (658 to 881)
| 1,602 ms (1,378 to 1,837) | 510 ms (470 to 589) |
| Main-thread busy time (event-loop utilization) | 2,938 ms | 858 ms |
1,673 ms | 597 ms |
| Total process CPU time | 6,735 ms | 4,762 ms | 5,564 ms | 3,306 ms |
| Wall time | 3,914 ms (3,656 to 4,253) | 2,580 ms (2,507 to 3,200) |
2,502 ms (1,756 to 2,922) | 1,799 ms (1,508 to 2,492) |
| Longest event-loop delay | 12.6 ms | 6.0 ms | 10.4 ms | 5.4 ms |
| Event-loop delay p99 | 9.2 ms | 2.0 ms | 2.1 ms | 1.8 ms |
| Unrelated request p99 (p50 is 0.2 to 0.3 ms in all runs) | 16.7 ms |
1.3 ms | 4.7 ms | 1.3 ms |
| Unrelated request, slowest | 18.5 ms | 4.5 ms | 10.0 ms | 3.5 ms |
| Peak resident memory | 495 MiB | 408 MiB | 524 MiB | 476 MiB |

200 MiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 571 ms | 181 ms | 378 ms | 141 ms |
| Main-thread busy time | 602 ms | 236 ms | 395 ms | 167 ms |
| Total process CPU time | 1,235 ms | 935 ms | 1,385 ms | 840 ms |
| Wall time | 842 ms | 679 ms | 538 ms | 382 ms |
| Longest event-loop delay | 12.4 ms | 4.0 ms | 7.3 ms | 2.9 ms |
| Unrelated request p99 | 16.5 ms | 2.0 ms | 1.9 ms | 1.9 ms |
| Peak resident memory | 437 MiB | 398 MiB | 510 MiB | 424 MiB |

How to read these numbers:

- The parent did not block the thread for seconds at one time. It worked
in 1 MiB slices, so the longest single delay was about 12 ms. The cost
was the sum: the thread was busy for about 75% of a 1 GiB save. That sum
is what the profile showed, and it is what decreases.
- A worker thread moves CPU time; it does not remove it. Total CPU time
also decreased here (about 29% for a save, 41% for a restore) because
the workers reuse two buffers in place of one new 1 MiB Buffer for each
chunk, and a restore reads the decrypted archive once in place of twice.
- Wall time decreased because the part digests no longer wait for the
thread that sends the parts, and a restore digests and decrypts at the
same time.
- What remains on the server's thread is network and file streaming:
socket writes of the parts, the download, and the database calls.
- Limits: the machine was shared with other work (1-minute load average
8 to 14 on 18 cores) and timings varied between runs, which is why the
pairs alternate. Earlier runs on a busier machine (load 15 to 28) showed
the same CPU-time ratios with wall times up to 4 times longer for both
implementations. The object store was on loopback without TLS.

Other checks:

- `cd server && pnpm exec vitest run src/storage
src/services/remote-workspace-snapshot-storage.test.ts
src/services/remote-workspace-snapshot-reuse.test.ts
src/__tests__/remote-workspace-file-resources.test.ts` and the 8
`src/__tests__/storage-s3-*.test.ts` files: 249 tests in 15 files pass.
- `tsc --noEmit` over the changed TypeScript files: clean. `pnpm
check:module-boundaries`: pass.
- The pool, bundled with the options of
`server/scripts/bundle-runtime.mjs` and run with plain `node`: encrypt,
decrypt and digest round trip, and the process exits.
- `pnpm verify` exits 2 (broader verification required, because of the
new file in `server/scripts`). Required CI supplies that coverage.
- Not verified: the hosted server after deploy, Linux x86 with Node 24
outside CI, a real object store with TLS, and archives larger than 1
GiB.

## Risks

- This is integrity and confidentiality code. The mitigation is that
each check and its failure is pinned by a test that passes on the parent
implementation and on this branch, and that the format is unchanged, so
objects saved before and after this change restore with either version.
- Order of work in a restore: the ciphertext is now decrypted into
private scratch while the object digest is computed, not after it. No
output is used before the digest, header, key and tag checks pass, and
the scratch directory is removed on every failure. A damaged object is
reported with the same error, but after the decrypt work.
- The staged copy beside the destination (`.workspace-snapshot-*.tmp`)
is now written before the plaintext digest is known, and after the GCM
tag is verified. It is removed on a mismatch. The destination itself is
linked only after the digest matches, as before.
- Resource trade-off: up to 2 more threads, each with about 2 MiB of
buffers and a small heap (64 MB limit), that stay for the life of the
process once used. In exchange the server's thread does about 70% less
work for each snapshot. With more than 2 snapshot passes at one time,
the others wait. A part digest can wait behind a long encryption.
- Run-log timing shifts: `storage.snapshot.digest` is now about 0 ms,
`storage.snapshot.verify_plaintext` includes the copy, and
`storage.snapshot.publish` is only the link. `doc/run-log-events.md`
describes this.
- A worker that stops unexpectedly fails its job with
`file_crypto_worker_exited`. The save or restore then fails as for any
other I/O error, and the next job starts a new worker.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:11:37 -06:00
..