mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Durable agent files live in object storage. Each server keeps a local copy (the projection), marked with the head generation it matches > - When the projection is stale, the next locked read or write restores it: it downloads every file from object storage. This happens for the first run of each agent on a new server (every deploy on hosted ephemeral servers) and after another server wrote > - The download ran inside the transaction that holds the `agents` row lock (`FOR UPDATE`), on the main pool for the run-start read (`readCommittedForRuntime`) and on the 2-connection file-storage pool for the store. A download can take up to 30 s > - After a deploy at 1,000 agents, this is one long row lock and one pinned connection per agent, all at once, and every writer of that agent waits behind it > - This pull request downloads the files outside the transaction and takes the lock again only to swap them in > - The benefit is that the row lock is held for milliseconds instead of the download time (0 ms observed instead of 321 ms with a 300 ms download in the test), and writers do not wait for a download ## Linked Issues or Issue Description **What existing behavior does this improve?** The restore of a stale local copy of an agent's durable files (the first run of each agent on a new server, or after another server wrote). **Subsystem affected** Server: durable agent files (`agent-file-durability.ts`, `agent-file-store.ts`, `agent-instruction-revisions.ts`). **Current behavior** The restore downloads every file from object storage inside the transaction that holds the `agents` row lock (`FOR UPDATE`) and a pooled connection. A download can take up to 30 s, and every writer of that agent waits behind it. **Proposed behavior** Under the lock, detect that a download is needed and roll back. Download the head's files, verified by digest, into a private staging directory with no lock. Run the transaction again, and swap the staged files in only when the head generation is unchanged. After two changed heads, restore under the lock as before. **Reason and benefit** After a deploy at 1,000 agents, every agent's first run restores at once. The row lock and connection are now held for milliseconds instead of the download time, and writers do not wait for a download. **Additional context** Refs the 1,000-concurrent-agents plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`). The save path had the same problem for uploads and was fixed earlier with uploads before the lock (`uploadAgentFilesBeforeCommit`); this is the restore side. Reading the head before every transaction was considered; it adds a query to every call, while the deferral costs nothing when the projection is current. ## What Changed - `agent-file-durability.ts`: the download loop moves into `downloadAgentFiles`, unchanged. `withAgentFilesRestoredBeforeLock(work)` runs `work` in a scope. Inside the scope, the first restore of an attempt that needs a download throws a private deferral error, which rolls back the transaction. The wrapper stages the download in `.agent-files-<uuid>` next to the root and runs `work` again. `restoreDurableAgentFiles` uses a staged directory only when its generation equals the head it reads under the lock. Staged directories are removed when the scope ends. - Only the first restore of each attempt can defer, so nested restores of other agents keep their old behavior. Outside the scope, the behavior is unchanged. - `agent-file-store.ts` (`locked`, so every store read, write, apply and working-copy preparation) and `agent-instruction-revisions.ts` (`readCommittedForRuntime`, `readCurrent`, `materializeCurrent`) run their transactions in the scope. ## Verification - New test `server/src/__tests__/agent-file-restore-locking.test.ts` on embedded PostgreSQL, with an object-storage fixture: - Hold time, with a 300 ms download, through `readCommittedForRuntime` (the run-start read). Before: the download ran with the agent row locked, and the row was locked 321 ms of 322 ms. After: the download ran with the row unlocked, and a sampler saw the row locked for 0 ms. - Concurrency: a board write of the same file runs while the reader's download is blocked. Before: the writer waited for the reader's download (5 s timeout). After: the writer commits a new head during the download. The reader then discards its stale staged copy and returns the writer's content. The projection is marked at the writer's generation, and no staging directory is left. - Failure: when object storage fails, the read fails once with the storage error, after one attempt made without the row lock, and leaves no staging directory. - Progress: when the head generation changes during each of two downloads, the third attempt restores under the lock as before (3 downloads), and the projection matches the final head. - `npx vitest run` for agent-file-checkpoint-locking, agent-file-checkpoints, agent-file-durability, agent-file-restore-locking, agent-instruction-revisions, agent-instruction-working-copies, agent-instructions-service, agent-directory-working-copies and agent-instructions-routes: 287 passed, 1 skipped. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - A deferred attempt runs the transaction body again. The deferral happens at the first restore, before the body writes files, and the transaction rollback undoes database work, so the second attempt starts from the same state. - A staged download uses disk next to the agent root until the scope ends, the same as the temporary directory of the old download. - If the head changes during every download, the third attempt downloads under the lock as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Server Tests
Server tests that need a real PostgreSQL process must use
./helpers/embedded-postgres.ts instead of constructing embedded-postgres
directly.
The shared helper creates a throwaway data directory and a reserved-safe loopback port for each test database. This protects the live Paperclip control-plane Postgres from server vitest runs; see PAP-2033 for the incident that introduced this guard.