mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent keeps its own files (AGENT_HOME). A warm native session saves these files at the end of each turn (an agent-file checkpoint) > - Each checkpoint took the agent row lock (`agents ... FOR UPDATE`) on the small file-storage pool, also when the run changed no file > - A checkpoint that changed files also uploaded the new bytes to object storage while it held that lock. Instruction edits and other runs of the same agent waited for the upload > - With many agents, these long transactions use the 2-connection file-storage pool and slow every agent > - This pull request removes the lock from an unchanged checkpoint and moves the uploads of a changed checkpoint before the lock > - The benefit is that an unchanged checkpoint takes no row lock, and the row lock is no longer held across object-storage I/O ## Linked Issues or Issue Description **What existing behavior does this improve?** The warm agent-file checkpoint: `checkpoint()` in `server/src/services/agent-directory-working-copies.ts`, `agentFileStore.locked` and `apply` in `agent-file-store.ts`, and `persistDurableAgentFiles` in `agent-file-durability.ts`. **Current behavior** - When the run changed no file, the checkpoint still ran `store.locked(..., write = true)`. That transaction locks the agent row (`FOR UPDATE` and `FOR NO KEY UPDATE`), restores and invalidates the local projection, walks the canonical tree, and compares the durable head. Its only result is the authorization check. - When the run changed files, `persistDurableAgentFiles` uploaded each new file to object storage inside the transaction that holds the agent row lock. **Proposed behavior** - An unchanged checkpoint (`saveUnchanged`) first reads, without a lock, whether the locked file-tree step would change anything (`agentFileProjectionCurrent`): the durable head exists and the local projection is marked at its generation, or durable files are off, no head exists and the local adoption is recorded. If so, it runs only the write authorization (`authorizeSave`): the agent interaction policy and `authorizeInstructionCommit`, in a short transaction without row locks that always rolls back, so it writes nothing. If not, it runs the locked save as on `origin/master`, which adopts, restores or heals the durable head. A de-authorized agent still gets the same error (the existing test "rechecks authorization even for an unchanged warm checkpoint" still passes). - A changed checkpoint runs the same read-only authorization (it consumes no accepted change confirmation), then uploads the run's changed files before it takes the lock (`uploadAgentFilesBeforeCommit`). A run that lost its save permission uploads nothing. The locked commit runs the authorization again, and only that commit consumes a confirmation. The locked commit uses an uploaded object only for a file with the same SHA-256 and size. It uploads any other file itself, as before. **Reason and benefit** The target is 1,000 agents in one controller process. Most warm turns change no agent file. These turns now take no row lock and send fewer statements. A changed turn holds the row lock for less time, because object-storage latency is outside the lock. ## What Changed - `agent-file-store.ts`: add `authorizeSave(companyId, agentId, actor)` and `saveUnchanged(companyId, agentId, actor)`. `apply` calls it for a checkpoint, then uploads the run's changed files before `locked`, and passes the uploaded objects to `persistDurableAgentFiles`. `locked` takes an optional map of uploaded objects. - `agent-file-durability.ts`: move the verified upload stream into `uploadAgentFile`. Add `uploadAgentFilesBeforeCommit`. It reads the durable head without a lock, skips files the head already has at the same path and digest, and uploads the others. If an upload fails, it stops and the locked commit uploads the rest, so errors are the same as before. `persistDurableAgentFiles` uses an uploaded object when the canonical file has the same digest and size. - `agent-directory-working-copies.ts`: an unchanged checkpoint calls `saveUnchanged` instead of `store.locked(..., true)`. - `agent-file-durability.ts`: add `agentFileProjectionCurrent` (one head read and two file checks, no lock). - The commit is still inside the agent row lock. The head cannot change between the read and the write of the commit, so it needs no separate compare-and-set. The upload before the lock does not need one either: an object is immutable and verified by digest, and the commit records it only for a file with that digest. ## Verification - New `server/src/__tests__/agent-file-checkpoint-locking.test.ts` (embedded PostgreSQL, object-storage fixture). All tests fail on the code before the change and pass with it: - An unchanged checkpoint completes while another transaction holds the agent row lock. It sends no `FOR UPDATE`, `FOR NO KEY UPDATE` or `FOR SHARE` statement and only one unlocked read of `agent_file_roots`. - Durable files on, the agent has local files and no head: an unchanged checkpoint creates the head, as on `origin/master` (added after review; failed without the fallback). - The local projection is not marked at the head: an unchanged checkpoint takes the locked path and marks it again (added after review; failed without the fallback). - A changed checkpoint uploads its file while the agent row is not locked (the fixture checks the lock with `FOR UPDATE NOWAIT` from another connection during each upload). The head then records that object. - A run loses its save permission before a changed checkpoint. The checkpoint is refused and uploads nothing (added after review; I saw it fail without the authorization before the upload). - An agent with `agents:suggest-changes` and one accepted change confirmation saves a changed checkpoint. An unchanged checkpoint first leaves the confirmation unconsumed. The changed save succeeds once and consumes the confirmation once; the next change is refused before it uploads (added after review; I saw it fail when the preview committed). - Two warm runs of the same agent save at the same time. Both files are in the canonical tree and in the head, and every head object has the correct digest. - An instruction edit commits while a checkpoint waits in its upload, without a wait for the checkpoint. Then the checkpoint commits. Both changes are in the head. - A save stops between upload and commit (the instruction entry changes while the bytes upload, so the commit refuses). The head does not change and stays restorable. The next save commits the same file. The unused object is never authoritative; the existing object collector removes it after its grace period. - Existing `agent-directory-working-copies.test.ts`, `agent-file-durability.test.ts` and `agent-file-checkpoints.test.ts` pass. - Measurement (opt-in, `PAPERCLIP_AGENT_FILE_CHECKPOINT_BENCHMARK=1`, test "measures checkpoint statements and agent row lock time"): one warm copy with 51 files; 10 unchanged checkpoints; then 5 checkpoints that each change one file, with 250 ms object-store latency. Wall times are noisy (load average about 40). Statement counts are exact. | | Before (`origin/master`) | After (final head) | | --- | --- | --- | | Unchanged checkpoint: statements | 26 | 25 (one unlocked head read, and the read-only authorization) | | Unchanged checkpoint: row-lock statements | 2 | 0 | | Unchanged checkpoint: median time | 214 ms | 26–126 ms | | Changed checkpoint: statements | 29 | 47 (the read-only authorization before the upload and one head read) | | Changed checkpoint: median time the agent row is locked | 924 ms | 51–227 ms | | Changed checkpoint: uploads while the agent row is locked | 1 | 0 | The "after" ranges come from two runs at different machine load (load average between about 18 and 40). ## Risks - Master side effects of the locked unchanged save, and what this change does with each: - Kept: the write authorization (`assertAgentInteraction`, `authorizeInstructionCommit`), so a run that lost its permission still gets the same error. - Kept through the fallback: one-time adoption (no head yet), restore of a projection that is not marked at the head, re-mark after a failed persist (a write path removes the marker before it changes files), and heal of a head that lags the root after such a failure. `agentFileProjectionCurrent` returns false in each of these cases, so the locked save runs. - Dropped when the projection is current: the re-mark at the same generation (no change), and the compare of the local tree with the head. With the marker at the head generation, every writer that changed the tree went through the locked save and moved the marker, so the compare finds no difference. An edit made outside Paperclip directly in the canonical directory is not picked up by an unchanged checkpoint any more; the next save that goes through the locked path still records it. - Dropped when the projection is current: the storage-limit check of the canonical tree in `persistDurableAgentFiles`. An over-quota agent whose run changed nothing no longer gets an error from that check; any save that changes files still enforces the limits. - Dropped: the `FOR NO KEY UPDATE` locks that `assertAgentInteraction` takes on the actor and target agents (`lock: "update"`). Those locks hold permission changes off a write until it commits. The read-only authorization writes nothing, so there is nothing to hold them off. The locked commit of a changed save still takes them. An unchanged save has no mention texts, so master locked no other agents for it. - Effects of `authorizeSave` outside the rolled-back transaction: none. I read `assertAgentInteraction` (`agentInteractionDenial`: reads only), `authorizeInstructionCommit`, `resolveInstructionActor` (reads only), `authorizationService.decide` (reads only; its memo maps live on the bound actor and on the service instance, both created for this call and dropped after it), and `changeConsentGateService.assertConsented` (one `UPDATE` of the confirmation row, rolled back). None of them publishes live events, queues notifications or wakes, writes activity through another connection, makes external calls, or changes an in-process cache that outlives the call. - Behavior change: on `origin/master`, an unchanged checkpoint of a consent-gated agent (`agents:suggest-changes`) consumed an accepted change confirmation, because the locked transaction committed. Now the fast path does not; only the commit of a changed save consumes it. When the fast path cannot be used (adoption, restore or heal), the locked save runs as on master and can still consume a confirmation. - Not in this change: the manifest is still stored twice in the receipt (`baseline` and `checkpointManifest`). Servers of the previous version read `receipt.baseline` during a deploy overlap, and the next capture needs the stamps in `checkpointManifest`, so neither can be removed safely now. - Not in this change: on the volume-free server, `serial()` still opens a dedicated connection with an advisory-lock transaction for each checkpoint. That lock serializes controllers and is not a row lock; changing it is a separate design decision. - A changed checkpoint can now leave an unused object in storage when its commit is refused after the upload (for example, the instruction entry changes during the upload) or the process stops before the commit. This was already possible for a failed commit. Such an object is never in the durable head, so no reader can reach it, and the existing `collectAgentFileObjects` maintenance removes it after `AGENT_FILE_OBJECT_GRACE_MS`. A permission that was removed before the save stops the save before any upload. - Old and new servers can run at the same time: the head format and the receipt format do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>