Files
PaperClipAI/server
DottaandClaude Opus 5.5 a48eaaef57 perf(server): skip the agent row lock for unchanged agent-file checkpoints and upload before the lock (#21457)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent keeps its own files (AGENT_HOME). A warm native session
saves these files at the end of each turn (an agent-file checkpoint)
> - Each checkpoint took the agent row lock (`agents ... FOR UPDATE`) on
the small file-storage pool, also when the run changed no file
> - A checkpoint that changed files also uploaded the new bytes to
object storage while it held that lock. Instruction edits and other runs
of the same agent waited for the upload
> - With many agents, these long transactions use the 2-connection
file-storage pool and slow every agent
> - This pull request removes the lock from an unchanged checkpoint and
moves the uploads of a changed checkpoint before the lock
> - The benefit is that an unchanged checkpoint takes no row lock, and
the row lock is no longer held across object-storage I/O

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The warm agent-file checkpoint: `checkpoint()` in
`server/src/services/agent-directory-working-copies.ts`,
`agentFileStore.locked` and `apply` in `agent-file-store.ts`, and
`persistDurableAgentFiles` in `agent-file-durability.ts`.

**Current behavior**
- When the run changed no file, the checkpoint still ran
`store.locked(..., write = true)`. That transaction locks the agent row
(`FOR UPDATE` and `FOR NO KEY UPDATE`), restores and invalidates the
local projection, walks the canonical tree, and compares the durable
head. Its only result is the authorization check.
- When the run changed files, `persistDurableAgentFiles` uploaded each
new file to object storage inside the transaction that holds the agent
row lock.

**Proposed behavior**
- An unchanged checkpoint (`saveUnchanged`) first reads, without a lock,
whether the locked file-tree step would change anything
(`agentFileProjectionCurrent`): the durable head exists and the local
projection is marked at its generation, or durable files are off, no
head exists and the local adoption is recorded. If so, it runs only the
write authorization (`authorizeSave`): the agent interaction policy and
`authorizeInstructionCommit`, in a short transaction without row locks
that always rolls back, so it writes nothing. If not, it runs the locked
save as on `origin/master`, which adopts, restores or heals the durable
head. A de-authorized agent still gets the same error (the existing test
"rechecks authorization even for an unchanged warm checkpoint" still
passes).
- A changed checkpoint runs the same read-only authorization (it
consumes no accepted change confirmation), then uploads the run's
changed files before it takes the lock (`uploadAgentFilesBeforeCommit`).
A run that lost its save permission uploads nothing. The locked commit
runs the authorization again, and only that commit consumes a
confirmation. The locked commit uses an uploaded object only for a file
with the same SHA-256 and size. It uploads any other file itself, as
before.

**Reason and benefit**
The target is 1,000 agents in one controller process. Most warm turns
change no agent file. These turns now take no row lock and send fewer
statements. A changed turn holds the row lock for less time, because
object-storage latency is outside the lock.

## What Changed

- `agent-file-store.ts`: add `authorizeSave(companyId, agentId, actor)`
and `saveUnchanged(companyId, agentId, actor)`. `apply` calls it for a
checkpoint, then uploads the run's changed files before `locked`, and
passes the uploaded objects to `persistDurableAgentFiles`. `locked`
takes an optional map of uploaded objects.
- `agent-file-durability.ts`: move the verified upload stream into
`uploadAgentFile`. Add `uploadAgentFilesBeforeCommit`. It reads the
durable head without a lock, skips files the head already has at the
same path and digest, and uploads the others. If an upload fails, it
stops and the locked commit uploads the rest, so errors are the same as
before. `persistDurableAgentFiles` uses an uploaded object when the
canonical file has the same digest and size.
- `agent-directory-working-copies.ts`: an unchanged checkpoint calls
`saveUnchanged` instead of `store.locked(..., true)`.
- `agent-file-durability.ts`: add `agentFileProjectionCurrent` (one head
read and two file checks, no lock).
- The commit is still inside the agent row lock. The head cannot change
between the read and the write of the commit, so it needs no separate
compare-and-set. The upload before the lock does not need one either: an
object is immutable and verified by digest, and the commit records it
only for a file with that digest.

## Verification

- New `server/src/__tests__/agent-file-checkpoint-locking.test.ts`
(embedded PostgreSQL, object-storage fixture). All tests fail on the
code before the change and pass with it:
- An unchanged checkpoint completes while another transaction holds the
agent row lock. It sends no `FOR UPDATE`, `FOR NO KEY UPDATE` or `FOR
SHARE` statement and only one unlocked read of `agent_file_roots`.
- Durable files on, the agent has local files and no head: an unchanged
checkpoint creates the head, as on `origin/master` (added after review;
failed without the fallback).
- The local projection is not marked at the head: an unchanged
checkpoint takes the locked path and marks it again (added after review;
failed without the fallback).
- A changed checkpoint uploads its file while the agent row is not
locked (the fixture checks the lock with `FOR UPDATE NOWAIT` from
another connection during each upload). The head then records that
object.
- A run loses its save permission before a changed checkpoint. The
checkpoint is refused and uploads nothing (added after review; I saw it
fail without the authorization before the upload).
- An agent with `agents:suggest-changes` and one accepted change
confirmation saves a changed checkpoint. An unchanged checkpoint first
leaves the confirmation unconsumed. The changed save succeeds once and
consumes the confirmation once; the next change is refused before it
uploads (added after review; I saw it fail when the preview committed).
- Two warm runs of the same agent save at the same time. Both files are
in the canonical tree and in the head, and every head object has the
correct digest.
- An instruction edit commits while a checkpoint waits in its upload,
without a wait for the checkpoint. Then the checkpoint commits. Both
changes are in the head.
- A save stops between upload and commit (the instruction entry changes
while the bytes upload, so the commit refuses). The head does not change
and stays restorable. The next save commits the same file. The unused
object is never authoritative; the existing object collector removes it
after its grace period.
- Existing `agent-directory-working-copies.test.ts`,
`agent-file-durability.test.ts` and `agent-file-checkpoints.test.ts`
pass.
- Measurement (opt-in, `PAPERCLIP_AGENT_FILE_CHECKPOINT_BENCHMARK=1`,
test "measures checkpoint statements and agent row lock time"): one warm
copy with 51 files; 10 unchanged checkpoints; then 5 checkpoints that
each change one file, with 250 ms object-store latency. Wall times are
noisy (load average about 40). Statement counts are exact.

| | Before (`origin/master`) | After (final head) |
| --- | --- | --- |
| Unchanged checkpoint: statements | 26 | 25 (one unlocked head read,
and the read-only authorization) |
| Unchanged checkpoint: row-lock statements | 2 | 0 |
| Unchanged checkpoint: median time | 214 ms | 26–126 ms |
| Changed checkpoint: statements | 29 | 47 (the read-only authorization
before the upload and one head read) |
| Changed checkpoint: median time the agent row is locked | 924 ms |
51–227 ms |
| Changed checkpoint: uploads while the agent row is locked | 1 | 0 |

The "after" ranges come from two runs at different machine load (load
average between about 18 and 40).

## Risks

- Master side effects of the locked unchanged save, and what this change
does with each:
- Kept: the write authorization (`assertAgentInteraction`,
`authorizeInstructionCommit`), so a run that lost its permission still
gets the same error.
- Kept through the fallback: one-time adoption (no head yet), restore of
a projection that is not marked at the head, re-mark after a failed
persist (a write path removes the marker before it changes files), and
heal of a head that lags the root after such a failure.
`agentFileProjectionCurrent` returns false in each of these cases, so
the locked save runs.
- Dropped when the projection is current: the re-mark at the same
generation (no change), and the compare of the local tree with the head.
With the marker at the head generation, every writer that changed the
tree went through the locked save and moved the marker, so the compare
finds no difference. An edit made outside Paperclip directly in the
canonical directory is not picked up by an unchanged checkpoint any
more; the next save that goes through the locked path still records it.
- Dropped when the projection is current: the storage-limit check of the
canonical tree in `persistDurableAgentFiles`. An over-quota agent whose
run changed nothing no longer gets an error from that check; any save
that changes files still enforces the limits.
- Dropped: the `FOR NO KEY UPDATE` locks that `assertAgentInteraction`
takes on the actor and target agents (`lock: "update"`). Those locks
hold permission changes off a write until it commits. The read-only
authorization writes nothing, so there is nothing to hold them off. The
locked commit of a changed save still takes them. An unchanged save has
no mention texts, so master locked no other agents for it.
- Effects of `authorizeSave` outside the rolled-back transaction: none.
I read `assertAgentInteraction` (`agentInteractionDenial`: reads only),
`authorizeInstructionCommit`, `resolveInstructionActor` (reads only),
`authorizationService.decide` (reads only; its memo maps live on the
bound actor and on the service instance, both created for this call and
dropped after it), and `changeConsentGateService.assertConsented` (one
`UPDATE` of the confirmation row, rolled back). None of them publishes
live events, queues notifications or wakes, writes activity through
another connection, makes external calls, or changes an in-process cache
that outlives the call.
- Behavior change: on `origin/master`, an unchanged checkpoint of a
consent-gated agent (`agents:suggest-changes`) consumed an accepted
change confirmation, because the locked transaction committed. Now the
fast path does not; only the commit of a changed save consumes it. When
the fast path cannot be used (adoption, restore or heal), the locked
save runs as on master and can still consume a confirmation.

- Not in this change: the manifest is still stored twice in the receipt
(`baseline` and `checkpointManifest`). Servers of the previous version
read `receipt.baseline` during a deploy overlap, and the next capture
needs the stamps in `checkpointManifest`, so neither can be removed
safely now.
- Not in this change: on the volume-free server, `serial()` still opens
a dedicated connection with an advisory-lock transaction for each
checkpoint. That lock serializes controllers and is not a row lock;
changing it is a separate design decision.
- A changed checkpoint can now leave an unused object in storage when
its commit is refused after the upload (for example, the instruction
entry changes during the upload) or the process stops before the commit.
This was already possible for a failed commit. Such an object is never
in the durable head, so no reader can reach it, and the existing
`collectAgentFileObjects` maintenance removes it after
`AGENT_FILE_OBJECT_GRACE_MS`. A permission that was removed before the
save stops the save before any upload.
- Old and new servers can run at the same time: the head format and the
receipt format do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:41:46 -06:00
..