mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute in remote sandboxes. Each run holds an
environment lease row that points to its environment, task, workspace
and company
> - When a run fails before it copies its files back, Paperclip keeps
the sandbox for a file repair. It writes a stop-and-retain intent and a
repair-hold policy on the lease
> - The recorder wrote the same lease row two times in one transaction.
Postgres runs the foreign-key checks again on a row that the current
transaction already wrote, and these checks take `FOR KEY SHARE` locks
on the referenced rows
> - Other runs lock the shared environment row `FOR UPDATE` when they
acquire a lease. Then the second write waits, meets the 1 second lock
timeout, and the run fails with "Database query failed (55P03)"
> - This pull request writes the stop intent and the repair hold in one
update
> - The benefit is that a busy shared environment no longer fails runs,
and the file repair hold is recorded
## Linked Issues or Issue Description
**What happened?**
On staging, native OpenCode runs failed with `adapter_failed` and the
error "Database query failed (55P03)". The failed statement was `update
"environment_leases" set "lease_policy" = 'retain_on_failure',
"metadata" = "metadata" || {workspaceRestoreRecovery...} where ...
"status" = 'pending_cleanup'` in
`preserveLegacyWorkspaceRestoreSources`. The cause was `canceling
statement due to lock timeout`. The statement does not wait on the lease
row, because the transaction already holds it `FOR UPDATE`. It waits on
the foreign-key check `SELECT 1 FROM ONLY "public"."environments" x
WHERE "id" = $1 FOR KEY SHARE OF x`. Postgres runs this check because
the transaction wrote the same row one statement before, in
`prepareSandboxStopAndRetain`.
**What did you expect?**
The run records its workspace restore failure and keeps its sandbox for
repair, also when another run holds the shared environment row.
**Steps to reproduce**
1. Start a run with an active, ephemeral sandbox lease.
2. In another session, lock the lease's `environments` row with `SELECT
... FOR UPDATE` and keep the transaction open.
3. Record a workspace restore failure for the run
(`terminalizeLegacyExecution`, legacy or native
`recordRestoreFailureOnly`).
4. Before this change, the second lease write fails after 1 second with
55P03. The new test
`server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts`
shows this for legacy and native runs.
No public issue exists. I searched open and recent pull requests for
this lock timeout and found no duplicate. #21071 bounds a different lock
wait (workspace source locks in run preparation) and does not overlap.
## What Changed
- `server/src/services/sandbox-stop-and-retain.ts`:
`prepareSandboxStopAndRetain` accepts an optional `retention` argument
(`leasePolicy` and extra `metadata`). The stop-and-retain update sets
these values in the same statement. Other callers do not change.
- `server/src/services/legacy-workspace-restore-recovery.ts`: the
active-source path passes the repair hold (`retain_on_failure` and
`workspaceRestoreRecovery` with `originalLeasePolicy`) to that update.
It no longer does a second update on the same row.
-
`server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts`:
a regression test for legacy and native runs. Another session holds the
environment row `FOR UPDATE` while the run records its restore failure.
The test checks the run's recovery evidence and the lease policy,
status, stop intent and repair metadata.
## Verification
- `cd server && npx vitest run
src/__tests__/legacy-workspace-restore-lock-contention.test.ts`: both
cases fail on `master` with `canceling statement due to lock timeout` on
the `environments` `FOR KEY SHARE` check. Both cases pass with this
change.
- `cd server && npx vitest run
src/__tests__/legacy-workspace-restore-recovery.test.ts`: 73 existing
tests pass.
- Related suites pass (10 files, 485 tests):
`heartbeat-observed-workspace-preservation`,
`heartbeat-native-remote-failure-retention`, `live-recovery-work`,
`workspace-recovery-stop-proof`, `native-failed-workspace-recovery`,
`native-cancelled-workspace-preservation`,
`workspace-recovery-admission`, `native-workspace-reconstruction`,
`environment-retained-orphan-sandbox`, `abandoned-workspace-files-hold`.
- I did not complete a local server typecheck. The process ran out of
memory in the 10 GB sandbox. CI runs the typecheck on this head.
## Risks
- Low risk. The lease ends with the same status, policy and metadata as
before. The values are written in one statement instead of two. The
update keeps its ownership guards (status, provider, provider lease ID
and exact metadata).
- Other code paths can still write a lease row two times in one
transaction (for example the shutdown settlement path, which sets a
lease back to `active` before it records the restore failure). That path
handles 55P03 as a lock timeout and retries. This pull request does not
change it.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, code execution, embedded Postgres tests, and
GitHub tools).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>