mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A sandbox run creates its sandbox with the provider first. Then `attachSandboxCreation` records the started sandbox as the run's `environment_leases` row in one transaction. > - That transaction locks the environment row, the creation operation row and the run row. Then the lease insert takes a foreign-key key-share lock on the parent issue (and execution workspace). > - Other writers lock the issue row first and then the run row. When such a writer holds the issue and asks for the run, a lock cycle occurs. PostgreSQL aborts the lease insert with `deadlock detected` (`40P01`). > - The transaction rolls back whole. But the run failed during setup with `Failed to acquire lease ... Failed query: insert into "environment_leases" ...`, and the sandbox that the provider had already started was discarded. The bootstrap retry then had to create a new sandbox. > - This pull request runs the `attachSandboxCreation` transaction through the shared bounded `retryTransientLockContention` helper. #22329, #21703 and #20971 use the same helper for the same kind of lock cycle. > - The benefit is that a short lock conflict no longer fails the run setup or discards a started sandbox. ## Linked Issues or Issue Description Refs #22329 Refs #21703 Refs #20971 **What happened?** On staging, a sandbox run failed during setup with `EnvironmentRunError: Failed to acquire lease for environment "..." (sandbox): Failed query: insert into "environment_leases" ...`. The cause chain was `EnvironmentProviderUnavailableError` ("a database query failed while recording the lease") and then `PostgresError: deadlock detected`. The provider had already started the sandbox (`sandboxState: started`). **What did you expect?** A lost lock race on the lease record must not fail the run. The run must keep the sandbox that the provider started and continue to its first turn. **Steps to reproduce** 1. Start a sandbox creation operation for a running heartbeat run on an issue. 2. In another transaction, lock the issue row `FOR UPDATE`. 3. Record the created sandbox lease (`environmentService(db).acquireLease({ creationOperationId, ... })`). It locks the run row and then waits for the issue key-share lock. 4. In the other transaction, lock the run row `FOR UPDATE`. 5. PostgreSQL aborts the lease insert with `40P01`. ## What Changed - `server/src/services/environments.ts`: `attachSandboxCreation` now runs its transaction through `retryTransientLockContention` (5 attempts, 100 ms × attempt backoff, only for `40P01`, `40001` and `55P03`). A fresh attempt re-reads the pending creation operation and the run row, and re-proves every check. A doc comment records the lock order and the rule that `attach` writes only through `tx`. - `server/src/services/environment-lease-attach-deadlock.integration.test.ts`: new embedded-Postgres regression that creates the real issue-then-run lock cycle, and a negative case that a deterministic failure is not retried and leaves the operation `pending`. ## Verification - Red before the fix: `npx vitest run src/services/environment-lease-attach-deadlock.integration.test.ts` failed with `PostgresError: deadlock detected` (`routine: DeadLockReport`) on the same `insert into "environment_leases"` statement as the staging event. - Green after the fix: the same file passes (2/2). The lease row is recorded once, `active`, with the provider lease id, and the creation operation is `attached` to it. - Related suites pass: `src/__tests__/sandbox-creation-operations.test.ts`, `src/__tests__/environment-task-authority.integration.test.ts`, `src/__tests__/plugin-warm-tasks.integration.test.ts`, `src/services/environment-provider-unavailable-error.test.ts` (158 tests), and `src/__tests__/environment-runtime.test.ts` (214 tests). - Not run locally: the server `tsc` typecheck. The sandbox memory limit killed it (exit 137). CI typecheck covers this revision. ## Risks - Low risk. Only a transaction that PostgreSQL already rolled back whole is retried, and the number of attempts is bounded. - Both callers of `attachSandboxCreation` pass the root database handle, and both `attach` callbacks write only through `tx`. If a future caller passes an open transaction, a retry runs inside it as a savepoint. That retry stays bounded and then throws the original error. - No change to the error classification of other lease acquisition failures. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, embedded-Postgres test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>