mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
93faca77ef
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When the control plane restarts, an interrupted run must continue on its own. A user must not have to press Retry because a deploy happened > - A graceful drain (SIGTERM) marks a preparing legacy run `interrupted` and exits, but the run keeps its legacy controller lease for up to 60 s more > - After the restart, `settleUnrecoverableExecutions` skips the run's recovery hold while that lease is live, in case a living controller is still unwinding a Stop > - The controller is the drained process, which is gone. For up to a minute, the task shows "Couldn't finish this attempt" with a Retry button, before the automatic continuation starts > - This pull request lets the disposition proceed when the shared owner prover proves the lease's recorded controller process gone, and expires that exact lease, so the held-preparation continuation also starts at once > - The benefit is that a run drained during workspace preparation continues automatically right after the restart, without a Retry click ## Linked Issues or Issue Description **Subsystem affected** Server execution recovery: the disposition of `legacy_execution_requires_reconciliation` holds (`server/src/services/execution-recovery-resolution.ts`). **Problem or motivation** The restart-chaos harness can now restart the server while it is in an S3 transfer (#20451). The wave ran on Daytona with `--storage s3 --storage-trigger snapshot-download`. A K2 drain interrupted a C1 follow-up run while its workspace snapshot was downloading, then a second K2 drain interrupted the automatic retry during its restore. The task then showed the board-owned "Couldn't finish this attempt. Your message is saved. Retry" state, and the harness had to click Retry (`R3:retry_needed:C1:K2`). A `--keep` replay reproduced it. The kept database shows a run drained at 07:08:05 with `controller_lease_expires_at` 07:09:04.98. The server was ready again at 07:08:45. Its disposition did not settle the hold until 07:09:14.8. The automatic continuation was then delegated, because the user's Retry had already started a run. **Proposed solution** Before the disposition transaction, ask the shared owner prover (`legacyControllerProvenGone`) whether the live lease's recorded controller process is gone. For that exact controller boot id only, skip the wait and expire the lease in the same transaction (compare-and-swap on run, company and boot id). The later checks (`isInterruptedNativePreparation`, `isFailedNativePreparation`, the held-preparation settlement) then see an expired lease and schedule the `interrupted_preparation` successor. A lease held by a live process, including this one, still blocks as before. **Alternatives considered** Expiring the lease in the drain itself. The drain runs before the process exits, so the owner proof after the restart is the safer evidence. ## What Changed - `legacy-controller-lease.ts`: new `provenGoneLegacyControllerBootId(db, companyId, runId)`. It returns the boot id of a legacy run's unexpired controller lease when the shared prover proves that controller's recorded process gone, else null. - `execution-recovery-resolution.ts`: `settleUnrecoverableExecutions` computes that boot id before its transaction. When the live lease's boot id matches, it expires that lease and settles the hold instead of waiting. - New `execution-recovery-dead-controller.test.ts` (embedded Postgres): a drained run whose controller process has exited is settled at once, its lease is expired, and `isInterruptedNativePreparation` turns true (continuation eligible) while the original lease would still run for about 55 s. A lease held by the current process still waits and keeps its expiry. ## Verification - Red/green: `npx vitest run src/services/execution-recovery-dead-controller.test.ts` fails on master (the hold stays `active`) and passes with the fix. - Related suites pass: `task-retry`, `execution-control-work`, `heartbeat-process-recovery` and `native-startup-retry.integration` with the new file (5 files, 573 tests). - Live: the Daytona `--storage s3 --storage-trigger snapshot-download` rerun on a build with this fix will be posted as a comment. ## Risks - Low. The change only removes a wait when the shared owner prover returns `gone` for the exact recorded controller. An unprovable or alive controller keeps the old behaviour. The prover runs outside the row-locked transaction, and only for candidates whose lease is still live. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>