mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
## Thinking Path - Restarts are product failures: a task interrupted by a restart must resume on its own, never wait for a user Retry. - A run interrupted during preparation, before any provider work, is safe to replay; the agent never touched the workspace. Paperclip already has the proof (`isInterruptedNativePreparation`) and the settle (`settleInterruptedNativeBootstrap`) that replay such a run in the uncharged interrupted-preparation lane. - But a graceful shutdown terminalizes that run inside the draining process, where the proof cannot hold yet: the owning controller is still that process, and its sandbox lease may still be cleaning up. So recovery records `preserve_without_replay_v1` with replay blocked. - When the proof becomes available (the next process boots, and the lease is released with a confirmed stop), nothing re-judges the hold. The only re-evaluation point runs after a pending-cleanup sweep, so the task waits for a manual Retry. - The coordinator set the rule: replay automatically with durable proof of no provider work; anything uncertain stays held. This PR adds the re-evaluation point and no new proof logic. ## Linked Issues or Issue Description Refs #20048 **What happened?** In Daytona warm wave w-20261108T142906-s9501, task t05: scheduled retry c99b3a2e was 2 s into preparation (legacy, stage `preparing`, no runtime mode, no events, no finalization) when the K2 graceful shutdown interrupted it. Terminalization in the draining process recorded `legacy_execution_requires_reconciliation` with `preserve_without_replay_v1` and replay blocked, while its lease was still `pending_cleanup`. The lease was released with a stop receipt 2 s later, and the next process started, but the hold was never re-judged. The harness clicked Retry at 14:44:27. **Steps to reproduce** 1. Start a run that is still in preparation (legacy, `preparing`, adapter `paperclip_runner`). 2. Gracefully shut down the server; terminalization records a preserve-without-replay hold. 3. Let its lease release with a confirmed stop, and restart the server. 4. The hold stays `blocked`; the task needs a manual Retry. **Expected behavior** Once the owning controller is proven gone and every lease is released with a confirmed stop, the ownership reconciler re-judges the hold with the existing proof. With no provider work and every cleanup confirmed, the hold is settled and the preparation is replayed with no attempt charged. Possible provider work, a live controller, or an unfinished cleanup keeps the hold. ## What Changed - New ownership kind `legacy_interrupted_preparation_hold`, in `server/src/services/interrupted-preparation-hold-settlement.ts` and registered in `recovery/ownership-registry.ts`. - **Listing:** legacy runs that are `interrupted` or `failed`, in stage `preparing` with no `runtimeModeResolvedAt`. Each must have a system-recorded `legacy-execution:<run>` hold: resolved/blocked, `preserve_without_replay_v1`, replay blocked, no owning user, and none of `ownerGoneAttestation`, `explicitUserContinuation` or `executionReconciliation`. Every lease must be released with no failed cleanup, so a lease that settles later is listed on the next pass. - **Owner:** the run's recorded controller (`legacyControllerOwnerStamp`). The shared reconciler settles only after `proveOwnerGone` proves it gone. A missing stamp, or one from another host or PID space, is unprovable and stays held. There is no age fallback. - **Settle:** re-checks the existing durable proof (`isInterruptedNativePreparation`: no coordinator, result or execution events; leases released with stop receipts; workspace repair when needed). It then resolves the hold through `settleInterruptedNativeBootstrap` as `false_positive` with replay allowed and continuation pending, delivered through the existing workspace-recovery queue in the uncharged interrupted-preparation lane. Its row locks make the transition happen once; the settle reports settled only when it resolved holds. - `server/src/services/legacy-execution-recovery.ts`: `settleInterruptedNativeBootstrap` takes an optional `proof`, recorded in the `issue.recovery_action_resolved` activity details as `ownerProof`, and an `onHoldsResolved` callback. - `server/src/__tests__/interrupted-preparation-hold-settle.test.ts`, K2-during-preparation shape, run through the reconciliation pass: - settled with a dead controller in this PID space, with the audit entry naming `pid_dead`; - kept on a `turn.started` event, a stamp from another process space, a missing stamp, the current process, a lease still cleaning up, an operator owner-gone attestation, and a user continuation; - two concurrent settles produce one resolution. - `ownership-reconciliation-report.test.ts`: adds the kind to the list of settling kinds. **Previous-release state:** holds written by the previous release (`preserve_without_replay_v1`, replay `blocked`) are what this re-judges, through the same proof. A run without a recorded controller stamp stays held. No saved state changes shape. ## Verification - `cd server && npx vitest run src/__tests__/interrupted-preparation-hold-settle.test.ts`: 9/9 pass. Removing the kind registration makes the settle case fail. - The ownership, retained-orphan and recovery suites (19 files) pass, after adding the kind to the report's settling-kinds list. - Not yet covered: the Daytona warm wave rerun, on measure/remote-warm with this change. ## Risks - **Replaying work that already ran.** Settlement requires both the shared owner proof and the existing durable no-provider-work proof. Anything unprovable stays held. - **Overriding an operator.** Only holds the system recorded are listed; attested, continued or owned holds are never touched. - **Load.** The listing is a filtered query on the 30 s reconciliation pass, gated so a run with an unsettled lease is not polled. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>