mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Restart recovery must continue a task after a server is killed. A person must never need to press Retry. > - When Paperclip cannot resume a cut-off run exactly, it records a "fresh continuation". For a remote workspace, the continuation abandons the old run's workspace epoch and restores the last save into a new root. It does not apply the old run's unsaved files. > - Workspace recovery can try to save the old run's files after that. The epoch is already abandoned, so the source is "unavailable". Recovery then wrote an active hold that asked a person to press Retry to continue in a new workspace, which the continuation had already arranged. > - That hold made the dispatch gate withhold every queued run of the task, and the task rested on a Retry banner. > - This pull request settles that hold into the recorded continuation at the moment workspace recovery would leave it exhausted. Dispatch and continuation delivery are unchanged. > - The benefit is that the queued continuation starts automatically after a hard kill, with no Retry banner. ## Linked Issues or Issue Description **What happened?** A restart-chaos release-swap wave hard-killed the old server. One task (of 6) then rested on a Retry banner for about 5 minutes. The other 5 tasks passed. The sequence for the failed task: 1. A run started on the incoming server. The wave's injected database delay held its process-identity write for 480 s, so its runner attach timed out. The run saved a same-run resume. 2. Restart recovery found a newer queued run for the task (a child-completion wake). It treated the same-run resume as superseded, and the run ended `failed` with `native_restart_recovery_blocked`. 3. The dispatch gate withheld the queued run (`execution_reconciliation_required`). In the same transaction, it recorded a fresh continuation with `remote_epoch` isolation, which abandoned the old run's remote workspace epoch. 4. 4 s later, workspace preservation for the old run found the abandoned epoch (`source_unavailable`, `use_alternative_strategy`). It left an active hold: "Paperclip cannot recover the unsaved files from the previous run ... Retry continues the task in a new workspace". The retained-source fallback also refused it later, with `workspace_authority_changed`, because the continuation owned the epoch. 5. The gate withheld both bounded retries because of that hold. The retries were spent, and the task showed a Retry banner. 6. A person pressed Retry. Retry started a run in a new workspace, which the recorded continuation had already decided to do. **Expected behavior** The queued continuation starts automatically. No Retry banner shows. **Steps to reproduce** 1. Run the restart-chaos K1 kill variant: 2 release swaps, and the old server is hard-killed 120 s after SIGTERM. 2. Use an OpenCode task on a remote sandbox whose run starts just before the second swap and that gets a child-completion wake while it runs. 3. Watch the task after the kill. Workspace recovery writes the unsaved-files hold, the gate cancels each queued run with `execution_reconciliation_required`, and the task rests on a Retry banner. `server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts` reproduces the database state at step 5. **Paperclip version or commit** `master` at25b5cad7f9(wave) andc41145e72a(base of this branch). ## What Changed - `server/src/services/fresh-continuation-fallback.ts`: new `settleHoldIntoAbandonedEpochContinuation()`. It settles one workspace-recovery hold with the existing `settleLateHolds()` when all of these are true: - the hold is active or escalated, and its saved-work recovery is exhausted; - no person owns it, it has no human-gate evidence, and it is not an unsafe-archive hold; - a fresh-continuation record for the same run has `remote_epoch` isolation and delivery `pending`, `delivered`, `delegated` or `not_requested`; - durable proof shows the run's prepared epoch was abandoned (`isAbandonedPreparedRemoteWorkspace`: an abandoned-epoch receipt for this run, epoch and location, and an authority that moved past it). The hold becomes `resolved` / `restored` (`continued_fresh_from_last_saved`) and is coalesced into the continuation record. Its files are not applied. - `server/src/services/workspace-recovery.ts` calls it at two points: when an attempt records an exhausted failure, and when a recovery pass starts on an already exhausted hold (holds written before this change). - `server/src/services/recovery/human-gate-evidence.ts`: one shared list of human-gate evidence keys (`executionReconciliation`, `OWNER_GONE_ATTESTATION_KEY`, `explicitUserContinuation`), with a SQL helper and a TypeScript helper. Both treat a present key as a person's decision, whatever its value. `isHumanGate`, `completed-turn-cleanup-hold-settlement.ts` and `interrupted-preparation-hold-settlement.ts` use them. - New tests: `server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts`. The dispatch gate, the execution-blocker predicate, the wake-queue lock release, and continuation delivery are unchanged from `master`. An earlier revision of this pull request changed them. That approach was removed because each review round found another dispatch case. `fresh_isolated_workspace` continuations are out of scope on purpose. They do not abandon a remote epoch, so they do not cause this failure. Only a run that carries the isolation marker is protected from the old local directory, so a late hold for such a run keeps blocking. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-isolated-continuation-hold.test.ts` gives 20 of 20 passing. - Recovery settles the late hold for each `remote_epoch` delivery state (`pending`, `delivered`, `delegated`, `not_requested`). The blocker is then empty, and the hold is coalesced into the continuation. - Assembled heartbeat test: after recovery settles the hold, the queued run is the only run that starts. The old run stays failed. With a hold that a person owns, the gate still withholds the queued run. - The hold stays for each of these cases: no continuation, a `remote_epoch` continuation while the run still owns its epoch, a run with no remote workspace, `fresh_isolated_workspace`, `writer_ended`, `invalidated`, `coalesced`, a hold that a person owns, an unsafe archive, each human-gate key, and a null-valued human-gate key. - Red/green: with the two `workspace-recovery.ts` calls disabled, 5 of 20 fail; without the abandoned-epoch check, the 2 epoch cases fail: the 4 settle cases and the assembled start test. - A focused regression run of the server test files for the blocker, fresh continuations, workspace recovery, both hold settlements, owner-gone attestation, and the run-dispatch and wake-queue adapters. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Only exhausted, system-owned workspace-recovery holds of a run whose remote epoch a recorded continuation replaced are settled. All other holds keep blocking. - After settlement, workspace recovery stops for that hold. Its files were already unreachable, because the epoch was abandoned. - A human-gate key with a null or false value now counts as a person's decision in TypeScript too. Before, TypeScript ignored it and SQL counted it. This makes the two sides agree and is more conservative. - No schema change, no migration, and no Runner change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edit, code search) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>