Files
PaperClipAI/server
DottaandClaude Opus 5.5 fe0c4aa95d fix(recovery): settle a late workspace hold when a remote-epoch continuation already took over (#22387)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Restart recovery must continue a task after a server is killed. A
person must never need to press Retry.
> - When Paperclip cannot resume a cut-off run exactly, it records a
"fresh continuation". For a remote workspace, the continuation abandons
the old run's workspace epoch and restores the last save into a new
root. It does not apply the old run's unsaved files.
> - Workspace recovery can try to save the old run's files after that.
The epoch is already abandoned, so the source is "unavailable". Recovery
then wrote an active hold that asked a person to press Retry to continue
in a new workspace, which the continuation had already arranged.
> - That hold made the dispatch gate withhold every queued run of the
task, and the task rested on a Retry banner.
> - This pull request settles that hold into the recorded continuation
at the moment workspace recovery would leave it exhausted. Dispatch and
continuation delivery are unchanged.
> - The benefit is that the queued continuation starts automatically
after a hard kill, with no Retry banner.

## Linked Issues or Issue Description

**What happened?**

A restart-chaos release-swap wave hard-killed the old server. One task
(of 6) then rested on a Retry banner for about 5 minutes. The other 5
tasks passed. The sequence for the failed task:

1. A run started on the incoming server. The wave's injected database
delay held its process-identity write for 480 s, so its runner attach
timed out. The run saved a same-run resume.
2. Restart recovery found a newer queued run for the task (a
child-completion wake). It treated the same-run resume as superseded,
and the run ended `failed` with `native_restart_recovery_blocked`.
3. The dispatch gate withheld the queued run
(`execution_reconciliation_required`). In the same transaction, it
recorded a fresh continuation with `remote_epoch` isolation, which
abandoned the old run's remote workspace epoch.
4. 4 s later, workspace preservation for the old run found the abandoned
epoch (`source_unavailable`, `use_alternative_strategy`). It left an
active hold: "Paperclip cannot recover the unsaved files from the
previous run ... Retry continues the task in a new workspace". The
retained-source fallback also refused it later, with
`workspace_authority_changed`, because the continuation owned the epoch.
5. The gate withheld both bounded retries because of that hold. The
retries were spent, and the task showed a Retry banner.
6. A person pressed Retry. Retry started a run in a new workspace, which
the recorded continuation had already decided to do.

**Expected behavior**

The queued continuation starts automatically. No Retry banner shows.

**Steps to reproduce**

1. Run the restart-chaos K1 kill variant: 2 release swaps, and the old
server is hard-killed 120 s after SIGTERM.
2. Use an OpenCode task on a remote sandbox whose run starts just before
the second swap and that gets a child-completion wake while it runs.
3. Watch the task after the kill. Workspace recovery writes the
unsaved-files hold, the gate cancels each queued run with
`execution_reconciliation_required`, and the task rests on a Retry
banner.

`server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts`
reproduces the database state at step 5.

**Paperclip version or commit**

`master` at 25b5cad7f9 (wave) and c41145e72a (base of this branch).

## What Changed

- `server/src/services/fresh-continuation-fallback.ts`: new
`settleHoldIntoAbandonedEpochContinuation()`. It settles one
workspace-recovery hold with the existing `settleLateHolds()` when all
of these are true:
- the hold is active or escalated, and its saved-work recovery is
exhausted;
- no person owns it, it has no human-gate evidence, and it is not an
unsafe-archive hold;
- a fresh-continuation record for the same run has `remote_epoch`
isolation and delivery `pending`, `delivered`, `delegated` or
`not_requested`;
- durable proof shows the run's prepared epoch was abandoned
(`isAbandonedPreparedRemoteWorkspace`: an abandoned-epoch receipt for
this run, epoch and location, and an authority that moved past it).
The hold becomes `resolved` / `restored`
(`continued_fresh_from_last_saved`) and is coalesced into the
continuation record. Its files are not applied.
- `server/src/services/workspace-recovery.ts` calls it at two points:
when an attempt records an exhausted failure, and when a recovery pass
starts on an already exhausted hold (holds written before this change).
- `server/src/services/recovery/human-gate-evidence.ts`: one shared list
of human-gate evidence keys (`executionReconciliation`,
`OWNER_GONE_ATTESTATION_KEY`, `explicitUserContinuation`), with a SQL
helper and a TypeScript helper. Both treat a present key as a person's
decision, whatever its value. `isHumanGate`,
`completed-turn-cleanup-hold-settlement.ts` and
`interrupted-preparation-hold-settlement.ts` use them.
- New tests:
`server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts`.

The dispatch gate, the execution-blocker predicate, the wake-queue lock
release, and continuation delivery are unchanged from `master`. An
earlier revision of this pull request changed them. That approach was
removed because each review round found another dispatch case.

`fresh_isolated_workspace` continuations are out of scope on purpose.
They do not abandon a remote epoch, so they do not cause this failure.
Only a run that carries the isolation marker is protected from the old
local directory, so a late hold for such a run keeps blocking.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-isolated-continuation-hold.test.ts` gives 20 of
20 passing.
- Recovery settles the late hold for each `remote_epoch` delivery state
(`pending`, `delivered`, `delegated`, `not_requested`). The blocker is
then empty, and the hold is coalesced into the continuation.
- Assembled heartbeat test: after recovery settles the hold, the queued
run is the only run that starts. The old run stays failed. With a hold
that a person owns, the gate still withholds the queued run.
- The hold stays for each of these cases: no continuation, a
`remote_epoch` continuation while the run still owns its epoch, a run
with no remote workspace, `fresh_isolated_workspace`, `writer_ended`,
`invalidated`, `coalesced`, a hold that a person owns, an unsafe
archive, each human-gate key, and a null-valued human-gate key.
- Red/green: with the two `workspace-recovery.ts` calls disabled, 5 of
20 fail; without the abandoned-epoch check, the 2 epoch cases fail: the
4 settle cases and the assembled start test.
- A focused regression run of the server test files for the blocker,
fresh continuations, workspace recovery, both hold settlements,
owner-gone attestation, and the run-dispatch and wake-queue adapters.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Only exhausted, system-owned workspace-recovery holds of a run whose
remote epoch a recorded continuation replaced are settled. All other
holds keep blocking.
- After settlement, workspace recovery stops for that hold. Its files
were already unreachable, because the epoch was abandoned.
- A human-gate key with a null or false value now counts as a person's
decision in TypeScript too. Before, TypeScript ignored it and SQL
counted it. This makes the two sides agree and is more conservative.
- No schema change, no migration, and no Runner change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code with tool use (shell, file edit, code search) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 04:34:53 -06:00
..