Files
PaperClipAI/doc
DottaandClaude Opus 5.5 96ab5e3d38 fix(recovery): read a retained Runner from its own allocation and continue unverifiable no-replay holds from the last save (#20567)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native warm runs on a remote sandbox continue a saved session in the
next run. The next run first reads the retained Runner lifecycle from
the sandbox.
> - #20346 abandons the workspace epoch of a run that died during
preparation. The next run then restores the last save into a new
directory of the same sandbox.
> - The retained-sandbox binding check required the old and the new
directory to be the same. The warm follow-up failed with
`native_controller_recovery_missing_evidence:retained_sandbox_binding`
before provider work.
> - The failed run's no-replay hold then refused its files with
`allocation_ownership_ambiguous`. The task stayed blocked. The text said
that Retry "succeeds only after Paperclip is updated", so no click could
help.
> - This pull request binds the Runner read to the directory that the
retained lease recorded, and lets the existing retained-source fallback
settle these holds from the last save.
> - The benefit is that a moved workspace continues its session, and a
hold that cannot be verified continues automatically instead of
dead-ending.

## Linked Issues or Issue Description

Refs #20346

**What happened?**
On staging, a warm native continuation ran about 10 minutes after a
deploy. The run before it died during preparation (database deadlock),
so its workspace epoch was abandoned. The continuation restored the last
save into a new allocation directory of the same Daytona sandbox. It
then failed with
`native_controller_recovery_missing_evidence:retained_sandbox_binding`,
because the retained lease of the committed source run recorded the old
directory. The failure recorded a no-replay hold
(`native_continuation_requires_reconciliation`, resolved with outcome
`blocked`). Workspace recovery refused the source with
`allocation_ownership_ambiguous`, because the sandbox's reusable-lease
chain had predecessors in another directory and a non-native
preparation-only run. The task stayed blocked with "Retry runs the check
again and succeeds only after Paperclip is updated."

**Expected behavior**
The continuation reads the retained Runner state from the directory
where that Runner wrote it, and continues. When retained files cannot be
verified, the task continues from the last committed save without a
person.

**Steps to reproduce**
1. Run a warm native session on a reusable remote sandbox and let it
commit.
2. Fail the next run during preparation, so that its epoch is abandoned
(#20346).
3. Start the next warm continuation. It restores the last save into a
new allocation directory and fails with `retained_sandbox_binding`.
4. See the hold refuse its source with `allocation_ownership_ambiguous`
and stay blocked.

**Paperclip version or commit**
Staging `c874cfa4dacc` (includes #20346).

## What Changed

- `assertNativeRetainedSandboxBinding` now returns the directory that
the retained lease recorded. It accepts a different target directory
only when both are canonical sibling allocations of the same execution
workspace under the sandbox's `.paperclip-workspaces` (`<workspaceId>`
or `<workspaceId>-<epoch>-<leaseId>`). Same sandbox (`providerLeaseId`),
environment, run, issue and committed owner are still required. Other
directories stay refused.
- `restoreRemoteSuspendedRunnerCheckpoint` reads the Runner state from
that recorded directory. A moved source must be suspended and settled. A
Runner left in the old allocation is never adopted live.
- The retained-source fallback (`continueFromLastSavedWorkspace`) and
its discovery now also cover the native no-replay hold
(`native_continuation_requires_reconciliation`, active or resolved with
outcome `blocked`, replay blocked for the same run). Rows saved by
earlier releases are discovered without migration.
- The fallback owns the Blocked status of such a hold only while the
recorded `nativeFailureBlock` status version still matches the task.
Otherwise it keeps the newer block and starts no continuation.
- The refusal text no longer says "succeeds only after Paperclip is
updated", and it promises no fallback.
- `doc/execution-semantics.md` describes both rules.

## Verification

- Red (unfixed code): `native-controller-recovery.integration.test.ts`
"binds a retained Runner read to the source's directory after the
workspace moved within the same sandbox" failed with
`native_controller_recovery_missing_evidence:retained_sandbox_binding`
at `native-controller-recovery.ts:145`, the staging error.
- Red (unfixed code): `workspace-recovery-fallback.test.ts` "continues a
resolved-but-blocked native no-replay hold from the last save,
discovered by the worker" failed: the worker found no due work for the
exact staging row. The newer-block safety case and the message case in
`system-recovery-state.test.ts` also failed.
- Provider boundary: a new test drives
`restoreRemoteSuspendedRunnerCheckpoint` through a sandbox runner that
refuses paths outside the sandbox workspace base. The suspended case
reads `runner-state.json` from the source directory, never the moved
root. The ready case is refused with `runner_not_settled` and caches
nothing.
- Green: `native-controller-recovery.integration.test.ts`,
`workspace-recovery-fallback.test.ts` and
`system-recovery-state.test.ts` pass (45 tests). The neighbouring suites
`workspace-recovery-retry-copyback`,
`workspace-recovery-preparation-gates` and
`workspace-recovery-stop-proof` pass (135 tests).
- Safety cases: another workspace, another parent, a non-canonical path
or name (`-0-`, trailing text), `/unrelated` and another sandbox stay
refused. A hold whose task moved to a newer Blocked decision keeps the
block and starts no continuation. Exactly one continuation is delivered.
Native replacement is denied for the abandoned source, so no second
executor starts.
- `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify`
returns exit 2 (a global doc input changed), so required CI supplies the
broad coverage.

## Risks

- Previous-release state: rows saved by earlier releases carry forward
without migration. The binding reads existing lease metadata. The
fallback discovers existing resolved-but-blocked holds by their existing
evidence fields.
- The binding now accepts a sibling allocation directory. The read still
binds to the exact sandbox, run, environment and committed owner, and it
reads only the source's recorded directory. A non-suspended Runner in a
moved allocation is refused, not adopted.
- The fallback abandons the unverified epoch and continues from the last
committed save with a fresh session. In the staging shape the run did no
provider work after it restored that save, so no work is lost.
Unverified files are attached to the task when they can be read, as
before.
- Not verified live: a real warm Codex resume after a moved allocation
on Daytona.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only browser access to staging).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 10:34:44 -06:00
..
…
…
…
…
…