Files
PaperClipAI/doc
DottaandClaude Opus 5.5 0986c3eb3a fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 keeps an unreachable remote Runner alive when the failed
attempt closes, so that the same-run reattachment can adopt it
> - The same close also leaves the session in the in-process cleanup
quarantine. The quarantine refuses every new session of the task, and
that includes the reattachment that the Runner was kept for
> - Close retries cannot settle it. After the first close fails, the
Runner transport returns the same rejected close for every retry
> - On staging, this was the largest failure class: 14 Codex runs in one
hour failed with `native_session_cleanup_quarantined`, while only 5 runs
succeeded. Each task then needed Retry
> - This pull request lets the same run's reattachment to that Runner,
in the same remote sandbox, take over its own retained cleanup
> - The benefit is that the task continues on its own, with no spent
attempt, no failed run and no Retry

## Linked Issues or Issue Description

Refs #20923, #21031

**What happened?**
A remote native run (Codex on Daytona) failed a turn with a recoverable
error, for example `runnerd did not report the provider turn identity`
or `native_finalization_missing: session returned no semantic result`.
The close then logged `runner did not prove durable suspension before
checkpoint` and `remote runner left alive unsettled for same-run
reattachment`. Each later same-run reattachment was refused at
admission:
- When the close error was retryable, admission waited out the grace and
failed with `native_session_cleanup_quarantined: prior session cleanup
remains incomplete`. That spent a same-run attempt. After the attempt
budget, the run failed. All close retries replayed the same failed close
within milliseconds (`runner transport close requested: native session
quarantined cleanup recovery`), so they could never settle the
quarantine.
- When the close error was `native_session_close_unrecoverable`
(operator-required), the next attempt failed at once with
`native_session_cleanup_quarantined: the previous session is still
shutting down; Paperclip finishes the cleanup automatically and the
saved work is kept.` That code is a permanent failure.

"Paperclip finishes the cleanup automatically" did not happen in the
process. Only a server restart (a deploy) removed the in-memory
quarantine, and by then the run had failed and the task waited for
Retry. Staging restarts every 20 to 45 minutes, so attempts often end
mid-cleanup and meet this gate.

**Expected behavior**
The same run's reattachment adopts the Runner that its earlier attempt
kept for it. It does not wait for a close that cannot settle, and it
does not spend an attempt. The task continues without Retry. Other runs
of the task and other sandboxes stay gated.

**Steps to reproduce**
1. Start a remote native run, and let its turn fail with a recoverable
transport error while the Runner cannot prove suspension.
2. The close keeps the Runner alive and quarantines the session.
3. The server schedules the same-run reattachment 30 s later.
4. The reattachment fails with `native_session_cleanup_quarantined`, and
the task needs Retry.

**Paperclip version or commit**
Deployed Core `c730872974` on staging. Reproduced on `master` at
`49f0030f06`.

**Deployment mode**
Hosted cloud (staging), Daytona remote sandboxes.

**Agent adapter(s) involved**
`paperclip_runner` native runtime with the Codex app server backend.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`executeNativeSession` takes a new option `retainedRunnerReattachment`.
With a remote cleanup scope, admission retires quarantine entries that
belong to the same run in the same remote scope. Removal stops later
close retries. A close attempt that is still in flight stays an
admission owner until it settles, so the old and new controllers do not
overlap. Other runs, other sandboxes and local backends keep the gate.
- `server/src/services/native-runtime/native-session-executor.ts`: sets
`retainedRunnerReattachment` only for a `reattach_remote_runner`
recovery of the same run.
- `doc/execution-semantics.md`: records the takeover.
- Tests:
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: four
new cases. An operator-required close with a retained Runner. A
retryable close whose retries replay one failure. A close that a deploy
interrupts while it is in flight (the reattachment waits for it, then
takes over). Controls: another run, a run without the takeover option
and a local backend stay refused.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
asserts that the option is set for a same-run remote reattachment and is
not set for a fresh sandbox run.

## Verification

- Red on `master` (runtime source restored): 3 of the 4 new runtime
cases fail with exactly the two staging errors (`the previous session is
still shutting down…` and `prior session cleanup remains incomplete`).
The control case passes.
- Red on `master` (executor source restored): the 3 executor assertions
fail.
- Green with this change:
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts` (294 passed)
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "exact
Stop cleanup|fresh sandbox|retirement"` (46 passed)
- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed)
- `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes.
- Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence
above in their events and logs.

## Risks

- The takeover applies only when the server recovers the same run with
`reattach_remote_runner` in the same remote cleanup scope. This is the
case that #20923 keeps the Runner alive for. After a deploy the
quarantine is already gone and the same reattachment runs today, so the
takeover gives the in-process retry the same behavior as a restart.
- A close attempt that is in flight is never overlapped. The
reattachment waits for it within the existing admission grace.
- The reattachment still verifies Runner ownership and its controller
generation before it adopts the Runner. If the Runner is gone, the
existing relaunch or reconciliation path handles it.
- No persisted shape changes. Saved state from the previous release is
not affected because the quarantine lives only in process memory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:15:40 -06:00
..
…
…