Files
PaperClipAI/server/src/services/native-runtime
DottaandClaude Opus 5.5 e680fa6814 fix(native): retry startup interrupted by a closed runtime API listener (#19462)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs prepare a remote workspace, then start a runner and a
provider session
> - A graceful server shutdown closes the HTTP listener before it drains
runs
> - A run that is still preparing then asks for the local runtime API
origin, and the lookup fails with
`local_runtime_api_listener_unavailable`
> - Recovery did not know this code. It treated the failure as an
ambiguous provider state and blocked the task behind a board-owned
action, although no runner or provider process existed
> - This pull request classifies the closed listener as a definite
pre-session failure
> - The benefit is that a deploy no longer strands a preparing task. The
same run gets a bounded bootstrap retry that the next server process
claims

## Linked Issues or Issue Description

**What happened**
A native Codex task on a remote sandbox spent about ten minutes in
workspace preparation. A deploy started a graceful shutdown during that
time. When preparation finished, the executor called
`resolveLocalRuntimeApiUrl()` to build the tool authority. The listener
was already closed, so it threw
`local_runtime_api_listener_unavailable`. No runner backend or provider
session had started. The failure classifier returned the generic
`native_session_interrupted` code. `nativeProviderRecoveryEvidence` then
returned `ambiguous_state`. The run became a terminal failure, the task
became blocked, and the recovery action told the board to resolve an
ambiguous session and to not open a replacement session. The same
signature occurred on three other runs during an earlier deploy.

**Expected behavior**
A failure that occurs before any runner or provider process exists is
safe to retry on the same run. The task must stay in progress with a
scheduled, bounded bootstrap retry. If a provider event or checkpoint
exists, recovery must stay operator-owned.

**Steps to reproduce**
Start a native run. Close the server's HTTP listener before the executor
creates the runner backend. Observe a `terminal_failure` with
`recoveryMode: "ambiguous_state"` and a blocked task. The new
integration test reproduces this with the real
`resolveLocalRuntimeApiUrl()` and a real Postgres database.

**Environment**
Server native runtime, remote sandbox target, `codex_app_server` driver.
Related recovery work: #19249, #19304, #19305.

## What Changed

- `nativeSessionFailureSourceCode` returns a new
`local_runtime_api_listener_unavailable` source code for this error.
- `nativeProviderRecoveryEvidence` treats that code as definitely
pre-session. The existing guard stays: the run must have no checkpoint
and no durable provider event.
- The chat failed-run retry accepts the code in its list of ordinary
pre-session failures, so an exhausted startup retry can still be retried
from chat.
- New integration tests in `native-startup-retry.integration.test.ts`
cover the retry path and the operator-owned path when a provider event
exists.

## Verification

- Red before the fix: the new test failed with `phase:
"terminal_failure"`, `recoveryMode: "ambiguous_state"`, `recoverable:
false` and a board-owned recovery. This matches the failed run on the
deployed instance.
- Green after the fix: `cd server && npx vitest run
src/services/native-runtime/native-startup-retry.integration.test.ts`
passes 26 of 26 tests. The retry test asserts `retryable_failure`,
`bootstrap_retry`, backoff timing, an agent-owned `resume_native_run`
wake, and a claim through `claimNativeSessionResumptions`.
- Safety case: when a durable `turn.started` event exists, the same
error stays `terminal_failure` with `ambiguous_state` and a blocked
task.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"listener|classif|pre-session"` passes.
- Not run: a live deploy that interrupts a preparing sandbox run. The
test uses the real listener lookup and the real recovery tables, but not
a real sandbox.

## Risks

- Low risk. The change only affects runs that fail with this exact error
and have no checkpoint and no provider event.
- The retry uses the existing bootstrap budget of three attempts with
backoff. Repeated failures still stop and hand the task to the board.
- The GitHub callback bridge that started in the sandbox before the
failure is a per-attempt helper. The existing cleanup path releases it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use, code execution and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 08:27:59 -06:00
..
…