Files
PaperClipAI/server/src/services/native-runtime
DottaandClaude Opus 5.5 a52f622be8 fix(native): retry a remote provider startup failure once its runner exit is proven (#20874)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a Runner in the
sandbox, and the Runner opens the provider session
> - When the provider rejects `session.open` at startup, no provider
work began
> - #20808 ends every such remote rejection at once with Retry, because
restart recovery could not prove that the failed runner was gone. A
fault that clears by itself therefore needs one human click
> - The Runner transport already settles the failed runner on close and
observes its exit. That is the missing proof
> - This pull request records that proof on the failure and lets the
same-run bootstrap retry use it. It stops a startup error that repeats,
so an image fault that a retry cannot fix does not loop
> - The benefit is that a transient remote startup failure recovers
without a person, and a permanent one still rests at once with a working
Retry that names the error

## Linked Issues or Issue Description

Refs #20808
Refs #20832
Refs #20820

**What happened?**
A remote (Daytona) native run whose provider rejected `session.open` at
startup ended as `native_provider_startup_rejected` with Retry. This
happened even when the Runner had settled and exited, and no provider
session, event or checkpoint existed. A person had to click Retry for a
fault that the durable record proves did no provider work.

**Expected behavior**
With a proven runner exit and no provider work, the run retries its
bootstrap on the same run automatically. A startup error that repeats on
the next attempt stops at once with Retry and names that error. Failures
saved without the proof keep today's behaviour.

**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote startup rejection|session.open rejection"`
2. On master, 3 cases fail: the proven remote rejection is
`terminal_failure`, the repeat does not name its error, and restart
recovery holds the proven failure on
`ambiguous_remote_recovery_evidence`.

**Paperclip version or commit**
master at 3ded7bbd91.

## What Changed

- Runner transport (`runnerd-codex-transport.ts`): new
`runnerExitConfirmed()`. It is true only after a completed close settled
the runner (provider stopped, runner suspended) and the runner process
exit was observed.
- Codex driver (`codex-app-server-driver-impl.ts`): after it closes the
transport for a failed session open, it sets `runnerExitConfirmed` on
the error when the transport proves the exit.
- Executor (`native-session-executor.ts`):
- A remote `provider_session_open_rejected` with a startup
`providerFailure` keeps the same-run `bootstrap_retry` only with that
proof. The saved failure records `runnerExitConfirmed: true`.
- Each failed attempt's recovery history entry records a startup failure
signature: provider, category and message, with numbers ignored. When
the next attempt fails with the same signature, it stops as
`native_provider_startup_rejected`. Its next action names the startup
error and keeps "Retry starts a new run".
- Deterministic rejections (`requires_action` authentication, permission
or invalid request) stop at once as before.
  - Local targets are unchanged.
- Restart recovery (`native-restart-recovery.ts`):
`remoteRunnerExitedBeforeProviderWork` also accepts a settled
`provider_session_open_rejected` with `runnerExitConfirmed: true`. All
its other checks still apply: no provider session, no provider event, no
checkpoint, no runner-sent event, no successor.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md`.

## Verification

- Red on master, green here:
- `native-startup-rejected-retry.integration.test.ts`: proven remote
rejection → `retryable_failure` with `runnerExitConfirmed`; unproven →
terminal; repeat → terminal, the next action names `bwrap: loopback`,
and Retry is offered.
- `native-runner-restart-recovery.integration.test.ts`: proven failure →
`bootstrap_incomplete` (`remote_runner_exited_before_provider_work`); an
older release's failure without the proof stays held.
- `codex-app-server-driver.lifecycle.test.ts`: the driver records the
proof only when the transport reports it after close.
- `native-session-executor.test.ts`: unit cases for the rule, the
`cause` chain, the repeat signature and deterministic rejections.
- Local runs: the 3 server files above pass (1380 tests); restart
recovery `-t remote` 75 pass; driver lifecycle 25 pass.
- `tsc --noEmit` clean for `server` and `packages/paperclip-runner`.
- Not run: a Daytona chaos wave. There is no in-sandbox startup fault
seam yet (see Risks).

## Risks

- The proof depends on the transport's close settling the runner after a
failed `session.open`. The Rust runner keeps serving after a failed
`session.open` (lifecycle `Continue`) and accepts `runner.suspend`, so
close should settle. If it does not settle, there is no proof and the
run ends at once with Retry, which is today's behaviour. It is never
less safe than today.
- Retries stay bounded: the startup retry budget (3 attempts) and the
repeat rule both apply. A repeated identical error stops after one
automatic retry.
- Saved state: failures saved by older releases have no
`runnerExitConfirmed` and keep the old rule. History entries without a
signature never match a repeat. A mid-run deploy is safe: an old
executor never writes the proof.
- Not covered by chaos: a remote "Codex fails at startup N times" fault
needs a hook inside the sandbox Runner or Codex. DF1's seam is
server-side only, and no open chaos PR adds a seam inside the sandbox.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file edit) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:55:15 -06:00
..
…