fix(native): retry a transient endpoint failure after a same-run bootstrap restart (#21488)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts a Runner in a remote sandbox for each
agent run. A bounded same-run retry (at most 3 startup attempts) handles
transient startup failures.
> - A server restart can stop a run during its first Runner bootstrap.
Restart recovery then claims `bootstrap_incomplete`. It stops and
quarantines the unfinished bootstrap and launches a fresh bootstrap on
the same run.
> - The executor builds its "unclaimed bootstrap" proof only when
`restartRecovery` is not set. Thus the fresh bootstrap of a
`bootstrap_incomplete` restart can never prove that a transient failure
is safe to retry.
> - A transient HTTP 502 from the Runner endpoint request in that fresh
bootstrap was then classified as `ambiguous_state`. The run stopped and
needed manual board recovery.
> - This pull request gives the unclaimed-bootstrap proof to a
`bootstrap_incomplete` restart as well as to a first attempt.
> - The benefit is that a transient endpoint failure after a restart
uses the bounded automatic retry and does not need a person to recover
the run.

## Linked Issues or Issue Description

Refs #20414 (related: it fixes executor admission of a
`bootstrap_incomplete` restart at state-root migration. It does not
change the retry evidence that this PR changes.)

**What happened?**
A server restart interrupted the first Runner bootstrap of a remote
native run. Restart recovery claimed `bootstrap_incomplete` and launched
a fresh bootstrap on the same run. The fresh bootstrap registered its
controller checkpoint. Then `runner.ingress.acquire` failed with HTTP
502, before any Runner connection or provider event. The run ended with
`nativePhase: terminal_failure`, `recoveryOwner: board`, attempt 1 of 3,
and the next action "resolve the ambiguous session state". The error was
`Runner connection endpoint could not be acquired (HTTP 502).`
(`runner_ingress_acquisition_failed`).

**What did you expect to happen?**
The transient 502 occurred before the Runner connected and before any
provider work. It should use the bounded same-run startup retry, as it
does on a first attempt.

**Steps to reproduce**
1. Start a native run on a remote sandbox target.
2. Restart the server during the first Runner bootstrap, so that restart
recovery claims `bootstrap_incomplete`.
3. Make the next `runner.ingress.acquire` call return HTTP 502.
4. See that the run stops as ambiguous state and does not schedule a
startup retry.

## What Changed

- Add `attemptLaunchesFreshBootstrap()` to `native-bootstrap-proof.ts`.
It returns true for a first attempt and for a `bootstrap_incomplete`
restart. It returns false for `reattach_existing_runner`,
`reattach_remote_runner` and `resume_dead_runner`, because these
continue an existing Runner or provider session.
- In `native-session-executor.ts`, use this helper in place of
`!input.restartRecovery` when the executor builds the
unclaimed-bootstrap evidence for a startup failure.
- Add `native-bootstrap-proof.test.ts` for the helper.
- Add a case to `native-startup-retry.integration.test.ts`. It runs the
real executor under a `bootstrap_incomplete` restart claim, and the
fresh bootstrap gets a typed HTTP 502 from the Runner endpoint before
the Runner connects.

## Verification

- `native-bootstrap-proof.test.ts`: the test "accepts a same-run restart
of an incomplete bootstrap" failed before the fix and passes after the
fix (5/5 tests pass, vitest 5.1.0).
- `native-startup-retry.integration.test.ts`, case "transient 502 after
a bootstrap_incomplete restart": the run must settle as
`retryable_failure` with `recoveryMode: "bootstrap_retry"`, attempt 1
and a set `nextAttemptAt`. Without the executor change, the executor
builds no bootstrap evidence and the run settles as `ambiguous_state`.
CI runs this suite; I could not run it locally.
- The other guards for a retry did not change. The checkpoint must be
unclaimed, there must be no provider events and no session checkpoint,
and the 3-attempt startup budget applies. If the earlier attempt opened
a provider session, provider events exist and the run still fails closed
as ambiguous.
- I could not run the full typecheck and the real-Runner integration
suites locally (no Rust toolchain in the sandbox). CI must run them.

## Risks

- Low risk. The change only affects a `bootstrap_incomplete` restart
whose fresh bootstrap fails during startup. Such a failure now gets the
same retry decision as a first attempt. All the other evidence checks
still apply, so a run with provider activity still stops for manual
recovery.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with
tool use (shell, file edit, and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-13 13:37:24 -08:00
1 parent 532e502791
commit 4cf04d64cb
4 files changed
+50 -3

No files matched your search

@@ -0,0 +1,21 @@
import { describe, expect, it } from "vitest";
import { attemptLaunchesFreshBootstrap } from "./native-bootstrap-proof.js";
describe("attemptLaunchesFreshBootstrap", () => {
it("accepts a first attempt", () => {
expect(attemptLaunchesFreshBootstrap(undefined)).toBe(true);
expect(attemptLaunchesFreshBootstrap(null)).toBe(true);
});
// The restart stops and quarantines the unfinished bootstrap, then launches a
// new one on the same run. A transient endpoint failure of that new bootstrap
// is as retryable as one on a first attempt.
it("accepts a same-run restart of an incomplete bootstrap", () => {
expect(attemptLaunchesFreshBootstrap({ kind: "bootstrap_incomplete" })).toBe(true);
});
it.each(["reattach_existing_runner", "reattach_remote_runner", "resume_dead_runner"])(
"refuses a %s recovery, which continues an existing Runner or provider session",
kind => { expect(attemptLaunchesFreshBootstrap({ kind })).toBe(false); },
);
});
@@ -17,6 +17,17 @@ function record(value: unknown): Record<string, unknown> {
: {};
}
/**
* Whether this attempt launches its own fresh Runner bootstrap, so that its
* unclaimed controller checkpoint can prove a startup failure retryable. A
* `bootstrap_incomplete` restart does: it stops and quarantines the unfinished
* bootstrap, then launches a new one on the same run. Other recoveries continue
* an existing Runner or provider session.
*/
export function attemptLaunchesFreshBootstrap(restartRecovery: { kind: string } | null | undefined): boolean {
return !restartRecovery || restartRecovery.kind === "bootstrap_incomplete";
}
/** The runner never connected, so every bootstrap command is still undelivered. */
export function runnerdStateIsUnconnectedBootstrap(state: Record<string, unknown>): boolean {
const commands = Array.isArray(state.commands) ? state.commands.map(record) : [];
@@ -109,7 +109,7 @@ import { createNativeProviderFailureObservation } from "./native-provider-failur
import { explainNativeTurnFailure } from "./native-run-failure-explanation.js";
import { nativeStartupRetryExhaustedAction, nativeStartupRetryWait, planNativeStartupRetry } from "./native-startup-retry.js";
import { admitUnclaimedNativeBootstrapRetry } from "./native-bootstrap-retry.js";
import { RUNNERD_CONTROL_PLANE_STATE_SCHEMAS, runnerdStateIsIncompleteBootstrap,
import { attemptLaunchesFreshBootstrap, RUNNERD_CONTROL_PLANE_STATE_SCHEMAS, runnerdStateIsIncompleteBootstrap,
runnerdStateHasNoProviderTurn, runnerdStateIsOpenedSessionBeforeTurn, runnerdStateIsUnconnectedBootstrap } from "./native-bootstrap-proof.js";
import { createNativeDeliverableTextObservation } from "./native-deliverable-text.js";
import { reconcileControllerJournalWithRunner, refreshRetiredWarmOwnerProjection } from "./warm-controller-journal.js";
@@ -12995,7 +12995,7 @@ async function executePaperclipNativeSessionAttempt(
: nativeRemoteStartupRejectionCode(classifiedFailureCode, error, input.runnerExecutionTarget?.kind === "remote",
previousStartupSignature);
const bootstrap = input.useRunnerd && input.runnerExecutionTarget?.kind === "remote"
&& !input.restartRecovery && !retainedTransition && !remoteWarmRecovery
&& attemptLaunchesFreshBootstrap(input.restartRecovery) && !retainedTransition && !remoteWarmRecovery
? {
binding: input.execution.binding,
identity: { runnerInstanceId: effectiveRunnerInstanceId,
@@ -35,7 +35,7 @@ import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js";
import { DurablePrpControlPlane, NativeProviderTerminalFailure } from "../../vendor/paperclip-runner/index.js";
import { createNativeRunnerExecutionDriver } from "./native-runner-execution-driver.js";
import { currentNativeControllerIdentity } from "./native-restart-recovery.js";
import { currentNativeControllerIdentity, type NativeRestartRecoveryClaim } from "./native-restart-recovery.js";
import { loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
import { environmentRunnerIngressErrorData, JsonRpcCallError, PluginEnvironmentRunnerIngressError } from "@paperclipai/plugin-sdk";
import { accountRunCost } from "../run-cost-accounting.js";
@@ -290,6 +290,7 @@ describe("native provider startup retry", () => {
it.each([
{ name: "transient 502", status: 502, retry: true },
{ name: "transient 502 after a bootstrap_incomplete restart", status: 502, retry: true, restart: true },
{ name: "credential 401", status: 401, retry: false },
{ name: "permission 403", status: 403, retry: false },
{ name: "invalid request 400", status: 400, retry: false },
@@ -305,6 +306,19 @@ describe("native provider startup retry", () => {
])("settles ingress failure safely: $name", async options => {
const seeded = await seedRun(await createAgent());
if (options.finalAttempt) await db.update(nativeRunFinalizations).set({ attempt: 2 }).where(eq(nativeRunFinalizations.runId, seeded.runId));
// A server restart interrupted the first bootstrap. The recovery scan's
// claim makes this attempt launch a fresh bootstrap on the same run.
let restartRecovery: NativeRestartRecoveryClaim | undefined;
if (options.restart) {
const controller = await currentNativeControllerIdentity();
restartRecovery = { kind: "bootstrap_incomplete", runId: seeded.runId, leaseOwner: `restart-${seeded.runId}`,
controllerGeneration: 2, providerAttempt: 0, restartKind: "hard", recoveryRequestId: null };
await db.update(nativeRunFinalizations).set({ leaseOwner: restartRecovery.leaseOwner,
leaseExpiresAt: new Date(Date.now() + 10 * 60_000), controllerBootId: controller.bootId,
controllerPid: controller.pid, controllerHostHash: controller.hostHash ?? null,
controllerProcessStartedAt: controller.processStartedAt, controllerGeneration: 2,
recoveryState: "bootstrap_incomplete" }).where(eq(nativeRunFinalizations.runId, seeded.runId));
}
const root = mkdtempSync(join(tmpdir(), "native-ingress-bootstrap-"));
const server = createServer((_request, response) => {
response.writeHead(options.status, { "content-type": "text/html" });
@@ -386,6 +400,7 @@ describe("native provider startup retry", () => {
try {
await expect(executePaperclipNativeSession({ db, execution: seeded.execution,
runnerInstanceId: seeded.runnerInstanceId, useRunnerd: true,
...(restartRecovery ? { leaseOwner: restartRecovery.leaseOwner, restartRecovery } : {}),
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", runner: {} } as never,
backend: { descriptor: async () => { throw new Error("unexpected descriptor"); },
openSession: async () => { throw new Error("unexpected provider session"); } },