From 4cf04d64cb07a201fdcef05928e49a463a8641a2 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 13 Nov 2026 13:37:24 -0800 Subject: [PATCH] fix(native): retry a transient endpoint failure after a same-run bootstrap restart (#21488) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime starts a Runner in a remote sandbox for each agent run. A bounded same-run retry (at most 3 startup attempts) handles transient startup failures. > - A server restart can stop a run during its first Runner bootstrap. Restart recovery then claims `bootstrap_incomplete`. It stops and quarantines the unfinished bootstrap and launches a fresh bootstrap on the same run. > - The executor builds its "unclaimed bootstrap" proof only when `restartRecovery` is not set. Thus the fresh bootstrap of a `bootstrap_incomplete` restart can never prove that a transient failure is safe to retry. > - A transient HTTP 502 from the Runner endpoint request in that fresh bootstrap was then classified as `ambiguous_state`. The run stopped and needed manual board recovery. > - This pull request gives the unclaimed-bootstrap proof to a `bootstrap_incomplete` restart as well as to a first attempt. > - The benefit is that a transient endpoint failure after a restart uses the bounded automatic retry and does not need a person to recover the run. ## Linked Issues or Issue Description Refs #20414 (related: it fixes executor admission of a `bootstrap_incomplete` restart at state-root migration. It does not change the retry evidence that this PR changes.) **What happened?** A server restart interrupted the first Runner bootstrap of a remote native run. Restart recovery claimed `bootstrap_incomplete` and launched a fresh bootstrap on the same run. The fresh bootstrap registered its controller checkpoint. Then `runner.ingress.acquire` failed with HTTP 502, before any Runner connection or provider event. The run ended with `nativePhase: terminal_failure`, `recoveryOwner: board`, attempt 1 of 3, and the next action "resolve the ambiguous session state". The error was `Runner connection endpoint could not be acquired (HTTP 502).` (`runner_ingress_acquisition_failed`). **What did you expect to happen?** The transient 502 occurred before the Runner connected and before any provider work. It should use the bounded same-run startup retry, as it does on a first attempt. **Steps to reproduce** 1. Start a native run on a remote sandbox target. 2. Restart the server during the first Runner bootstrap, so that restart recovery claims `bootstrap_incomplete`. 3. Make the next `runner.ingress.acquire` call return HTTP 502. 4. See that the run stops as ambiguous state and does not schedule a startup retry. ## What Changed - Add `attemptLaunchesFreshBootstrap()` to `native-bootstrap-proof.ts`. It returns true for a first attempt and for a `bootstrap_incomplete` restart. It returns false for `reattach_existing_runner`, `reattach_remote_runner` and `resume_dead_runner`, because these continue an existing Runner or provider session. - In `native-session-executor.ts`, use this helper in place of `!input.restartRecovery` when the executor builds the unclaimed-bootstrap evidence for a startup failure. - Add `native-bootstrap-proof.test.ts` for the helper. - Add a case to `native-startup-retry.integration.test.ts`. It runs the real executor under a `bootstrap_incomplete` restart claim, and the fresh bootstrap gets a typed HTTP 502 from the Runner endpoint before the Runner connects. ## Verification - `native-bootstrap-proof.test.ts`: the test "accepts a same-run restart of an incomplete bootstrap" failed before the fix and passes after the fix (5/5 tests pass, vitest 5.1.0). - `native-startup-retry.integration.test.ts`, case "transient 502 after a bootstrap_incomplete restart": the run must settle as `retryable_failure` with `recoveryMode: "bootstrap_retry"`, attempt 1 and a set `nextAttemptAt`. Without the executor change, the executor builds no bootstrap evidence and the run settles as `ambiguous_state`. CI runs this suite; I could not run it locally. - The other guards for a retry did not change. The checkpoint must be unclaimed, there must be no provider events and no session checkpoint, and the 3-attempt startup budget applies. If the earlier attempt opened a provider session, provider events exist and the run still fails closed as ambiguous. - I could not run the full typecheck and the real-Runner integration suites locally (no Rust toolchain in the sandbox). CI must run them. ## Risks - Low risk. The change only affects a `bootstrap_incomplete` restart whose fresh bootstrap fails during startup. Such a failure now gets the same retry decision as a first attempt. All the other evidence checks still apply, so a run with provider activity still stops for manual recovery. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with tool use (shell, file edit, and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- .../native-bootstrap-proof.test.ts | 21 +++++++++++++++++++ .../native-runtime/native-bootstrap-proof.ts | 11 ++++++++++ .../native-runtime/native-session-executor.ts | 4 ++-- .../native-startup-retry.integration.test.ts | 17 ++++++++++++++- 4 files changed, 50 insertions(+), 3 deletions(-) create mode 100644 server/src/services/native-runtime/native-bootstrap-proof.test.ts diff --git a/server/src/services/native-runtime/native-bootstrap-proof.test.ts b/server/src/services/native-runtime/native-bootstrap-proof.test.ts new file mode 100644 index 00000000000..9c2b25d4377 --- /dev/null +++ b/server/src/services/native-runtime/native-bootstrap-proof.test.ts @@ -0,0 +1,21 @@ +import { describe, expect, it } from "vitest"; +import { attemptLaunchesFreshBootstrap } from "./native-bootstrap-proof.js"; + +describe("attemptLaunchesFreshBootstrap", () => { + it("accepts a first attempt", () => { + expect(attemptLaunchesFreshBootstrap(undefined)).toBe(true); + expect(attemptLaunchesFreshBootstrap(null)).toBe(true); + }); + + // The restart stops and quarantines the unfinished bootstrap, then launches a + // new one on the same run. A transient endpoint failure of that new bootstrap + // is as retryable as one on a first attempt. + it("accepts a same-run restart of an incomplete bootstrap", () => { + expect(attemptLaunchesFreshBootstrap({ kind: "bootstrap_incomplete" })).toBe(true); + }); + + it.each(["reattach_existing_runner", "reattach_remote_runner", "resume_dead_runner"])( + "refuses a %s recovery, which continues an existing Runner or provider session", + kind => { expect(attemptLaunchesFreshBootstrap({ kind })).toBe(false); }, + ); +}); diff --git a/server/src/services/native-runtime/native-bootstrap-proof.ts b/server/src/services/native-runtime/native-bootstrap-proof.ts index 90d169be4cc..acc47705c07 100644 --- a/server/src/services/native-runtime/native-bootstrap-proof.ts +++ b/server/src/services/native-runtime/native-bootstrap-proof.ts @@ -17,6 +17,17 @@ function record(value: unknown): Record { : {}; } +/** + * Whether this attempt launches its own fresh Runner bootstrap, so that its + * unclaimed controller checkpoint can prove a startup failure retryable. A + * `bootstrap_incomplete` restart does: it stops and quarantines the unfinished + * bootstrap, then launches a new one on the same run. Other recoveries continue + * an existing Runner or provider session. + */ +export function attemptLaunchesFreshBootstrap(restartRecovery: { kind: string } | null | undefined): boolean { + return !restartRecovery || restartRecovery.kind === "bootstrap_incomplete"; +} + /** The runner never connected, so every bootstrap command is still undelivered. */ export function runnerdStateIsUnconnectedBootstrap(state: Record): boolean { const commands = Array.isArray(state.commands) ? state.commands.map(record) : []; diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 8c6c95f78a2..2ff0253168c 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -109,7 +109,7 @@ import { createNativeProviderFailureObservation } from "./native-provider-failur import { explainNativeTurnFailure } from "./native-run-failure-explanation.js"; import { nativeStartupRetryExhaustedAction, nativeStartupRetryWait, planNativeStartupRetry } from "./native-startup-retry.js"; import { admitUnclaimedNativeBootstrapRetry } from "./native-bootstrap-retry.js"; -import { RUNNERD_CONTROL_PLANE_STATE_SCHEMAS, runnerdStateIsIncompleteBootstrap, +import { attemptLaunchesFreshBootstrap, RUNNERD_CONTROL_PLANE_STATE_SCHEMAS, runnerdStateIsIncompleteBootstrap, runnerdStateHasNoProviderTurn, runnerdStateIsOpenedSessionBeforeTurn, runnerdStateIsUnconnectedBootstrap } from "./native-bootstrap-proof.js"; import { createNativeDeliverableTextObservation } from "./native-deliverable-text.js"; import { reconcileControllerJournalWithRunner, refreshRetiredWarmOwnerProjection } from "./warm-controller-journal.js"; @@ -12995,7 +12995,7 @@ async function executePaperclipNativeSessionAttempt( : nativeRemoteStartupRejectionCode(classifiedFailureCode, error, input.runnerExecutionTarget?.kind === "remote", previousStartupSignature); const bootstrap = input.useRunnerd && input.runnerExecutionTarget?.kind === "remote" - && !input.restartRecovery && !retainedTransition && !remoteWarmRecovery + && attemptLaunchesFreshBootstrap(input.restartRecovery) && !retainedTransition && !remoteWarmRecovery ? { binding: input.execution.binding, identity: { runnerInstanceId: effectiveRunnerInstanceId, diff --git a/server/src/services/native-runtime/native-startup-retry.integration.test.ts b/server/src/services/native-runtime/native-startup-retry.integration.test.ts index f32a6bfd65f..7f420fc2c6b 100644 --- a/server/src/services/native-runtime/native-startup-retry.integration.test.ts +++ b/server/src/services/native-runtime/native-startup-retry.integration.test.ts @@ -35,7 +35,7 @@ import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js"; import { DurablePrpControlPlane, NativeProviderTerminalFailure } from "../../vendor/paperclip-runner/index.js"; import { createNativeRunnerExecutionDriver } from "./native-runner-execution-driver.js"; -import { currentNativeControllerIdentity } from "./native-restart-recovery.js"; +import { currentNativeControllerIdentity, type NativeRestartRecoveryClaim } from "./native-restart-recovery.js"; import { loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; import { environmentRunnerIngressErrorData, JsonRpcCallError, PluginEnvironmentRunnerIngressError } from "@paperclipai/plugin-sdk"; import { accountRunCost } from "../run-cost-accounting.js"; @@ -290,6 +290,7 @@ describe("native provider startup retry", () => { it.each([ { name: "transient 502", status: 502, retry: true }, + { name: "transient 502 after a bootstrap_incomplete restart", status: 502, retry: true, restart: true }, { name: "credential 401", status: 401, retry: false }, { name: "permission 403", status: 403, retry: false }, { name: "invalid request 400", status: 400, retry: false }, @@ -305,6 +306,19 @@ describe("native provider startup retry", () => { ])("settles ingress failure safely: $name", async options => { const seeded = await seedRun(await createAgent()); if (options.finalAttempt) await db.update(nativeRunFinalizations).set({ attempt: 2 }).where(eq(nativeRunFinalizations.runId, seeded.runId)); + // A server restart interrupted the first bootstrap. The recovery scan's + // claim makes this attempt launch a fresh bootstrap on the same run. + let restartRecovery: NativeRestartRecoveryClaim | undefined; + if (options.restart) { + const controller = await currentNativeControllerIdentity(); + restartRecovery = { kind: "bootstrap_incomplete", runId: seeded.runId, leaseOwner: `restart-${seeded.runId}`, + controllerGeneration: 2, providerAttempt: 0, restartKind: "hard", recoveryRequestId: null }; + await db.update(nativeRunFinalizations).set({ leaseOwner: restartRecovery.leaseOwner, + leaseExpiresAt: new Date(Date.now() + 10 * 60_000), controllerBootId: controller.bootId, + controllerPid: controller.pid, controllerHostHash: controller.hostHash ?? null, + controllerProcessStartedAt: controller.processStartedAt, controllerGeneration: 2, + recoveryState: "bootstrap_incomplete" }).where(eq(nativeRunFinalizations.runId, seeded.runId)); + } const root = mkdtempSync(join(tmpdir(), "native-ingress-bootstrap-")); const server = createServer((_request, response) => { response.writeHead(options.status, { "content-type": "text/html" }); @@ -386,6 +400,7 @@ describe("native provider startup retry", () => { try { await expect(executePaperclipNativeSession({ db, execution: seeded.execution, runnerInstanceId: seeded.runnerInstanceId, useRunnerd: true, + ...(restartRecovery ? { leaseOwner: restartRecovery.leaseOwner, restartRecovery } : {}), runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", runner: {} } as never, backend: { descriptor: async () => { throw new Error("unexpected descriptor"); }, openSession: async () => { throw new Error("unexpected provider session"); } },