mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
fix(native): retry a transient endpoint failure after a same-run bootstrap restart (#21488)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime starts a Runner in a remote sandbox for each agent run. A bounded same-run retry (at most 3 startup attempts) handles transient startup failures. > - A server restart can stop a run during its first Runner bootstrap. Restart recovery then claims `bootstrap_incomplete`. It stops and quarantines the unfinished bootstrap and launches a fresh bootstrap on the same run. > - The executor builds its "unclaimed bootstrap" proof only when `restartRecovery` is not set. Thus the fresh bootstrap of a `bootstrap_incomplete` restart can never prove that a transient failure is safe to retry. > - A transient HTTP 502 from the Runner endpoint request in that fresh bootstrap was then classified as `ambiguous_state`. The run stopped and needed manual board recovery. > - This pull request gives the unclaimed-bootstrap proof to a `bootstrap_incomplete` restart as well as to a first attempt. > - The benefit is that a transient endpoint failure after a restart uses the bounded automatic retry and does not need a person to recover the run. ## Linked Issues or Issue Description Refs #20414 (related: it fixes executor admission of a `bootstrap_incomplete` restart at state-root migration. It does not change the retry evidence that this PR changes.) **What happened?** A server restart interrupted the first Runner bootstrap of a remote native run. Restart recovery claimed `bootstrap_incomplete` and launched a fresh bootstrap on the same run. The fresh bootstrap registered its controller checkpoint. Then `runner.ingress.acquire` failed with HTTP 502, before any Runner connection or provider event. The run ended with `nativePhase: terminal_failure`, `recoveryOwner: board`, attempt 1 of 3, and the next action "resolve the ambiguous session state". The error was `Runner connection endpoint could not be acquired (HTTP 502).` (`runner_ingress_acquisition_failed`). **What did you expect to happen?** The transient 502 occurred before the Runner connected and before any provider work. It should use the bounded same-run startup retry, as it does on a first attempt. **Steps to reproduce** 1. Start a native run on a remote sandbox target. 2. Restart the server during the first Runner bootstrap, so that restart recovery claims `bootstrap_incomplete`. 3. Make the next `runner.ingress.acquire` call return HTTP 502. 4. See that the run stops as ambiguous state and does not schedule a startup retry. ## What Changed - Add `attemptLaunchesFreshBootstrap()` to `native-bootstrap-proof.ts`. It returns true for a first attempt and for a `bootstrap_incomplete` restart. It returns false for `reattach_existing_runner`, `reattach_remote_runner` and `resume_dead_runner`, because these continue an existing Runner or provider session. - In `native-session-executor.ts`, use this helper in place of `!input.restartRecovery` when the executor builds the unclaimed-bootstrap evidence for a startup failure. - Add `native-bootstrap-proof.test.ts` for the helper. - Add a case to `native-startup-retry.integration.test.ts`. It runs the real executor under a `bootstrap_incomplete` restart claim, and the fresh bootstrap gets a typed HTTP 502 from the Runner endpoint before the Runner connects. ## Verification - `native-bootstrap-proof.test.ts`: the test "accepts a same-run restart of an incomplete bootstrap" failed before the fix and passes after the fix (5/5 tests pass, vitest 5.1.0). - `native-startup-retry.integration.test.ts`, case "transient 502 after a bootstrap_incomplete restart": the run must settle as `retryable_failure` with `recoveryMode: "bootstrap_retry"`, attempt 1 and a set `nextAttemptAt`. Without the executor change, the executor builds no bootstrap evidence and the run settles as `ambiguous_state`. CI runs this suite; I could not run it locally. - The other guards for a retry did not change. The checkpoint must be unclaimed, there must be no provider events and no session checkpoint, and the 3-attempt startup budget applies. If the earlier attempt opened a provider session, provider events exist and the run still fails closed as ambiguous. - I could not run the full typecheck and the real-Runner integration suites locally (no Rust toolchain in the sandbox). CI must run them. ## Risks - Low risk. The change only affects a `bootstrap_incomplete` restart whose fresh bootstrap fails during startup. Such a failure now gets the same retry decision as a first attempt. All the other evidence checks still apply, so a run with provider activity still stops for manual recovery. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with tool use (shell, file edit, and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
4 files changed
+50
-3
No files matched your search
@@ -0,0 +1,21 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { attemptLaunchesFreshBootstrap } from "./native-bootstrap-proof.js";
|
||||
|
||||
describe("attemptLaunchesFreshBootstrap", () => {
|
||||
it("accepts a first attempt", () => {
|
||||
expect(attemptLaunchesFreshBootstrap(undefined)).toBe(true);
|
||||
expect(attemptLaunchesFreshBootstrap(null)).toBe(true);
|
||||
});
|
||||
|
||||
// The restart stops and quarantines the unfinished bootstrap, then launches a
|
||||
// new one on the same run. A transient endpoint failure of that new bootstrap
|
||||
// is as retryable as one on a first attempt.
|
||||
it("accepts a same-run restart of an incomplete bootstrap", () => {
|
||||
expect(attemptLaunchesFreshBootstrap({ kind: "bootstrap_incomplete" })).toBe(true);
|
||||
});
|
||||
|
||||
it.each(["reattach_existing_runner", "reattach_remote_runner", "resume_dead_runner"])(
|
||||
"refuses a %s recovery, which continues an existing Runner or provider session",
|
||||
kind => { expect(attemptLaunchesFreshBootstrap({ kind })).toBe(false); },
|
||||
);
|
||||
});
|
||||
@@ -17,6 +17,17 @@ function record(value: unknown): Record<string, unknown> {
|
||||
: {};
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether this attempt launches its own fresh Runner bootstrap, so that its
|
||||
* unclaimed controller checkpoint can prove a startup failure retryable. A
|
||||
* `bootstrap_incomplete` restart does: it stops and quarantines the unfinished
|
||||
* bootstrap, then launches a new one on the same run. Other recoveries continue
|
||||
* an existing Runner or provider session.
|
||||
*/
|
||||
export function attemptLaunchesFreshBootstrap(restartRecovery: { kind: string } | null | undefined): boolean {
|
||||
return !restartRecovery || restartRecovery.kind === "bootstrap_incomplete";
|
||||
}
|
||||
|
||||
/** The runner never connected, so every bootstrap command is still undelivered. */
|
||||
export function runnerdStateIsUnconnectedBootstrap(state: Record<string, unknown>): boolean {
|
||||
const commands = Array.isArray(state.commands) ? state.commands.map(record) : [];
|
||||
|
||||
@@ -109,7 +109,7 @@ import { createNativeProviderFailureObservation } from "./native-provider-failur
|
||||
import { explainNativeTurnFailure } from "./native-run-failure-explanation.js";
|
||||
import { nativeStartupRetryExhaustedAction, nativeStartupRetryWait, planNativeStartupRetry } from "./native-startup-retry.js";
|
||||
import { admitUnclaimedNativeBootstrapRetry } from "./native-bootstrap-retry.js";
|
||||
import { RUNNERD_CONTROL_PLANE_STATE_SCHEMAS, runnerdStateIsIncompleteBootstrap,
|
||||
import { attemptLaunchesFreshBootstrap, RUNNERD_CONTROL_PLANE_STATE_SCHEMAS, runnerdStateIsIncompleteBootstrap,
|
||||
runnerdStateHasNoProviderTurn, runnerdStateIsOpenedSessionBeforeTurn, runnerdStateIsUnconnectedBootstrap } from "./native-bootstrap-proof.js";
|
||||
import { createNativeDeliverableTextObservation } from "./native-deliverable-text.js";
|
||||
import { reconcileControllerJournalWithRunner, refreshRetiredWarmOwnerProjection } from "./warm-controller-journal.js";
|
||||
@@ -12995,7 +12995,7 @@ async function executePaperclipNativeSessionAttempt(
|
||||
: nativeRemoteStartupRejectionCode(classifiedFailureCode, error, input.runnerExecutionTarget?.kind === "remote",
|
||||
previousStartupSignature);
|
||||
const bootstrap = input.useRunnerd && input.runnerExecutionTarget?.kind === "remote"
|
||||
&& !input.restartRecovery && !retainedTransition && !remoteWarmRecovery
|
||||
&& attemptLaunchesFreshBootstrap(input.restartRecovery) && !retainedTransition && !remoteWarmRecovery
|
||||
? {
|
||||
binding: input.execution.binding,
|
||||
identity: { runnerInstanceId: effectiveRunnerInstanceId,
|
||||
|
||||
@@ -35,7 +35,7 @@ import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
||||
import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js";
|
||||
import { DurablePrpControlPlane, NativeProviderTerminalFailure } from "../../vendor/paperclip-runner/index.js";
|
||||
import { createNativeRunnerExecutionDriver } from "./native-runner-execution-driver.js";
|
||||
import { currentNativeControllerIdentity } from "./native-restart-recovery.js";
|
||||
import { currentNativeControllerIdentity, type NativeRestartRecoveryClaim } from "./native-restart-recovery.js";
|
||||
import { loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||
import { environmentRunnerIngressErrorData, JsonRpcCallError, PluginEnvironmentRunnerIngressError } from "@paperclipai/plugin-sdk";
|
||||
import { accountRunCost } from "../run-cost-accounting.js";
|
||||
@@ -290,6 +290,7 @@ describe("native provider startup retry", () => {
|
||||
|
||||
it.each([
|
||||
{ name: "transient 502", status: 502, retry: true },
|
||||
{ name: "transient 502 after a bootstrap_incomplete restart", status: 502, retry: true, restart: true },
|
||||
{ name: "credential 401", status: 401, retry: false },
|
||||
{ name: "permission 403", status: 403, retry: false },
|
||||
{ name: "invalid request 400", status: 400, retry: false },
|
||||
@@ -305,6 +306,19 @@ describe("native provider startup retry", () => {
|
||||
])("settles ingress failure safely: $name", async options => {
|
||||
const seeded = await seedRun(await createAgent());
|
||||
if (options.finalAttempt) await db.update(nativeRunFinalizations).set({ attempt: 2 }).where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
// A server restart interrupted the first bootstrap. The recovery scan's
|
||||
// claim makes this attempt launch a fresh bootstrap on the same run.
|
||||
let restartRecovery: NativeRestartRecoveryClaim | undefined;
|
||||
if (options.restart) {
|
||||
const controller = await currentNativeControllerIdentity();
|
||||
restartRecovery = { kind: "bootstrap_incomplete", runId: seeded.runId, leaseOwner: `restart-${seeded.runId}`,
|
||||
controllerGeneration: 2, providerAttempt: 0, restartKind: "hard", recoveryRequestId: null };
|
||||
await db.update(nativeRunFinalizations).set({ leaseOwner: restartRecovery.leaseOwner,
|
||||
leaseExpiresAt: new Date(Date.now() + 10 * 60_000), controllerBootId: controller.bootId,
|
||||
controllerPid: controller.pid, controllerHostHash: controller.hostHash ?? null,
|
||||
controllerProcessStartedAt: controller.processStartedAt, controllerGeneration: 2,
|
||||
recoveryState: "bootstrap_incomplete" }).where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
}
|
||||
const root = mkdtempSync(join(tmpdir(), "native-ingress-bootstrap-"));
|
||||
const server = createServer((_request, response) => {
|
||||
response.writeHead(options.status, { "content-type": "text/html" });
|
||||
@@ -386,6 +400,7 @@ describe("native provider startup retry", () => {
|
||||
try {
|
||||
await expect(executePaperclipNativeSession({ db, execution: seeded.execution,
|
||||
runnerInstanceId: seeded.runnerInstanceId, useRunnerd: true,
|
||||
...(restartRecovery ? { leaseOwner: restartRecovery.leaseOwner, restartRecovery } : {}),
|
||||
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", runner: {} } as never,
|
||||
backend: { descriptor: async () => { throw new Error("unexpected descriptor"); },
|
||||
openSession: async () => { throw new Error("unexpected provider session"); } },
|
||||
|
||||
Reference in new issue
Block a user