From 85bb942aad686c197f858af1a037e955e91129f0 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 20:07:02 -0600 Subject: [PATCH] fix(native): prove an unconnected bootstrap from the sandbox's own Runner state on a volume-free server (#20992) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through a Runner (`paperclip-runnerd`) in a remote sandbox. When a Runner exits before `session.open`, the run restarts its bootstrap on the same run. > - #20972 lets host preparation restart that bootstrap when the Runner wrote `runner-state.json` but no provider state. It needs a proof that this run's own interrupted Runner is stopped or gone. > - That proof is recorded only when the server still has the run's local Runner controller root. A volume-free server, or a root that held no identity, skips it. The live chaos wave after #20972 still failed with `runner_harness_state_mismatch: resumed_unavailable`. > - This pull request lets host preparation establish the same proof from durable sandbox evidence when no local proof exists. > - The benefit is that the restart also works on volume-free servers, which is how Cloud runs. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261112T002811-s6402` ran #20972 (Claude through ACPX, Daytona warm, `--ephemeral-server`, K1/K2/K15 with `--k15-fresh-home`, seed 6402, `--runner-bind-fault`). 5 of 6 tasks needed Retry, with 6 `resumed_unavailable` failures. Run `fbb1666d` in t01: the sandbox held `runner-state.json` for this run with `lifecycle: connecting`, the same Runner instance and session, `lastConnectionProtocolVersion: null`, and no provider state. That is exactly the state #20972 restarts, but its proof was absent. **Root cause** `recordRemoteBootstrapRestartProof` is called only inside the `existsSync(scoped)` block of the restart path in `server/src/services/native-runtime/native-session-executor.ts`, after the local controller identity is read. With no local root (volume-free, a fresh home, or an emptied root), `provenOwnBootstrapRestart` stays false, so `inspectOwnUnpreparedBootstrap` from #20972 returns nothing and the resumed branch throws `resumed_unavailable`. **Expected behavior** A run with a `bootstrap_incomplete` recovery claim whose sandbox Runner state proves it never connected restarts its bootstrap without Retry. ## What Changed - `inspectOwnUnpreparedBootstrap` now runs for any `bootstrap_incomplete` claim of this run. When no proof was recorded, it records one only if the sandbox Runner state proves it never connected (`remoteRunnerStateNeverConnected`: `lastConnectionProtocolVersion: null`, `lastControllerCommandSeq: 0`, `reconnectCount: 0`, empty outbox and processed commands, no pending terminal delivery). The exact Runner process must also be stopped or proven absent by the existing `stopStaleRemoteRunnerBeforeBootstrap`, which uses the Runner state's own identity. All earlier conditions from #20972 still apply: same run, Runner instance and session, a lifecycle other than `suspended`, no failover backup and no archived rotation snapshot. ## Verification - Red on master (#20972 merged): `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"`. The new `interrupted_bootstrap_unprepared_unrecorded` cells failed: resumed lease `Error: runner_harness_state_mismatch: resumed_unavailable`, created lease no restart. - Green: those cells pass. `_connected` (a Runner state that connected, no local proof) and `_unproven` (the stop is not proven) keep the refusal. The full file passes (1409 of 1409). Server `tsc --noEmit` passes. - Chaos, same cell with `--runner-bind-fault`. Wave `w-20261112T002811-s6402` (#20972 only): 1 of 6 passed, 6 `resumed_unavailable`. Wave `w-20261112T010800-s6402` (this change): 3 of 6 passed, 2 `resumed_unavailable`. In this wave, run `0a080e83` restarted its bootstrap three times on the same sandbox: three `.bootstrap-restart-*` roots were moved aside. The baseline wave on master before #20972 (`w-20261111T230537-s6402`, no injected fault) passed 2 of 6. ## Risks - The new path runs only for a `bootstrap_incomplete` claim of this run when the full harness check fails, so the normal path is unchanged and adds no wait before `turn.start`. - A Runner state that ever connected, received a command or holds outbox data keeps today's refusal. The moved-aside root is kept for inspection. - Not fixed here, and seen in both waves: every Runner on a sandbox listens on one fixed port (43127 in all captures). A live Runner of another run or session on the same sandbox therefore makes each relaunch fail with `runner_ingress_bind_conflict`, until the bounded retry is exhausted (`native_session_retry_exhausted`). A later attempt can still end with `resumed_unavailable` or `resumed_not_suspended`. That needs its own fix in predecessor retirement or port selection. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20972, #20928, #20953, #20863. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../native-session-executor.test.ts | 17 +++++++++++--- .../native-runtime/native-session-executor.ts | 22 ++++++++++++++----- 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 1bf347ec6ba..b0450623921 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -21109,10 +21109,12 @@ describe("runnerd provider runtime wiring", () => { ...["interrupted_bootstrap", "interrupted_bootstrap_other_run", "interrupted_bootstrap_unproven", "interrupted_bootstrap_symlink_parent", "interrupted_bootstrap_unopened", "interrupted_bootstrap_unopened_unproven", "interrupted_bootstrap_unopened_other_run", "interrupted_bootstrap_unopened_active_turn", "interrupted_bootstrap_unprepared", "interrupted_bootstrap_unprepared_unproven", - "interrupted_bootstrap_unprepared_other_run", "interrupted_bootstrap_unprepared_suspended", "interrupted_bootstrap_unprepared_backup"] + "interrupted_bootstrap_unprepared_other_run", "interrupted_bootstrap_unprepared_suspended", "interrupted_bootstrap_unprepared_backup", + "interrupted_bootstrap_unprepared_unrecorded", "interrupted_bootstrap_unprepared_connected"] .map(scenario => ({ outcome: "resumed", scenario, homeMode: "isolated" })), { outcome: "created", scenario: "interrupted_bootstrap", homeMode: "isolated" }, { outcome: "created", scenario: "interrupted_bootstrap_unprepared", homeMode: "isolated" }, + { outcome: "created", scenario: "interrupted_bootstrap_unprepared_unrecorded", homeMode: "isolated" }, { outcome: "created", scenario: "interrupted_bootstrap_symlink_parent", homeMode: "isolated" }, { outcome: "created", scenario: "interrupted_bootstrap_unopened", homeMode: "isolated" }, { outcome: "created", scenario: "interrupted_bootstrap_unopened_active_turn", homeMode: "isolated" }, @@ -21162,6 +21164,11 @@ describe("runnerd provider runtime wiring", () => { } if (command.args?.[1]?.includes("base64") && scenario.startsWith("interrupted_bootstrap_unprepared") && command.args[1].includes("codex-provider-state.json")) return { exitCode: 1, timedOut: false, stdout: "", stderr: "" }; + if (command.args?.[2] === "paperclip-runner-recovery-identity" && scenario === "interrupted_bootstrap_unprepared_unrecorded") return { + exitCode: 0, timedOut: false, stderr: "", + stdout: "exitednonce\n4242\n2026-11-12T00:32:10.725Z\nrunner-new-in-retained-workspace\nlinux:abcd-1234:100\n" }; + if (command.args?.[2] === "paperclip-runner-recovery-check" && scenario === "interrupted_bootstrap_unprepared_unrecorded") return { + exitCode: 45, timedOut: false, stdout: "", stderr: "" }; if (command.args?.[1]?.includes("base64") && scenario.startsWith("interrupted_bootstrap_unopened") && command.args[1].includes("codex-provider-state.json") && !command.args[1].includes("runner-state.json")) return { exitCode: 0, timedOut: false, stderr: "", stdout: Buffer.from(JSON.stringify({ @@ -21175,7 +21182,10 @@ describe("runnerd provider runtime wiring", () => { runId: scenario.endsWith("_other_run") ? "previous-warm-run" : execution.binding.runId, lifecycle: scenario.startsWith("interrupted_bootstrap_unopened") || scenario.endsWith("_suspended") ? "suspended" : scenario.startsWith("interrupted_bootstrap_unprepared") ? "connecting" : "ready", - outbox: [], pendingTerminalDelivery: null })).toString("base64"), + outbox: [], pendingTerminalDelivery: null, + ...(scenario.startsWith("interrupted_bootstrap_unprepared") ? { lastControllerCommandSeq: 0, reconnectCount: 0, + processedCommands: {}, lastConnectionProtocolVersion: scenario.endsWith("_connected") ? 2 : null, + environmentLeaseId: "lease-resumed", turnId: `turn-${execution.binding.runId}`, itemId: `item-${execution.binding.runId}` } : {}) })).toString("base64"), }; if (command.args?.[1]?.includes("base64")) return { exitCode: 1, timedOut: false, stdout: "", stderr: "", @@ -21272,7 +21282,8 @@ describe("runnerd provider runtime wiring", () => { "interrupted_bootstrap_unprepared_backup"].includes(scenario)) { noteRemoteBootstrapRestartProven(execution.binding.runId, "stopped"); } - if (scenario === "interrupted_bootstrap" || scenario === "interrupted_bootstrap_unopened" || scenario === "interrupted_bootstrap_unprepared") { + if (scenario === "interrupted_bootstrap" || scenario === "interrupted_bootstrap_unopened" || scenario === "interrupted_bootstrap_unprepared" + || scenario === "interrupted_bootstrap_unprepared_unrecorded") { await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined(); const siblings = await readdir(dirname(sessionRoot)); expect(siblings.filter(name => name.startsWith(`${basename(sessionRoot)}.bootstrap-restart-`))).toHaveLength(1); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 7fbcab9816e..e1e347fed8d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -14686,6 +14686,14 @@ export async function readRemoteRunnerState(input: { } } +export function remoteRunnerStateNeverConnected(state: Record): boolean { + const empty = (value: unknown) => value !== null && typeof value === "object" && !Array.isArray(value) + && Object.keys(value).length === 0; + return state.lastConnectionProtocolVersion === null && state.lastControllerCommandSeq === 0 + && state.reconnectCount === 0 && Array.isArray(state.outbox) && state.outbox.length === 0 + && state.pendingTerminalDelivery === null && empty(state.processedCommands); +} + async function readRemoteRunnerProviderState(input: { runner: CommandManagedRuntimeRunner; stateDirectory: string; @@ -16551,16 +16559,20 @@ async function createRunnerdBackendWithinSessionClaim( (state.incompleteReason === "not_suspended" || state.incompleteReason === "provider_unopened") && provenOwnBootstrapRestart(state.notSuspendedRunId); const inspectOwnUnpreparedBootstrap = async (): Promise => { + const runId = input.execution.binding.runId; if (!remoteCommandRunner || !remoteStateDirectory || archivedRotationRunnerStatePath - || !provenOwnBootstrapRestart(input.execution.binding.runId) + || input.restartRecovery?.kind !== "bootstrap_incomplete" || input.restartRecovery.runId !== runId || harnessBackupCandidates(root).some(candidate => existsSync(resolve(candidate, "manifest.json")))) return null; const runnerState = await readRemoteRunnerState({ runner: remoteCommandRunner, stateDirectory: remoteStateDirectory }) .then(record, () => null); - if (!runnerState || runnerState.runnerInstanceId !== input.runnerInstanceId + if (!runnerState || runnerState.runnerInstanceId !== input.runnerInstanceId || runnerState.runId !== runId || runnerState.normalizedSessionId !== nativeSessionKey(input.execution) - || typeof runnerState.lifecycle !== "string" || runnerState.lifecycle === "suspended" - || !provenOwnBootstrapRestart(typeof runnerState.runId === "string" ? runnerState.runId : null)) return null; - return input.execution.binding.runId; + || typeof runnerState.lifecycle !== "string" || runnerState.lifecycle === "suspended") return null; + if (!hasRemoteBootstrapRestartProof(runId) && !(remoteRunnerStateNeverConnected(runnerState) + && recordRemoteBootstrapRestartProof(runId, await stopStaleRemoteRunnerBeforeBootstrap({ + execution: input.execution, runnerExecutionTarget: input.runnerExecutionTarget, identity: runnerState, onLog: input.onLog, + })))) return null; + return provenOwnBootstrapRestart(runId) ? runId : null; }; const inspectRemoteHarnessState = async ( source: "live" | "host_preparation" = "live",