From 66344b76ae380f263ea3d1786fc9298acf0c3780 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 18:36:33 -0600 Subject: [PATCH] fix(native): restart a bootstrap whose Runner exited before writing provider state on a resumed sandbox (#20972) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through a Runner (`paperclip-runnerd`) inside a remote sandbox. A warm Daytona sandbox is reused by later runs. > - When a Runner exits before `session.open`, the run does a bounded same-run bootstrap retry. Durable evidence proves that no provider session and no provider event exist. > - The retry fails when the Runner wrote `runner-state.json` but exited before it wrote its provider state. Host preparation then calls the sandbox state "unavailable" and refuses with `runner_harness_state_mismatch: resumed_unavailable`. The task needs Retry. > - This pull request reads the Runner state on its own when the full check fails, but only for a proven bootstrap restart of the same run. A non-suspended Runner state of this run is then the run's own interrupted bootstrap, and the existing restart path moves it aside and claims a new session root. > - The benefit is that a Runner that exits before `session.open` on a reused sandbox recovers without a person. Saved state, a verified backup and every other mismatch keep their refusal. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX, Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with `--k15-fresh-home`, seed 6402) needed Retry on 4 of 6 tasks, with the same signature after each restart kind. The first attempt of each run reused the warm sandbox. Its Runner failed with `runner_ingress_bind_conflict: failed to bind listener: Address already in use` and exited with `lifecycle=connecting`. Recovery logged "Restarting an incomplete native runner bootstrap on the same heartbeat run". The second attempt reused the environment and failed with `runner_harness_state_mismatch: resumed_unavailable`. The earlier wave `w-20261111T212429-s6402` had one run with the same error code. **Root cause** `inspectRemoteHarnessState` in `server/src/services/native-runtime/native-session-executor.ts` checks the persistence directories, `runner-state.json` and the provider state file in one shell test. A Runner that exits at bind time wrote `runner-state.json` but no provider state, so the one test fails and the result is `incompleteReason: "unavailable"` before the Runner state is read. The restart path (`restartableBootstrapState`) accepts only `not_suspended` and `provider_unopened`, so it is skipped. `claimUntouchedRemoteSession` then cannot create the session root that the first attempt created, and the resumed branch throws `resumed_unavailable`. On a created lease the same state fell through without a restart. **Expected behavior** A proven, interrupted bootstrap of the same run restarts on the same sandbox. The run finishes without Retry. **Steps to reproduce** Run the s6402 cell above, or add `--runner-bind-fault` (this pull request) to force the Runner exit on every run's first launch. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: when the full harness check fails during host preparation, `inspectOwnUnpreparedBootstrap` reads `runner-state.json` alone. It returns `not_suspended` for this run only when all of these are true: this run has a proven bootstrap restart (`bootstrap_incomplete` recovery plus the stop proof), no failover backup exists, no archived rotation snapshot is in use, and the Runner state names this run, this Runner instance and this session with a lifecycle other than `suspended`. Otherwise the result stays `unavailable`, as before. The existing restart path then moves the session root aside (it is kept, not deleted) and claims a new one. - Chaos coverage: `--runner-bind-fault` in `tests/restart-chaos` gives every server of the wave `PAPERCLIP_TEST_RUNNER_BIND_FAULT=first_launch`. With `NODE_ENV=test` only, the remote Runner launcher starts a short-lived listener on the Runner's listen port before a Runner's first launch on that server, so the Runner exits with a real `runner_ingress_bind_conflict`. Documented in `tests/restart-chaos/README.md`. ## Verification - Red on master: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"` failed 2 of the new cells. Resumed lease: `Error: runner_harness_state_mismatch: resumed_unavailable` (the staging error). Created lease: no restart happened (`expected [] to have a length of 1 but got +0`). - Green: the same cells pass. The negative cells keep `resumed_unavailable`: no restart proof, a Runner state of another run, a `suspended` Runner state with missing provider state, and a failover backup present. - `native-session-executor.test.ts`: 1406 of 1406 pass. A new test proves the bind fault holds the listen port (a second bind gets `EADDRINUSE`) and fires once per Runner instance. - `tests/restart-chaos`: `runner-bind-fault.test.ts`, `question-handoff.test.ts`, `shape-cells.test.ts` pass. Server and harness `tsc --noEmit` pass. - Chaos wave: see the comment below for the before/after counts. ## Risks - The new read runs only for a proven bootstrap restart of the same run with a failed full check, so the normal path has no extra command and no wait before `turn.start`. - A partly saved session (`suspended` Runner state without provider state), a Runner state of another run and any failover backup keep today's refusal. The moved-aside root is kept for inspection. - A Runner that started under an older server and is restarted by this version takes the same path, because it reads only files that older Runners already write. - The bind fault is test instrumentation. It is inert unless `NODE_ENV=test` and the variable are both set. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20928 (restart a Runner that exited before session.open), #20874, #20863, #20953. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-session-executor.test.ts | 70 +++++++++++++++++-- .../native-runtime/native-session-executor.ts | 46 ++++++++++++ tests/restart-chaos/README.md | 20 +++++- tests/restart-chaos/config.ts | 7 +- tests/restart-chaos/main.ts | 3 +- tests/restart-chaos/runner-bind-fault.test.ts | 25 +++++++ tests/restart-chaos/session-faults.ts | 4 ++ tests/restart-chaos/types.ts | 1 + 8 files changed, 167 insertions(+), 9 deletions(-) create mode 100644 tests/restart-chaos/runner-bind-fault.test.ts diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index f798686a118..1bf347ec6ba 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -79,6 +79,7 @@ import { } from "@paperclipai/paperclip-runner"; import { existsSync } from "node:fs"; import { createHash, generateKeyPairSync, randomUUID } from "node:crypto"; +import { createServer as createNetServer } from "node:net"; import { DatabaseSync } from "node:sqlite"; import { nativeSha256 } from "./canonical.js"; import { resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; @@ -1393,6 +1394,54 @@ describe("remote runner process supervision", () => { } }); + it("holds the runner listen port once per runner before its first launch under the chaos bind fault", async () => { + vi.stubEnv("PAPERCLIP_TEST_RUNNER_BIND_FAULT", "first_launch"); + onTestFinished(() => { vi.unstubAllEnvs(); }); + const port = await new Promise((resolvePort, reject) => { + const probe = createNetServer().listen(0, "127.0.0.1", () => { + const address = probe.address(); + probe.close(() => typeof address === "object" && address ? resolvePort(address.port) : reject(new Error("no port"))); + }); + }); + const execute = vi.fn(async (request: { args: string[] }) => { + if (request.args[2] === "paperclip-runner-test-bind-fault") { + return { exitCode: spawnSync("sh", request.args, { stdio: "ignore" }).status ?? 1, timedOut: false, stdout: "", stderr: "" }; + } + return { exitCode: 1, timedOut: false, stdout: "", stderr: "" }; + }); + const onLog = vi.fn(async () => undefined); + const launcher = createRemoteRunnerProcessLauncher({ + target: { kind: "remote", transport: "sandbox", environmentId: "environment", leaseId: "lease", remoteCwd: "/workspace" }, + runner: { execute } as never, + remoteBinary: "/runtime/runnerd", processIdentityPath: "/runtime/identity", + stateDirectory: "/runtime", diagnosticsDirectory: "/runtime/diagnostics", runnerInstanceId: `runner-bind-fault-${randomUUID()}`, + onLog, + }); + const spec = { command: "/controller/runnerd", args: ["--listen-address", "127.0.0.1", "--listen-port", String(port)], + cwd: "/controller", environment: {} }; + await expect(launcher(spec).completion).rejects.toThrow("runner_remote_process_launch_failed"); + const bindFailure = await new Promise(resolveBind => { + const contender = createNetServer(); + contender.once("error", (error: NodeJS.ErrnoException) => resolveBind(error.code ?? "error")); + contender.listen(port, "127.0.0.1", () => contender.close(() => resolveBind(null))); + }); + expect(bindFailure).toBe("EADDRINUSE"); + expect(onLog).toHaveBeenCalledWith("stderr", `[paperclip-runner] test bind fault armed on port ${port}\n`); + await expect(launcher(spec).completion).rejects.toThrow("runner_remote_process_launch_failed"); + expect(execute.mock.calls.filter(([request]) => request.args[2] === "paperclip-runner-test-bind-fault")).toHaveLength(1); + expect(execute.mock.calls.map(([request]) => request.args[2])).toEqual([ + "paperclip-runner-test-bind-fault", "paperclip-runner-launch", "paperclip-runner-launch"]); + const blockedLog = vi.fn(async () => undefined); + await expect(createRemoteRunnerProcessLauncher({ + target: { kind: "remote", transport: "sandbox", environmentId: "environment", leaseId: "lease", remoteCwd: "/workspace" }, + runner: { execute } as never, + remoteBinary: "/runtime/runnerd", processIdentityPath: "/runtime/identity", + stateDirectory: "/runtime", diagnosticsDirectory: "/runtime/diagnostics", runnerInstanceId: `runner-bind-fault-${randomUUID()}`, + onLog: blockedLog, + })(spec).completion).rejects.toThrow("runner_remote_process_launch_failed"); + expect(blockedLog).toHaveBeenCalledWith("stderr", `[paperclip-runner] test bind fault failed on port ${port}\n`); + }); + it("confirms no replacement process exists when staging fails before the launch RPC", async () => { // A full sandbox disk rejects the runtime-context upload before dispatch. const stagingError = new Error( @@ -21059,9 +21108,11 @@ describe("runnerd provider runtime wiring", () => { ...["resumed", "replacement"].map(outcome => ({ outcome, scenario: "fresh", homeMode: "preconfigured" })), ...["interrupted_bootstrap", "interrupted_bootstrap_other_run", "interrupted_bootstrap_unproven", "interrupted_bootstrap_symlink_parent", "interrupted_bootstrap_unopened", "interrupted_bootstrap_unopened_unproven", "interrupted_bootstrap_unopened_other_run", - "interrupted_bootstrap_unopened_active_turn"] + "interrupted_bootstrap_unopened_active_turn", "interrupted_bootstrap_unprepared", "interrupted_bootstrap_unprepared_unproven", + "interrupted_bootstrap_unprepared_other_run", "interrupted_bootstrap_unprepared_suspended", "interrupted_bootstrap_unprepared_backup"] .map(scenario => ({ outcome: "resumed", scenario, homeMode: "isolated" })), { outcome: "created", scenario: "interrupted_bootstrap", homeMode: "isolated" }, + { outcome: "created", scenario: "interrupted_bootstrap_unprepared", homeMode: "isolated" }, { outcome: "created", scenario: "interrupted_bootstrap_symlink_parent", homeMode: "isolated" }, { outcome: "created", scenario: "interrupted_bootstrap_unopened", homeMode: "isolated" }, { outcome: "created", scenario: "interrupted_bootstrap_unopened_active_turn", homeMode: "isolated" }, @@ -21109,6 +21160,8 @@ describe("runnerd provider runtime wiring", () => { try { execFileSync("sh", command.args, { stdio: "pipe" }); return { exitCode: 0, timedOut: false, stdout: "", stderr: "" }; } catch { return { exitCode: 1, timedOut: false, stdout: "", stderr: "" }; } } + if (command.args?.[1]?.includes("base64") && scenario.startsWith("interrupted_bootstrap_unprepared") + && command.args[1].includes("codex-provider-state.json")) return { exitCode: 1, timedOut: false, stdout: "", stderr: "" }; if (command.args?.[1]?.includes("base64") && scenario.startsWith("interrupted_bootstrap_unopened") && command.args[1].includes("codex-provider-state.json") && !command.args[1].includes("runner-state.json")) return { exitCode: 0, timedOut: false, stderr: "", stdout: Buffer.from(JSON.stringify({ @@ -21120,7 +21173,8 @@ describe("runnerd provider runtime wiring", () => { exitCode: 0, timedOut: false, stderr: "", stdout: Buffer.from(JSON.stringify({ runnerInstanceId: "runner-new-in-retained-workspace", normalizedSessionId: execution.session.normalizedSessionId, runId: scenario.endsWith("_other_run") ? "previous-warm-run" : execution.binding.runId, - lifecycle: scenario.startsWith("interrupted_bootstrap_unopened") ? "suspended" : "ready", + lifecycle: scenario.startsWith("interrupted_bootstrap_unopened") || scenario.endsWith("_suspended") ? "suspended" + : scenario.startsWith("interrupted_bootstrap_unprepared") ? "connecting" : "ready", outbox: [], pendingTerminalDelivery: null })).toString("base64"), }; if (command.args?.[1]?.includes("base64")) return { @@ -21208,15 +21262,17 @@ describe("runnerd provider runtime wiring", () => { if (scenario === "missing_control_plane") { await rm(join(options.stateDirectory!, "control-plane", "control-plane-state.json")); } - if (scenario === "backup_present") { + if (scenario === "backup_present" || scenario === "interrupted_bootstrap_unprepared_backup") { await mkdir(join(options.stateDirectory!, "failover-backups", "current"), { recursive: true }); await writeFile(join(options.stateDirectory!, "failover-backups", "current", "manifest.json"), "{}"); } if (["interrupted_bootstrap", "interrupted_bootstrap_other_run", "interrupted_bootstrap_symlink_parent", - "interrupted_bootstrap_unopened", "interrupted_bootstrap_unopened_other_run", "interrupted_bootstrap_unopened_active_turn"].includes(scenario)) { + "interrupted_bootstrap_unopened", "interrupted_bootstrap_unopened_other_run", "interrupted_bootstrap_unopened_active_turn", + "interrupted_bootstrap_unprepared", "interrupted_bootstrap_unprepared_other_run", "interrupted_bootstrap_unprepared_suspended", + "interrupted_bootstrap_unprepared_backup"].includes(scenario)) { noteRemoteBootstrapRestartProven(execution.binding.runId, "stopped"); } - if (scenario === "interrupted_bootstrap" || scenario === "interrupted_bootstrap_unopened") { + if (scenario === "interrupted_bootstrap" || scenario === "interrupted_bootstrap_unopened" || scenario === "interrupted_bootstrap_unprepared") { await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined(); const siblings = await readdir(dirname(sessionRoot)); expect(siblings.filter(name => name.startsWith(`${basename(sessionRoot)}.bootstrap-restart-`))).toHaveLength(1); @@ -21276,7 +21332,9 @@ describe("runnerd provider runtime wiring", () => { expect(remoteExecute.mock.calls.filter(([command]) => command.args?.[2] === "paperclip-runner-claim-unstarted-session")).toHaveLength(1); } else { await expect(options.prepareExternalRunnerState()).rejects.toThrow( - outcome === "replacement" ? "runner_harness_backup_unavailable" : "runner_harness_state_mismatch", + outcome === "replacement" ? "runner_harness_backup_unavailable" + : scenario.startsWith("interrupted_bootstrap_unprepared") ? "runner_harness_state_mismatch: resumed_unavailable" + : "runner_harness_state_mismatch", ); expect(remoteExecute.mock.calls.some(([command]) => command.args?.[1]?.includes("install -d"))).toBe(false); expect(syncIn).not.toHaveBeenCalled(); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index c6d56c6d8af..e6f6e4e729c 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -15113,6 +15113,29 @@ async function cleanupRemoteRunnerAfterIdentityFailure(input: { return result?.exitCode === 0 && result.timedOut === false; } +const injectedRunnerBindConflicts = new Set(); + +async function injectRunnerBindConflictForTest(runner: CommandManagedRuntimeRunner, runnerInstanceId: string, + args: readonly string[], onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise): Promise { + if (process.env.NODE_ENV !== "test" || process.env.PAPERCLIP_TEST_RUNNER_BIND_FAULT !== "first_launch" + || injectedRunnerBindConflicts.has(runnerInstanceId)) return; + const option = (name: string) => args.includes(name) ? args[args.indexOf(name) + 1] ?? null : null; + const port = option("--listen-port"); + const address = option("--listen-address") ?? "127.0.0.1"; + if (!port || !/^\d+$/.test(port)) return; + injectedRunnerBindConflicts.add(runnerInstanceId); + const result = await runner.execute({ + command: "sh", + args: ["-c", + 'ready=$(mktemp); rm -f "$ready"; nohup node -e "const s = require(\'net\').createServer().listen(Number(process.argv[1]), process.argv[2], () => { require(\'fs\').writeFileSync(process.argv[3], \'\'); setTimeout(() => s.close(), 15000); })" "$1" "$2" "$ready" /dev/null 2>&1 & i=0; while [ ! -e "$ready" ] && [ $i -lt 50 ]; do sleep 0.1; i=$((i + 1)); done; test -e "$ready" && rm -f "$ready"', + "paperclip-runner-test-bind-fault", port, address], + bypassSession: true, + timeoutMs: 10_000, + }).catch(() => null); + const armed = result?.exitCode === 0 && !result.timedOut; + await onLog?.("stderr", `[paperclip-runner] test bind fault ${armed ? "armed" : "failed"} on port ${port}\n`).catch(() => undefined); +} + export function createRemoteRunnerProcessLauncher(input: { target: Extract; runner: CommandManagedRuntimeRunner; @@ -15251,6 +15274,7 @@ export function createRemoteRunnerProcessLauncher(input: { // into its own session instead; its own bounded diagnostics directory and // durable PRP state remain the authorities, and the controller monitors // the exact persisted process identity below. + await injectRunnerBindConflictForTest(runner, input.runnerInstanceId, remoteArgs, input.onLog); launchDispatched = true; const launchResult = await runner.execute({ command: "sh", @@ -16525,6 +16549,18 @@ async function createRunnerdBackendWithinSessionClaim( const restartableBootstrapState = (state: { incompleteReason: string | null; notSuspendedRunId?: string | null }) => (state.incompleteReason === "not_suspended" || state.incompleteReason === "provider_unopened") && provenOwnBootstrapRestart(state.notSuspendedRunId); + const inspectOwnUnpreparedBootstrap = async (): Promise => { + if (!remoteCommandRunner || !remoteStateDirectory || archivedRotationRunnerStatePath + || !provenOwnBootstrapRestart(input.execution.binding.runId) + || harnessBackupCandidates(root).some(candidate => existsSync(resolve(candidate, "manifest.json")))) return null; + const runnerState = await readRemoteRunnerState({ runner: remoteCommandRunner, stateDirectory: remoteStateDirectory }) + .then(record, () => null); + if (!runnerState || runnerState.runnerInstanceId !== input.runnerInstanceId + || runnerState.normalizedSessionId !== nativeSessionKey(input.execution) + || typeof runnerState.lifecycle !== "string" || runnerState.lifecycle === "suspended" + || !provenOwnBootstrapRestart(typeof runnerState.runId === "string" ? runnerState.runId : null)) return null; + return input.execution.binding.runId; + }; const inspectRemoteHarnessState = async ( source: "live" | "host_preparation" = "live", ): Promise<{ @@ -16569,6 +16605,16 @@ async function createRunnerdBackendWithinSessionClaim( timeoutMs: 10_000, }); if (inspected.exitCode !== 0 || inspected.timedOut) { + const unpreparedBootstrapRunId = source === "host_preparation" ? await inspectOwnUnpreparedBootstrap() : null; + if (unpreparedBootstrapRunId) { + return { + complete: false, + runnerState: null, + providerSessionIdentity: null, + incompleteReason: "not_suspended", + notSuspendedRunId: unpreparedBootstrapRunId, + }; + } return { complete: false, runnerState: null, diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index d270b74ba1a..f3302cc4f4f 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -26,7 +26,7 @@ Options: `--triggers time,comment,retry,queued,tool,result` (fault timing, below default 3000), `--poll-ms` (2000–5000), `--keep` (keep the temporary instance), `--k4-hold-max-seconds` (raise the seeded K4 hold upper end from 300 s, up to the 600 s outage cap). `--faults K0` plans cells with no fault, for workloads whose shape is the fault (W2, below). `--fail-recovery-step`, -`--long-turn-minutes`, `--k15-overlap-seconds`, `--k15-fresh-home` and `--ephemeral-server` are described in their own sections. +`--long-turn-minutes`, `--k15-overlap-seconds`, `--k15-fresh-home`, `--ephemeral-server` and `--runner-bind-fault` are described in their own sections. `--fault-after-seconds ` fires every planned fault `n` seconds after its task launches (5–3600; with `--triggers time`), keeping the rest of the seeded plan, for example to restart a Daytona R1 task after its workspace preparation. `--matrix` crosses every selected workload with every fault kind (below), and `--server-from ` @@ -518,6 +518,24 @@ workloads and refused when listed. loses the agents' instruction bundles, which staging keeps in Postgres and object storage. Every restart replays the server's startup eligibility check, so a replacement that refuses to start fails the wave. +### Runner exits before session.open (`--runner-bind-fault`) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-warm --storage s3 \ + --ephemeral-server --adapters claude --workloads W2 --faults K1,K2,K15 --k15-fresh-home --runner-bind-fault --seed 6402 +``` + +Staging's volume-free shape (wave `w-20261111T230537-s6402`) showed a reused warm sandbox whose Runner listen port +was still held by another Runner. The new Runner wrote `runner-state.json` (`lifecycle: connecting`), failed with +`runner_ingress_bind_conflict` and exited before `session.open`, so it never wrote its provider state. The same-run +bootstrap retry then found that partial session root and failed with `runner_harness_state_mismatch: resumed_unavailable`. +`--runner-bind-fault` gives every server of the wave `PAPERCLIP_TEST_RUNNER_BIND_FAULT=first_launch` (honoured only +with `NODE_ENV=test`): before a run's first Runner launch on that server, a short-lived process in the sandbox holds +the Runner's listen port for 15 s. The server waits until that listener has bound the port and writes +`[paperclip-runner] test bind fault armed on port ` (or `failed`) to the run log, so a wave can count the runs that +really took this path. Every run then takes it once per server generation, and the run must still finish without +Retry. It needs `--env daytona`. + ## Absolute symlink in the workspace (F1) ```sh diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts index f52210945b2..84ccf192008 100644 --- a/tests/restart-chaos/config.ts +++ b/tests/restart-chaos/config.ts @@ -145,6 +145,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC "k15-overlap-seconds": { type: "string" }, "k15-fresh-home": { type: "boolean" }, "ephemeral-server": { type: "boolean" }, + "runner-bind-fault": { type: "boolean" }, "tm1-hold-seconds": { type: "string" }, "fail-recovery-step": { type: "string" }, "long-turn-minutes": { type: "string" }, @@ -158,7 +159,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000); const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000); if (values.replay !== undefined) { - for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-pack-mismatch", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "k15-overlap-seconds", "k15-fresh-home", "ephemeral-server", "tm1-hold-seconds", "fail-recovery-step", "long-turn-minutes", "cr1-hold-seconds", "cr2-hold-seconds", "cr3-stall-seconds", "sandbox-watch-seconds"] as const) { + for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-pack-mismatch", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "k15-overlap-seconds", "k15-fresh-home", "ephemeral-server", "runner-bind-fault", "tm1-hold-seconds", "fail-recovery-step", "long-turn-minutes", "cr1-hold-seconds", "cr2-hold-seconds", "cr3-stall-seconds", "sandbox-watch-seconds"] as const) { if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`); } return { mode: "replay", ref: parseReplayRef(values.replay), keep, @@ -396,6 +397,10 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC if (repository.length) throw new Error(`${flag} cannot run ${repository.join(", ")}: the fixture repository is a local_path project source, which a volume-free server refuses; pick other --workloads`); options.ephemeralServer = true; } + if (values["runner-bind-fault"]) { + if (env !== "daytona") throw new Error("--runner-bind-fault holds a port inside a remote sandbox: run it with --env daytona"); + options.runnerBindFault = true; + } if (values["storage-faults"] !== undefined) { if (storage !== "s3") throw new Error("--storage-faults needs --storage s3"); options.storageFaults = list(values["storage-faults"], STORAGE_FAULT_KINDS, "storage-faults", []); diff --git a/tests/restart-chaos/main.ts b/tests/restart-chaos/main.ts index ded4a0efb10..16dc725ab9d 100644 --- a/tests/restart-chaos/main.ts +++ b/tests/restart-chaos/main.ts @@ -52,7 +52,7 @@ import type { RestartStep } from "./repeated-restarts.js"; import { isBrokenStdoutError, markTimelineEchoBroken, Timeline } from "./timeline.js"; import { guardedRunnerOutputs, worktreeFingerprint } from "./wave-guard.js"; import { checkCrossServer, readCrossServerEvidence } from "./multi-server.js"; -import { sessionFaultServerEnvironment } from "./session-faults.js"; +import { runnerBindFaultServerEnvironment, sessionFaultServerEnvironment } from "./session-faults.js"; import { controllerLogNames, gradeWarmShutdown, isWarmWave, warmShutdownMarkdown, type WarmShutdownCleanup } from "./warm-shutdown.js"; import type { FaultKind, FaultReceipt, OracleArtifacts, TriggerId, WavePlan } from "./types.js"; @@ -427,6 +427,7 @@ async function main(): Promise { if (plan.options.timeAgeMs) supervisor.timeAgeMs = plan.options.timeAgeMs; if (plan.options.rrStopOn) supervisor.rrStopOn = plan.options.rrStopOn; Object.assign(serverEnvironment, sessionFaultServerEnvironment(plan.options.sessionFaults)); + Object.assign(serverEnvironment, runnerBindFaultServerEnvironment(plan.options.runnerBindFault)); await supervisor.start(serverEnvironment); // TM2/TM3 age the database; the observer and grader read run times back at their real times. const api: ChaosApi = new TimeShiftedApi(supervisor.baseUrl, supervisor.agingShifts); diff --git a/tests/restart-chaos/runner-bind-fault.test.ts b/tests/restart-chaos/runner-bind-fault.test.ts new file mode 100644 index 00000000000..b9ba331d225 --- /dev/null +++ b/tests/restart-chaos/runner-bind-fault.test.ts @@ -0,0 +1,25 @@ +import { describe, expect, it } from "vitest"; +import { parseCommand } from "./config.js"; +import { runnerBindFaultServerEnvironment } from "./session-faults.js"; + +const image = `img@sha256:${"a".repeat(64)}`; + +describe("--runner-bind-fault", () => { + it("arms the server's first-launch bind fault on the s6402 volume-free shape", () => { + const command = parseCommand(["--env", "daytona", "--daytona-image", image, "--daytona-warm", "--storage", "s3", + "--ephemeral-server", "--adapters", "claude", "--workloads", "W2", "--faults", "K1,K2,K15", "--k15-fresh-home", + "--seed", "6402", "--runner-bind-fault"]); + if (command.mode !== "wave") throw new Error("expected a wave"); + expect(command.options.runnerBindFault).toBe(true); + expect(runnerBindFaultServerEnvironment(command.options.runnerBindFault)) + .toEqual({ PAPERCLIP_TEST_RUNNER_BIND_FAULT: "first_launch" }); + }); + + it("leaves the server unarmed without the flag and refuses a local wave", () => { + const command = parseCommand(["--env", "daytona", "--daytona-image", image, "--workloads", "W2", "--faults", "K2"]); + if (command.mode !== "wave") throw new Error("expected a wave"); + expect(command.options.runnerBindFault).toBeUndefined(); + expect(runnerBindFaultServerEnvironment(command.options.runnerBindFault)).toEqual({}); + expect(() => parseCommand(["--workloads", "W2", "--faults", "K2", "--runner-bind-fault"])).toThrow("--env daytona"); + }); +}); diff --git a/tests/restart-chaos/session-faults.ts b/tests/restart-chaos/session-faults.ts index 4f51f9dca75..923607d3c4b 100644 --- a/tests/restart-chaos/session-faults.ts +++ b/tests/restart-chaos/session-faults.ts @@ -36,6 +36,10 @@ export function sessionFaultServerEnvironment(faults: readonly SessionFaultKind[ return faults?.includes("SR1U") ? { PAPERCLIP_TEST_WARM_RETIREMENT_FAULT: "unconfirmed" } : {}; } +export function runnerBindFaultServerEnvironment(armed: boolean | undefined): Record { + return armed ? { PAPERCLIP_TEST_RUNNER_BIND_FAULT: "first_launch" } : {}; +} + /** What the board changed, recorded on the task's question. */ export interface SessionRotation { at: number; diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts index 8ad98fd558b..be413df9d46 100644 --- a/tests/restart-chaos/types.ts +++ b/tests/restart-chaos/types.ts @@ -87,6 +87,7 @@ export interface WaveOptions { /** `--k15-fresh-home`: each deploy generation gets a new, empty instance home and no volume (redeploy.ts). */ k15FreshHome?: boolean; ephemeralServer?: boolean; + runnerBindFault?: boolean; /** `--faults K0`: cells with no fault, for workloads whose shape is the fault (W2, W2Q, W2C). */ baseline?: boolean; /**