From bd97a3e6f7bf1bf8ab5cd2dc7593a189b76927ad Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 19:21:32 -0600 Subject: [PATCH] fix(native): retire a finished per-turn owner past its own quarantined interrupted bootstrap (#20140) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A follow-up turn on a per-turn session must retire the previous turn's runner state before it starts a new runner. Retirement proves the previous owner finished, and it also proves that every quarantined copy of that session root is settled > - If restart recovery interrupts a run's own start (for example a hard kill during `session.open`), recovery proves the half-started bootstrap held no provider work, quarantines that root, and bootstraps the same run again. The quarantined copy keeps the run's identity > - After that run finished, retirement checked the old bootstrap copy as if it were a finished owner. An unfinished `session.open` (or an opened session without a turn) is never "settled", so the follow-up failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate` and needed a Retry > - This pull request skips a quarantined copy of the same owner when the same bootstrap proof that let recovery quarantine it still holds > - The benefit is that a follow-up after a restart-interrupted start continues on its own ## Linked Issues or Issue Description Refs #20137 **What happened?** Local restart-chaos K13 regression storm w-20261108T235957-s9401 (master 4420a5adcf): 102 of 105 tasks passed. Two of the three failures share this cause. - **t58 (codex C1):** - A K1 at 00:15:42 interrupted run `78dd97f1` during its first `session.open`. Recovery quarantined that root (`run.prepare` completed, `session.open` pending, no events) and bootstrapped again. - The run then completed. A second K1 hit 190 ms after its result was accepted, and recovery finalized the result. - Follow-up `def668a6` failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=quarantine_journal_commands_unsettled(session.open.pending)`. - **t67 (claude C1):** the same pattern. The quarantined copy of run `990936c0` holds a completed `session.open` with only session-open events and no turn, so the refusal was `quarantine_journal_terminal_unsettled`. Both copies are in the kept instance's quarantine directory under the previous run's own identity. Each Retry succeeded. **Expected behavior** A quarantined copy of the finished owner's own earlier bootstrap that provably held no provider work does not hold the follow-up. Retirement proceeds and the follow-up starts without a Retry. **Steps to reproduce** 1. Let a per-turn run's first bootstrap be quarantined after a restart (the run bootstraps again and finishes). 2. Start a follow-up turn. 3. Before this change, retirement refuses with `quarantine_journal_commands_unsettled(session.open.pending)` or `quarantine_journal_terminal_unsettled`. This is not a code regression. The quarantine-copy check dates from #19061. Today's recovery fixes let more interrupted starts bootstrap again within the same run instead of blocking, and each such restart leaves this kind of copy behind. **Paperclip version or commit** 4420a5adcf (wave). Branch rebased on current master. ## What Changed - `native-session-executor.ts`: the quarantine loop in `inspectFinishedPerTurnRunnerdOwner` skips a copy when both of these hold: - its durable identity matches the finished owner exactly; - `runnerdStateProvesIncompleteBootstrap` holds for it. That is the proof restart recovery uses before it quarantines an interrupted bootstrap: no provider turn, and for a local runner, its journal and processes proven gone. Every other copy still has to pass the settled-projection proof. - `native-recovered-owner-followup.test.ts`: a red/green test covering both cases: - an interrupted bootstrap copy is retired past; - a copy that had already issued a turn still holds the follow-up. ## Verification - Red/green: on master, the new admitted case fails with exactly the t58 error. With the change it is admitted. The turn-issued copy is refused on both. - `vitest run` of `native-recovered-owner-followup.test.ts`, `native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts`: 3 files, 1504 tests passed (14 skipped). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed. ## Risks - Low risk. The skip reuses the existing bootstrap-incomplete proof and requires an exact identity match, so a foreign or work-holding copy still blocks. Nothing is deleted; the copy stays in quarantine. - Not covered: t22 in the same storm, a follow-up whose own unconnected `run.attach` root was hit by a K1 (`authority_indeterminate`), has a different shape and is tracked separately. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-recovered-owner-followup.test.ts | 62 ++++++++++++++++++- .../native-runtime/native-session-executor.ts | 10 +++ 2 files changed, 71 insertions(+), 1 deletion(-) diff --git a/server/src/services/native-runtime/native-recovered-owner-followup.test.ts b/server/src/services/native-runtime/native-recovered-owner-followup.test.ts index 009f20b6afd..40520c67178 100644 --- a/server/src/services/native-runtime/native-recovered-owner-followup.test.ts +++ b/server/src/services/native-runtime/native-recovered-owner-followup.test.ts @@ -39,7 +39,8 @@ vi.mock("./current-wake-comments.js", async importOriginal => ({ resolveCurrentWakeCommentsBinding: async () => null, assertCurrentWakeCommentsRead: async () => undefined, })); -import { buildNativeHarnessBackupManifest, createRunnerdBackend, resolveNativeHarnessPersistenceProfile, scopedRunnerdStateRoot } from "./native-session-executor.js"; +import { buildNativeHarnessBackupManifest, createRunnerdBackend, quarantinedBootstrapOfOwnerHeldNoProviderWork, resolveNativeHarnessPersistenceProfile, + scopedRunnerdStateRoot } from "./native-session-executor.js"; // Durable-root admission is the boundary under test. Reaching the run's tool // authority proves admission passed; no provider or runner process starts. @@ -648,6 +649,34 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered expect(await quarantined()).toHaveLength(1); }, 60_000); + it.each([ + { label: "an interrupted bootstrap that never opened its session", turnIssued: false, sessionOpened: false }, + { label: "a remote bootstrap that opened its session without a turn", turnIssued: false, sessionOpened: true }, + { label: "a bootstrap copy that already issued a turn", turnIssued: true, sessionOpened: false }, + ])("retires a finished owner past its own quarantined earlier bootstrap: $label", async ({ turnIssued, sessionOpened }) => { + const prior = await seedRecoveredPriorRun(); + const followUp = await admitFollowUp(prior); + await db.update(nativeRunFinalizations).set({ phase: "observed" }).where(eq(nativeRunFinalizations.runId, prior.runId)); + expect(await followUp.start()).toBe(stranded("owner_observed")); + const [copy] = await quarantined(); + const copyRoot = join(stateDirectory, "quarantine", copy!); + const path = join(copyRoot, "control-plane", "control-plane-state.json"); + const state = JSON.parse(await readFile(path, "utf8")); + const bootstrap = state.commands.filter((command: { type: string }) => command.type === "run.prepare" || command.type === "session.open" + || (turnIssued && command.type === "turn.start")); + for (const command of bootstrap) { + command.status = command.type === "run.prepare" || (sessionOpened && command.type === "session.open") ? "completed" : "pending"; + } + const opened = sessionOpened ? ["session.started", "session.capabilities.updated"].map((eventType, index) => ({ + sourceSeq: index + 1, sourceEventId: `opened-${index + 1}`, eventType, priority: 1, envelope: { ...prior.identity }, + deliveryCount: 1, logicalEffectCount: 1 })) : []; + await writeFile(path, JSON.stringify({ ...state, commands: bootstrap, committedEvents: opened, ackedSourceSeq: opened.length })); + await rm(join(copyRoot, "runner"), { recursive: true, force: true }); + await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, prior.runId)); + expect(await followUp.start()).toBe(turnIssued + ? stranded("quarantine_journal_commands_unsettled(session.open.pending,turn.start.pending)") : ADMITTED); + }, 60_000); + it("keeps an unsuspended Runner receipt decisive after a retry rebuilds the root", async () => { const prior = await seedRecoveredPriorRun(); const followUp = await admitFollowUp(prior); @@ -770,6 +799,37 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered return { prior, followUp, root }; } + it.each(["exited", "alive", "another owner"] as const)( + "skips a local quarantined bootstrap that opened a session only when it is this owner's and its processes %s", + async scenario => { + const runner = scenario === "alive" ? await runnerProcess() : await stoppedRunnerProcess(); + const root = await mkdtemp(join(tmpdir(), "quarantined-opened-bootstrap-")); + const identity = { runnerInstanceId: "runner-1", environmentLeaseId: "lease-1", runId: "owner-run", + normalizedSessionId: "session-1", turnId: "turn-owner-run", itemId: "item-owner-run" }; + try { + await mkdir(join(root, "control-plane"), { recursive: true, mode: 0o700 }); + await mkdir(join(root, "runner"), { recursive: true, mode: 0o700 }); + await writeFile(join(root, "control-plane", "control-plane-state.json"), JSON.stringify({ + schema: "paperclip.runner.durable.control-plane-state.v2", identity, + bootstrapHistory: { schema: "paperclip.runner.bootstrap-history.v1", originIdentity: identity, credentialsIssued: 1, adopted: false }, + commands: [ + { commandId: "prepare", type: "run.prepare", status: "completed", result: { status: "prepared" } }, + { commandId: "open", type: "session.open", status: "completed", result: { result: { status: "started", processId: runner.pid } } }, + ], + committedEvents: [{ sourceSeq: 1, sourceEventId: "opened-1", eventType: "session.started", priority: 1, envelope: identity }], + ackedSourceSeq: 1, connectionCount: 1 })); + await writeFile(join(root, "runner", "runner-state.json"), JSON.stringify({ schema: "paperclip.runner.durable.state.v1", + ...identity, lifecycle: "ready", outbox: [], processedCommands: { + prepare: { commandType: "run.prepare", status: "completed", result: { status: "prepared" } }, + open: { commandType: "session.open", status: "completed", result: { status: "started", processId: runner.pid } } } })); + const owner = scenario === "another owner" ? { ...identity, runId: "another-run" } : identity; + expect(quarantinedBootstrapOfOwnerHeldNoProviderWork(root, owner, true)).toBe(scenario === "exited"); + } finally { + if (scenario === "alive") runner.child.kill("SIGKILL"); + await rm(root, { recursive: true, force: true }); + } + }, 60_000); + it("retires a finished local owner whose Runner stopped before it proved suspension", async () => { const { prior, followUp, root } = await seedUnsuspendedLocalOwner(await stoppedRunnerProcess()); expect(await followUp.startLocal()).toBe(ADMITTED); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index bdc1a2c5b92..42d038a2508 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -5612,6 +5612,7 @@ async function inspectFinishedPerTurnRunnerdOwner(input: { if (!isSafeNativeStateDirectory(quarantine)) return refuse("quarantine_unsafe"); for (const name of readdirSync(quarantine)) { if (!name.startsWith(`${basename(input.root)}.`)) continue; + if (quarantinedBootstrapOfOwnerHeldNoProviderWork(resolve(quarantine, name), identity, localRunner)) continue; const copy = readSettledRunnerdProjection(resolve(quarantine, name), identity, { localRunner }); if (copy.refused !== undefined) return refuse(`quarantine_${copy.refused}`); } @@ -5627,6 +5628,15 @@ async function inspectFinishedPerTurnRunnerdOwner(input: { return { finished: true, run: prior.run, recheck }; } +export function quarantinedBootstrapOfOwnerHeldNoProviderWork(root: string, identity: Record, + localRunner: boolean): boolean { + if (!isSafeNativeStateDirectory(root)) return false; + const copied = readRunnerdDurableIdentity(root); + if (!copied || Object.entries(identity).some(([key, value]) => copied[key] !== value)) return false; + return runnerdStateProvesIncompleteBootstrap(root, { localProcesses: localRunner }) + || (!localRunner && runnerdStateProvesNoRemoteProviderTurn(root)); +} + async function retireFinishedPerTurnRunnerdOwner(input: Parameters[0]) : Promise<{ retired: true } | { retired: false; refused: string }> { const proof = await inspectFinishedPerTurnRunnerdOwner(input);