From 3eea8b87bd61c5249063749cc70cbe20532a8675 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 20:47:05 -0600 Subject: [PATCH] fix(native): settle a finished owner whose journal holds a definitively answered command (#20148) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a per-turn follow-up starts a new runner, it retires the previous turn's finished owner. Retirement requires that owner's controller journal to be settled: every command answered, then a run terminal > - A restart can cut off a continuation turn. The recovered runner answers that `turn.start` with `failed` and the nested code `execution_indeterminate` ("it will not execute twice"), and the run continues on a later turn to an accepted result > - Retirement counted only `completed` commands as answered. A finished run whose journal held that answered `turn.start` never counted as settled, so the follow-up failed with `retirement_refused=journal_commands_unsettled(turn.start.failed)` and needed a Retry > - This pull request counts definitive `failed` and `rejected` answers as settled, except for runner lifecycle commands. An unknown-outcome `turn.start` counts as settled only once a later `turn.start` completed, which shows the run moved past it. Every other unknown or pending command stays unsettled > - The benefit is that a follow-up after a restart-interrupted, then finished, run continues on its own ## Linked Issues or Issue Description Refs #20140 **What happened?** Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master 8e3b64a257), t07 (claude C1, K6): - The previous run `9cb2c0b3` lost a continuation turn to a restart. The runner answered that `turn.start` as `failed` with `result.result.code: "execution_indeterminate"`. - After the next restore it ran a new continuation to an accepted result, and the run terminal was committed at 00:48:18. - A hard kill then landed before the runner suspended, and recovery finalized the result. - Follow-up `372991c8` failed at 00:48:37 with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`. **Expected behavior** A command the runner definitively answered (`failed` or `rejected`) leaves nothing outstanding, so it counts as settled. A `turn.start` with an unknown outcome is never executed again by the runner, so once a later `turn.start` completed, the run has moved past it and it also counts as settled. The follow-up then retires the finished owner and continues. The following still count as unsettled: - an unknown-outcome `turn.start` with no later completed turn; - any other unknown-outcome command; - a `pending` command; - a failed runner lifecycle command, such as a failed `runner.suspend` (the runner did not suspend). **Steps to reproduce** 1. Finish a per-turn run whose controller journal holds a `turn.start` answered `failed`, followed by a committed run terminal. 2. Start a follow-up. Before this change, retirement refuses with `journal_commands_unsettled(turn.start.failed)`. **Paperclip version or commit** 8e3b64a257 (wave). Branch rebased on current master. ## What Changed - `remote-warm-session-recovery.ts`: the new `unsettledControllerCommands` decides which commands are settled. `isSettledRemoteWarmControlState` uses it. - `completed` is settled. - An unknown outcome (raw `indeterminate`, or `failed` with `execution_indeterminate` at `result.code` or `result.result.code`) is settled only for a `turn.start` with a later completed `turn.start`. - Other `failed` or `rejected` answers are settled, except for `runner.suspend`, `runner.drain` and `runner.shutdown`. - `native-session-executor.ts`: the retirement refusal message lists unsettled commands with the same rule. - `native-recovered-owner-followup.test.ts`: a red/green test whose journals are in real lifecycle order. - Admitted: a `turn.start` answered `failed` or `rejected`; a superseded unknown-outcome `turn.start` in the runner's stored nested shape; a superseded raw `indeterminate`. - Held: the same unknown-outcome answer with no later completed turn; a failed `runner.suspend`. ## Verification - Red/green: on master, the admitted cases fail with exactly the t07 error. With the change they are admitted. The held cases stay held. If only `result.code` is checked, the unsuperseded case fails. - `vitest run` of `native-recovered-owner-followup.test.ts`, `remote-warm-session-recovery.test.ts` and `native-session-executor.test.ts`: 3 files, 1403 tests passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed. ## Risks - Low risk. Definitive answers to non-lifecycle commands, and unknown-outcome turn starts that a completed later turn superseded, become settled. - The run terminal requirement, post-terminal live-work checks, and every lifecycle and process proof are unchanged. - The same rule now also applies to remote warm recovery's settled check, where a definitively answered command is equally finished. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-recovered-owner-followup.test.ts | 34 +++++++++++++++++-- .../native-runtime/native-session-executor.ts | 4 +-- .../remote-warm-session-recovery.ts | 24 ++++++++++++- 3 files changed, 57 insertions(+), 5 deletions(-) diff --git a/server/src/services/native-runtime/native-recovered-owner-followup.test.ts b/server/src/services/native-runtime/native-recovered-owner-followup.test.ts index 178faddc960..ae9be61febe 100644 --- a/server/src/services/native-runtime/native-recovered-owner-followup.test.ts +++ b/server/src/services/native-runtime/native-recovered-owner-followup.test.ts @@ -90,7 +90,9 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered * later committed its result and released the controller lease, but no * suspended Runner receipt or failover backup ever reached the controller. * Only the database controller checkpoint and recovery pointer survive. */ - async function seedRecoveredPriorRun(control: "settled" | "pending-command" | "unanswered-tool" = "settled", + async function seedRecoveredPriorRun(control: "settled" | "pending-command" | "unanswered-tool" + | "failed-turn-start" | "rejected-turn-start" | "indeterminate-turn-start" | "unknown-outcome-turn-start" + | "unsuperseded-unknown-outcome-turn-start" | "failed-suspend" = "settled", options: { controller?: { bootId: string; pid: number; processStartedAt: Date }; provider?: "codex" | "acpx" | "opencode" } = {}) { const issueId = randomUUID(), workspaceId = randomUUID(), runId = randomUUID(); await db.insert(issues).values({ id: issueId, companyId, title: "Continue after recovery", assigneeAgentId: agentId, status: "in_progress" }); @@ -127,10 +129,25 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered try { // Shape the file-backed journal first; the database checkpoint captures // it when persistence moves to the controller's external writer. - const commands = ["run.prepare", "session.open", "turn.start", "runner.suspend"].map(type => controller.queueCommand(type)); + const unknownOutcome = { status: "failed", result: { commandType: "turn.start", status: "failed", + result: { code: "execution_indeterminate", message: "runner recovered after journaling this command; it will not execute twice" } } }; + const answers = { + "failed-turn-start": { status: "failed", result: { status: "failed" } }, + "rejected-turn-start": { status: "rejected", result: { status: "rejected" } }, + "indeterminate-turn-start": { status: "indeterminate", result: { status: "indeterminate" } }, + "unknown-outcome-turn-start": unknownOutcome, + } as const; + const answered = answers as Record; + const commands = ["run.prepare", "session.open", ...(control in answered ? ["turn.start"] : []), "turn.start", "runner.suspend"] + .map(type => controller.queueCommand(type)); for (const command of control === "pending-command" ? commands.slice(0, -1) : commands) { Object.assign(command, { status: "completed", result: { status: "completed" } }); } + if (control in answered) Object.assign(commands[2]!, answers[control as keyof typeof answers]); + if (control === "unsuperseded-unknown-outcome-turn-start") { + Object.assign(commands.find(command => command.type === "turn.start")!, unknownOutcome); + } + if (control === "failed-suspend") Object.assign(commands.at(-1)!, { status: "failed", result: { status: "failed" } }); const event = (sourceSeq: number, sourceEventId: string, eventType: string) => ({ sourceSeq, sourceEventId, eventType, priority: 0 as const, envelope: { ...identity }, deliveryCount: 1, logicalEffectCount: 1 }); controller.store.state.committedEvents.push( @@ -628,6 +645,19 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered expect(await retry.start()).toBe("native_controller_recovery_unsupported:metadata_format"); }, 60_000); + it.each([ + { control: "failed-turn-start", outcome: ADMITTED }, + { control: "rejected-turn-start", outcome: ADMITTED }, + { control: "indeterminate-turn-start", outcome: ADMITTED }, + { control: "unknown-outcome-turn-start", outcome: ADMITTED }, + { control: "unsuperseded-unknown-outcome-turn-start", outcome: stranded("journal_commands_unsettled(turn.start.failed)") }, + { control: "failed-suspend", outcome: stranded("journal_commands_unsettled(runner.suspend.failed)") }, + ] as const)("retires a finished owner whose journal holds an answered $control command", async ({ control, outcome }) => { + const prior = await seedRecoveredPriorRun(control); + const followUp = await admitFollowUp(prior); + expect(await followUp.start()).toBe(outcome); + }, 60_000); + it("admits again when a retry rebuilds the same projection from the database", async () => { const prior = await seedRecoveredPriorRun(); const followUp = await admitFollowUp(prior); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index c74977841dd..4782abec0a4 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -267,7 +267,7 @@ import { remoteRecoveryProbeReason, } from "./remote-runner-recovery-diagnostics.js"; import { readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS } from "./remote-runner-recovery-probe.js"; -import { describeOwnFailedAttachment, inspectRemoteWarmSessionRecovery, isSettledRemoteWarmControlState, summarizePostTerminalEvents, type RemoteWarmSessionRecovery } from "./remote-warm-session-recovery.js"; +import { describeOwnFailedAttachment, inspectRemoteWarmSessionRecovery, isSettledRemoteWarmControlState, summarizePostTerminalEvents, type RemoteWarmSessionRecovery, unsettledControllerCommands } from "./remote-warm-session-recovery.js"; import { assertNativeWarmIngressRetirement, nativeWarmIngressAllocation, nativeWarmIngressIdentity, type NativeWarmIngressClaim, type NativeWarmIngressOwner } from "./native-warm-ingress-ownership.js"; import { resolveNativeStagedAttachmentRecovery, type NativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js"; import { emitStartupReuse, environmentStartupReuse } from "@paperclipai/adapter-utils/startup-reuse"; @@ -5458,7 +5458,7 @@ function readSettledRunnerdProjection(root: string, identity: Record command.status !== "completed") + const unsettled = unsettledControllerCommands(settledCommands) .map(command => `${retirementReasonName("commandType", command.type)}.${retirementReasonName("commandStatus", command.status)}`); return refuse(unsettled.length > 0 ? `journal_commands_unsettled(${[...new Set(unsettled)].slice(0, 4).join(",")})` : "journal_terminal_unsettled"); diff --git a/server/src/services/native-runtime/remote-warm-session-recovery.ts b/server/src/services/native-runtime/remote-warm-session-recovery.ts index 40324702407..d1759466ed7 100644 --- a/server/src/services/native-runtime/remote-warm-session-recovery.ts +++ b/server/src/services/native-runtime/remote-warm-session-recovery.ts @@ -31,10 +31,32 @@ function postTerminalEvents(control: Record): unknown[] { /** A session is settled when its commands finished and nothing is live or waiting. * Events after the terminal describe the past, so refusing cannot undo them: they * are reported instead. Every event must still authenticate as this session's. */ +const RUNNER_LIFECYCLE_COMMAND_TYPES = new Set(["runner.suspend", "runner.drain", "runner.shutdown"]); + +function outcomeUnknown(command: Record): boolean { + return command.status === "indeterminate" || + (command.status === "failed" && (record(command.result).code === "execution_indeterminate" || + record(record(command.result).result).code === "execution_indeterminate")); +} + +export function unsettledControllerCommands(commands: readonly Record[]): Record[] { + const lastCompletedTurnStart = Math.max(-1, ...commands + .filter(command => command.type === "turn.start" && command.status === "completed") + .map(command => Number(command.controllerSeq))); + return commands.filter(command => { + if (command.status === "completed") return false; + if (outcomeUnknown(command)) { + return !(command.type === "turn.start" && Number(command.controllerSeq) < lastCompletedTurnStart); + } + return !((command.status === "failed" || command.status === "rejected") && + !RUNNER_LIFECYCLE_COMMAND_TYPES.has(String(command.type))); + }); +} + export function isSettledRemoteWarmControlState(control: Record, identity: Record): boolean { const commands = control.commands; const events = control.committedEvents; - if (!Array.isArray(commands) || commands.some(command => record(command).status !== "completed") || + if (!Array.isArray(commands) || unsettledControllerCommands(commands.map(record)).length > 0 || !Array.isArray(events) || Object.entries(identity).some(([key, value]) => record(control.identity)[key] !== value)) return false; const terminalIndex = events.findLastIndex(event => record(event).eventType === "run.terminal"); if (terminalIndex < 0) return false;