fix(native): settle a finished owner whose journal holds a definitively answered command (#20148)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a per-turn follow-up starts a new runner, it retires the
previous turn's finished owner. Retirement requires that owner's
controller journal to be settled: every command answered, then a run
terminal
> - A restart can cut off a continuation turn. The recovered runner
answers that `turn.start` with `failed` and the nested code
`execution_indeterminate` ("it will not execute twice"), and the run
continues on a later turn to an accepted result
> - Retirement counted only `completed` commands as answered. A finished
run whose journal held that answered `turn.start` never counted as
settled, so the follow-up failed with
`retirement_refused=journal_commands_unsettled(turn.start.failed)` and
needed a Retry
> - This pull request counts definitive `failed` and `rejected` answers
as settled, except for runner lifecycle commands. An unknown-outcome
`turn.start` counts as settled only once a later `turn.start` completed,
which shows the run moved past it. Every other unknown or pending
command stays unsettled
> - The benefit is that a follow-up after a restart-interrupted, then
finished, run continues on its own

## Linked Issues or Issue Description

Refs #20140

**What happened?**
Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master
8e3b64a257), t07 (claude C1, K6):
- The previous run `9cb2c0b3` lost a continuation turn to a restart. The
runner answered that `turn.start` as `failed` with `result.result.code:
"execution_indeterminate"`.
- After the next restore it ran a new continuation to an accepted
result, and the run terminal was committed at 00:48:18.
- A hard kill then landed before the runner suspended, and recovery
finalized the result.
- Follow-up `372991c8` failed at 00:48:37 with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=journal_commands_unsettled(turn.start.failed)`.

**Expected behavior**
A command the runner definitively answered (`failed` or `rejected`)
leaves nothing outstanding, so it counts as settled. A `turn.start` with
an unknown outcome is never executed again by the runner, so once a
later `turn.start` completed, the run has moved past it and it also
counts as settled. The follow-up then retires the finished owner and
continues.

The following still count as unsettled:
- an unknown-outcome `turn.start` with no later completed turn;
- any other unknown-outcome command;
- a `pending` command;
- a failed runner lifecycle command, such as a failed `runner.suspend`
(the runner did not suspend).

**Steps to reproduce**
1. Finish a per-turn run whose controller journal holds a `turn.start`
answered `failed`, followed by a committed run terminal.
2. Start a follow-up. Before this change, retirement refuses with
`journal_commands_unsettled(turn.start.failed)`.

**Paperclip version or commit**
8e3b64a257 (wave). Branch rebased on current master.

## What Changed

- `remote-warm-session-recovery.ts`: the new
`unsettledControllerCommands` decides which commands are settled.
`isSettledRemoteWarmControlState` uses it.
  - `completed` is settled.
- An unknown outcome (raw `indeterminate`, or `failed` with
`execution_indeterminate` at `result.code` or `result.result.code`) is
settled only for a `turn.start` with a later completed `turn.start`.
- Other `failed` or `rejected` answers are settled, except for
`runner.suspend`, `runner.drain` and `runner.shutdown`.
- `native-session-executor.ts`: the retirement refusal message lists
unsettled commands with the same rule.
- `native-recovered-owner-followup.test.ts`: a red/green test whose
journals are in real lifecycle order.
- Admitted: a `turn.start` answered `failed` or `rejected`; a superseded
unknown-outcome `turn.start` in the runner's stored nested shape; a
superseded raw `indeterminate`.
- Held: the same unknown-outcome answer with no later completed turn; a
failed `runner.suspend`.

## Verification

- Red/green: on master, the admitted cases fail with exactly the t07
error. With the change they are admitted. The held cases stay held. If
only `result.code` is checked, the unsuperseded case fails.
- `vitest run` of `native-recovered-owner-followup.test.ts`,
`remote-warm-session-recovery.test.ts` and
`native-session-executor.test.ts`: 3 files, 1403 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.

## Risks

- Low risk. Definitive answers to non-lifecycle commands, and
unknown-outcome turn starts that a completed later turn superseded,
become settled.
- The run terminal requirement, post-terminal live-work checks, and
every lifecycle and process proof are unchanged.
- The same rule now also applies to remote warm recovery's settled
check, where a definitively answered command is equally finished.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/native-runtime/native-recovered-owner-followup.test.ts

@@ -90,7 +90,9 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
    * later committed its result and released the controller lease, but no
    * suspended Runner receipt or failover backup ever reached the controller.
    * Only the database controller checkpoint and recovery pointer survive. */
-  async function seedRecoveredPriorRun(control: "settled" | "pending-command" | "unanswered-tool" = "settled",
+  async function seedRecoveredPriorRun(control: "settled" | "pending-command" | "unanswered-tool"
+    | "failed-turn-start" | "rejected-turn-start" | "indeterminate-turn-start" | "unknown-outcome-turn-start"
+    | "unsuperseded-unknown-outcome-turn-start" | "failed-suspend" = "settled",
     options: { controller?: { bootId: string; pid: number; processStartedAt: Date }; provider?: "codex" | "acpx" | "opencode" } = {}) {
     const issueId = randomUUID(), workspaceId = randomUUID(), runId = randomUUID();
     await db.insert(issues).values({ id: issueId, companyId, title: "Continue after recovery", assigneeAgentId: agentId, status: "in_progress" });
@@ -127,10 +129,25 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
     try {
       // Shape the file-backed journal first; the database checkpoint captures
       // it when persistence moves to the controller's external writer.
-      const commands = ["run.prepare", "session.open", "turn.start", "runner.suspend"].map(type => controller.queueCommand(type));
+      const unknownOutcome = { status: "failed", result: { commandType: "turn.start", status: "failed",
+        result: { code: "execution_indeterminate", message: "runner recovered after journaling this command; it will not execute twice" } } };
+      const answers = {
+        "failed-turn-start": { status: "failed", result: { status: "failed" } },
+        "rejected-turn-start": { status: "rejected", result: { status: "rejected" } },
+        "indeterminate-turn-start": { status: "indeterminate", result: { status: "indeterminate" } },
+        "unknown-outcome-turn-start": unknownOutcome,
+      } as const;
+      const answered = answers as Record<string, unknown>;
+      const commands = ["run.prepare", "session.open", ...(control in answered ? ["turn.start"] : []), "turn.start", "runner.suspend"]
+        .map(type => controller.queueCommand(type));
       for (const command of control === "pending-command" ? commands.slice(0, -1) : commands) {
         Object.assign(command, { status: "completed", result: { status: "completed" } });
       }
+      if (control in answered) Object.assign(commands[2]!, answers[control as keyof typeof answers]);
+      if (control === "unsuperseded-unknown-outcome-turn-start") {
+        Object.assign(commands.find(command => command.type === "turn.start")!, unknownOutcome);
+      }
+      if (control === "failed-suspend") Object.assign(commands.at(-1)!, { status: "failed", result: { status: "failed" } });
       const event = (sourceSeq: number, sourceEventId: string, eventType: string) => ({ sourceSeq, sourceEventId, eventType,
         priority: 0 as const, envelope: { ...identity }, deliveryCount: 1, logicalEffectCount: 1 });
       controller.store.state.committedEvents.push(
@@ -628,6 +645,19 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
       expect(await retry.start()).toBe("native_controller_recovery_unsupported:metadata_format");
     }, 60_000);
 
+  it.each([
+    { control: "failed-turn-start", outcome: ADMITTED },
+    { control: "rejected-turn-start", outcome: ADMITTED },
+    { control: "indeterminate-turn-start", outcome: ADMITTED },
+    { control: "unknown-outcome-turn-start", outcome: ADMITTED },
+    { control: "unsuperseded-unknown-outcome-turn-start", outcome: stranded("journal_commands_unsettled(turn.start.failed)") },
+    { control: "failed-suspend", outcome: stranded("journal_commands_unsettled(runner.suspend.failed)") },
+  ] as const)("retires a finished owner whose journal holds an answered $control command", async ({ control, outcome }) => {
+    const prior = await seedRecoveredPriorRun(control);
+    const followUp = await admitFollowUp(prior);
+    expect(await followUp.start()).toBe(outcome);
+  }, 60_000);
+
   it("admits again when a retry rebuilds the same projection from the database", async () => {
     const prior = await seedRecoveredPriorRun();
     const followUp = await admitFollowUp(prior);

server/src/services/native-runtime/native-session-executor.ts

@@ -267,7 +267,7 @@ import {
   remoteRecoveryProbeReason,
 } from "./remote-runner-recovery-diagnostics.js";
 import { readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS } from "./remote-runner-recovery-probe.js";
-import { describeOwnFailedAttachment, inspectRemoteWarmSessionRecovery, isSettledRemoteWarmControlState, summarizePostTerminalEvents, type RemoteWarmSessionRecovery } from "./remote-warm-session-recovery.js";
+import { describeOwnFailedAttachment, inspectRemoteWarmSessionRecovery, isSettledRemoteWarmControlState, summarizePostTerminalEvents, type RemoteWarmSessionRecovery, unsettledControllerCommands } from "./remote-warm-session-recovery.js";
 import { assertNativeWarmIngressRetirement, nativeWarmIngressAllocation, nativeWarmIngressIdentity, type NativeWarmIngressClaim, type NativeWarmIngressOwner } from "./native-warm-ingress-ownership.js";
 import { resolveNativeStagedAttachmentRecovery, type NativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js";
 import { emitStartupReuse, environmentStartupReuse } from "@paperclipai/adapter-utils/startup-reuse";
@@ -5458,7 +5458,7 @@ function readSettledRunnerdProjection(root: string, identity: Record<string, str
     }
     if (!isSettledRemoteWarmControlState(settledControl, identity)) {
       // Closed vocabulary only: known command types and statuses, never payloads.
-      const unsettled = settledCommands.filter(command => command.status !== "completed")
+      const unsettled = unsettledControllerCommands(settledCommands)
         .map(command => `${retirementReasonName("commandType", command.type)}.${retirementReasonName("commandStatus", command.status)}`);
       return refuse(unsettled.length > 0 ? `journal_commands_unsettled(${[...new Set(unsettled)].slice(0, 4).join(",")})`
         : "journal_terminal_unsettled");

server/src/services/native-runtime/remote-warm-session-recovery.ts

@@ -31,10 +31,32 @@ function postTerminalEvents(control: Record<string, unknown>): unknown[] {
 /** A session is settled when its commands finished and nothing is live or waiting.
  * Events after the terminal describe the past, so refusing cannot undo them: they
  * are reported instead. Every event must still authenticate as this session's. */
+const RUNNER_LIFECYCLE_COMMAND_TYPES = new Set(["runner.suspend", "runner.drain", "runner.shutdown"]);
+
+function outcomeUnknown(command: Record<string, unknown>): boolean {
+  return command.status === "indeterminate" ||
+    (command.status === "failed" && (record(command.result).code === "execution_indeterminate" ||
+      record(record(command.result).result).code === "execution_indeterminate"));
+}
+
+export function unsettledControllerCommands(commands: readonly Record<string, unknown>[]): Record<string, unknown>[] {
+  const lastCompletedTurnStart = Math.max(-1, ...commands
+    .filter(command => command.type === "turn.start" && command.status === "completed")
+    .map(command => Number(command.controllerSeq)));
+  return commands.filter(command => {
+    if (command.status === "completed") return false;
+    if (outcomeUnknown(command)) {
+      return !(command.type === "turn.start" && Number(command.controllerSeq) < lastCompletedTurnStart);
+    }
+    return !((command.status === "failed" || command.status === "rejected") &&
+      !RUNNER_LIFECYCLE_COMMAND_TYPES.has(String(command.type)));
+  });
+}
+
 export function isSettledRemoteWarmControlState(control: Record<string, unknown>, identity: Record<string, unknown>): boolean {
   const commands = control.commands;
   const events = control.committedEvents;
-  if (!Array.isArray(commands) || commands.some(command => record(command).status !== "completed") ||
+  if (!Array.isArray(commands) || unsettledControllerCommands(commands.map(record)).length > 0 ||
       !Array.isArray(events) || Object.entries(identity).some(([key, value]) => record(control.identity)[key] !== value)) return false;
   const terminalIndex = events.findLastIndex(event => record(event).eventType === "run.terminal");
   if (terminalIndex < 0) return false;