fix(native): retire a finished per-turn owner past its own quarantined interrupted bootstrap (#20140)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A follow-up turn on a per-turn session must retire the previous
turn's runner state before it starts a new runner. Retirement proves the
previous owner finished, and it also proves that every quarantined copy
of that session root is settled
> - If restart recovery interrupts a run's own start (for example a hard
kill during `session.open`), recovery proves the half-started bootstrap
held no provider work, quarantines that root, and bootstraps the same
run again. The quarantined copy keeps the run's identity
> - After that run finished, retirement checked the old bootstrap copy
as if it were a finished owner. An unfinished `session.open` (or an
opened session without a turn) is never "settled", so the follow-up
failed with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate` and needed a Retry
> - This pull request skips a quarantined copy of the same owner when
the same bootstrap proof that let recovery quarantine it still holds
> - The benefit is that a follow-up after a restart-interrupted start
continues on its own

## Linked Issues or Issue Description

Refs #20137

**What happened?**
Local restart-chaos K13 regression storm w-20261108T235957-s9401 (master
4420a5adcf): 102 of 105 tasks passed. Two of the three failures share
this cause.

- **t58 (codex C1):**
- A K1 at 00:15:42 interrupted run `78dd97f1` during its first
`session.open`. Recovery quarantined that root (`run.prepare` completed,
`session.open` pending, no events) and bootstrapped again.
- The run then completed. A second K1 hit 190 ms after its result was
accepted, and recovery finalized the result.
- Follow-up `def668a6` failed with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=quarantine_journal_commands_unsettled(session.open.pending)`.
- **t67 (claude C1):** the same pattern. The quarantined copy of run
`990936c0` holds a completed `session.open` with only session-open
events and no turn, so the refusal was
`quarantine_journal_terminal_unsettled`.

Both copies are in the kept instance's quarantine directory under the
previous run's own identity. Each Retry succeeded.

**Expected behavior**
A quarantined copy of the finished owner's own earlier bootstrap that
provably held no provider work does not hold the follow-up. Retirement
proceeds and the follow-up starts without a Retry.

**Steps to reproduce**
1. Let a per-turn run's first bootstrap be quarantined after a restart
(the run bootstraps again and finishes).
2. Start a follow-up turn.
3. Before this change, retirement refuses with
`quarantine_journal_commands_unsettled(session.open.pending)` or
`quarantine_journal_terminal_unsettled`.

This is not a code regression. The quarantine-copy check dates from
#19061. Today's recovery fixes let more interrupted starts bootstrap
again within the same run instead of blocking, and each such restart
leaves this kind of copy behind.

**Paperclip version or commit**
4420a5adcf (wave). Branch rebased on current master.

## What Changed

- `native-session-executor.ts`: the quarantine loop in
`inspectFinishedPerTurnRunnerdOwner` skips a copy when both of these
hold:
  - its durable identity matches the finished owner exactly;
- `runnerdStateProvesIncompleteBootstrap` holds for it. That is the
proof restart recovery uses before it quarantines an interrupted
bootstrap: no provider turn, and for a local runner, its journal and
processes proven gone.
  
  Every other copy still has to pass the settled-projection proof.
- `native-recovered-owner-followup.test.ts`: a red/green test covering
both cases:
  - an interrupted bootstrap copy is retired past;
  - a copy that had already issued a turn still holds the follow-up.

## Verification

- Red/green: on master, the new admitted case fails with exactly the t58
error. With the change it is admitted. The turn-issued copy is refused
on both.
- `vitest run` of `native-recovered-owner-followup.test.ts`,
`native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: 3 files, 1504
tests passed (14 skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.

## Risks

- Low risk. The skip reuses the existing bootstrap-incomplete proof and
requires an exact identity match, so a foreign or work-holding copy
still blocks. Nothing is deleted; the copy stays in quarantine.
- Not covered: t22 in the same storm, a follow-up whose own unconnected
`run.attach` root was hit by a K1 (`authority_indeterminate`), has a
different shape and is tracked separately.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-08 19:21:32 -06:00
1 parent 546cfee919
commit bd97a3e6f7
2 files changed
+71 -1

No files matched your search

@@ -39,7 +39,8 @@ vi.mock("./current-wake-comments.js", async importOriginal => ({
resolveCurrentWakeCommentsBinding: async () => null,
assertCurrentWakeCommentsRead: async () => undefined,
}));
import { buildNativeHarnessBackupManifest, createRunnerdBackend, resolveNativeHarnessPersistenceProfile, scopedRunnerdStateRoot } from "./native-session-executor.js";
import { buildNativeHarnessBackupManifest, createRunnerdBackend, quarantinedBootstrapOfOwnerHeldNoProviderWork, resolveNativeHarnessPersistenceProfile,
scopedRunnerdStateRoot } from "./native-session-executor.js";
// Durable-root admission is the boundary under test. Reaching the run's tool
// authority proves admission passed; no provider or runner process starts.
@@ -648,6 +649,34 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
expect(await quarantined()).toHaveLength(1);
}, 60_000);
it.each([
{ label: "an interrupted bootstrap that never opened its session", turnIssued: false, sessionOpened: false },
{ label: "a remote bootstrap that opened its session without a turn", turnIssued: false, sessionOpened: true },
{ label: "a bootstrap copy that already issued a turn", turnIssued: true, sessionOpened: false },
])("retires a finished owner past its own quarantined earlier bootstrap: $label", async ({ turnIssued, sessionOpened }) => {
const prior = await seedRecoveredPriorRun();
const followUp = await admitFollowUp(prior);
await db.update(nativeRunFinalizations).set({ phase: "observed" }).where(eq(nativeRunFinalizations.runId, prior.runId));
expect(await followUp.start()).toBe(stranded("owner_observed"));
const [copy] = await quarantined();
const copyRoot = join(stateDirectory, "quarantine", copy!);
const path = join(copyRoot, "control-plane", "control-plane-state.json");
const state = JSON.parse(await readFile(path, "utf8"));
const bootstrap = state.commands.filter((command: { type: string }) => command.type === "run.prepare" || command.type === "session.open"
|| (turnIssued && command.type === "turn.start"));
for (const command of bootstrap) {
command.status = command.type === "run.prepare" || (sessionOpened && command.type === "session.open") ? "completed" : "pending";
}
const opened = sessionOpened ? ["session.started", "session.capabilities.updated"].map((eventType, index) => ({
sourceSeq: index + 1, sourceEventId: `opened-${index + 1}`, eventType, priority: 1, envelope: { ...prior.identity },
deliveryCount: 1, logicalEffectCount: 1 })) : [];
await writeFile(path, JSON.stringify({ ...state, commands: bootstrap, committedEvents: opened, ackedSourceSeq: opened.length }));
await rm(join(copyRoot, "runner"), { recursive: true, force: true });
await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, prior.runId));
expect(await followUp.start()).toBe(turnIssued
? stranded("quarantine_journal_commands_unsettled(session.open.pending,turn.start.pending)") : ADMITTED);
}, 60_000);
it("keeps an unsuspended Runner receipt decisive after a retry rebuilds the root", async () => {
const prior = await seedRecoveredPriorRun();
const followUp = await admitFollowUp(prior);
@@ -770,6 +799,37 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
return { prior, followUp, root };
}
it.each(["exited", "alive", "another owner"] as const)(
"skips a local quarantined bootstrap that opened a session only when it is this owner's and its processes %s",
async scenario => {
const runner = scenario === "alive" ? await runnerProcess() : await stoppedRunnerProcess();
const root = await mkdtemp(join(tmpdir(), "quarantined-opened-bootstrap-"));
const identity = { runnerInstanceId: "runner-1", environmentLeaseId: "lease-1", runId: "owner-run",
normalizedSessionId: "session-1", turnId: "turn-owner-run", itemId: "item-owner-run" };
try {
await mkdir(join(root, "control-plane"), { recursive: true, mode: 0o700 });
await mkdir(join(root, "runner"), { recursive: true, mode: 0o700 });
await writeFile(join(root, "control-plane", "control-plane-state.json"), JSON.stringify({
schema: "paperclip.runner.durable.control-plane-state.v2", identity,
bootstrapHistory: { schema: "paperclip.runner.bootstrap-history.v1", originIdentity: identity, credentialsIssued: 1, adopted: false },
commands: [
{ commandId: "prepare", type: "run.prepare", status: "completed", result: { status: "prepared" } },
{ commandId: "open", type: "session.open", status: "completed", result: { result: { status: "started", processId: runner.pid } } },
],
committedEvents: [{ sourceSeq: 1, sourceEventId: "opened-1", eventType: "session.started", priority: 1, envelope: identity }],
ackedSourceSeq: 1, connectionCount: 1 }));
await writeFile(join(root, "runner", "runner-state.json"), JSON.stringify({ schema: "paperclip.runner.durable.state.v1",
...identity, lifecycle: "ready", outbox: [], processedCommands: {
prepare: { commandType: "run.prepare", status: "completed", result: { status: "prepared" } },
open: { commandType: "session.open", status: "completed", result: { status: "started", processId: runner.pid } } } }));
const owner = scenario === "another owner" ? { ...identity, runId: "another-run" } : identity;
expect(quarantinedBootstrapOfOwnerHeldNoProviderWork(root, owner, true)).toBe(scenario === "exited");
} finally {
if (scenario === "alive") runner.child.kill("SIGKILL");
await rm(root, { recursive: true, force: true });
}
}, 60_000);
it("retires a finished local owner whose Runner stopped before it proved suspension", async () => {
const { prior, followUp, root } = await seedUnsuspendedLocalOwner(await stoppedRunnerProcess());
expect(await followUp.startLocal()).toBe(ADMITTED);
@@ -5612,6 +5612,7 @@ async function inspectFinishedPerTurnRunnerdOwner(input: {
if (!isSafeNativeStateDirectory(quarantine)) return refuse("quarantine_unsafe");
for (const name of readdirSync(quarantine)) {
if (!name.startsWith(`${basename(input.root)}.`)) continue;
if (quarantinedBootstrapOfOwnerHeldNoProviderWork(resolve(quarantine, name), identity, localRunner)) continue;
const copy = readSettledRunnerdProjection(resolve(quarantine, name), identity, { localRunner });
if (copy.refused !== undefined) return refuse(`quarantine_${copy.refused}`);
}
@@ -5627,6 +5628,15 @@ async function inspectFinishedPerTurnRunnerdOwner(input: {
return { finished: true, run: prior.run, recheck };
}
export function quarantinedBootstrapOfOwnerHeldNoProviderWork(root: string, identity: Record<string, string>,
localRunner: boolean): boolean {
if (!isSafeNativeStateDirectory(root)) return false;
const copied = readRunnerdDurableIdentity(root);
if (!copied || Object.entries(identity).some(([key, value]) => copied[key] !== value)) return false;
return runnerdStateProvesIncompleteBootstrap(root, { localProcesses: localRunner })
|| (!localRunner && runnerdStateProvesNoRemoteProviderTurn(root));
}
async function retireFinishedPerTurnRunnerdOwner(input: Parameters<typeof inspectFinishedPerTurnRunnerdOwner>[0])
: Promise<{ retired: true } | { retired: false; refused: string }> {
const proof = await inspectFinishedPerTurnRunnerdOwner(input);