fix(native): name the proof that holds a recovered per-turn owner (#19061)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner per-turn sessions in a remote sandbox keep one
conversation across task turns.
> - A follow-up must prove that the previous Runner owner is finished
before it starts a new Runner. #18846 added a retirement path for an
owner that finished without a saved Runner checkpoint.
> - On a test deployment, follow-ups still fail with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate`. The retirement path has about
twenty separate fences, and each refusal gives the same error.
> - The read-only run data cannot tell which fence held these runs. A
code fix without that evidence would be a guess.
> - This pull request names the first failed retirement proof in the
error. Admission behavior does not change.
> - The benefit is that the next failure on a deployment identifies the
exact fence, so the follow-up fix can target it.

## Linked Issues or Issue Description

Refs #18846 (retirement of recovered per-turn owners). Refs #18992
(settlement from live state).

**What happened?**

On a test deployment, 16 runs in one day failed before provider work
with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate`. All were per-turn
`acpx_runtime` runs in a Daytona sandbox, and each got a new sandbox. In
the runs that I inspected, the previous run committed its result. In one
of them, `harness_state.backup.persist` and `session.checkpoint.persist`
ran for about 346 seconds and failed, so the Runner left no suspension
receipt or failover backup. That is the case that #18846 addresses, but
the follow-ups still failed. The run log does not show the retirement
message, and the error is the same for each fence. Thus the cause cannot
be found from the run data.

**Expected behavior**

When admission holds a follow-up, the error names the proof that failed.

**Steps to reproduce**

1. Seed a per-turn sandbox owner that committed its result without a
Runner suspension receipt.
2. Make one retirement proof fail. For example, leave the owner phase
`observed`, or leave a `runner.suspend` command pending.
3. Admit a follow-up in a new sandbox.
4. Before this change, every case fails with the same error text.

**Paperclip version or commit**

Observed on a deployment before `e107a75755`. The same error text is on
`master`.

## What Changed

- `retireFinishedPerTurnRunnerdOwner` returns the refusal reason instead
of `false`. `readSettledRunnerdProjection` returns the journal refusal
reason instead of `null`.
- When admission tries the retirement and it fails, the error is
`prior_owner_terminal_state_indeterminate; retirement_refused=<reason>`.
A warm session gets no suffix, because retirement does not apply to it.
- Reasons use only fixed lists of known names: owner phase, recovery
state, controller fence reason, unsettled command type and status,
sandbox outcome, and quarantine copy result. Any other stored value
appears as `other`. Reasons do not include payloads, file names, or
paths.
- The same proofs apply in the same order. Admission accepts and refuses
the same states as before.
- `doc/execution-semantics.md` describes the suffix.

## Verification

- **Failure before the change:** I ran the updated
`native-recovered-owner-followup.test.ts` against the unchanged
executor. 18 tests failed. Example: expected `...indeterminate;
retirement_refused=owner_observed`, received `...indeterminate`.
- **After the change:** `cd server && pnpm exec vitest run
src/services/native-runtime/native-recovered-owner-followup.test.ts`: 22
passed. Each held case now asserts its reason:
`owner_retryable_failure`, `controller_remote_controller_lease_live`,
`owner_recovery_resuming_session`,
`journal_commands_unsettled(runner.suspend.pending)`,
`journal_tool_input_unanswered`, `quarantine_journal_identity_mismatch`,
`runner_lifecycle_not_suspended`, and
`quarantine_runner_lifecycle_not_suspended`.
- The success cases still retire the owner and admit the follow-up.
- `pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/remote-warm-session-recovery.test.ts`: 1227
passed, 20 skipped.
- `pnpm exec vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
src/__tests__/task-workspace-binding.test.ts`: passed.
- `tsc --noEmit` reports no errors in the changed lines. This worktree
did not build every workspace package, so the full-package typecheck
shows unrelated missing-module errors. CI runs the full typecheck.
- **Not verified live:** no deployment has run this change.

## Risks

- Low risk. Admission logic is unchanged. Only the error text gets a
suffix.
- Consumers that match the error by prefix still match.
`failed-run-retry.ts` uses `startsWith("runner_state_identity_mismatch:
")`. No code compares the full string outside tests.
- This change does not fix the held runs. It finds the fence. A second
change is needed after a deployment reports the reason.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use,
shell execution, and read-only browser access to a test deployment. The
harness does not expose the context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-04 18:50:01 -06:00
1 parent 3a72efebe5
commit 11f2a5293d
3 files changed
+121 -54

No files matched your search

@@ -44,6 +44,8 @@ import { createRunnerdBackend } from "./native-session-executor.js";
// authority proves admission passed; no provider or runner process starts.
const ADMITTED = "follow-up admission reached tool authority";
const STRANDED = "runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate";
/** The held follow-up names the retirement proof that failed. */
const stranded = (refused: string) => `${STRANDED}; retirement_refused=${refused}`;
describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered per-turn native owner", () => {
let database: Awaited<ReturnType<typeof startRecoveryPostgres>>;
@@ -223,7 +225,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
async function keepLocalRootWithoutPointer(prior: Awaited<ReturnType<typeof seedRecoveredPriorRun>>,
followUp: Awaited<ReturnType<typeof admitFollowUp>>) {
await db.update(nativeRunFinalizations).set({ phase: "observed" }).where(eq(nativeRunFinalizations.runId, prior.runId));
expect(await followUp.start()).toBe(STRANDED);
expect(await followUp.start()).toBe(stranded("owner_observed"));
const [copy] = await quarantined();
await rename(join(stateDirectory, "quarantine", copy!), join(stateDirectory, copy!.split(".")[0]!));
await rm(join(stateDirectory, "quarantine"), { recursive: true, force: true });
@@ -264,7 +266,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
const prior = await seedRecoveredPriorRun("settled", { controller });
const followUp = await admitFollowUp(prior);
await keepLocalRootWithoutPointer(prior, followUp);
expect(await followUp.start()).toBe(STRANDED);
expect(await followUp.start()).toMatch(/; retirement_refused=controller_/);
expect(followUp.logs.join("")).not.toContain("finished without a saved runner checkpoint");
} finally {
child.kill("SIGKILL");
@@ -284,7 +286,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, prior.runId));
}
})();
expect(error).toBe(STRANDED);
expect(error).toBe(stranded("owner_observed"));
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "adapter_failed", error, finishedAt: new Date(),
nativePhase: "observed", resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } })
.where(eq(heartbeatRuns.id, followUp.runId));
@@ -371,7 +373,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
const prior = await seedRecoveredPriorRun();
const followUp = await admitFollowUp(prior);
await db.update(nativeRunFinalizations).set({ phase: "observed" }).where(eq(nativeRunFinalizations.runId, prior.runId));
expect(await followUp.start()).toBe(STRANDED);
expect(await followUp.start()).toBe(stranded("owner_observed"));
expect(await quarantined()).toHaveLength(1);
await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, prior.runId));
expect(await followUp.start()).toBe(ADMITTED);
@@ -384,7 +386,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
// Learn the scope key from a held attempt, then give the active root the
// local evidence of an owner whose Runner never suspended.
await db.update(nativeRunFinalizations).set({ phase: "observed" }).where(eq(nativeRunFinalizations.runId, prior.runId));
expect(await followUp.start()).toBe(STRANDED);
expect(await followUp.start()).toBe(stranded("owner_observed"));
const [copy] = await quarantined();
const root = join(stateDirectory, copy!.split(".")[0]!);
await rename(join(stateDirectory, "quarantine", copy!), root);
@@ -392,16 +394,16 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
await writeFile(join(root, "runner", "runner-state.json"), JSON.stringify({
schema: "paperclip.runner.durable.state.v1", lifecycle: "ready", ...prior.identity }));
await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, prior.runId));
expect(await followUp.start()).toBe(STRANDED);
expect(await followUp.start()).toBe(stranded("runner_lifecycle_not_suspended"));
expect(await quarantined()).toHaveLength(1);
// The retry restores only the database journal. The quarantined receipt
// must still hold it.
expect(await followUp.start()).toBe(STRANDED);
expect(await followUp.start()).toBe(stranded("quarantine_runner_lifecycle_not_suspended"));
expect(await quarantined()).toHaveLength(2);
expect(followUp.logs.join("")).not.toContain("finished without a saved runner checkpoint");
}, 60_000);
it.each(["uncommitted result", "live controller lease", "unexpired controller lease", "recovery pending", "pending command",
it.each(["uncommitted result", "committed without result", "live controller lease", "unexpired controller lease", "recovery pending", "pending command",
"unanswered tool input", "foreign quarantine"] as const)("keeps the prior owner held with %s", async scenario => {
const prior = await seedRecoveredPriorRun(scenario === "pending command" ? "pending-command"
: scenario === "unanswered tool input" ? "unanswered-tool" : "settled");
@@ -411,6 +413,9 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
await db.update(nativeRunFinalizations).set({ phase: "retryable_failure", resultId: null })
.where(eq(nativeRunFinalizations.runId, prior.runId));
}
if (scenario === "committed without result") {
await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, prior.runId));
}
if (scenario === "live controller lease") {
await db.update(nativeRunFinalizations).set({ leaseOwner: "another-controller", leaseExpiresAt: new Date(Date.now() + 60_000) })
.where(eq(nativeRunFinalizations.runId, prior.runId));
@@ -427,7 +432,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
// A copy of this projection is harmless; a different owner saved under
// the same scope is unresolved evidence that must not be overtaken.
await db.update(nativeRunFinalizations).set({ phase: "observed" }).where(eq(nativeRunFinalizations.runId, prior.runId));
expect(await followUp.start()).toBe(STRANDED);
expect(await followUp.start()).toBe(stranded("owner_observed"));
const [copy] = await quarantined();
const foreign = join(stateDirectory, "quarantine", copy!.replace(/\.[0-9a-f-]{36}$/, `.${randomUUID()}`));
await cp(join(stateDirectory, "quarantine", copy!), foreign, { recursive: true });
@@ -438,8 +443,18 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
expectedQuarantine = 3;
}
const outcome = await followUp.start();
// A live owner is still waited for before any projection is rebuilt.
expect(outcome).toBe(scenario === "live controller lease" ? "native_controller_recovery_waiting_for_owner" : STRANDED);
// A live owner is still waited for before any projection is rebuilt. Every
// other hold names its failed proof instead of one indeterminate state.
expect(outcome).toBe({
"uncommitted result": stranded("owner_retryable_failure"),
"committed without result": stranded("owner_result_missing"),
"live controller lease": "native_controller_recovery_waiting_for_owner",
"unexpired controller lease": stranded("controller_remote_controller_lease_live"),
"recovery pending": stranded("owner_recovery_resuming_session"),
"pending command": stranded("journal_commands_unsettled(runner.suspend.pending)"),
"unanswered tool input": stranded("journal_tool_input_unanswered"),
"foreign quarantine": stranded("quarantine_journal_identity_mismatch"),
}[scenario]);
expect(await quarantined()).toHaveLength(scenario === "live controller lease" ? 0 : expectedQuarantine);
expect(followUp.logs.join("")).not.toContain("finished without a saved runner checkpoint");
}, 60_000);
@@ -5198,30 +5198,67 @@ function moveNativeContinuityState(root: string, archiveRoot: string): void {
* evidence, even an unsuspended receipt, and unknown entries are owner
* claims that this proof never overrides.
*/
function readSettledRunnerdProjection(root: string, identity: Record<string, string>): string | null {
type SettledRunnerdProjection = { bytes: string; refused?: never } | { bytes?: never; refused: string };
function readSettledRunnerdProjection(root: string, identity: Record<string, string>): SettledRunnerdProjection {
const refuse = (refused: string): SettledRunnerdProjection => ({ refused });
try {
if (!isSafeNativeStateDirectory(root) ||
runnerdAuthorityLifecycle(root, identity as RunnerdDurableIdentity) !== "absent" ||
readdirSync(root).some(name => name !== "continuity-breaks" && !NATIVE_CONTINUITY_STATE_ENTRIES.includes(name))) {
return null;
}
if (!isSafeNativeStateDirectory(root)) return refuse("root_unsafe");
const lifecycle = runnerdAuthorityLifecycle(root, identity as RunnerdDurableIdentity);
if (lifecycle !== "absent") return refuse(`runner_lifecycle_${lifecycle}`);
const unexpected = readdirSync(root).find(name => name !== "continuity-breaks" && !NATIVE_CONTINUITY_STATE_ENTRIES.includes(name));
if (unexpected !== undefined) return refuse("unexpected_root_entry");
const bytes = readBoundedNativeFile(resolve(root, "control-plane", "control-plane-state.json"),
NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, "runner_durable_identity_too_large").toString("utf8");
const control = record(JSON.parse(bytes));
const answered = new Set((Array.isArray(control.commands) ? control.commands : []).map(record)
const commands = (Array.isArray(control.commands) ? control.commands : []).map(record);
const answered = new Set(commands
.filter(command => command.type === "semantic_tool.result" && command.status === "completed")
.map(command => record(command.payload).sourceEventId));
if (!RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(control.schema)) || control.warmTransition != null ||
control.stagedRunAttachment != null || !isSettledRemoteWarmControlState(control, identity) ||
(control.committedEvents as unknown[]).map(record).some(event =>
(event.eventType === "semantic_tool.input" || event.eventType === "mcp_app.tool_input") &&
!answered.has(event.sourceEventId))) return null;
return bytes;
if (!RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(control.schema))) return refuse("journal_schema_unsupported");
if (control.warmTransition != null) return refuse("journal_warm_transition");
if (control.stagedRunAttachment != null) return refuse("journal_staged_attachment");
if (Object.entries(identity).some(([key, value]) => record(control.identity)[key] !== value)) {
return refuse("journal_identity_mismatch");
}
if (!isSettledRemoteWarmControlState(control, identity)) {
// Closed vocabulary only: known command types and statuses, never payloads.
const pending = commands.filter(command => command.status !== "completed")
.map(command => `${retirementReasonName("commandType", command.type)}.${retirementReasonName("commandStatus", command.status)}`);
return refuse(pending.length > 0 ? `journal_commands_unsettled(${[...new Set(pending)].slice(0, 4).join(",")})`
: "journal_terminal_unsettled");
}
if ((control.committedEvents as unknown[]).map(record).some(event =>
(event.eventType === "semantic_tool.input" || event.eventType === "mcp_app.tool_input") &&
!answered.has(event.sourceEventId))) return refuse("journal_tool_input_unanswered");
return { bytes };
} catch {
return null;
return refuse("journal_unreadable");
}
}
// Refusal reasons reach run errors, so they draw only from these closed sets.
// Any other persisted value reports as "other".
const RETIREMENT_REASON_NAMES = {
// Mirrors the durable controller's stored-journal contract (commandTypes and
// the command status union in paperclip-runner's durable PRP control plane).
commandType: new Set(["run.prepare", "run.attach", "session.open", "turn.start", "turn.steer", "turn.interrupt",
"turn.stop", "request.resolve", "interaction.receipt", "semantic_tool.result", "external_provider.operation",
"session.snapshot", "session.goal.get", "session.goal.set", "session.goal.clear", "session.close",
"session.budget.increase", "session.destroy", "run.cancel", "runner.drain", "runner.suspend", "runner.shutdown"]),
commandStatus: new Set(["pending", "failed", "rejected", "indeterminate"]),
ownerPhase: new Set(["observed", "retryable_failure", "terminal_failure", "workspace_finalizing",
"recovery_needed"]),
recoveryState: new Set(["blocked", "resuming_session", "awaiting_evidence", "awaiting_runner_reattach"]),
controllerReason: new Set(["controller_still_alive", "coordinated_controller_still_alive", "remote_controller_lease_live",
"expired_lease_without_controller_identity", "live_legacy_lease_without_controller_identity"]),
sandboxOutcome: new Set(["created", "resumed", "replacement"]),
} as const;
function retirementReasonName(kind: keyof typeof RETIREMENT_REASON_NAMES, value: unknown): string {
return typeof value === "string" && RETIREMENT_REASON_NAMES[kind].has(value) ? value : "other";
}
/**
* A per-turn sandbox turn can finish through recovery after its post-turn
* checkpoint copy failed: the result is committed and the controller lease is
@@ -5240,21 +5277,27 @@ async function retireFinishedPerTurnRunnerdOwner(input: {
execution: NativeExecutionInput;
runnerExecutionTarget?: AdapterExecutionTarget | null;
onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
}): Promise<boolean> {
}): Promise<{ retired: true } | { retired: false; refused: string }> {
const target = input.runnerExecutionTarget;
// A resumed sandbox still holds the prior Runner's own lifecycle, which its
// existing recovery path reads. Plugin tasks own a separate saved checkpoint.
if (input.execution.session.lifecyclePolicy.mode !== "per_turn" || target?.kind !== "remote" ||
target.transport !== "sandbox" || !target.runner || target.runnerTask ||
!["created", "replacement"].includes(target.sandboxLeaseAcquisition?.outcome ?? "")) return false;
const refuse = (refused: string) => ({ retired: false as const, refused });
if (input.execution.session.lifecyclePolicy.mode !== "per_turn") return refuse("lifecycle_not_per_turn");
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask) {
return refuse("target_not_sandbox_runner");
}
if (!["created", "replacement"].includes(target.sandboxLeaseAcquisition?.outcome ?? "")) {
return refuse(`sandbox_${retirementReasonName("sandboxOutcome", target.sandboxLeaseAcquisition?.outcome)}`);
}
const identity: Record<string, string> = {};
for (const key of ["runId", "normalizedSessionId", "runnerInstanceId", "environmentLeaseId", "turnId", "itemId"]) {
const value = input.identity[key];
if (typeof value !== "string" || !value) return false;
if (typeof value !== "string" || !value) return refuse("durable_identity_incomplete");
identity[key] = value;
}
const controlBytes = readSettledRunnerdProjection(input.root, identity);
if (controlBytes === null) return false;
const projection = readSettledRunnerdProjection(input.root, identity);
if (projection.refused !== undefined) return refuse(projection.refused);
const controlBytes = projection.bytes;
const { binding } = input.execution;
let prior: { run: typeof heartbeatRuns.$inferSelect; owner: typeof nativeRunFinalizations.$inferSelect } | undefined;
try {
@@ -5266,36 +5309,39 @@ async function retireFinishedPerTurnRunnerdOwner(input: {
eq(nativeRunFinalizations.issueId, binding.issueId)))
.limit(1);
} catch {
return false;
}
if (!prior || prior.run.runtimeMode !== "native" || !TERMINAL_HEARTBEAT_RUN_STATUSES.has(prior.run.status) ||
prior.owner.phase !== "committed" || !prior.owner.resultId || prior.owner.leaseOwner || prior.owner.recoveryState) {
return false;
return refuse("owner_unavailable");
}
if (!prior || prior.run.runtimeMode !== "native") return refuse("owner_missing");
if (!TERMINAL_HEARTBEAT_RUN_STATUSES.has(prior.run.status)) return refuse("owner_run_not_terminal");
if (prior.owner.phase !== "committed") return refuse(`owner_${retirementReasonName("ownerPhase", prior.owner.phase)}`);
if (!prior.owner.resultId) return refuse("owner_result_missing");
if (prior.owner.leaseOwner) return refuse("owner_lease_held");
if (prior.owner.recoveryState) return refuse(`owner_recovery_${retirementReasonName("recoveryState", prior.owner.recoveryState)}`);
// A durable controller is fenced by its released database lease; a legacy
// controller still needs process evidence, exactly as for warm recovery.
const finishedHere = () => isCurrentNativeControllerProcess(prior!.owner) &&
!activeNativeSessions.has(identity.runId) && !nativeSessionStartups.has(identity.runId);
const profile = record(prior.run.runnerProfileJson);
const stopped = finishedHere() ? { allowed: true } : await evaluateNativeControllerTakeover({ owner: prior.owner,
const stopped: { allowed: boolean; reason?: string } = finishedHere() ? { allowed: true } : await evaluateNativeControllerTakeover({ owner: prior.owner,
now: new Date(), requireStoppedController: true,
remoteLeaseFenced: readNativeControllerRecovery(profile.nativeControllerRecovery, profile.sessionCheckpoint) !== null });
if (!stopped.allowed) return false;
if (!stopped.allowed) return refuse(`controller_${retirementReasonName("controllerReason", stopped.reason)}`);
const provenFinishedHere = finishedHere();
// Earlier rejected admissions can quarantine the same projection. Each copy
// must pass the same proof: a rebuilt root never carries the local Runner
// evidence that held an earlier attempt, so that evidence stays decisive.
const quarantine = resolve(runnerdStateBase(), "quarantine");
if (existsSync(quarantine)) {
if (!isSafeNativeStateDirectory(quarantine)) return false;
if (!isSafeNativeStateDirectory(quarantine)) return refuse("quarantine_unsafe");
for (const name of readdirSync(quarantine)) {
if (name.startsWith(`${basename(input.root)}.`) &&
readSettledRunnerdProjection(resolve(quarantine, name), identity) === null) return false;
if (!name.startsWith(`${basename(input.root)}.`)) continue;
const copy = readSettledRunnerdProjection(resolve(quarantine, name), identity);
if (copy.refused !== undefined) return refuse(`quarantine_${copy.refused}`);
}
}
// Database reads can await. Retire only the exact evidence proven above.
if (readSettledRunnerdProjection(input.root, identity) !== controlBytes ||
(provenFinishedHere && !finishedHere())) return false;
if (readSettledRunnerdProjection(input.root, identity).bytes !== controlBytes) return refuse("journal_changed");
if (provenFinishedHere && !finishedHere()) return refuse("controller_restarted_here");
const archivedAt = Date.now();
const archiveRoot = resolve(input.root, "continuity-breaks", `${archivedAt}-${randomUUID()}`);
mkdirSync(archiveRoot, { recursive: true, mode: 0o700 });
@@ -5305,7 +5351,7 @@ async function retireFinishedPerTurnRunnerdOwner(input: {
"stdout",
`[paperclip-runner] Run ${identity.runId} finished without a saved runner checkpoint; starting a new runner for this conversation.\n`,
);
return true;
return { retired: true };
}
async function migrateRunnerdStateRootForExecution(input: {
@@ -5445,13 +5491,13 @@ async function migrateRunnerdStateRootForExecution(input: {
allowVerifiedBackup: input.allowVerifiedBackup,
retainedWarmOwnerRunId: input.retainedWarmOwnerRunId,
});
if (
verification === "terminal_state_indeterminate" &&
!input.restartRecovery &&
!input.retainedWarmOwnerRunId &&
(await retireFinishedPerTurnRunnerdOwner({ ...input, root: scoped, identity }))
) {
return;
// Name the failed retirement proof: the shared error otherwise hides
// which of several independent fences held the follow-up.
let retirementRefusal = "";
if (verification === "terminal_state_indeterminate" && !input.restartRecovery && !input.retainedWarmOwnerRunId) {
const retirement = await retireFinishedPerTurnRunnerdOwner({ ...input, root: scoped, identity });
if (retirement.retired) return;
if (retirement.refused !== "lifecycle_not_per_turn") retirementRefusal = `; retirement_refused=${retirement.refused}`;
}
if (
verification !== "verified" &&
@@ -5465,7 +5511,7 @@ async function migrateRunnerdStateRootForExecution(input: {
: "identity_indeterminate",
);
}
throw new Error(`runner_state_identity_mismatch: prior_owner_${verification}`);
throw new Error(`runner_state_identity_mismatch: prior_owner_${verification}${retirementRefusal}`);
}
}
return;