mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
fix(native): retire a finished per-turn owner past its own quarantined interrupted bootstrap (#20140)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A follow-up turn on a per-turn session must retire the previous turn's runner state before it starts a new runner. Retirement proves the previous owner finished, and it also proves that every quarantined copy of that session root is settled > - If restart recovery interrupts a run's own start (for example a hard kill during `session.open`), recovery proves the half-started bootstrap held no provider work, quarantines that root, and bootstraps the same run again. The quarantined copy keeps the run's identity > - After that run finished, retirement checked the old bootstrap copy as if it were a finished owner. An unfinished `session.open` (or an opened session without a turn) is never "settled", so the follow-up failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate` and needed a Retry > - This pull request skips a quarantined copy of the same owner when the same bootstrap proof that let recovery quarantine it still holds > - The benefit is that a follow-up after a restart-interrupted start continues on its own ## Linked Issues or Issue Description Refs #20137 **What happened?** Local restart-chaos K13 regression storm w-20261108T235957-s9401 (master4420a5adcf): 102 of 105 tasks passed. Two of the three failures share this cause. - **t58 (codex C1):** - A K1 at 00:15:42 interrupted run `78dd97f1` during its first `session.open`. Recovery quarantined that root (`run.prepare` completed, `session.open` pending, no events) and bootstrapped again. - The run then completed. A second K1 hit 190 ms after its result was accepted, and recovery finalized the result. - Follow-up `def668a6` failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=quarantine_journal_commands_unsettled(session.open.pending)`. - **t67 (claude C1):** the same pattern. The quarantined copy of run `990936c0` holds a completed `session.open` with only session-open events and no turn, so the refusal was `quarantine_journal_terminal_unsettled`. Both copies are in the kept instance's quarantine directory under the previous run's own identity. Each Retry succeeded. **Expected behavior** A quarantined copy of the finished owner's own earlier bootstrap that provably held no provider work does not hold the follow-up. Retirement proceeds and the follow-up starts without a Retry. **Steps to reproduce** 1. Let a per-turn run's first bootstrap be quarantined after a restart (the run bootstraps again and finishes). 2. Start a follow-up turn. 3. Before this change, retirement refuses with `quarantine_journal_commands_unsettled(session.open.pending)` or `quarantine_journal_terminal_unsettled`. This is not a code regression. The quarantine-copy check dates from #19061. Today's recovery fixes let more interrupted starts bootstrap again within the same run instead of blocking, and each such restart leaves this kind of copy behind. **Paperclip version or commit**4420a5adcf(wave). Branch rebased on current master. ## What Changed - `native-session-executor.ts`: the quarantine loop in `inspectFinishedPerTurnRunnerdOwner` skips a copy when both of these hold: - its durable identity matches the finished owner exactly; - `runnerdStateProvesIncompleteBootstrap` holds for it. That is the proof restart recovery uses before it quarantines an interrupted bootstrap: no provider turn, and for a local runner, its journal and processes proven gone. Every other copy still has to pass the settled-projection proof. - `native-recovered-owner-followup.test.ts`: a red/green test covering both cases: - an interrupted bootstrap copy is retired past; - a copy that had already issued a turn still holds the follow-up. ## Verification - Red/green: on master, the new admitted case fails with exactly the t58 error. With the change it is admitted. The turn-issued copy is refused on both. - `vitest run` of `native-recovered-owner-followup.test.ts`, `native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts`: 3 files, 1504 tests passed (14 skipped). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed. ## Risks - Low risk. The skip reuses the existing bootstrap-incomplete proof and requires an exact identity match, so a foreign or work-holding copy still blocks. Nothing is deleted; the copy stays in quarantine. - Not covered: t22 in the same storm, a follow-up whose own unconnected `run.attach` root was hit by a K1 (`authority_indeterminate`), has a different shape and is tracked separately. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
546cfee919
commit
bd97a3e6f7
2 files changed
+71
-1
No files matched your search
@@ -39,7 +39,8 @@ vi.mock("./current-wake-comments.js", async importOriginal => ({
|
||||
resolveCurrentWakeCommentsBinding: async () => null,
|
||||
assertCurrentWakeCommentsRead: async () => undefined,
|
||||
}));
|
||||
import { buildNativeHarnessBackupManifest, createRunnerdBackend, resolveNativeHarnessPersistenceProfile, scopedRunnerdStateRoot } from "./native-session-executor.js";
|
||||
import { buildNativeHarnessBackupManifest, createRunnerdBackend, quarantinedBootstrapOfOwnerHeldNoProviderWork, resolveNativeHarnessPersistenceProfile,
|
||||
scopedRunnerdStateRoot } from "./native-session-executor.js";
|
||||
|
||||
// Durable-root admission is the boundary under test. Reaching the run's tool
|
||||
// authority proves admission passed; no provider or runner process starts.
|
||||
@@ -648,6 +649,34 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
|
||||
expect(await quarantined()).toHaveLength(1);
|
||||
}, 60_000);
|
||||
|
||||
it.each([
|
||||
{ label: "an interrupted bootstrap that never opened its session", turnIssued: false, sessionOpened: false },
|
||||
{ label: "a remote bootstrap that opened its session without a turn", turnIssued: false, sessionOpened: true },
|
||||
{ label: "a bootstrap copy that already issued a turn", turnIssued: true, sessionOpened: false },
|
||||
])("retires a finished owner past its own quarantined earlier bootstrap: $label", async ({ turnIssued, sessionOpened }) => {
|
||||
const prior = await seedRecoveredPriorRun();
|
||||
const followUp = await admitFollowUp(prior);
|
||||
await db.update(nativeRunFinalizations).set({ phase: "observed" }).where(eq(nativeRunFinalizations.runId, prior.runId));
|
||||
expect(await followUp.start()).toBe(stranded("owner_observed"));
|
||||
const [copy] = await quarantined();
|
||||
const copyRoot = join(stateDirectory, "quarantine", copy!);
|
||||
const path = join(copyRoot, "control-plane", "control-plane-state.json");
|
||||
const state = JSON.parse(await readFile(path, "utf8"));
|
||||
const bootstrap = state.commands.filter((command: { type: string }) => command.type === "run.prepare" || command.type === "session.open"
|
||||
|| (turnIssued && command.type === "turn.start"));
|
||||
for (const command of bootstrap) {
|
||||
command.status = command.type === "run.prepare" || (sessionOpened && command.type === "session.open") ? "completed" : "pending";
|
||||
}
|
||||
const opened = sessionOpened ? ["session.started", "session.capabilities.updated"].map((eventType, index) => ({
|
||||
sourceSeq: index + 1, sourceEventId: `opened-${index + 1}`, eventType, priority: 1, envelope: { ...prior.identity },
|
||||
deliveryCount: 1, logicalEffectCount: 1 })) : [];
|
||||
await writeFile(path, JSON.stringify({ ...state, commands: bootstrap, committedEvents: opened, ackedSourceSeq: opened.length }));
|
||||
await rm(join(copyRoot, "runner"), { recursive: true, force: true });
|
||||
await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, prior.runId));
|
||||
expect(await followUp.start()).toBe(turnIssued
|
||||
? stranded("quarantine_journal_commands_unsettled(session.open.pending,turn.start.pending)") : ADMITTED);
|
||||
}, 60_000);
|
||||
|
||||
it("keeps an unsuspended Runner receipt decisive after a retry rebuilds the root", async () => {
|
||||
const prior = await seedRecoveredPriorRun();
|
||||
const followUp = await admitFollowUp(prior);
|
||||
@@ -770,6 +799,37 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
|
||||
return { prior, followUp, root };
|
||||
}
|
||||
|
||||
it.each(["exited", "alive", "another owner"] as const)(
|
||||
"skips a local quarantined bootstrap that opened a session only when it is this owner's and its processes %s",
|
||||
async scenario => {
|
||||
const runner = scenario === "alive" ? await runnerProcess() : await stoppedRunnerProcess();
|
||||
const root = await mkdtemp(join(tmpdir(), "quarantined-opened-bootstrap-"));
|
||||
const identity = { runnerInstanceId: "runner-1", environmentLeaseId: "lease-1", runId: "owner-run",
|
||||
normalizedSessionId: "session-1", turnId: "turn-owner-run", itemId: "item-owner-run" };
|
||||
try {
|
||||
await mkdir(join(root, "control-plane"), { recursive: true, mode: 0o700 });
|
||||
await mkdir(join(root, "runner"), { recursive: true, mode: 0o700 });
|
||||
await writeFile(join(root, "control-plane", "control-plane-state.json"), JSON.stringify({
|
||||
schema: "paperclip.runner.durable.control-plane-state.v2", identity,
|
||||
bootstrapHistory: { schema: "paperclip.runner.bootstrap-history.v1", originIdentity: identity, credentialsIssued: 1, adopted: false },
|
||||
commands: [
|
||||
{ commandId: "prepare", type: "run.prepare", status: "completed", result: { status: "prepared" } },
|
||||
{ commandId: "open", type: "session.open", status: "completed", result: { result: { status: "started", processId: runner.pid } } },
|
||||
],
|
||||
committedEvents: [{ sourceSeq: 1, sourceEventId: "opened-1", eventType: "session.started", priority: 1, envelope: identity }],
|
||||
ackedSourceSeq: 1, connectionCount: 1 }));
|
||||
await writeFile(join(root, "runner", "runner-state.json"), JSON.stringify({ schema: "paperclip.runner.durable.state.v1",
|
||||
...identity, lifecycle: "ready", outbox: [], processedCommands: {
|
||||
prepare: { commandType: "run.prepare", status: "completed", result: { status: "prepared" } },
|
||||
open: { commandType: "session.open", status: "completed", result: { status: "started", processId: runner.pid } } } }));
|
||||
const owner = scenario === "another owner" ? { ...identity, runId: "another-run" } : identity;
|
||||
expect(quarantinedBootstrapOfOwnerHeldNoProviderWork(root, owner, true)).toBe(scenario === "exited");
|
||||
} finally {
|
||||
if (scenario === "alive") runner.child.kill("SIGKILL");
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
it("retires a finished local owner whose Runner stopped before it proved suspension", async () => {
|
||||
const { prior, followUp, root } = await seedUnsuspendedLocalOwner(await stoppedRunnerProcess());
|
||||
expect(await followUp.startLocal()).toBe(ADMITTED);
|
||||
|
||||
@@ -5612,6 +5612,7 @@ async function inspectFinishedPerTurnRunnerdOwner(input: {
|
||||
if (!isSafeNativeStateDirectory(quarantine)) return refuse("quarantine_unsafe");
|
||||
for (const name of readdirSync(quarantine)) {
|
||||
if (!name.startsWith(`${basename(input.root)}.`)) continue;
|
||||
if (quarantinedBootstrapOfOwnerHeldNoProviderWork(resolve(quarantine, name), identity, localRunner)) continue;
|
||||
const copy = readSettledRunnerdProjection(resolve(quarantine, name), identity, { localRunner });
|
||||
if (copy.refused !== undefined) return refuse(`quarantine_${copy.refused}`);
|
||||
}
|
||||
@@ -5627,6 +5628,15 @@ async function inspectFinishedPerTurnRunnerdOwner(input: {
|
||||
return { finished: true, run: prior.run, recheck };
|
||||
}
|
||||
|
||||
export function quarantinedBootstrapOfOwnerHeldNoProviderWork(root: string, identity: Record<string, string>,
|
||||
localRunner: boolean): boolean {
|
||||
if (!isSafeNativeStateDirectory(root)) return false;
|
||||
const copied = readRunnerdDurableIdentity(root);
|
||||
if (!copied || Object.entries(identity).some(([key, value]) => copied[key] !== value)) return false;
|
||||
return runnerdStateProvesIncompleteBootstrap(root, { localProcesses: localRunner })
|
||||
|| (!localRunner && runnerdStateProvesNoRemoteProviderTurn(root));
|
||||
}
|
||||
|
||||
async function retireFinishedPerTurnRunnerdOwner(input: Parameters<typeof inspectFinishedPerTurnRunnerdOwner>[0])
|
||||
: Promise<{ retired: true } | { retired: false; refused: string }> {
|
||||
const proof = await inspectFinishedPerTurnRunnerdOwner(input);
|
||||
|
||||
Reference in new issue
Block a user