From 36643d93783e6f41f1ceb485217e964ceb1da1ab Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 5 Nov 2026 19:37:01 -0600 Subject: [PATCH] fix(native): retire finished owners whose continued turn rotated authority (#19373) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native runner runs per-turn agent sessions in managed sandboxes. A controller keeps a durable session root for each session. > - When a per-turn owner finishes without a saved runner checkpoint, the next turn retires that owner. Retirement first checks that the session root holds only known entries. > - The runner transport writes two entries that the check does not know: `authority-epochs/` and `runtime-context.json`. A continued turn writes `authority-epochs/` when it rotates authority from the turn before it. > - So the third turn of an ephemeral-lease task cannot retire the second turn's owner. The run fails with `native_continuation_requires_reconciliation` and the task waits for a user message. > - This pull request classifies every session-root entry in one contract module, and adds a test that scans the writers so the list cannot drift again. > - The benefit is that follow-up turns continue without manual help, and any future refusal names the unexpected entry. ## Linked Issues or Issue Description Refs #18846, Refs #19061, Refs #19355 **What happened?** On staging, the third turn of every task for one agent (gpt-5.6-sol, native runner, ephemeral sandbox lease) failed. The error was `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=unexpected_root_entry`. The first two turns succeeded. The task then went to `blocked` with "Saved work recovered. Send a message to continue." **Expected behavior** The third turn retires the finished second-turn owner and continues the conversation. **Steps to reproduce** 1. Assign a task to a native agent that uses an ephemeral sandbox lease. 2. Let the first turn finish. Reopen the task with a comment so a second turn runs and finishes. 3. Reopen the task again. The third turn fails with the error above. **Paperclip version or commit** Staging near 9a79384958; reproduced locally on master 7c0922a7b6. ## What Changed - Add `server/src/services/native-runtime/native-session-root-entries.ts`. It is the single contract for top-level session-root entries: - Continuity state (moves with a continuity break): the existing entries, plus `authority-epochs` and `launch-preparation.json`. - Launch entries (stay in place, carry no authority): `runtime-context.json`, `diagnostics`. - Held entries (retirement still refuses): `plugin-runner-state.json`, `cleanup-activation.json`, `cursor-cloud`. - `readSettledRunnerdProjection` uses this contract. The refusal now names the entries. A held entry reports as `held_root_entry(cursor-cloud)`. An unclassified entry reports as `unexpected_root_entry(unrecognized-state)`. Names are bounded to three, limited to safe characters, and have identifiers replaced with ``. - Add `native-session-root-entries.test.ts`. It scans every local root writer (the executor, the Cursor Cloud runtime, the provider-pack authority, the controller migration, the runner transport, and the launch-preparation store). It reads string and template literals, same-file constants, `posix.join` and `path.join`, and the harness persistence directory names. It fails when a writer adds an unclassified entry, or reaches the root through a new unresolved variable. - Add three cases to `native-recovered-owner-followup.test.ts`: a finished owner whose root has `runtime-context.json`, an archived authority epoch, `diagnostics`, and `launch-preparation.json` is retired, and the epoch and launch preparation move into the archive; an unknown entry still holds the owner and is named; a held entry is named as held. ## Verification - Red before the fix (same string as staging): `retires a finished owner whose continued turn left runtime context and an archived authority epoch` received `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=unexpected_root_entry`. - Green after the fix: - `cd server && npx vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts src/services/native-runtime/native-session-root-entries.test.ts` (34 passed) - `cd server && npx vitest run src/__tests__/native-harness-backup-expiration.test.ts src/services/native-runtime/native-session-resume.test.ts src/services/native-runtime/native-session-executor.test.ts` (1313 passed) - Live confirmation on staging is pending. After deploy, run a three-turn task on an ephemeral-lease codex agent. The third turn must succeed. Its log must show "finished without a saved runner checkpoint; starting a new runner". ## Risks - Retirement now accepts two more entries. `authority-epochs` only holds epochs that the transport archived after it verified a suspended runner, so it carries no live owner claim. `runtime-context.json` and `diagnostics` are launch inputs and logs. Unknown entries still refuse retirement. - A continuity break now also moves `authority-epochs` and `launch-preparation.json` into the archive. This is the intended owner: these files belong to the retired authority lineage. If they stayed, a later cold rotation could read an archived epoch of the retired lineage. - I could not read the staging sandbox. The evidence for the exact entry is from code: the second turn's transport writes `authority-epochs/` during authority rotation, and every launch writes `runtime-context.json`. The refusal now names the entry, so any other entry will be visible in the next failure. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-recovered-owner-followup.test.ts | 62 ++++++++++++ .../native-runtime/native-session-executor.ts | 26 +++-- .../native-session-root-entries.test.ts | 95 +++++++++++++++++++ .../native-session-root-entries.ts | 72 ++++++++++++++ 4 files changed, 240 insertions(+), 15 deletions(-) create mode 100644 server/src/services/native-runtime/native-session-root-entries.test.ts create mode 100644 server/src/services/native-runtime/native-session-root-entries.ts diff --git a/server/src/services/native-runtime/native-recovered-owner-followup.test.ts b/server/src/services/native-runtime/native-recovered-owner-followup.test.ts index 96c4bd9290e..f12aad1871f 100644 --- a/server/src/services/native-runtime/native-recovered-owner-followup.test.ts +++ b/server/src/services/native-runtime/native-recovered-owner-followup.test.ts @@ -403,6 +403,68 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered expect(followUp.logs.join("")).not.toContain("finished without a saved runner checkpoint"); }, 60_000); + /** Learn the scope key from a held attempt and restore its root as the + * live session root, as it stands after the prior turn finished. */ + async function restoreHeldRoot(prior: Awaited>, + followUp: Awaited>) { + await db.update(nativeRunFinalizations).set({ phase: "observed" }).where(eq(nativeRunFinalizations.runId, prior.runId)); + expect(await followUp.start()).toBe(stranded("owner_observed")); + const [copy] = await quarantined(); + const root = join(stateDirectory, copy!.split(".")[0]!); + await rename(join(stateDirectory, "quarantine", copy!), root); + await rm(join(stateDirectory, "quarantine"), { recursive: true, force: true }); + await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, prior.runId)); + return root; + } + + // A continued turn's transport leaves these in the controller root: the + // runtime context it wrote before launch, and the settled epoch it archived + // when it rotated authority from the turn before it (S49: the third turn of + // an ephemeral-lease task could never retire the second turn's owner). + it("retires a finished owner whose continued turn left runtime context and an archived authority epoch", async () => { + const prior = await seedRecoveredPriorRun(); + const followUp = await admitFollowUp(prior); + const root = await restoreHeldRoot(prior, followUp); + await writeFile(join(root, "runtime-context.json"), "{}\n", { mode: 0o600 }); + const epoch = join(root, "authority-epochs", "epoch-0123456789abcdef01234567"); + await mkdir(join(epoch, "control-plane"), { recursive: true, mode: 0o700 }); + await cp(join(root, "control-plane", "control-plane-state.json"), join(epoch, "control-plane", "control-plane-state.json")); + await writeFile(join(epoch, "runner-state.json"), "{}\n", { mode: 0o600 }); + await mkdir(join(root, "diagnostics"), { mode: 0o700 }); + // The launch preparation saved for the retired identity. + const preparation = JSON.stringify({ schema: "paperclip.runner.launch-preparation-store.v1", entries: [] }); + await writeFile(join(root, "launch-preparation.json"), preparation, { mode: 0o600 }); + expect(await followUp.start()).toBe(ADMITTED); + expect(await archivedOwners(root)).toEqual([prior.runId]); + // Archived epochs belong to the retired lineage; launch inputs stay. + const [archive] = await readdir(join(root, "continuity-breaks")); + expect(existsSync(join(root, "continuity-breaks", archive!, "authority-epochs", "epoch-0123456789abcdef01234567"))).toBe(true); + expect(existsSync(join(root, "authority-epochs"))).toBe(false); + expect(await readFile(join(root, "continuity-breaks", archive!, "launch-preparation.json"), "utf8")).toBe(preparation); + expect(existsSync(join(root, "launch-preparation.json"))).toBe(false); + expect(existsSync(join(root, "runtime-context.json"))).toBe(true); + expect(followUp.logs.join("")).toContain(`Run ${prior.runId} finished without a saved runner checkpoint`); + }, 60_000); + + it("keeps a finished owner held by an unclassified root entry and names it", async () => { + const prior = await seedRecoveredPriorRun(); + const followUp = await admitFollowUp(prior); + const root = await restoreHeldRoot(prior, followUp); + await writeFile(join(root, "unrecognized-state"), "{}\n"); + expect(await followUp.start()).toBe(stranded("unexpected_root_entry(unrecognized-state)")); + expect(followUp.logs.join("")).not.toContain("finished without a saved runner checkpoint"); + }, 60_000); + + it("keeps a finished owner held by a provider checkpoint and names it as held", async () => { + const prior = await seedRecoveredPriorRun(); + const followUp = await admitFollowUp(prior); + const root = await restoreHeldRoot(prior, followUp); + await mkdir(join(root, "cursor-cloud"), { mode: 0o700 }); + await writeFile(join(root, "unrecognized-state"), "{}\n"); + expect(await followUp.start()).toBe(stranded("held_root_entry(cursor-cloud)")); + expect(followUp.logs.join("")).not.toContain("finished without a saved runner checkpoint"); + }, 60_000); + it.each(["uncommitted result", "committed without result", "live controller lease", "unexpired controller lease", "recovery pending", "pending command", "unanswered tool input", "foreign quarantine"] as const)("keeps the prior owner held with %s", async scenario => { const prior = await seedRecoveredPriorRun(scenario === "pending command" ? "pending-command" diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index c40cbcf8d7e..5e9a8b38384 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -52,6 +52,10 @@ import { museRunnerBroker } from "../muse-runner-broker.js"; import { nativeCaseHandoffResult } from "./native-case-handoff.js"; import { assertExternalNativeRunAuthority } from "./external-native-authority.js"; import { stageRemoteLaunchAssets } from "./remote-launch-assets.js"; +import { + NATIVE_CONTINUITY_STATE_ENTRIES, NATIVE_SESSION_ROOT_HELD_ENTRIES, describeUnexpectedNativeSessionRootEntries, + isKnownNativeSessionRootEntry, +} from "./native-session-root-entries.js"; import { createRunPreparationTiming } from "../execution-preparation-timing.js"; import { configuredEnvironment } from "../../vendor/paperclip-runner/index.js"; import { parseProviderPerformanceSpans } from "../../vendor/paperclip-runner/index.js"; @@ -5270,19 +5274,6 @@ async function recoverPluginRunnerCheckpoint(input: { } } -// Session-root entries owned by one provider session's authority. A continuity -// break moves them aside together. Backups belong to the retired session too: -// left active, they make a fresh replacement look like lost harness state. -const NATIVE_CONTINUITY_STATE_ENTRIES: readonly string[] = [ - "control-plane", - "runner", - "codex-home", - "opencode", - "acpx", - "failover-backups", - "provider-pack-authority.json", -]; - function moveNativeContinuityState(root: string, archiveRoot: string): void { for (const name of NATIVE_CONTINUITY_STATE_ENTRIES) { const source = resolve(root, name); @@ -5306,8 +5297,13 @@ function readSettledRunnerdProjection(root: string, identity: Record name !== "continuity-breaks" && !NATIVE_CONTINUITY_STATE_ENTRIES.includes(name)); - if (unexpected !== undefined) return refuse("unexpected_root_entry"); + // Every legitimate writer's entry is classified in native-session-root-entries.ts. + // Held entries are known evidence that needs its own settlement; anything + // else is writer drift. Name each kind separately. + const unknown = readdirSync(root).filter(name => !isKnownNativeSessionRootEntry(name)); + const held = unknown.filter(name => NATIVE_SESSION_ROOT_HELD_ENTRIES.includes(name)); + if (held.length > 0) return refuse(`held_root_entry(${describeUnexpectedNativeSessionRootEntries(held)})`); + if (unknown.length > 0) return refuse(`unexpected_root_entry(${describeUnexpectedNativeSessionRootEntries(unknown)})`); const bytes = readBoundedNativeFile(resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, "runner_durable_identity_too_large").toString("utf8"); const control = record(JSON.parse(bytes)); diff --git a/server/src/services/native-runtime/native-session-root-entries.test.ts b/server/src/services/native-runtime/native-session-root-entries.test.ts new file mode 100644 index 00000000000..8ac6ab64dad --- /dev/null +++ b/server/src/services/native-runtime/native-session-root-entries.test.ts @@ -0,0 +1,95 @@ +import { readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; +import { + NATIVE_CONTINUITY_ARCHIVE_ENTRY, NATIVE_CONTINUITY_STATE_ENTRIES, NATIVE_SESSION_ROOT_HELD_ENTRIES, + NATIVE_SESSION_ROOT_LAUNCH_ENTRIES, describeUnexpectedNativeSessionRootEntries, isKnownNativeSessionRootEntry, +} from "./native-session-root-entries.js"; + +const source = (path: string) => readFileSync(new URL(path, import.meta.url), "utf8"); +const escape = (value: string) => value.replace(/[.*+?^${}()|[\]\\#]/g, "\\$&"); + +// Each module that writes a native session's durable state root, with the +// expressions naming that root in the file. A new top-level entry written by +// any of them must be classified before retirement can see it. +const WRITERS: Array<{ file: string; roots: string[]; dynamic?: string[] }> = [ + // Dynamic arguments iterate over classified lists: continuity moves and + // cleanup/recovery reads of control-plane and runner files. + { file: "./native-session-executor.ts", roots: ["root", "scoped", "input.root"], dynamic: ["file", "name", "directory"] }, + { file: "./cursor-cloud-runtime.ts", roots: ["input.root"] }, + { file: "./remote-provider-pack-authority.ts", roots: ["input.stateDirectory"] }, + { file: "./native-controller-migration.ts", roots: ["paths.root"] }, + { file: "../../../../packages/paperclip-runner/src/live/runnerd-codex-transport.ts", roots: ["this.#root", "root"], + dynamic: ["file"] }, + { file: "../../../../packages/paperclip-runner/src/live/runner-launch-preparation-store.ts", roots: ["directory"] }, +]; + +/** + * Every top-level entry a writer names: string and template literals (their + * leading segment) and identifiers resolved through a same-file string + * constant, across resolve/join/posix.join/path.join. Any other identifier + * must be listed as dynamic, so a writer that reaches the root through a new + * variable fails here instead of passing silently. + */ +function writtenRootEntries(): { names: Set; unresolved: string[] } { + const names = new Set(); + const unresolved: string[] = []; + for (const writer of WRITERS) { + const text = source(writer.file); + const roots = writer.roots.map(escape).join("|"); + const call = new RegExp(`(?:\\bresolve|(?:\\bposix\\.|\\bpath\\.|\\b)join)\\((?:${roots}),\\s*([^,)]+)`, "g"); + for (const [, raw] of text.matchAll(call)) { + const argument = raw!.trim(); + const literal = /^["'`]([^"'`/$]+)/.exec(argument); + if (literal) { + names.add(literal[1]!); + continue; + } + const identifier = /^[A-Za-z_$][\w$.]*$/.exec(argument)?.[0]; + const constant = identifier + ? new RegExp(`const ${escape(identifier)} = ["'\`]([^"'\`/]+)["'\`]`).exec(text) + : null; + if (constant) { + names.add(constant[1]!); + continue; + } + if (identifier && writer.dynamic?.includes(identifier.split(".")[0]!)) continue; + unresolved.push(`${writer.file}: ${argument}`); + } + } + // Harness persistence directories are restored into the root by name. + for (const [, name] of source("./native-session-executor.ts").matchAll(/name: "([^"]+)",\s*location: "filesystem"/g)) { + names.add(name!); + } + return { names, unresolved }; +} + +describe("native session root entry contract", () => { + it("classifies every entry a known writer puts in the session root", () => { + const { names: written, unresolved } = writtenRootEntries(); + expect(unresolved).toEqual([]); + // Guard the scan itself: these writers exist today. + for (const name of ["control-plane", "runner", "runtime-context.json", "authority-epochs", "launch-preparation.json", + "provider-pack-authority.json", "cleanup-activation.json", "plugin-runner-state.json", "codex-home", "opencode", + "acpx", "cursor-cloud", "diagnostics", "failover-backups", "continuity-breaks"]) { + expect(written).toContain(name); + } + const unclassified = [...written].filter(name => !isKnownNativeSessionRootEntry(name) && + !NATIVE_SESSION_ROOT_HELD_ENTRIES.includes(name)); + expect(unclassified).toEqual([]); + }); + + it("keeps the classes disjoint so an entry has one retirement meaning", () => { + const all = [...NATIVE_CONTINUITY_STATE_ENTRIES, ...NATIVE_SESSION_ROOT_LAUNCH_ENTRIES, + ...NATIVE_SESSION_ROOT_HELD_ENTRIES, NATIVE_CONTINUITY_ARCHIVE_ENTRY]; + expect(new Set(all).size).toBe(all.length); + for (const held of NATIVE_SESSION_ROOT_HELD_ENTRIES) expect(isKnownNativeSessionRootEntry(held)).toBe(false); + }); + + it("names unexpected entries in a bounded, redacted form", () => { + expect(describeUnexpectedNativeSessionRootEntries(["unrecognized-state"])).toBe("unrecognized-state"); + expect(describeUnexpectedNativeSessionRootEntries([".state.123e4567-e89b-12d3-a456-426614174000.tmp"])) + .toBe(".state..tmp"); + expect(describeUnexpectedNativeSessionRootEntries(["has space", "a/b", "x".repeat(65)])).toBe("other"); + expect(describeUnexpectedNativeSessionRootEntries(["d", "c", "b", "a"])).toBe("a,b,c,+1"); + }); +}); diff --git a/server/src/services/native-runtime/native-session-root-entries.ts b/server/src/services/native-runtime/native-session-root-entries.ts new file mode 100644 index 00000000000..0f477497865 --- /dev/null +++ b/server/src/services/native-runtime/native-session-root-entries.ts @@ -0,0 +1,72 @@ +// Contract for the top-level entries of a native session's durable state root +// (`/`). Every writer that targets that root +// must have its entry classified here; native-session-root-entries.test.ts +// scans the known writers so a new entry cannot drift past retirement. + +/** + * Entries owned by one provider session's authority. A continuity break moves + * them aside together. Backups and archived authority epochs belong to the + * retired session too: left active, they make a fresh replacement look like + * lost harness state, or let a later cold rotation revive retired authority. + */ +export const NATIVE_CONTINUITY_STATE_ENTRIES: readonly string[] = [ + "control-plane", + "runner", + "codex-home", + "opencode", + "acpx", + "failover-backups", + "provider-pack-authority.json", + // Runner transport: each cold authority rotation archives the settled prior + // epoch here (rotateLocalAuthorityEpoch/rotateExternalAuthorityEpoch). + "authority-epochs", + // Runner transport: execution-owned launch preparation keyed by identity. + "launch-preparation.json", +]; + +/** + * Per-launch inputs and process logs that carry no session authority. Every + * launch rewrites them, so retirement leaves them in place. + */ +export const NATIVE_SESSION_ROOT_LAUNCH_ENTRIES: readonly string[] = [ + // Runner transport writes the resolved runtime context before each launch. + "runtime-context.json", + // Local runnerd stdout/stderr diagnostics. + "diagnostics", +]; + +/** + * Known entries that are evidence a finished-owner proof must never override: + * a plugin Runner receipt, in-flight backup maintenance, and Cursor Cloud's + * own checkpoint (archived only by its provider runtime). Retirement refuses + * while any of them is present and reports it as `held_root_entry`, apart from + * unclassified drift (`unexpected_root_entry`). + */ +export const NATIVE_SESSION_ROOT_HELD_ENTRIES: readonly string[] = [ + "plugin-runner-state.json", + "cleanup-activation.json", + "cursor-cloud", +]; + +/** Holds archived continuity state; never an owner claim by itself. */ +export const NATIVE_CONTINUITY_ARCHIVE_ENTRY = "continuity-breaks"; + +export function isKnownNativeSessionRootEntry(name: string): boolean { + return name === NATIVE_CONTINUITY_ARCHIVE_ENTRY || + NATIVE_CONTINUITY_STATE_ENTRIES.includes(name) || + NATIVE_SESSION_ROOT_LAUNCH_ENTRIES.includes(name); +} + +const REPORTABLE_ENTRY_NAME = /^[A-Za-z0-9._-]{1,64}$/; +const UUID_LIKE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{16,}/gi; + +/** + * Bounded, redacted names for a refusal message: at most three entries, safe + * characters only, identifiers collapsed. Anything else reports as "other". + */ +export function describeUnexpectedNativeSessionRootEntries(names: readonly string[]): string { + const described = [...new Set(names.map(name => REPORTABLE_ENTRY_NAME.test(name) + ? name.replace(UUID_LIKE, "") : "other"))].sort(); + const shown = described.slice(0, 3); + return described.length > shown.length ? `${shown.join(",")},+${described.length - shown.length}` : shown.join(","); +}