diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index cd96c833aa4..6efb240dc2b 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -2939,8 +2939,11 @@ loss. The marker may describe the first turn, so it cannot identify every later warm checkpoint. Legacy file-backed roots retain their existing checks. A restored checkpoint must match the normalized session before writing its local projection. This adds a scoped checkpoint read and local reconstruction only -when warm admission encounters an external marker or an older projection; it -adds no streaming checkpoint writes or recurring idle work. +when warm admission encounters an external marker or a prior-run projection. +A prior-run projection refreshes even when its run ID already matches the saved +supervisor: the canonical checkpoint can have advanced after an earlier read. +Current-run state and a retained in-memory warm owner keep their existing +authority. This adds no streaming checkpoint writes or recurring idle work. Ordinary remote `per_turn` continuations restore a valid external controller marker from the referenced run's company/task-bound checkpoint before reading diff --git a/doc/testing-native-remote-continuity.md b/doc/testing-native-remote-continuity.md index 65671f91edf..65b00378d69 100644 --- a/doc/testing-native-remote-continuity.md +++ b/doc/testing-native-remote-continuity.md @@ -12,6 +12,11 @@ checkpoints and changed process configuration require verified suspension and replacement, while preserving the conversation and starting each turn once. Recovery can restart a provider to refresh its tool contract; process reuse is required for ordinary healthy follow-ups. +The `stale-projection` case leaves a full local control projection from just +before the prior run's terminal while its database checkpoint is complete. It +must refresh that same-run projection before the next turn, then preserve the +same Runner and provider history through another healthy follow-up. This is a +controlled cache fault, not a reconstruction of a historical incident. Each recovery case includes another healthy turn after the recovered turn and asserts that it does not start another runner or provider process. The same-server case also starts preparation lease renewal during recovery and diff --git a/maps/agent-access-map/runs-adapters.md b/maps/agent-access-map/runs-adapters.md index 74ba6a7c22f..9c9f142b7ff 100644 --- a/maps/agent-access-map/runs-adapters.md +++ b/maps/agent-access-map/runs-adapters.md @@ -351,6 +351,14 @@ guard. This changes no actor permission or session-reset authority. Follow the [session recipe](../feature-map/runs-adapters.md#session) for recovery followed by ordinary turns; deterministic runner checks do not establish live provider access. +Warm continuation without a retained controller owner refreshes a prior-run local +projection from its company/task/session-bound canonical checkpoint, including +later commits under the same run ID. Current-run state keeps its owner. Pending +canonical commands and mismatched checkpoint bindings still hold admission. This +adds no agent permission, session-reset authority, or provider replay path. The +executor and remote continuity regressions cover this internal recovery boundary; +live provider access remains separate evidence. + Sandbox export carries shallow Git boundaries through the full-bundle fallback and uses normal Git fetch connectivity checks. Existing host ancestry and merge guards remain effective. This grants no task retry or operator recovery authority; diff --git a/maps/feature-map/runs-adapters.md b/maps/feature-map/runs-adapters.md index b2c786031c3..ce07cbb01bd 100644 --- a/maps/feature-map/runs-adapters.md +++ b/maps/feature-map/runs-adapters.md @@ -102,6 +102,11 @@ The [remote continuity regression](../../doc/testing-native-remote-continuity.md uses a real Linux runner and deterministic provider. Live provider acceptance is a separate check. +Repeat with a full prior-run controller projection that predates its committed +database checkpoint under the same run ID. Admission must refresh that projection +before reuse. A pending canonical command must still hold execution; current-run +state and a retained warm owner must keep their existing authority. + In a fresh sandbox workspace, fetch a different published branch with depth one, then save a local result and finish. Verify export imports the shallow history, retains host-only edits, and completes finalization without another provider turn. diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 4430e7c2121..aa23a2d35b0 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -150,7 +150,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( expect(next.kind).toBe("bootstrap_incomplete"); }); - it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "legacy", "changed", "same-server", "diskless", "diskless-per-turn", "diskless-per-turn-read-retry", "diskless-warm-suspended", "diskless-warm-suspended-read-retry", "diskless-live-inspection"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { + it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "stale-projection", "legacy", "changed", "same-server", "diskless", "diskless-per-turn", "diskless-per-turn-read-retry", "diskless-warm-suspended", "diskless-warm-suspended-read-retry", "diskless-live-inspection"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { const hardRestart = !["warm", "same-server"].includes(scenario); const diskless = scenario.startsWith("diskless"); const perTurn = scenario.startsWith("diskless-per-turn"); @@ -364,6 +364,28 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( } expect(changed).toBe(1); } + if (scenario === "stale-projection") { + const checkpoint = await loadNativeRunnerCheckpoint(fixture.db, { companyId, + issueId: fixture.issueId, runId: fixture.runId }); + expect(checkpoint).not.toBeNull(); + const sessions = resolve(resolvePaperclipInstanceRoot(), "runtime/paperclip-runner/durable-sessions"); + let projected = 0; + for (const name of await readdir(sessions)) { + const root = resolve(sessions, name), path = resolve(root, "control-plane/control-plane-state.json"); + if (!existsSync(path) || JSON.parse(await readFile(path, "utf8")).identity?.runId !== fixture.runId) continue; + checkpoint!.restoreSessionDirectory(root); + const state = JSON.parse(await readFile(path, "utf8")); + const terminalIndex = state.committedEvents.findLastIndex((event: { eventType: string }) => event.eventType === "run.terminal"); + expect(terminalIndex).toBeGreaterThan(0); + // Retain a full local projection from just before the terminal. The + // database keeps the complete same-run checkpoint and all receipts. + state.committedEvents = state.committedEvents.slice(0, terminalIndex); + state.ackedSourceSeq = state.committedEvents.at(-1).sourceSeq; + await writeFile(path, JSON.stringify(state), { mode: 0o600 }); + projected++; + } + expect(projected).toBe(1); + } if (diskless) { // Preserve sandbox/provider storage and workspace dependencies. Delete // both native controller projections; admission must discover the source. diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index bd1844e8dd7..57b549bffd1 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -3,6 +3,7 @@ import { readIdleLocalWork, markIdleStartupComplete, markIdleIngressTracked } fr import * as runnerCheckpoint from "./native-runner-checkpoint.js"; import * as stagedAttachmentRecovery from "./native-staged-attachment-recovery.js"; import * as controllerRecovery from "./native-controller-recovery.js"; +import { isSettledRemoteWarmControlState } from "./remote-warm-session-recovery.js"; import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js"; import * as environmentTaskRuntime from "../environment-task-runtime.js"; import * as pluginTaskRuntime from "./plugin-runner-task.js"; @@ -36,7 +37,8 @@ import { import { execFile, execFileSync, spawn, spawnSync } from "node:child_process"; import { tmpdir } from "node:os"; import { inspect } from "node:util"; -import { dirname, join } from "node:path"; +import { basename, dirname, join } from "node:path"; +import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; import { activityLog, budgetReservations, @@ -14581,6 +14583,78 @@ describe("runnerd provider runtime wiring", () => { } }); + it.each(["settled", "pending", "current-run", "legacy-file", "binding-mismatch"] as const)("refreshes a prior warm controller projection after its same-run checkpoint advances: %s", async mode => { + const prior = scopedNativeExecution("warm-same-run-checkpoint-advance"); + prior.session.lifecyclePolicy = { mode: "warm", idleTimeoutMs: 60_000 }; + const current = { ...prior, binding: { ...prior.binding, + runId: mode === "current-run" ? prior.binding.runId : `${prior.binding.runId}-next` } }; + const identity = { runId: prior.binding.runId, normalizedSessionId: prior.session.normalizedSessionId!, + runnerInstanceId: "runner-warm-checkpoint-advance", environmentLeaseId: "lease-warm-checkpoint-advance", + turnId: "prior-turn", itemId: "prior-item" }; + const root = await initializeScopedRunnerRoot(prior, identity.runnerInstanceId); + const controller = new DurablePrpControlPlane({ stateDirectory: join(root, "control-plane"), identity, + expectedRunnerVersion: "warm-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + controller.queueCommand("run.prepare", { provider: prior.provider }); + const path = join(root, "control-plane/control-plane-state.json"); + const staleProjection = await readFile(path, "utf8"); + if (mode !== "pending") { + controller.store.state.commands[0]!.status = "completed"; + controller.store.state.commands[0]!.result = { status: "completed" }; + controller.store.state.committedEvents.push({ sourceSeq: 1, sourceEventId: "warm-terminal", + eventType: "run.terminal", priority: 0, envelope: { ...identity }, deliveryCount: 1, logicalEffectCount: 1 }); + controller.store.state.ackedSourceSeq = 1; + } + controller.persistRunAttachTemplate({ provider: prior.provider, normalizedSessionId: identity.normalizedSessionId }); + let saved!: import("../../vendor/paperclip-runner/index.js").DurablePrpCheckpoint; + controller.setCheckpointPersistence(async checkpoint => { saved = checkpoint; }); + await controller.flushCheckpoint(); + // A previous admission can leave a full local projection while the owner + // later commits its completed command and terminal under the same run ID. + await writeFile(path, staleProjection); + await mkdir(join(root, "runner"), { recursive: true }); + await writeFile(join(root, "runner/runner-state.json"), JSON.stringify(durableRunnerState(identity, "suspended"))); + const supervisorDirectory = join(resolvePaperclipInstanceRoot(), "runtime/paperclip-runner/sessions"); + await mkdir(supervisorDirectory, { recursive: true }); + const supervisorPath = join(supervisorDirectory, `${basename(root)}.json`); + await writeFile(supervisorPath, JSON.stringify({ schema: "paperclip.native-session-supervisor.v1", + snapshot: { identity: { ...prior.binding, sessionId: identity.normalizedSessionId } } })); + const restore = vi.spyOn(runnerCheckpoint, "restoreNativeRunnerCheckpoint").mockImplementation(async (_db, binding, directory, sessionId) => { + expect(binding).toEqual({ companyId: prior.binding.companyId, issueId: prior.binding.issueId, runId: prior.binding.runId }); + expect(sessionId).toBe(identity.normalizedSessionId); + if (mode === "legacy-file") return null; + if (mode === "binding-mismatch") throw new Error("native_runner_checkpoint_binding_mismatch"); + saved.restoreSessionDirectory(directory); + return saved; + }); + const admitted = new Error("warm resume reached current tool authority"); + state.toolAuthorityDefinitions.mockRejectedValueOnce(admitted); + state.createBackend.mockClear(); + state.createTransport.mockClear(); + try { + await expect(createRunnerdBackend({ db: priorNativeRunDb(prior), execution: current, + runnerInstanceId: "next-warm-runner", runnerExecutionTarget: { kind: "remote", transport: "sandbox", + remoteCwd: "/workspace", runner: { execute: vi.fn() } } as never })).rejects.toThrow( + mode === "binding-mismatch" ? "native_runner_checkpoint_binding_mismatch" : admitted.message); + const refreshed = JSON.parse(await readFile(path, "utf8")); + const refreshedFromDatabase = mode === "settled" || mode === "pending"; + expect(refreshed.commands).toEqual([expect.objectContaining({ type: "run.prepare", + status: mode === "settled" ? "completed" : "pending" })]); + expect(refreshed.committedEvents).toEqual(mode === "settled" + ? [expect.objectContaining({ eventType: "run.terminal" })] : []); + expect(isSettledRemoteWarmControlState(refreshed, identity)).toBe(mode === "settled"); + if (refreshedFromDatabase) expect(refreshed.runAttachTemplate).not.toBeNull(); + else expect(await readFile(path, "utf8")).toBe(staleProjection); + expect(restore).toHaveBeenCalledTimes(mode === "current-run" ? 0 : 1); + expect(state.createBackend).not.toHaveBeenCalled(); + expect(state.createTransport).not.toHaveBeenCalled(); + } finally { + state.toolAuthorityDefinitions.mockReset().mockResolvedValue([]); + restore.mockRestore(); + await controller.stop(); + await rm(supervisorPath, { force: true }); + } + }); + it.each(["suspended", "backup", "codex", "file-projection", "legacy-file-no-checkpoint", "file-checkpoint-binding-error", "missing-checkpoint", "checkpoint-binding-error", "foreign-task", "foreign-marker-session", "missing-marker", "malformed-marker", "unknown-marker-schema", "active-owner", "foreign-prior-scope", "unsuspended-runner"] as const)( diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index fd3d9ccc29b..715e7c599a7 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -5966,7 +5966,10 @@ async function restoreWarmRunnerdController(input: { issueId = prior.issueId; } } - if (identity?.runId !== runId) { + // A prior admission can leave a full projection before later commits under + // the same run ID. Without a retained owner, refresh it from its canonical + // checkpoint; current-run state still belongs to the active controller. + if (identity?.runId !== runId || (!retainedRunId && runId !== input.execution.binding.runId)) { const restored = await restoreNativeRunnerCheckpoint(input.db, { companyId: input.execution.binding.companyId, issueId, runId }, root, nativeSessionKey(input.execution)); // A legacy file-backed root still follows its existing prior-owner checks.