fix: recover native instructions from the admitted run checkpoint

Co-Authored-By: Paperclip <noreply@paperclip.ing>

doc/plans/2026-10-31-zero-downtime-web-deploys.md

@@ -1,30 +1,36 @@
 # Zero-downtime web deployments
 
-Date: 2026-10-31. Status (2026-11-01 13:28 UTC): existing-volume staging enrollment
+Date: 2026-10-31. Status (2026-11-01 13:54 UTC): existing-volume staging enrollment
 is complete and its read/save/file-integrity checks pass. Active-agent recovery,
-idle behavior and the ten-deploy staging campaign remain unqualified. No production
-canary yet.
+natural database idling and the ten-deploy staging campaign remain unqualified.
+No production canary yet.
 
-Latest critical path: Core #18543 includes the runtime admission correction and
-owned immutable-cache retirement. Review exposed a resolved host-source fallback;
-its fix passes all 122 workspace-preparation tests. The live test at `e14ca5f3e9`
-now reconstructs the agent-file projection, deletes A's entire home after graceful
-shutdown (22.8 seconds), and transfers ownership of the same remote execution.
-It then fails because adoption tries to rematerialize launch skills from A's
-deleted controller directory. That failure remains failed evidence; all three
-owned Daytona sandboxes were subsequently confirmed absent and the temporary
-credential revoked. Adoption now skips preparation of launch assets for the
-already-running provider. The regression fails before the fix and passes after
-it; all 71 related adoption/recovery transport cases pass with real local Runner
-processes and simulated providers. No paid-model success is inferred.
+Latest critical path: Core #18543 includes runtime admission, immutable-cache
+retirement, remote agent-file projection, and adoption without rematerializing
+launch assets. The live test at `67714af9ff` transferred the same runner/session to
+controller generation two after deleting A's entire home, then failed in the full
+backend constructor: composing system instructions still read A's retired bundle.
+All three owned Daytona sandboxes were independently confirmed absent; the child
+credential was revoked and the test object bucket deleted. This remains failed
+evidence, not a successful deployment qualification.
+
+The backend now accepts admitted instructions from a validated same-run Runner
+checkpoint. Runner owns checkpoint/template inspection; Core binds it to the
+current run, normalized session and context digest. Staged attachments select the
+validated target template. Fresh runs and ordinary later continuations still
+compose current instructions. The deleted-bundle reproduction failed before the
+fix, then passed with real backend session recovery and no new thread/turn start.
+All 59 backend factory tests and the focused checkpoint validation tests pass.
+The next step is a clean rebuilt live retry and exact-head PR review.
 
 Review also identified save quotas incorrectly applied during cache recovery.
-Recovery now uses the existing inbound remote projection and preserves path/link
-validation without enforcing save quotas on admission. The over-quota case still
-rejects an ordinary save. A broad directory run passed 110 cases but hit three
-cleanup timing failures; all 20 cleanup/cache-loss cases passed in isolation with
-the same timeouts. Server and Runner TypeScript checks pass. A rebuilt live retry
-and exact-head PR review are next.
+Recovery uses the existing inbound remote projection and preserves path/link
+validation without enforcing save quotas on admission. Ordinary saves still reject
+oversized changes. Related transport recovery tests passed 71 cases; directory
+cleanup/cache-loss tests passed 20 cases after three timing failures in a broad
+run. A separate CI attention-retention timeout reproduced on clean master: its
+fixed clock fell behind database-assigned updatedAt. Advancing across the saved
+deadline fixes it; all 53 tests in that file pass with original timeouts.
 
 Cloud #1102 merged as `461c4aea28` and is serving on the staging controller.
 The original enrollment completed using its original qualified image, a retained
@@ -34,10 +40,13 @@ The original browser tab reconnected and persisted a task title. Server logs lat
 showed old Agents chunks returned 404 and navigation recovered via a page reload:
 this is migration continuity evidence, not uninterrupted old-tab asset loading.
 
-The first eight-minute natural database-idle observation failed. A bounded SQL
-attribution window is underway and is explicitly not an idle observation. Normal
-automatic sleep was correctly refused because this development fixture is not
-eligible for the free/demo idle-sleep policy; manual sleep/wake is still pending.
+Two eight-minute natural database-idle observations failed. A separate bounded SQL
+attribution window saw no other app-database sessions; it is diagnostic stimulus,
+not idle proof, and does not establish the source of provider activity. Automatic
+idle sleep was correctly refused because this development fixture is not eligible
+for the free/demo policy. Manual sleep with force disabled passed: the registry
+reports sleeping, Railway has no active web deployment or attached volume, and
+Neon is idle. Browser wake is being verified through the normal organization entry.
 Private receipts remain under the task's live-test, staging enrollment and hosted
 probe directories; no credential or live target identity belongs here.
 

packages/paperclip-runner/src/backends/codex-native-backend.ts

@@ -21,6 +21,8 @@ import {
 } from "./runtime-context.js";
 
 export interface CodexNativeSessionBackendOptions {
+  /** Admitted instructions from a validated same-run checkpoint, supplied by the recovery owner. */
+  retainedSystemInstructions?: string;
   /** Effective provider environment, including the assigned workspace boundary. */
   environment?: NodeJS.ProcessEnv;
   observability?: import("../drivers/agent-observability.js").ProviderObservabilityLaunch;
@@ -149,7 +151,7 @@ function createTransportBackedNativeSessionBackend(
     ...nativeTaskConstraints(input),
   ];
 
-  const baseInstructions = nativeSystemInstructions(input);
+  const baseInstructions = options.retainedSystemInstructions ?? nativeSystemInstructions(input);
   const transportFactory = options.transportFactory;
   return new HarnessDriverBackend(
     new CodexAppServerDriver({

packages/paperclip-runner/src/backends/native-backend-factory.test.ts

@@ -360,6 +360,36 @@ describe("native backend factory", () => {
     await session.close({ reason: "test complete" });
   });
 
+  it("recovers with admitted instructions after the old bundle is removed", async () => {
+    const input = preparedExecution(execution());
+    if (!("runtimeContext" in input)) throw new Error("fixture requires runtime context");
+    const retainedSystemInstructions = composeNativeSystemInstructions(input.runtimeContext, "Keep assigned instructions.");
+    const original = await createNativeSessionBackend(input, {
+      codexTransportFactory: () => new FakeCodexTransport(),
+      environment: { ...process.env, PAPERCLIP_WORKSPACE_CWD: WORKSPACE },
+    }).openSession({
+      identity: { runId: "run", sessionId: "session", companyId: "company", agentId: "agent", issueId: "issue" },
+      workingDirectory: WORKSPACE,
+    });
+    const snapshot = await original.snapshot();
+    await original.close({ reason: "controller replacement" });
+    rmSync(input.runtimeContext.instructions.bundle.rootPath, { recursive: true });
+    const transport = new FakeCodexTransport();
+    transport.readResponse = { thread: { id: transport.threadId, sessionId: transport.providerSessionId, cwd: WORKSPACE, turns: [] } };
+    const factory = vi.fn(() => transport);
+    expect(() => createNativeSessionBackend(input, { codexTransportFactory: factory })).toThrow();
+    expect(factory).not.toHaveBeenCalled();
+    const backend = createNativeSessionBackend(input, {
+      retainedSystemInstructions, codexTransportFactory: factory,
+      workingDirectoryAuthority: "remote_runner", environment: { PAPERCLIP_WORKSPACE_CWD: WORKSPACE },
+    });
+    const recovered = await backend.recoverSession!({ ...snapshot, activeTurnId: null }, { signal: new AbortController().signal });
+    expect(recovered).toMatchObject({ recovered: true });
+    expect(factory).toHaveBeenCalledWith(expect.objectContaining({ baseInstructions: retainedSystemInstructions }));
+    expect(transport.calls.some(call => call.method === "thread/start" || call.method === "turn/start")).toBe(false);
+    if (recovered.recovered) await recovered.session.close({ reason: "test complete" });
+  });
+
   describe("runner-prepared launch working directory", () => {
     // The transport reports the prepared directory; it exists only on the VM.
     class PreparedLaunchTransport extends FakeCodexTransport {

packages/paperclip-runner/src/backends/native-backend-factory.ts

@@ -68,6 +68,7 @@ export function createNativeSessionBackend(
   }
   if (options.codexTransportFactory) {
     return createRunnerdNativeSessionBackend(input, {
+      retainedSystemInstructions: options.retainedSystemInstructions,
       completionFeedback: options.completionFeedback,
       runnerInstanceId: options.runnerInstanceId,
       onSpawn: options.onSpawn,

packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts

@@ -33,6 +33,7 @@ import {
   decodeDurablePrpCheckpoint,
   inspectDurablePrpCheckpointHeader,
   inspectStagedRunAttachment,
+  inspectRetainedRunInstructions,
   inspectStagedRunAttachmentHeader,
   type DurablePrpCheckpoint,
   durableRecoveryInternals,
@@ -55,6 +56,33 @@ const identity: DurableRecoveryIdentity = {
 const expectedRunnerVersion = "0.3.0";
 const expectedRunnerDigest = `sha256:${"a".repeat(64)}`;
 
+it("recovers only the matching run's admitted instructions from a valid checkpoint", async () => {
+  const root = mkdtempSync(resolve(tmpdir(), "runner-retained-instructions-"));
+  const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest });
+  const digest = "b".repeat(64);
+  const expected = { runId: identity.runId, normalizedSessionId: identity.normalizedSessionId, runtimeContextDigest: digest };
+  const template = { provider: { instructions: "Exact admitted instructions.\n", runtimeContext: { aggregateDigest: digest } } };
+  try {
+    core.persistRunAttachTemplate(template);
+    expect(inspectRetainedRunInstructions(core.store.state, expected)).toBe(template.provider.instructions);
+    for (const field of ["runId", "normalizedSessionId", "runtimeContextDigest"] as const) {
+      expect(inspectRetainedRunInstructions(core.store.state, { ...expected, [field]: "foreign" })).toBeNull();
+    }
+    expect(inspectRetainedRunInstructions({ ...core.store.state, commands: null }, expected)).toBeNull();
+    expect(inspectRetainedRunInstructions({ ...core.store.state, runAttachTemplate: { provider: { ...template.provider, instructions: "" } } }, expected)).toBeNull();
+    const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" };
+    const targetTemplate = { provider: { instructions: "Target instructions", runtimeContext: { aggregateDigest: "c".repeat(64) } } };
+    core.stageRunAttachment(target, targetTemplate);
+    const targetExpected = { ...expected, runId: target.runId, runtimeContextDigest: "c".repeat(64) };
+    expect(inspectRetainedRunInstructions(core.store.state, targetExpected)).toBe("Target instructions");
+    expect(inspectRetainedRunInstructions(core.store.state, { ...targetExpected, runtimeContextDigest: digest })).toBeNull();
+    expect(inspectRetainedRunInstructions(core.store.state, expected)).toBe(template.provider.instructions);
+    const corrupted = structuredClone(core.store.state);
+    corrupted.stagedRunAttachment!.templateDigest = `sha256:${"f".repeat(64)}`;
+    expect(inspectRetainedRunInstructions(corrupted, targetExpected)).toBeNull();
+  } finally { await core.stop(); rmSync(root, { recursive: true, force: true }); }
+});
+
 it("persists an undispatched source-to-target attachment before bootstrap and preserves it after cache loss", async () => {
   const root = mkdtempSync(resolve(tmpdir(), "runner-staged-attachment-"));
   const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" };

packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts

@@ -923,6 +923,26 @@ export function inspectStagedRunAttachment(state: unknown, expectedTarget?: Dura
   } catch { return null; }
 }
 
+/** Recover admitted instructions without opening the retired controller's assets.
+ * Inspection is evidence only; the caller still authorizes the runner and lease. */
+export function inspectRetainedRunInstructions(state: unknown, expected: {
+  runId: string; normalizedSessionId: string; runtimeContextDigest: string;
+}): string | null {
+  try {
+    if (!isRecord(state) || !exactIdentity(state.identity) || !isStoredCoreState(state, state.identity)) return null;
+    const matches = (identity: DurableRecoveryIdentity) => identity.runId === expected.runId
+      && identity.normalizedSessionId === expected.normalizedSessionId;
+    const staged = state.stagedRunAttachment;
+    const template = staged && matches(staged.targetIdentity)
+      ? stagedAttachmentTemplate(state)
+      : matches(state.identity) ? state.runAttachTemplate : null;
+    if (!isRecord(template) || !isRecord(template.provider)) return null;
+    const { instructions, runtimeContext } = template.provider;
+    return isRecord(runtimeContext) && runtimeContext.aggregateDigest === expected.runtimeContextDigest
+      && typeof instructions === "string" && instructions.trim().length > 0 ? instructions : null;
+  } catch { return null; }
+}
+
 function isStoredCoreState(
   value: unknown,
   identity: DurableRecoveryIdentity,

packages/paperclip-runner/src/index.ts

@@ -39,6 +39,7 @@ export {
   RUNNER_PRP_PROTOCOL_MAX,
   inspectWarmRunTransition,
   inspectStagedRunAttachment,
+  inspectRetainedRunInstructions,
   inspectStagedRunAttachmentHeader,
   type StagedRunAttachment,
   type DurablePrpControlPlaneOptions,

server/src/services/native-runtime/native-session-executor.ts

@@ -145,6 +145,7 @@ import {
   executeNativeSession,
   applyNativeSessionGoalControl,
   inspectStagedRunAttachment,
+  inspectRetainedRunInstructions,
   inspectWarmRunTransition,
   parseNativeExecutionInput,
   parsePaperclipQuestionSet,
@@ -12545,6 +12546,19 @@ async function createRunnerdBackendWithinSessionClaim(
       : null;
   const stagedRecovery = input.restartRecovery?.kind === "reattach_remote_runner"
     ? await verifyStagedAttachmentRestart(input) : null;
+  const retainedSystemInstructions = input.restartRecovery?.kind === "reattach_remote_runner"
+    && "runtimeContext" in input.execution
+    && input.execution.schema !== "paperclip.native-execution-input.v6"
+    && input.execution.schema !== "paperclip.native-execution-input.v8"
+    ? inspectRetainedRunInstructions(JSON.parse(readBoundedNativeFile(
+        resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
+        "native_runner_recovery_instructions_unavailable").toString("utf8")), {
+        runId: input.execution.binding.runId,
+        normalizedSessionId: nativeSessionKey(input.execution),
+        runtimeContextDigest: input.execution.runtimeContext.aggregateDigest,
+      })
+    : undefined;
+  if (retainedSystemInstructions === null) throw new Error("native_runner_recovery_instructions_unavailable");
   let stagedRecoveryPending = stagedRecovery !== null;
   const recoveryUsageIdentity = stagedRecovery?.sourceIdentity ?? input.remoteWarmRecovery?.identity ?? durableIdentity;
   const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity
@@ -12647,6 +12661,7 @@ async function createRunnerdBackendWithinSessionClaim(
       return current.execute(call);
     };
     const backend = createNativeSessionBackend(input.execution, {
+      retainedSystemInstructions,
       // The plugin resolves the VM directory while preparing the launch. Core keeps
       // its host environment out of the plugin and records the prepared directory.
       runnerInstanceId: prior?.runnerInstanceId ?? effectiveRunnerInstanceId, environment: {},
@@ -13981,6 +13996,7 @@ async function createRunnerdBackendWithinSessionClaim(
       runId: input.execution.binding.runId, prepareRemote: prepareRemoteRunner });
   }
   const backend = createNativeSessionBackend(runnerExecution, {
+    retainedSystemInstructions,
     runnerInstanceId:
       retainedTransition?.runnerInstanceId ?? input.runnerInstanceId,
     environment: effectiveRunnerEnvironment,

server/src/vendor/paperclip-runner/index.ts

@@ -95,6 +95,7 @@ export const RUNNER_PRP_PROTOCOL_MAX = runner.RUNNER_PRP_PROTOCOL_MAX;
 export const runnerCodexDynamicToolsFit = runner.runnerCodexDynamicToolsFit;
 export const inspectWarmRunTransition = runner.inspectWarmRunTransition;
 export const inspectStagedRunAttachment = runner.inspectStagedRunAttachment;
+export const inspectRetainedRunInstructions = runner.inspectRetainedRunInstructions;
 export const inspectStagedRunAttachmentHeader = runner.inspectStagedRunAttachmentHeader;
 export type StagedRunAttachment = import("@paperclipai/paperclip-runner").StagedRunAttachment;
 export const readRunnerdArtifactBinding = runner.readRunnerdArtifactBinding;