diff --git a/doc/plans/2026-10-31-zero-downtime-web-deploys.md b/doc/plans/2026-10-31-zero-downtime-web-deploys.md index 27c3be61b2d..322439952b0 100644 --- a/doc/plans/2026-10-31-zero-downtime-web-deploys.md +++ b/doc/plans/2026-10-31-zero-downtime-web-deploys.md @@ -1,30 +1,36 @@ # Zero-downtime web deployments -Date: 2026-10-31. Status (2026-11-01 13:28 UTC): existing-volume staging enrollment +Date: 2026-10-31. Status (2026-11-01 13:54 UTC): existing-volume staging enrollment is complete and its read/save/file-integrity checks pass. Active-agent recovery, -idle behavior and the ten-deploy staging campaign remain unqualified. No production -canary yet. +natural database idling and the ten-deploy staging campaign remain unqualified. +No production canary yet. -Latest critical path: Core #18543 includes the runtime admission correction and -owned immutable-cache retirement. Review exposed a resolved host-source fallback; -its fix passes all 122 workspace-preparation tests. The live test at `e14ca5f3e9` -now reconstructs the agent-file projection, deletes A's entire home after graceful -shutdown (22.8 seconds), and transfers ownership of the same remote execution. -It then fails because adoption tries to rematerialize launch skills from A's -deleted controller directory. That failure remains failed evidence; all three -owned Daytona sandboxes were subsequently confirmed absent and the temporary -credential revoked. Adoption now skips preparation of launch assets for the -already-running provider. The regression fails before the fix and passes after -it; all 71 related adoption/recovery transport cases pass with real local Runner -processes and simulated providers. No paid-model success is inferred. +Latest critical path: Core #18543 includes runtime admission, immutable-cache +retirement, remote agent-file projection, and adoption without rematerializing +launch assets. The live test at `67714af9ff` transferred the same runner/session to +controller generation two after deleting A's entire home, then failed in the full +backend constructor: composing system instructions still read A's retired bundle. +All three owned Daytona sandboxes were independently confirmed absent; the child +credential was revoked and the test object bucket deleted. This remains failed +evidence, not a successful deployment qualification. + +The backend now accepts admitted instructions from a validated same-run Runner +checkpoint. Runner owns checkpoint/template inspection; Core binds it to the +current run, normalized session and context digest. Staged attachments select the +validated target template. Fresh runs and ordinary later continuations still +compose current instructions. The deleted-bundle reproduction failed before the +fix, then passed with real backend session recovery and no new thread/turn start. +All 59 backend factory tests and the focused checkpoint validation tests pass. +The next step is a clean rebuilt live retry and exact-head PR review. Review also identified save quotas incorrectly applied during cache recovery. -Recovery now uses the existing inbound remote projection and preserves path/link -validation without enforcing save quotas on admission. The over-quota case still -rejects an ordinary save. A broad directory run passed 110 cases but hit three -cleanup timing failures; all 20 cleanup/cache-loss cases passed in isolation with -the same timeouts. Server and Runner TypeScript checks pass. A rebuilt live retry -and exact-head PR review are next. +Recovery uses the existing inbound remote projection and preserves path/link +validation without enforcing save quotas on admission. Ordinary saves still reject +oversized changes. Related transport recovery tests passed 71 cases; directory +cleanup/cache-loss tests passed 20 cases after three timing failures in a broad +run. A separate CI attention-retention timeout reproduced on clean master: its +fixed clock fell behind database-assigned updatedAt. Advancing across the saved +deadline fixes it; all 53 tests in that file pass with original timeouts. Cloud #1102 merged as `461c4aea28` and is serving on the staging controller. The original enrollment completed using its original qualified image, a retained @@ -34,10 +40,13 @@ The original browser tab reconnected and persisted a task title. Server logs lat showed old Agents chunks returned 404 and navigation recovered via a page reload: this is migration continuity evidence, not uninterrupted old-tab asset loading. -The first eight-minute natural database-idle observation failed. A bounded SQL -attribution window is underway and is explicitly not an idle observation. Normal -automatic sleep was correctly refused because this development fixture is not -eligible for the free/demo idle-sleep policy; manual sleep/wake is still pending. +Two eight-minute natural database-idle observations failed. A separate bounded SQL +attribution window saw no other app-database sessions; it is diagnostic stimulus, +not idle proof, and does not establish the source of provider activity. Automatic +idle sleep was correctly refused because this development fixture is not eligible +for the free/demo policy. Manual sleep with force disabled passed: the registry +reports sleeping, Railway has no active web deployment or attached volume, and +Neon is idle. Browser wake is being verified through the normal organization entry. Private receipts remain under the task's live-test, staging enrollment and hosted probe directories; no credential or live target identity belongs here. diff --git a/packages/paperclip-runner/src/backends/codex-native-backend.ts b/packages/paperclip-runner/src/backends/codex-native-backend.ts index d5e94f5c62c..69bcbf411ee 100644 --- a/packages/paperclip-runner/src/backends/codex-native-backend.ts +++ b/packages/paperclip-runner/src/backends/codex-native-backend.ts @@ -21,6 +21,8 @@ import { } from "./runtime-context.js"; export interface CodexNativeSessionBackendOptions { + /** Admitted instructions from a validated same-run checkpoint, supplied by the recovery owner. */ + retainedSystemInstructions?: string; /** Effective provider environment, including the assigned workspace boundary. */ environment?: NodeJS.ProcessEnv; observability?: import("../drivers/agent-observability.js").ProviderObservabilityLaunch; @@ -149,7 +151,7 @@ function createTransportBackedNativeSessionBackend( ...nativeTaskConstraints(input), ]; - const baseInstructions = nativeSystemInstructions(input); + const baseInstructions = options.retainedSystemInstructions ?? nativeSystemInstructions(input); const transportFactory = options.transportFactory; return new HarnessDriverBackend( new CodexAppServerDriver({ diff --git a/packages/paperclip-runner/src/backends/native-backend-factory.test.ts b/packages/paperclip-runner/src/backends/native-backend-factory.test.ts index 23e2b6fd89d..91e3a46b958 100644 --- a/packages/paperclip-runner/src/backends/native-backend-factory.test.ts +++ b/packages/paperclip-runner/src/backends/native-backend-factory.test.ts @@ -360,6 +360,36 @@ describe("native backend factory", () => { await session.close({ reason: "test complete" }); }); + it("recovers with admitted instructions after the old bundle is removed", async () => { + const input = preparedExecution(execution()); + if (!("runtimeContext" in input)) throw new Error("fixture requires runtime context"); + const retainedSystemInstructions = composeNativeSystemInstructions(input.runtimeContext, "Keep assigned instructions."); + const original = await createNativeSessionBackend(input, { + codexTransportFactory: () => new FakeCodexTransport(), + environment: { ...process.env, PAPERCLIP_WORKSPACE_CWD: WORKSPACE }, + }).openSession({ + identity: { runId: "run", sessionId: "session", companyId: "company", agentId: "agent", issueId: "issue" }, + workingDirectory: WORKSPACE, + }); + const snapshot = await original.snapshot(); + await original.close({ reason: "controller replacement" }); + rmSync(input.runtimeContext.instructions.bundle.rootPath, { recursive: true }); + const transport = new FakeCodexTransport(); + transport.readResponse = { thread: { id: transport.threadId, sessionId: transport.providerSessionId, cwd: WORKSPACE, turns: [] } }; + const factory = vi.fn(() => transport); + expect(() => createNativeSessionBackend(input, { codexTransportFactory: factory })).toThrow(); + expect(factory).not.toHaveBeenCalled(); + const backend = createNativeSessionBackend(input, { + retainedSystemInstructions, codexTransportFactory: factory, + workingDirectoryAuthority: "remote_runner", environment: { PAPERCLIP_WORKSPACE_CWD: WORKSPACE }, + }); + const recovered = await backend.recoverSession!({ ...snapshot, activeTurnId: null }, { signal: new AbortController().signal }); + expect(recovered).toMatchObject({ recovered: true }); + expect(factory).toHaveBeenCalledWith(expect.objectContaining({ baseInstructions: retainedSystemInstructions })); + expect(transport.calls.some(call => call.method === "thread/start" || call.method === "turn/start")).toBe(false); + if (recovered.recovered) await recovered.session.close({ reason: "test complete" }); + }); + describe("runner-prepared launch working directory", () => { // The transport reports the prepared directory; it exists only on the VM. class PreparedLaunchTransport extends FakeCodexTransport { diff --git a/packages/paperclip-runner/src/backends/native-backend-factory.ts b/packages/paperclip-runner/src/backends/native-backend-factory.ts index f50bd0c9a4f..f603fb0a8c8 100644 --- a/packages/paperclip-runner/src/backends/native-backend-factory.ts +++ b/packages/paperclip-runner/src/backends/native-backend-factory.ts @@ -68,6 +68,7 @@ export function createNativeSessionBackend( } if (options.codexTransportFactory) { return createRunnerdNativeSessionBackend(input, { + retainedSystemInstructions: options.retainedSystemInstructions, completionFeedback: options.completionFeedback, runnerInstanceId: options.runnerInstanceId, onSpawn: options.onSpawn, diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts index c4f29bdbb6a..b55f34797cf 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts @@ -33,6 +33,7 @@ import { decodeDurablePrpCheckpoint, inspectDurablePrpCheckpointHeader, inspectStagedRunAttachment, + inspectRetainedRunInstructions, inspectStagedRunAttachmentHeader, type DurablePrpCheckpoint, durableRecoveryInternals, @@ -55,6 +56,33 @@ const identity: DurableRecoveryIdentity = { const expectedRunnerVersion = "0.3.0"; const expectedRunnerDigest = `sha256:${"a".repeat(64)}`; +it("recovers only the matching run's admitted instructions from a valid checkpoint", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-retained-instructions-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + const digest = "b".repeat(64); + const expected = { runId: identity.runId, normalizedSessionId: identity.normalizedSessionId, runtimeContextDigest: digest }; + const template = { provider: { instructions: "Exact admitted instructions.\n", runtimeContext: { aggregateDigest: digest } } }; + try { + core.persistRunAttachTemplate(template); + expect(inspectRetainedRunInstructions(core.store.state, expected)).toBe(template.provider.instructions); + for (const field of ["runId", "normalizedSessionId", "runtimeContextDigest"] as const) { + expect(inspectRetainedRunInstructions(core.store.state, { ...expected, [field]: "foreign" })).toBeNull(); + } + expect(inspectRetainedRunInstructions({ ...core.store.state, commands: null }, expected)).toBeNull(); + expect(inspectRetainedRunInstructions({ ...core.store.state, runAttachTemplate: { provider: { ...template.provider, instructions: "" } } }, expected)).toBeNull(); + const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" }; + const targetTemplate = { provider: { instructions: "Target instructions", runtimeContext: { aggregateDigest: "c".repeat(64) } } }; + core.stageRunAttachment(target, targetTemplate); + const targetExpected = { ...expected, runId: target.runId, runtimeContextDigest: "c".repeat(64) }; + expect(inspectRetainedRunInstructions(core.store.state, targetExpected)).toBe("Target instructions"); + expect(inspectRetainedRunInstructions(core.store.state, { ...targetExpected, runtimeContextDigest: digest })).toBeNull(); + expect(inspectRetainedRunInstructions(core.store.state, expected)).toBe(template.provider.instructions); + const corrupted = structuredClone(core.store.state); + corrupted.stagedRunAttachment!.templateDigest = `sha256:${"f".repeat(64)}`; + expect(inspectRetainedRunInstructions(corrupted, targetExpected)).toBeNull(); + } finally { await core.stop(); rmSync(root, { recursive: true, force: true }); } +}); + it("persists an undispatched source-to-target attachment before bootstrap and preserves it after cache loss", async () => { const root = mkdtempSync(resolve(tmpdir(), "runner-staged-attachment-")); const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" }; diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index ea82cd674e0..fad19bfada3 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -923,6 +923,26 @@ export function inspectStagedRunAttachment(state: unknown, expectedTarget?: Dura } catch { return null; } } +/** Recover admitted instructions without opening the retired controller's assets. + * Inspection is evidence only; the caller still authorizes the runner and lease. */ +export function inspectRetainedRunInstructions(state: unknown, expected: { + runId: string; normalizedSessionId: string; runtimeContextDigest: string; +}): string | null { + try { + if (!isRecord(state) || !exactIdentity(state.identity) || !isStoredCoreState(state, state.identity)) return null; + const matches = (identity: DurableRecoveryIdentity) => identity.runId === expected.runId + && identity.normalizedSessionId === expected.normalizedSessionId; + const staged = state.stagedRunAttachment; + const template = staged && matches(staged.targetIdentity) + ? stagedAttachmentTemplate(state) + : matches(state.identity) ? state.runAttachTemplate : null; + if (!isRecord(template) || !isRecord(template.provider)) return null; + const { instructions, runtimeContext } = template.provider; + return isRecord(runtimeContext) && runtimeContext.aggregateDigest === expected.runtimeContextDigest + && typeof instructions === "string" && instructions.trim().length > 0 ? instructions : null; + } catch { return null; } +} + function isStoredCoreState( value: unknown, identity: DurableRecoveryIdentity, diff --git a/packages/paperclip-runner/src/index.ts b/packages/paperclip-runner/src/index.ts index 9f628c39f0a..72e5aa5eb1e 100644 --- a/packages/paperclip-runner/src/index.ts +++ b/packages/paperclip-runner/src/index.ts @@ -39,6 +39,7 @@ export { RUNNER_PRP_PROTOCOL_MAX, inspectWarmRunTransition, inspectStagedRunAttachment, + inspectRetainedRunInstructions, inspectStagedRunAttachmentHeader, type StagedRunAttachment, type DurablePrpControlPlaneOptions, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index b2799be84cb..452a290b23d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -145,6 +145,7 @@ import { executeNativeSession, applyNativeSessionGoalControl, inspectStagedRunAttachment, + inspectRetainedRunInstructions, inspectWarmRunTransition, parseNativeExecutionInput, parsePaperclipQuestionSet, @@ -12545,6 +12546,19 @@ async function createRunnerdBackendWithinSessionClaim( : null; const stagedRecovery = input.restartRecovery?.kind === "reattach_remote_runner" ? await verifyStagedAttachmentRestart(input) : null; + const retainedSystemInstructions = input.restartRecovery?.kind === "reattach_remote_runner" + && "runtimeContext" in input.execution + && input.execution.schema !== "paperclip.native-execution-input.v6" + && input.execution.schema !== "paperclip.native-execution-input.v8" + ? inspectRetainedRunInstructions(JSON.parse(readBoundedNativeFile( + resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, + "native_runner_recovery_instructions_unavailable").toString("utf8")), { + runId: input.execution.binding.runId, + normalizedSessionId: nativeSessionKey(input.execution), + runtimeContextDigest: input.execution.runtimeContext.aggregateDigest, + }) + : undefined; + if (retainedSystemInstructions === null) throw new Error("native_runner_recovery_instructions_unavailable"); let stagedRecoveryPending = stagedRecovery !== null; const recoveryUsageIdentity = stagedRecovery?.sourceIdentity ?? input.remoteWarmRecovery?.identity ?? durableIdentity; const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity @@ -12647,6 +12661,7 @@ async function createRunnerdBackendWithinSessionClaim( return current.execute(call); }; const backend = createNativeSessionBackend(input.execution, { + retainedSystemInstructions, // The plugin resolves the VM directory while preparing the launch. Core keeps // its host environment out of the plugin and records the prepared directory. runnerInstanceId: prior?.runnerInstanceId ?? effectiveRunnerInstanceId, environment: {}, @@ -13981,6 +13996,7 @@ async function createRunnerdBackendWithinSessionClaim( runId: input.execution.binding.runId, prepareRemote: prepareRemoteRunner }); } const backend = createNativeSessionBackend(runnerExecution, { + retainedSystemInstructions, runnerInstanceId: retainedTransition?.runnerInstanceId ?? input.runnerInstanceId, environment: effectiveRunnerEnvironment, diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index 63a918160ea..c2122cb1c6e 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -95,6 +95,7 @@ export const RUNNER_PRP_PROTOCOL_MAX = runner.RUNNER_PRP_PROTOCOL_MAX; export const runnerCodexDynamicToolsFit = runner.runnerCodexDynamicToolsFit; export const inspectWarmRunTransition = runner.inspectWarmRunTransition; export const inspectStagedRunAttachment = runner.inspectStagedRunAttachment; +export const inspectRetainedRunInstructions = runner.inspectRetainedRunInstructions; export const inspectStagedRunAttachmentHeader = runner.inspectStagedRunAttachmentHeader; export type StagedRunAttachment = import("@paperclipai/paperclip-runner").StagedRunAttachment; export const readRunnerdArtifactBinding = runner.readRunnerdArtifactBinding;