test(upgrades): qualify controller replacement with immutable artifacts and phase evidence (#17455)

<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/105/changes/paperclip/105
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip needs evidence that an admitted execution survives a
main-server replacement.
> - Runner mechanics and server adoption belong to #17097 and #17439.
> - A healthy HTTP endpoint does not prove recovery of the original
execution.
> - Qualification must identify the serving artifact, natural ownership,
authenticated attachment and resumed processing separately.
> - It must also retain failed runs and measure normal-operation costs.
> - This PR extends the existing qualification machinery and acceptance
record.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operators can assess upgrade continuity from phase evidence tied to
exact artifacts and profiles.

**Current behavior**

A same-artifact restart or synthetic executor pass cannot establish
different-artifact upgrade and rollback support. Idle SQL gates also do
not measure active-run capacity.

**Proposed behavior**

Use immutable A/B artifacts and strict phase receipts through the
existing supervisor. Preserve failed evidence and add measured
before/after workload costs.

Builds on merged #17097 and #17439. The base is now `master`. #17456
follows for review convenience. This PR lands qualification machinery
and the evidence record. The remaining hosted gates stay open for
deployment qualification; merging this machinery does not claim those
experiments passed.

## What Changed

- Extend the main-server supervisor and installed-artifact verification
for actual serving revisions, forward migrations and compatible
rollback.
- Distinguish HTTP readiness, natural ownership, authenticated
attachment, resumed processing and backlog settlement.
- Retain the original reconnect grace and execution identity checks.
Never substitute a new task or edited lease for continuity.
- Reuse real-harness and Daytona qualification paths, with exact profile
and artifact provenance.
- Bind current Copilot attestation inputs in the Daytona image
fingerprint.
- Keep one acceptance record, including both failed file-only stress
phases and the final passing bounded-pool comparison.

## Verification

After retargeting to master, all 158 focused qualification cases across
13 files pass, with E2E typecheck. The first 12-file batch has four
report-lock test failures and 133 passes. Master #17811 had committed
the exact already-approved output lock, while that fixture assumed HEAD
must be the uncorrected input. The fixture now checks both pinned forms,
executes the committed-lock path, and reconstructs the pinned input to
retain all rejection tests. The production helper, policy hashes, patch
and dependency resolution are unchanged. The corrected batch explicitly
includes both process-preservation suites; the prior 146-case receipt
has its own source and selection.

Current candidate `8f1d6ec02f` integrates master `09ad9adce2`, including
the merged server slice. Its diff contains only qualification code and
documentation. The [qualification
record](https://github.com/paperclipai/paperclip/blob/8f1d6ec02f5bf6c90e7b27da8156953cabd9a180/doc/plans/2026-10-22-hot-upgrade-qualification.md)
is the acceptance ledger and retains source-attributed failures and
hosted limits.

On the eighth integration, all 146 cases across 12 focused qualification
files pass, including report-lock preparation, workflow security and
workflow targeting. E2E typecheck passes. The earlier 118-case result
remains attributed to its prior source. The first sandbox invocation has
117 passes and one failure because process inspection was denied during
inert fixture cleanup. The permitted repeat passes all 118 and a
targeted process check finds no remaining fixture daemon. The earlier
96-case receipt remains source-attributed in the ledger. The latest two
review regressions first fail: an intermediate replacement can cite
another runner/session despite a valid final step, and malformed receipt
JSON aborts evidence packaging. Each replacement now binds
run/company/issue/runner/session, and corrupt receipts retain explicit
invalid metadata plus sanitized diagnostics while other logs and
receipts still package. These checks prove qualification machinery with
injected supervisor operations, configuration subprocesses and real
filesystem packaging. They do not qualify a hosted harness.

Three request-reader cases, 14 artifact/configuration/cache/history
cases, and three lost-partial-receipt cases fail before their respective
corrections. The final collector preserves sanitized startup, readiness
and recovery phase receipts through packaging and temporary-root
deletion. Earlier embedded-Postgres initialization, report-catalog
drift, CI scheduling/cleanup failures and fixture collisions remain
recorded with their corrections or limits. The previous Greptile service
error is retained as historical review evidence, not a current review
pass.

The record includes prior source `9bb9d3417e`'s actual Linux
contention/fairness successes and the ordinary-warm regression proof
from the server slice. The historical 300-session component campaign
passes four after phases and retains earlier burst failures. Those
component costs do not establish hosted capacity. The hosted API-only
pair has zero agents/viewers, fails both absolute latency gates and the
candidate memory gate, and does not complete the active-agent/viewer
matrix.

The strict ten-agent same-artifact restart remains attributed to its
original artifact. It does not qualify this updated stack. Actual
A→B/B→A, the normal drain-free Cloud rollout, and other hosted harnesses
remain open. No new provider turn, deployment or hosted qualification
was run during merge preparation.

Prior source `57d3767eac` jobs 122590643944, 122590643983 and
122590644161 ended after runner shutdown/cancellation without final
Vitest results. The shutdown cause is unproven. No retry was issued
before this source was superseded by the required server recovery fix.
These attempts remain failures/incomplete evidence. Fresh exact-head CI,
legacy status, completed Greptile 5/5, resolved feedback and normal
merge eligibility remain required.

## Risks

Different-artifact A→B/B→A, the normal drain-free Cloud rollout path,
other hosted harnesses and the complete concurrency/viewer matrix remain
open. Frozen branch builds are development-build evidence. This PR owns
qualification and evidence; production defects must be fixed in their
earlier owning slice.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The runtime does not expose a more specific model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

tests/runner-e2e/EVERYDAY-WORKFLOWS.md

@@ -1,5 +1,43 @@
 # Everyday Paperclip workflow evals
 
+## Main-server outage qualification
+
+The remote native Codex and Claude `recover-controller` cells default to SIGKILL,
+an observed 125-second offline interval and retirement of the old controller cache. They use
+natural ownership timing. The wrapper emits `controller-outage.json`; the grader
+requires distinct server processes, confirmed HTTP loss, cache retirement and the
+requested interval, in addition to same-run/provider-turn and finalization proof.
+Partial fault receipts remain in the attempt's private `control/` directory on
+failure. A successful task without the fault receipt cannot qualify.
+
+Use `PAPERCLIP_RUNNER_E2E_CONTROLLER_STOP_SIGNAL=SIGTERM` for graceful shutdown,
+`PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS=1000` for an explicit one-second
+experiment, or `PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS=300000` for the
+five-minute contract. These settings are recorded; compare only matching configurations. The configured
+duration is a minimum hold after confirmed HTTP loss. Shutdown and startup add
+to total recovery time; they do not extend the runner's seven-minute grace or the
+workflow's attempt deadline. `recover-controller` permits no automatic retries.
+
+```sh
+PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS=300000 pnpm test:e2e:runner -- --id everyday-workflows.runner-codex.daytona.recover-controller --max-automatic-retries 0
+```
+
+The credential-free wrapper smoke starts the actual main server and a separate
+disposable Postgres, kills/replaces the server, checks that Postgres never
+restarted, reads existing data, accepts a new HTTP mutation, and confirms cleanup:
+
+```sh
+node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/controller-replacement-smoke.ts
+```
+
+It prints a private JSON evidence path. It uses no model and **does not qualify
+active-agent continuity, queued launches, or an A-to-B version upgrade**. See the
+[qualification plan](../../doc/plans/2026-10-22-hot-upgrade-qualification.md) for the
+separate gates and outstanding work. Hosted qualification still requires a valid
+pinned Daytona image and the existing credential/budget admission.
+
+## Workflow suite
+
 This manual suite tests useful work through the production browser, public API,
 native runner, and normal agent instructions. It complements the tightly
 scripted runner contract fixtures. It does not add a scheduled or default paid
@@ -53,6 +91,12 @@ for 125 seconds after its health listener stops, then start its replacement.
 Before and after restart each pins
 the same run, normalized session, Runner identity, PID/start time, and active
 sandbox lease. The original receipt prefix must remain unchanged. A run that
+uses this native Codex remote cell also loses the old controller runtime cache
+between stopping and starting the server. The harness moves it into private
+evidence outside product lookup paths and requires an acknowledgement of that
+fault. Postgres and the instance encryption key survive. This checks deployment
+recovery without relying on the old controller journal or local warm-session cache.
+A run that
 ends before either active observation leaves the fault boundary unexercised;
 it cannot pass from the final ZIP alone. The cell no longer exempts an original
 Codex remote run that fails with `process_lost` and is replaced.
@@ -290,3 +334,24 @@ consumed response cannot extend that wait.
 `declineGradeEvidence` retain the exact cloned assertion input and original
 checks. Later final-state or cleanup observations cannot overwrite that input.
 Earlier campaign grades remain unchanged when the evaluator is corrected.
+# Upgrade qualification receipts
+
+`PAPERCLIP_RUNNER_E2E_UPGRADE_ARTIFACTS` selects frozen installed A/B artifacts
+and an alternating A/B/A sequence. The remote controller-recovery workflow
+executes every requested replacement on the original active run. If that run
+ends before a later fault can be injected, that boundary is unexercised; starting
+a replacement task does not qualify it.
+
+The supervisor reads existing Postgres ownership and Runner-owned checkpoint
+evidence. It records HTTP readiness, natural lease takeover, a new authenticated
+connection, and ingestion past the stopped controller's saved event cursor.
+Final native and workspace receipts establish backlog settlement. No lease
+timestamps are edited. Authentication and processing must be observed within
+the original 420-second grace, including artifact verification and startup.
+
+For a credential-free startup/migration experiment, run
+`controller-replacement-smoke.ts` with the manifest and an explicit disposable
+`PAPERCLIP_TEST_DATABASE_URL`. The driver creates and drops a unique empty
+database; it does not change the supplied database. This smoke qualifies zero
+agent turns. The [qualification record](../../doc/plans/2026-10-22-hot-upgrade-qualification.md)
+is the acceptance ledger for exact artifacts, attempts, and remaining gates.

tests/runner-e2e/OPERATIONS.md

@@ -958,7 +958,7 @@ The trusted report job applies a reviewed `report-lock.patch` before its frozen
 install. `report-lock.json` pins the input lock, patch and resulting lock hashes.
 This updates existing patch identities and the already declared yaml dependency
 without registry resolution. The worker target and its lock artifact never
-control report dependencies. An unknown committed lock fails before install;
+control report dependencies. An already-reviewed output lock is used unchanged. An unknown committed lock fails before install;
 update and review this correction when dependency inputs change. The original
 failed run remains failed when retained evidence gets a new presentation.
 

tests/runner-e2e/README.md

@@ -1282,7 +1282,7 @@ stay unchanged. The correction adds exactly yaml@2.9.1 with pinned integrity. Th
 trusted authorization job rejects unknown locks before paid work. Manual history
 publication uses the same reviewed correction. Qualification image builds apply
 it only to a target equal to the workflow revision; other targets keep their own
-lock. Unknown lock contents fail before installation. The
+lock. An already-reviewed output lock is used unchanged. Unknown lock contents fail before installation. The
 workflow definition, runner-group permission, and protected-environment
 deployment still come from the default branch. Do not select the target branch
 in GitHub's **Use workflow from** control.

tests/runner-e2e/catalog.ts

@@ -1516,7 +1516,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
     groups: ["native"], profiles: [...everydayProfiles,
       productionStoryProfile(runnerProfiles.find(profile => profile.id === "runner-opencode")!),
     ], environments: [localEnvironment, daytonaWarmEnvironment],
-    tasks: everydayTasks.map(task => ["hire-reuse", "delegate-feedback"].includes(task.id)
+    tasks: everydayTasks.map(task => ["hire-reuse", "delegate-feedback", "recover-controller"].includes(task.id)
       ? { ...task, automaticRetryPolicy: "single_attempt" as const } : task), expectedMatrixSize: 52,
     excludedExecutionIds: [...everydayProfiles.flatMap(profile => everydayTasks
       .filter(task => !["build-revise", "delegate-feedback", "recover-controller", "create-skill-studio"].includes(task.id))
@@ -1526,9 +1526,9 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
         ...(["hire-reuse", "delegate-feedback"].includes(task.id) ? [] : [`everyday-workflows.runner-opencode.local.${task.id}`]),
       ]),
     ],
-    definitionMetadata: { version: 9, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only",
-      remoteControllerDowntimeMs: 125_000,
-      remoteCodexRestart: "same-active-process-lease-native-prefix-and-workspace-receipt",
+    definitionMetadata: { version: 10, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only",
+      controllerDowntimeMs: 125_000,
+      remoteCodexRestart: "bounded-main-server-outage-receipt-empty-cache-same-active-process-and-finalization",
       procedureCases: { ids: ["hire-reuse", "delegate-feedback"], maximumAttemptsPerCell: 1, companyAndLeadBudgetCents: 1_000 } },
   },
   {

tests/runner-e2e/controller-database.test.ts

@@ -0,0 +1,79 @@
+import { spawn } from "node:child_process";
+import { once } from "node:events";
+import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
+import { createRequire } from "node:module";
+import path from "node:path";
+import { expect, it } from "vitest";
+import { createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "../../packages/db/src/index.js";
+import { assertEmbeddedDatabaseIsolation } from "./instance-isolation.js";
+import { prepareControllerDatabase } from "./controller-database.js";
+import { createRunnerE2ETemporaryRoot, prepareRunnerE2EServerConfig } from "./server-config.js";
+
+it("keeps an unmigrated fixture database alive when its serving client dies", async () => {
+  const root = await createRunnerE2ETemporaryRoot();
+  const configPath = path.join(root, "instance", "config.json");
+  let database: Awaited<ReturnType<typeof prepareControllerDatabase>> | undefined;
+  let db: ReturnType<typeof createDb> | undefined;
+  let child: ReturnType<typeof spawn> | undefined;
+  try {
+    await (await prepareRunnerE2EServerConfig({ temporaryRoot: root, configPath, serverPort: 3100 }))!.close();
+    database = await prepareControllerDatabase(configPath, root);
+    const config = JSON.parse(await readFile(configPath, "utf8"));
+    expect(config.database.mode).toBe("postgres");
+    await mkdir(path.dirname(config.secrets.localEncrypted.keyFilePath), { recursive: true });
+    await writeFile(config.secrets.localEncrypted.keyFilePath, "fixture-key");
+    await expect(assertEmbeddedDatabaseIsolation(configPath, root)).resolves.toBeUndefined();
+    db = createDb(config.database.connectionString);
+    const before = await db.$client`select pg_postmaster_start_time()::text as started`;
+    expect(await db.$client`select tablename from pg_tables where schemaname = 'public'`).toEqual([]);
+    const require = createRequire(new URL("../../packages/db/package.json", import.meta.url));
+    child = spawn(process.execPath, ["--input-type=module", "-e", `
+      const { default: postgres } = await import(process.argv[1]);
+      const sql = postgres(process.argv[2]);
+      await sql.unsafe('select 1');
+      process.stdout.write('connected');
+      setInterval(() => {}, 1000);
+    `, require.resolve("postgres"), config.database.connectionString], { stdio: ["ignore", "pipe", "pipe"] });
+    let errors = "";
+    child.stderr!.on("data", chunk => { errors += chunk; });
+    const connected = await Promise.race([
+      once(child.stdout!, "data").then(([chunk]) => String(chunk)),
+      once(child, "exit").then(() => { throw new Error(`Database client exited before connection: ${errors}`); }),
+    ]);
+    expect(connected).toBe("connected");
+    const stopped = once(child, "exit");
+    child.kill("SIGKILL");
+    await stopped;
+    expect(await db.$client`select pg_postmaster_start_time()::text as started`).toEqual(before);
+    await db.$client`create table replacement_probe (id integer primary key)`;
+    await db.$client`insert into replacement_probe values (1)`;
+    expect(await db.$client`select count(*)::integer as count from replacement_probe`).toEqual([{ count: 1 }]);
+    await db.$client.end({ timeout: 1 }); db = undefined;
+    await database.cleanup(); database = undefined;
+    // The launcher grades isolation after the supervisor has shut down.
+    await expect(assertEmbeddedDatabaseIsolation(configPath, root)).resolves.toBeUndefined();
+  } finally {
+    if (child && child.exitCode === null && child.signalCode === null) child.kill("SIGKILL");
+    await db?.$client.end({ timeout: 1 });
+    await database?.cleanup();
+    await rm(root, { recursive: true, force: true });
+  }
+}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
+
+it("rejects database directories outside the fixture and leaves external databases alone", async () => {
+  const root = await createRunnerE2ETemporaryRoot();
+  const configPath = path.join(root, "config.json");
+  try {
+    await (await prepareRunnerE2EServerConfig({ temporaryRoot: root, configPath, serverPort: 3100 }))!.close();
+    const config = JSON.parse(await readFile(configPath, "utf8"));
+    config.database.embeddedPostgresDataDir = path.join(root, "..", "unrelated");
+    await writeFile(configPath, JSON.stringify(config));
+    await expect(prepareControllerDatabase(configPath, root)).rejects.toThrow("must belong to its qualification fixture");
+    config.database.mode = "postgres";
+    config.database.connectionString = "postgres://unused@127.0.0.1:1/never-connect";
+    const encoded = JSON.stringify(config);
+    await writeFile(configPath, encoded);
+    await (await prepareControllerDatabase(configPath, root)).cleanup();
+    expect(await readFile(configPath, "utf8")).toBe(encoded);
+  } finally { await rm(root, { recursive: true, force: true }); }
+});

tests/runner-e2e/controller-database.ts

@@ -0,0 +1,43 @@
+/** Qualification owns Postgres outside the server process tree. No migrations
+ * run here: each serving artifact must apply its own schema on startup. */
+import { existsSync } from "node:fs";
+import { readFile, writeFile } from "node:fs/promises";
+import { createRequire } from "node:module";
+import path from "node:path";
+import { pathToFileURL } from "node:url";
+import { createEmbeddedPostgresLogBuffer, ensurePostgresDatabase, loadWithoutEmbeddedPostgresExitHooks,
+  prepareEmbeddedPostgresNativeRuntime } from "../../packages/db/src/index.js";
+import { recordControllerDatabaseOwnership } from "./instance-isolation.js";
+import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js";
+
+export async function prepareControllerDatabase(configPath: string, temporaryRoot: string) {
+  const config = paperclipConfigSchema.parse(JSON.parse(await readFile(configPath, "utf8")));
+  if (config.database.mode === "postgres") return { cleanup: async () => {} };
+  const directory = path.resolve(config.database.embeddedPostgresDataDir);
+  const relative = path.relative(path.resolve(temporaryRoot), directory);
+  if (!relative || relative.startsWith("..") || path.isAbsolute(relative)) throw new Error("Controller database must belong to its qualification fixture");
+  const require = createRequire(new URL("../../packages/db/package.json", import.meta.url));
+  const module = await loadWithoutEmbeddedPostgresExitHooks(() => import(pathToFileURL(require.resolve("embedded-postgres")).href));
+  await prepareEmbeddedPostgresNativeRuntime();
+  const logs = createEmbeddedPostgresLogBuffer();
+  const database = new module.default({ databaseDir: directory, user: "paperclip", password: "paperclip",
+    port: config.database.embeddedPostgresPort, persistent: true,
+    initdbFlags: ["--encoding=UTF8", "--locale=C", "--lc-messages=C"],
+    onLog: (message: unknown) => logs.append(message), onError: (message: unknown) => logs.append(message) });
+  let started = false;
+  try {
+    if (!existsSync(path.join(directory, "PG_VERSION"))) await database.initialise();
+    await database.start();
+    started = true;
+    const base = `postgres://paperclip:paperclip@127.0.0.1:${config.database.embeddedPostgresPort}`;
+    await ensurePostgresDatabase(`${base}/postgres`, "paperclip");
+    await recordControllerDatabaseOwnership({ configPath, temporaryRoot, dataDirectory: directory,
+      connectionString: `${base}/paperclip` });
+    config.database = { ...config.database, mode: "postgres", connectionString: `${base}/paperclip` };
+    await writeFile(configPath, `${JSON.stringify(config, null, 2)}\n`, { mode: 0o600 });
+    return { cleanup: async () => { await database.stop(); } };
+  } catch (error) {
+    if (started) await database.stop().catch(() => undefined);
+    throw new Error(`Qualification database startup failed: ${logs.getRecentLogs().join(" | ")}`, { cause: error });
+  }
+}

tests/runner-e2e/controller-outage.test.ts

@@ -0,0 +1,141 @@
+import { mkdtemp, mkdir, writeFile, readFile, rm, lstat } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { describe, expect, it, vi } from "vitest";
+import { configuredControllerOutage, controllerOutageForExecution, parseControllerOutage, replaceController, retireControllerCache, validControllerOutageReceipt } from "./controller-outage.js";
+import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
+
+function fixture() {
+  const order: string[] = [];
+  const receipts: unknown[] = [];
+  return { order, receipts, input: {
+    requestId: "calibration", previousPid: 100, outage: { signal: "SIGKILL" as const, holdMs: 20 },
+    stop: async (signal: string) => { order.push(signal); },
+    waitForUnhealthy: async () => { order.push("http-down"); },
+    discardCache: async () => { order.push("empty-cache"); },
+    start: vi.fn(async () => { order.push("start"); return { pid: 200 }; }),
+    waitForReady: async () => { order.push("ready"); }, cancelled: () => false,
+    record: async (receipt: unknown) => { receipts.push(receipt); },
+  } };
+}
+
+describe("controller outage qualification", () => {
+  it("keeps original-execution fault evidence enabled for both dedicated continuity cells", async () => {
+    const executions = selectRunnerExecutions(parseRunnerSelectors(["--suite", "daytona-restart-continuity"]));
+    expect(executions).toHaveLength(2);
+    for (const execution of executions) {
+      const outage = controllerOutageForExecution(execution, {});
+      expect(outage).toEqual({ signal: "SIGKILL", holdMs: 125_000 });
+      expect(controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "300000" }))
+        .toEqual({ signal: "SIGKILL", holdMs: 300_000 });
+      expect(() => controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "1000" }))
+        .toThrow("shorter than the dedicated continuity contract");
+    }
+  });
+
+  it("uses the everyday suite's declared default while retaining explicit short profiles", () => {
+    const [execution] = selectRunnerExecutions(parseRunnerSelectors([
+      "--id", "everyday-workflows.runner-codex.daytona.recover-controller",
+    ]));
+    expect(controllerOutageForExecution(execution!, {})).toEqual({ signal: "SIGKILL", holdMs: 125_000 });
+    expect(controllerOutageForExecution(execution!, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "1000" }))
+      .toEqual({ signal: "SIGKILL", holdMs: 1_000 });
+  });
+
+  it.each(["runner-codex", "runner-acpx-claude"])("supplies replacement fault evidence to the %s Daytona restart cell", async profile => {
+    const [execution] = selectRunnerExecutions(parseRunnerSelectors([
+      "--id", `everyday-workflows.${profile}.daytona.recover-controller`,
+    ]));
+    expect(execution).toBeDefined();
+    const f = fixture();
+    const outage = controllerOutageForExecution(execution!, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "20" });
+    const receipt = outage ? await replaceController({ ...f.input, outage }) : undefined;
+    expect(validControllerOutageReceipt(receipt, { signal: "SIGKILL", holdMs: 20 })).toBe(true);
+    expect(receipt).toMatchObject({ previousPid: 100, replacementPid: 200, cacheDiscarded: true });
+    expect(f.order).toEqual(["SIGKILL", "http-down", "empty-cache", "start", "ready"]);
+    for (const other of [
+      { ...execution!, environment: { ...execution!.environment, id: "local" as const } },
+      { ...execution!, profile: { ...execution!.profile, generation: "legacy" as const } },
+      { ...execution!, task: { ...execution!.task, id: "stop-redirect" } },
+    ]) expect(controllerOutageForExecution(other, {})).toBeUndefined();
+  });
+  it("records an actual bounded outage before starting the successor", async () => {
+    const f = fixture();
+    const receipt = await replaceController(f.input);
+    expect(f.order).toEqual(["SIGKILL", "http-down", "empty-cache", "start", "ready"]);
+    expect(receipt.replacementStartedAfterMs! - receipt.stoppedAfterMs!).toBeGreaterThanOrEqual(20);
+    expect(validControllerOutageReceipt(receipt, f.input.outage)).toBe(true);
+    expect(validControllerOutageReceipt(receipt, { signal: "SIGTERM", holdMs: 20 })).toBe(false);
+    expect(validControllerOutageReceipt(receipt, { signal: "SIGKILL", holdMs: 300_000 })).toBe(false);
+    expect(validControllerOutageReceipt(receipt, f.input.outage, "another-attempt")).toBe(false);
+    expect(f.receipts[0]).not.toHaveProperty("readyAfterMs");
+    expect(f.receipts.at(-1)).toEqual(receipt);
+  });
+
+  it.each(["stop", "waitForUnhealthy", "discardCache", "start", "waitForReady"] as const)("retains incomplete evidence when %s fails", async stage => {
+    const f = fixture();
+    f.input[stage] = vi.fn(async () => { throw new Error(stage); });
+    await expect(replaceController(f.input)).rejects.toThrow(stage);
+    expect(f.receipts.length).toBeGreaterThan(0);
+    expect(f.receipts.every(receipt => !validControllerOutageReceipt(receipt))).toBe(true);
+  });
+
+  it("cancels during the offline interval without launching another server", async () => {
+    const f = fixture();
+    f.input.cancelled = () => f.order.includes("empty-cache");
+    await expect(replaceController(f.input)).rejects.toThrow("shutdown interrupted");
+    expect(f.input.start).not.toHaveBeenCalled();
+  });
+
+  it.each([-1, 300001, 0.5, NaN, Infinity, "300000", null])("rejects invalid duration %s before signaling", holdMs => {
+    expect(() => parseControllerOutage({ signal: "SIGKILL", holdMs })).toThrow();
+  });
+  it.each(["", "-1", "1e3", "NaN", " 1000", "300001"])("rejects invalid environment duration %s", value => {
+    expect(() => configuredControllerOutage({ PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: value })).toThrow();
+  });
+  it("accepts the five-minute contract without extending the runner grace", () => {
+    expect(configuredControllerOutage({ PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "300000" }))
+      .toEqual({ signal: "SIGKILL", holdMs: 300000 });
+    expect(validControllerOutageReceipt({ schema: "paperclip.e2e.controller-outage/v1", requestId: "late",
+      signal: "SIGKILL", holdMs: 300000, previousPid: 1, replacementPid: 2, startedAt: new Date().toISOString(),
+      stoppedAfterMs: 0, cacheDiscarded: true, replacementStartedAfterMs: 300000, readyAfterMs: 420000 })).toBe(false);
+  });
+});
+
+it.each(["absent", "present"])("retires a %s old controller cache", async state => {
+  const root = await mkdtemp(join(tmpdir(), "controller-cache-"));
+  const cache = join(root, "cache"), retired = join(root, "retired");
+  try {
+    if (state === "present") { await mkdir(cache); await writeFile(join(cache, "evidence"), "saved"); }
+    await retireControllerCache(cache, retired);
+    await expect(lstat(cache)).rejects.toMatchObject({ code: "ENOENT" });
+    if (state === "present") expect(await readFile(join(retired, "evidence"), "utf8")).toBe("saved");
+  } finally { await rm(root, { recursive: true, force: true }); }
+});
+it("does not mistake a missing retirement parent for an already absent cache", async () => {
+  const root = await mkdtemp(join(tmpdir(), "controller-cache-")), cache = join(root, "cache");
+  try {
+    await mkdir(cache); await writeFile(join(cache, "evidence"), "saved");
+    await expect(retireControllerCache(cache, join(root, "missing", "retired"))).rejects.toMatchObject({ code: "ENOENT" });
+    expect(await readFile(join(cache, "evidence"), "utf8")).toBe("saved");
+  } finally { await rm(root, { recursive: true, force: true }); }
+});
+it.each(["missing", "previous-revision", "previous-tree", "replacement-revision", "replacement-tree", "reversed"])(
+  "requires the requested upgrade artifact pair when evidence is %s", problem => {
+    const A = { revision: "a".repeat(40), treeSha256: "a".repeat(64), version: "1", evidence: "development-build" as const };
+    const B = { ...A, revision: "b".repeat(40), treeSha256: "b".repeat(64) };
+    const receipt = { schema: "paperclip.e2e.controller-outage/v1", requestId: "upgrade", previousPid: 1, replacementPid: 2,
+      startedAt: new Date().toISOString(), signal: "SIGKILL", holdMs: 0, stoppedAfterMs: 0,
+      replacementStartedAfterMs: 1, readyAfterMs: 2, cacheDiscarded: true,
+      previousArtifact: { ...A }, replacementArtifact: { ...B } };
+    expect(validControllerOutageReceipt(receipt, undefined, "upgrade", { previous: A, replacement: B })).toBe(true);
+    expect(validControllerOutageReceipt({ ...receipt, previousArtifact: B, replacementArtifact: A }, undefined,
+      "upgrade", { previous: B, replacement: A })).toBe(true);
+    if (problem === "missing") Object.assign(receipt, { previousArtifact: undefined, replacementArtifact: undefined });
+    if (problem === "previous-revision") receipt.previousArtifact.revision = "c".repeat(40);
+    if (problem === "previous-tree") receipt.previousArtifact.treeSha256 = "c".repeat(64);
+    if (problem === "replacement-revision") receipt.replacementArtifact.revision = "c".repeat(40);
+    if (problem === "replacement-tree") receipt.replacementArtifact.treeSha256 = "c".repeat(64);
+    if (problem === "reversed") Object.assign(receipt, { previousArtifact: B, replacementArtifact: A });
+    expect(validControllerOutageReceipt(receipt, undefined, "upgrade", { previous: A, replacement: B })).toBe(false);
+  });

tests/runner-e2e/controller-outage.ts

@@ -0,0 +1,159 @@
+import { lstat, rename } from "node:fs/promises";
+import type { ServingArtifact } from "./upgrade-artifacts.js";
+import type { ControllerRecoveryReceipt } from "./controller-recovery-observer.js";
+import { setTimeout as delay } from "node:timers/promises";
+import type { MatrixExecution } from "./types.js";
+
+export interface ControllerOutage {
+  signal: "SIGTERM" | "SIGKILL";
+  holdMs: number;
+}
+
+export interface ControllerOutageReceipt extends ControllerOutage {
+  schema: "paperclip.e2e.controller-outage/v1";
+  requestId: string;
+  previousPid: number;
+  replacementPid?: number;
+  startedAt: string;
+  stoppedAfterMs?: number;
+  cacheDiscarded?: boolean;
+  replacementStartedAfterMs?: number;
+  readyAfterMs?: number;
+  previousArtifact?: ServingArtifact;
+  replacementArtifact?: ServingArtifact;
+  recovery?: ControllerRecoveryReceipt;
+  recoveryProgress?: Partial<ControllerRecoveryReceipt>;
+}
+
+export function parseControllerOutage(value: unknown): ControllerOutage {
+  if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("Invalid controller outage");
+  const { signal, holdMs } = value as Record<string, unknown>;
+  if ((signal !== "SIGTERM" && signal !== "SIGKILL") || !Number.isSafeInteger(holdMs)
+    || Number(holdMs) < 0 || Number(holdMs) > 300_000) throw new Error("Controller outage requires SIGTERM/SIGKILL and 0..300000 integer milliseconds");
+  return { signal, holdMs: Number(holdMs) };
+}
+
+export function configuredControllerOutage(env: NodeJS.ProcessEnv, defaultHoldMs = 1_000): ControllerOutage {
+  const raw = env.PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS ?? String(defaultHoldMs);
+  if (!/^\d+$/.test(raw)) throw new Error("Invalid PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS");
+  return parseControllerOutage({ signal: env.PAPERCLIP_RUNNER_E2E_CONTROLLER_STOP_SIGNAL ?? "SIGKILL", holdMs: Number(raw) });
+}
+
+export function controllerOutageForExecution(execution: MatrixExecution, env: NodeJS.ProcessEnv): ControllerOutage | undefined {
+  if (execution.task.flow !== "everyday_workflow" || execution.task.id !== "recover-controller"
+    || execution.environment.id !== "daytona" || execution.profile.generation !== "native"
+    || (execution.profile.provider !== "codex" && execution.profile.provider !== "acpx")) return undefined;
+  const declaredHoldMs = execution.suite.definitionMetadata?.controllerDowntimeMs;
+  if (typeof declaredHoldMs !== "number") throw new Error("Controller recovery suite must declare its outage duration");
+  const outage = configuredControllerOutage(env, declaredHoldMs);
+  if (execution.suite.id === "daytona-restart-continuity" && outage.holdMs < declaredHoldMs) {
+    throw new Error("Controller outage is shorter than the dedicated continuity contract");
+  }
+  return outage;
+}
+
+/** A fault receipt is required in addition to the runner's continuity receipts. */
+export function validControllerOutageReceipt(value: unknown, expected?: ControllerOutage, requestId?: string,
+  expectedArtifacts?: { previous: Pick<ServingArtifact, "revision" | "treeSha256">;
+    replacement: Pick<ServingArtifact, "revision" | "treeSha256"> }): value is ControllerOutageReceipt {
+  if (!value || typeof value !== "object" || Array.isArray(value)) return false;
+  const r = value as ControllerOutageReceipt;
+  try { parseControllerOutage(r); } catch { return false; }
+  const durations = [r.stoppedAfterMs, r.replacementStartedAfterMs, r.readyAfterMs];
+  return r.schema === "paperclip.e2e.controller-outage/v1"
+    && typeof r.requestId === "string" && /^[A-Za-z0-9._:-]{1,200}$/.test(r.requestId)
+    && typeof r.startedAt === "string" && Number.isFinite(Date.parse(r.startedAt))
+    && Number.isSafeInteger(r.previousPid) && r.previousPid > 0
+    && Number.isSafeInteger(r.replacementPid) && Number(r.replacementPid) > 0 && r.previousPid !== r.replacementPid
+    && r.cacheDiscarded === true
+    && durations.every(ms => typeof ms === "number" && Number.isFinite(ms) && ms >= 0)
+    && r.replacementStartedAfterMs! - r.stoppedAfterMs! >= r.holdMs
+    && r.readyAfterMs! >= r.replacementStartedAfterMs!
+    && r.readyAfterMs! < 420_000
+    && ((!r.previousArtifact && !r.replacementArtifact) || Boolean(r.previousArtifact && r.replacementArtifact
+      && [r.previousArtifact, r.replacementArtifact].every(a => /^[a-f0-9]{40}$/.test(a.revision)
+        && /^[a-f0-9]{64}$/.test(a.treeSha256) && typeof a.version === "string"
+        && ["released", "development-build"].includes(a.evidence))
+      && r.previousArtifact.revision !== r.replacementArtifact.revision
+      && r.previousArtifact.treeSha256 !== r.replacementArtifact.treeSha256))
+    && (!expectedArtifacts || Boolean(r.previousArtifact && r.replacementArtifact
+      && r.previousArtifact.revision === expectedArtifacts.previous.revision
+      && r.previousArtifact.treeSha256 === expectedArtifacts.previous.treeSha256
+      && r.replacementArtifact.revision === expectedArtifacts.replacement.revision
+      && r.replacementArtifact.treeSha256 === expectedArtifacts.replacement.treeSha256))
+    && (requestId === undefined || r.requestId === requestId)
+    && (!expected || (r.signal === expected.signal && r.holdMs === expected.holdMs));
+}
+
+/** Test harness only. Owners supply process cleanup, HTTP probing and cache paths.
+ * No database mutation, runner command or production recovery hook is involved.
+ */
+export async function replaceController<T extends { pid?: number }>(input: {
+  requestId: string; previousPid: number; outage: ControllerOutage;
+  stop: (signal: ControllerOutage["signal"]) => Promise<void>;
+  waitForUnhealthy: () => Promise<void>;
+  discardCache: () => Promise<void>;
+  start: () => Promise<T>;
+  waitForReady: (replacement: T) => Promise<void>;
+  cancelled: () => boolean;
+  record: (receipt: ControllerOutageReceipt) => Promise<void>;
+  previousArtifact?: ServingArtifact;
+  replacementArtifact?: () => ServingArtifact | undefined;
+  afterStopped?: (receipt: ControllerOutageReceipt) => Promise<void>;
+  afterReady?: (receipt: ControllerOutageReceipt, record: () => Promise<void>) => Promise<void>;
+}): Promise<ControllerOutageReceipt> {
+  const outage = parseControllerOutage(input.outage);
+  const started = performance.now();
+  const receipt: ControllerOutageReceipt = { schema: "paperclip.e2e.controller-outage/v1",
+    requestId: input.requestId, previousPid: input.previousPid, startedAt: new Date().toISOString(), ...outage,
+    ...(input.previousArtifact ? { previousArtifact: input.previousArtifact } : {}) };
+  const check = () => { if (input.cancelled()) throw new Error("Wrapper shutdown interrupted the controller outage"); };
+  const record = () => input.record({ ...receipt });
+  check();
+  await record(); // Failed attempts retain the boundary they actually reached.
+  await input.stop(outage.signal);
+  await input.waitForUnhealthy();
+  receipt.stoppedAfterMs = performance.now() - started;
+  await input.afterStopped?.(receipt);
+  await record();
+  await input.discardCache();
+  receipt.cacheDiscarded = true;
+  await record();
+  while (performance.now() - started - receipt.stoppedAfterMs < outage.holdMs) {
+    check();
+    await delay(Math.min(100, outage.holdMs - (performance.now() - started - receipt.stoppedAfterMs)));
+  }
+  check();
+  receipt.replacementStartedAfterMs = performance.now() - started;
+  await record();
+  const replacement = await input.start();
+  receipt.replacementPid = replacement.pid;
+  await record();
+  await input.waitForReady(replacement);
+  check();
+  receipt.readyAfterMs = performance.now() - started;
+  if (input.previousArtifact) {
+    receipt.replacementArtifact = input.replacementArtifact?.();
+    if (!receipt.replacementArtifact) throw new Error("Replacement has no verified serving artifact");
+  }
+  await record();
+  if (!validControllerOutageReceipt(receipt, outage, input.requestId)) throw new Error("Controller outage did not produce a qualifying replacement receipt");
+  await input.afterReady?.(receipt, record);
+  return receipt;
+}
+
+/** Retire only the controller-owned cache; this is not Runner state. */
+export async function retireControllerCache(cache: string, retired: string): Promise<void> {
+  try { await rename(cache, retired); }
+  catch (error) {
+    if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
+    // ENOENT can also mean the destination parent is missing. Only absence of
+    // the old cache satisfies retirement; preserve every other failure.
+    try { await lstat(cache); }
+    catch (missing) {
+      if ((missing as NodeJS.ErrnoException).code === "ENOENT") return;
+      throw missing;
+    }
+    throw error;
+  }
+}

tests/runner-e2e/controller-recovery-observer.integration.test.ts

@@ -0,0 +1,84 @@
+import { randomUUID } from "node:crypto";
+import path from "node:path";
+import { execFile } from "node:child_process";
+import { writeFile } from "node:fs/promises";
+import { promisify } from "node:util";
+import { pathToFileURL } from "node:url";
+import { afterEach, expect, it, vi } from "vitest";
+import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations,
+  nativeRunnerCheckpoints, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "../../packages/db/src/index.js";
+import { DurablePrpControlPlane } from "../../server/src/vendor/paperclip-runner/index.js";
+import { registerNativeRunnerCheckpoint } from "../../server/src/services/native-runtime/native-runner-checkpoint.js";
+import { recoveryPostgresSupport, startRecoveryPostgres } from "../../server/src/__tests__/helpers/recovery-postgres.js";
+import { createIsolatedTestHome } from "../../server/src/__tests__/helpers/isolated-test-home.js";
+import { readControllerRecoverySnapshot } from "./controller-recovery-observer.js";
+import { buildRunnerE2EProcessEnvironment } from "./harness-env.js";
+import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js";
+
+afterEach(() => vi.unstubAllEnvs());
+(recoveryPostgresSupport.supported ? it : it.skip)("reads real UUID runner ownership and text event identity in one read-only snapshot", async () => {
+  const database = await startRecoveryPostgres("controller-observer-");
+  const db = createDb(database.connectionString);
+  const home = createIsolatedTestHome();
+  vi.stubEnv("PAPERCLIP_HOME", home.home);
+  vi.stubEnv("PAPERCLIP_INSTANCE_ID", "observer-fixture");
+  vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", undefined);
+  vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", undefined);
+  const binding = { companyId: randomUUID(), issueId: randomUUID(), runId: randomUUID() };
+  const agentId = randomUUID(), runnerInstanceId = randomUUID(), normalizedSessionId = randomUUID();
+  const authority = new DurablePrpControlPlane({ stateDirectory: path.join(home.home, "authority"),
+    identity: { runnerInstanceId, normalizedSessionId, runId: binding.runId,
+      environmentLeaseId: randomUUID(), turnId: randomUUID(), itemId: randomUUID() },
+    expectedRunnerVersion: "observer-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
+  try {
+    await db.insert(companies).values({ id: binding.companyId, name: "Qualification observer" });
+    await db.insert(agents).values({ id: agentId, companyId: binding.companyId, name: "Observer fixture", adapterType: "paperclip_runner" });
+    await db.insert(issues).values({ id: binding.issueId, companyId: binding.companyId, title: "Observe original execution" });
+    await db.insert(heartbeatRuns).values({ id: binding.runId, companyId: binding.companyId, agentId,
+      invocationSource: "manual", status: "running", runtimeMode: "native", nativeIssueId: binding.issueId,
+      runnerInstanceId, nativeSessionId: normalizedSessionId });
+    await db.insert(nativeRunFinalizations).values({ ...binding, phase: "observed", controllerBootId: "observer-controller",
+      controllerGeneration: 1, leaseOwner: "observer-owner", leaseExpiresAt: new Date(Date.now() + 60_000) });
+    await registerNativeRunnerCheckpoint({ db, binding, authority, bootId: "observer-controller" });
+    await db.insert(heartbeatRunEvents).values([runnerInstanceId, randomUUID()].map((sourceInstanceId, index) => ({
+      companyId: binding.companyId, runId: binding.runId, agentId, seq: index + 1,
+      eventType: "fixture", sourceSeq: index === 0 ? 7 : 99, sourceInstanceId,
+    })));
+    const checkpointBefore = await db.select().from(nativeRunnerCheckpoints);
+    expect(await readControllerRecoverySnapshot(db, binding)).toMatchObject({
+      runId: binding.runId, runnerInstanceId, nativeSessionId: normalizedSessionId,
+      bootId: "observer-controller", generation: 1, sourceSeq: 7, authenticatedConnectionCount: 0,
+    });
+    expect(await db.select().from(nativeRunnerCheckpoints)).toEqual(checkpointBefore);
+    // Exercise the actual launcher boundary in a separate process. Neither an
+    // inline shell key nor a shell key file may replace this fixture's key.
+    const repositoryRoot = path.resolve(import.meta.dirname, "../..");
+    const childScript = path.join(home.home, "observe.mts");
+    const wrongKeyFile = path.join(home.home, "ambient-master.key");
+    await writeFile(wrongKeyFile, "b".repeat(64), { mode: 0o600 });
+    await writeFile(childScript, `
+      import { createDb } from ${JSON.stringify(pathToFileURL(path.join(repositoryRoot, "packages/db/src/index.ts")).href)};
+      import { readControllerRecoverySnapshot } from ${JSON.stringify(pathToFileURL(path.join(repositoryRoot, "tests/runner-e2e/controller-recovery-observer.ts")).href)};
+      const db = createDb(process.env.OBSERVER_TEST_DATABASE_URL);
+      try { console.log(JSON.stringify(await readControllerRecoverySnapshot(db, ${JSON.stringify(binding)}))); }
+      finally { await db.$client.end(); }
+    `, { mode: 0o600 });
+    for (const override of [{ PAPERCLIP_SECRETS_MASTER_KEY: "a".repeat(64) },
+      { PAPERCLIP_SECRETS_MASTER_KEY_FILE: wrongKeyFile }]) {
+      const { stdout } = await promisify(execFile)(process.execPath,
+        runnerE2ETypeScriptProcessArgs(repositoryRoot, childScript), {
+          env: buildRunnerE2EProcessEnvironment({ ...process.env, ...override,
+            OBSERVER_TEST_DATABASE_URL: database.connectionString }, []), timeout: 20_000,
+        });
+      expect(JSON.parse(stdout)).toMatchObject({ runId: binding.runId, sourceSeq: 7,
+        runnerInstanceId, authenticatedConnectionCount: 0 });
+    }
+    expect(await db.select().from(nativeRunnerCheckpoints)).toEqual(checkpointBefore);
+    await expect(readControllerRecoverySnapshot(db, { ...binding, companyId: randomUUID() }))
+      .rejects.toThrow("no admitted native owner");
+  } finally {
+    await authority.stop();
+    await database.cleanup();
+    home.cleanup();
+  }
+}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);

tests/runner-e2e/controller-recovery-observer.test.ts

@@ -0,0 +1,46 @@
+import type { Db } from "../../packages/db/src/client.js";
+import { describe, expect, it } from "vitest";
+import { parseRecoveryBinding, readControllerRecoverySnapshot, recoveryObservation, validControllerRecoveryReceipt, type RecoverySnapshot } from "./controller-recovery-observer.js";
+import type { ControllerOutageReceipt } from "./controller-outage.js";
+
+const binding = { companyId: "10000000-0000-0000-0000-000000000001", issueId: "10000000-0000-0000-0000-000000000002", runId: "10000000-0000-0000-0000-000000000003" };
+const before: RecoverySnapshot = { runId: binding.runId, runnerInstanceId: "runner", nativeSessionId: "session",
+  bootId: "old", generation: 1, attempt: 1, leaseExpiresAt: "2026-10-24T00:01:00.000Z", sourceSeq: 42, history: [],
+  authenticatedConnectionCount: 1, checkpointSourceSeq: 42 };
+const fault: ControllerOutageReceipt = { schema: "paperclip.e2e.controller-outage/v1", requestId: "restart", previousPid: 1,
+  replacementPid: 2, signal: "SIGKILL", holdMs: 1000, startedAt: "2026-10-24T00:00:00.000Z", stoppedAfterMs: 1,
+  replacementStartedAfterMs: 1001, readyAfterMs: 2000, cacheDiscarded: true };
+const after = (): RecoverySnapshot => ({ ...before, bootId: "new", generation: 2, sourceSeq: 47, authenticatedConnectionCount: 2,
+  history: [{ controllerBootId: "new", controllerGeneration: 2, disposition: "reattach_remote_runner", at: "2026-10-24T00:01:00.001Z" }] });
+describe("read-only controller recovery evidence", () => {
+  it("requires natural ownership and new original-run receipts after the stopped-process cursor", () => {
+    expect(recoveryObservation(before, before, fault, binding, 2500)).toBeNull();
+    expect(recoveryObservation(before, { ...after(), sourceSeq: 42 }, fault, binding, 61_000)).toBeNull();
+    expect(recoveryObservation(before, { ...after(), authenticatedConnectionCount: 1 }, fault, binding, 61_000)).toBeNull();
+    const recovery = recoveryObservation(before, after(), fault, binding, 62_000)!;
+    expect(validControllerRecoveryReceipt({ ...fault, recovery })).toBe(true);
+    expect(recovery.authenticatedAttachmentObservedAfterMs).toBe(62_000);
+    expect(recovery.ownershipAcquiredAt).toBe("2026-10-24T00:01:00.001Z");
+    expect(validControllerRecoveryReceipt(fault)).toBe(false);
+    expect(validControllerRecoveryReceipt({ ...fault, recovery: { ...recovery, resumedProcessingObservedAfterMs: 420_000 } })).toBe(false);
+  });
+  it.each(["runId", "runnerInstanceId", "nativeSessionId", "attempt"] as const)("rejects replacement of %s", field => {
+    expect(() => recoveryObservation(before, { ...after(), [field]: field === "attempt" ? 2 : "replacement" }, fault, binding, 62_000))
+      .toThrow("changed the admitted execution");
+  });
+  it("rejects shortened leases, replacement launches and recovery nudges", () => {
+    for (const change of [{ at: "2026-10-24T00:00:30.000Z" }, { disposition: "resume_dead_runner" }, { disposition: "bootstrap_incomplete" }]) {
+      const current = after(); Object.assign(current.history[0]!, change);
+      expect(() => recoveryObservation(before, current, fault, binding, 62_000)).toThrow();
+    }
+  });
+  it("validates the exact fixture binding", () => {
+    expect(parseRecoveryBinding(binding)).toEqual(binding);
+    for (const invalid of [null, {}, { ...binding, runId: "other" }, { ...binding, extra: true }]) expect(() => parseRecoveryBinding(invalid)).toThrow();
+  });
+});
+
+it.each([null, {}, "not-history", [null], [42]])("rejects malformed database recovery history %j before reading checkpoints", async history => {
+  const db = { transaction: async (work: (tx: unknown) => Promise<unknown>) => work({ execute: async () => [{ ...before, history }] }) } as unknown as Db;
+  await expect(readControllerRecoverySnapshot(db, binding)).rejects.toThrow("Recovery history must be an array of records");
+});

tests/runner-e2e/controller-recovery-observer.ts

@@ -0,0 +1,169 @@
+/** Read-only qualification observer. This never calls recovery, renews a lease,
+ * parses checkpoint records, or interprets an underlying harness protocol. */
+import { readFile } from "node:fs/promises";
+import { setTimeout as delay } from "node:timers/promises";
+import { createRequire } from "node:module";
+const { sql }: typeof import("../../packages/db/node_modules/drizzle-orm/index.js") =
+  createRequire(new URL("../../packages/db/package.json", import.meta.url))("drizzle-orm");
+import { createDb, type Db } from "../../packages/db/src/index.js";
+import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js";
+import { loadNativeRunnerCheckpoint } from "../../server/src/services/native-runtime/native-runner-checkpoint.js";
+import type { ControllerOutageReceipt } from "./controller-outage.js";
+
+export interface RecoveryBinding { companyId: string; issueId: string; runId: string }
+export interface RecoverySnapshot {
+  runId: string; runnerInstanceId: string; nativeSessionId: string;
+  bootId: string; generation: number; attempt: number; leaseExpiresAt: string | null;
+  sourceSeq: number; history: Array<Record<string, unknown>>;
+  authenticatedConnectionCount: number;
+  checkpointSourceSeq: number;
+}
+export interface ControllerRecoveryReceipt {
+  binding: RecoveryBinding;
+  runnerInstanceId: string;
+  nativeSessionId: string;
+  previousGeneration: number;
+  generation: number;
+  previousBootId: string;
+  bootId: string;
+  previousLeaseExpiresAt: string | null;
+  ownershipAcquiredAt: string;
+  ownershipObservedAfterMs: number;
+  authenticatedAttachmentObservedAfterMs: number;
+  resumedProcessingObservedAfterMs: number;
+  runnerSourceSeqBefore: number;
+  runnerSourceSeqAfter: number;
+  previousAuthenticatedConnectionCount: number;
+  authenticatedConnectionCount: number;
+  attachmentEvidence: "runner_checkpoint_authenticated_connection";
+}
+export function parseRecoveryBinding(value: unknown): RecoveryBinding {
+  if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("Invalid controller recovery binding");
+  const row = value as Record<string, unknown>;
+  if (Object.keys(row).length !== 3 || !["companyId", "issueId", "runId"].every(key =>
+    typeof row[key] === "string" && /^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(row[key] as string))) {
+    throw new Error("Invalid controller recovery binding");
+  }
+  return row as unknown as RecoveryBinding;
+}
+
+export function recoveryObservation(before: RecoverySnapshot, after: RecoverySnapshot, fault: ControllerOutageReceipt,
+  binding: RecoveryBinding, observedAfterMs: number): ControllerRecoveryReceipt | null {
+  if (before.runId !== binding.runId || after.runId !== before.runId || after.runnerInstanceId !== before.runnerInstanceId
+    || after.nativeSessionId !== before.nativeSessionId || after.attempt !== before.attempt) {
+    throw new Error("Controller replacement changed the admitted execution");
+  }
+  if (after.bootId === before.bootId || after.generation <= before.generation) return null;
+  const claim = after.history.findLast(entry => entry.controllerBootId === after.bootId
+    && entry.controllerGeneration === after.generation);
+  if (!claim || claim.disposition !== "reattach_remote_runner" || typeof claim.at !== "string") {
+    throw new Error("Replacement did not naturally adopt the original remote runner");
+  }
+  const claimedAt = Date.parse(claim.at);
+  if (!Number.isFinite(claimedAt) || claimedAt < Date.parse(fault.startedAt)
+    || (fault.signal === "SIGKILL" && before.leaseExpiresAt && claimedAt < Date.parse(before.leaseExpiresAt))) {
+    throw new Error("Replacement bypassed natural controller lease expiry");
+  }
+  if (after.authenticatedConnectionCount <= before.authenticatedConnectionCount || after.sourceSeq <= before.sourceSeq) return null;
+  // The runner-owned projection counts successful authentications. New canonical
+  // receipts separately prove ingestion beyond both the saved checkpoint and DB
+  // cursor. Replaying an old saved event alone cannot pass either boundary.
+  return { binding, runnerInstanceId: before.runnerInstanceId, nativeSessionId: before.nativeSessionId,
+    previousGeneration: before.generation, generation: after.generation, previousBootId: before.bootId,
+    bootId: after.bootId, previousLeaseExpiresAt: before.leaseExpiresAt, ownershipAcquiredAt: claim.at,
+    ownershipObservedAfterMs: observedAfterMs, authenticatedAttachmentObservedAfterMs: observedAfterMs,
+    resumedProcessingObservedAfterMs: observedAfterMs, runnerSourceSeqBefore: before.sourceSeq,
+    runnerSourceSeqAfter: after.sourceSeq, previousAuthenticatedConnectionCount: before.authenticatedConnectionCount,
+    authenticatedConnectionCount: after.authenticatedConnectionCount, attachmentEvidence: "runner_checkpoint_authenticated_connection" };
+}
+
+export function validControllerRecoveryReceipt(fault: ControllerOutageReceipt | undefined): boolean {
+  const r = fault?.recovery;
+  return Boolean(r && r.bootId !== r.previousBootId && r.generation > r.previousGeneration
+    && r.runnerSourceSeqAfter > r.runnerSourceSeqBefore && r.authenticatedConnectionCount > r.previousAuthenticatedConnectionCount
+    && r.attachmentEvidence === "runner_checkpoint_authenticated_connection"
+    && [r.ownershipObservedAfterMs, r.authenticatedAttachmentObservedAfterMs, r.resumedProcessingObservedAfterMs]
+      .every(ms => Number.isFinite(ms) && ms >= fault!.readyAfterMs! && ms < 420_000));
+}
+
+export async function readControllerRecoverySnapshot(db: Db, binding: RecoveryBinding): Promise<RecoverySnapshot> {
+  return db.transaction(async tx => {
+    const [row] = await tx.execute(sql`select r.id as "runId", r.runner_instance_id as "runnerInstanceId",
+      r.native_session_id as "nativeSessionId", f.controller_boot_id as "bootId", f.controller_generation as generation,
+      f.attempt, f.lease_expires_at::text as "leaseExpiresAt", f.recovery_history as history,
+      coalesce((select max(e.source_seq) from heartbeat_run_events e where e.run_id = r.id
+        and e.company_id = r.company_id and e.source_instance_id = r.runner_instance_id::text), 0)::float8 as "sourceSeq"
+      from heartbeat_runs r join native_run_finalizations f on f.run_id = r.id and f.company_id = r.company_id
+      where r.id = ${binding.runId} and r.company_id = ${binding.companyId} and r.native_issue_id = ${binding.issueId}`);
+    if (!row?.runnerInstanceId || !row.nativeSessionId || !row.bootId) throw new Error("Recovery binding has no admitted native owner");
+    if (!Array.isArray(row.history) || row.history.some(entry => entry === null
+      || typeof entry !== "object" || Array.isArray(entry))) {
+      throw new Error("Recovery history must be an array of records");
+    }
+    // Use the existing server loader and Runner's public diagnostic projection.
+    // The qualification code never reads encrypted material or private formats.
+    const checkpoint = await loadNativeRunnerCheckpoint(tx as unknown as Db, binding);
+    const evidence = checkpoint?.recoveryEvidence;
+    if (!evidence || evidence.runnerInstanceId !== row.runnerInstanceId || evidence.normalizedSessionId !== row.nativeSessionId) {
+      throw new Error("Recovery checkpoint has no matching runner evidence");
+    }
+    return { ...row, authenticatedConnectionCount: evidence.authenticatedConnectionCount,
+      checkpointSourceSeq: evidence.highestRecordedSourceSeq } as RecoverySnapshot;
+  }, { isolationLevel: "repeatable read", accessMode: "read only" });
+}
+
+export async function createControllerRecoveryObserver(configPath: string, binding: RecoveryBinding) {
+  const config = paperclipConfigSchema.parse(JSON.parse(await readFile(configPath, "utf8")));
+  const url = config.database.mode === "postgres" ? config.database.connectionString
+    : `postgres://paperclip:paperclip@127.0.0.1:${config.database.embeddedPostgresPort}/paperclip`;
+  if (!url) throw new Error("Qualification database is unavailable");
+  const db = createDb(url);
+  let before: RecoverySnapshot | undefined;
+  const read = () => readControllerRecoverySnapshot(db, binding);
+  return {
+    // Called only after the old server is confirmed dead. This also excludes
+    // receipts committed while it was stopping from replacement evidence.
+    stopped: async (fault: ControllerOutageReceipt) => {
+      before = await read();
+      before.sourceSeq = Math.max(before.sourceSeq, before.checkpointSourceSeq);
+      fault.recoveryProgress = { binding, previousBootId: before.bootId, previousGeneration: before.generation,
+        previousLeaseExpiresAt: before.leaseExpiresAt, runnerSourceSeqBefore: before.sourceSeq,
+        previousAuthenticatedConnectionCount: before.authenticatedConnectionCount,
+        runnerInstanceId: before.runnerInstanceId, nativeSessionId: before.nativeSessionId };
+    },
+    recovered: async (fault: ControllerOutageReceipt, record: () => Promise<void>, cancelled: () => boolean) => {
+      if (!before) throw new Error("Missing stopped-controller observation");
+      const deadline = Date.parse(fault.startedAt) + 420_000;
+      let ownershipObservedAfterMs: number | undefined;
+      let authenticatedObservedAfterMs: number | undefined;
+      while (Date.now() < deadline && !cancelled()) {
+        const current = await read();
+        const elapsed = Date.now() - Date.parse(fault.startedAt);
+        if (current.bootId !== before.bootId && current.generation > before.generation && ownershipObservedAfterMs === undefined) {
+          ownershipObservedAfterMs = elapsed;
+          fault.recoveryProgress = { ...fault.recoveryProgress, bootId: current.bootId, generation: current.generation,
+            ownershipObservedAfterMs };
+          await record();
+        }
+        if (ownershipObservedAfterMs !== undefined && current.authenticatedConnectionCount > before.authenticatedConnectionCount
+          && authenticatedObservedAfterMs === undefined) {
+          authenticatedObservedAfterMs = elapsed;
+          fault.recoveryProgress = { ...fault.recoveryProgress, authenticatedConnectionCount: current.authenticatedConnectionCount,
+            authenticatedAttachmentObservedAfterMs: elapsed };
+          await record();
+        }
+        const recovered = recoveryObservation(before, current, fault, binding, elapsed);
+        if (recovered) {
+          recovered.ownershipObservedAfterMs = ownershipObservedAfterMs!;
+          recovered.authenticatedAttachmentObservedAfterMs = authenticatedObservedAfterMs!;
+          fault.recovery = recovered;
+          await record();
+          return;
+        }
+        await delay(250);
+      }
+      throw new Error("Original runner did not reattach and resume ingestion within its original 420-second grace");
+    },
+    close: () => db.$client.end(),
+  };
+}

tests/runner-e2e/controller-replacement-smoke.test.ts

@@ -0,0 +1,51 @@
+import { tmpdir } from "node:os";
+import { execFile } from "node:child_process";
+import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { promisify } from "node:util";
+import { expect, it } from "vitest";
+import { upgradeArtifactKey } from "./upgrade-artifacts.js";
+import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js";
+
+const execute = promisify(execFile);
+const repositoryRoot = path.resolve(import.meta.dirname, "../..");
+
+it.each(["invalid-database-url", "postgres://fixture:fixture@127.0.0.1:1/fixture?connect_timeout=1", "invalid-outage", "invalid-manifest"])(
+  "retains failed replacement evidence when setup rejects %s", async connectionString => {
+    const configRoot = await mkdtemp(path.join(tmpdir(), "replacement-config-"));
+    const manifest = path.join(configRoot, "malformed.json");
+    await writeFile(manifest, "{broken JSON");
+    const fixtureEnv: NodeJS.ProcessEnv = { ...process.env, PAPERCLIP_TEST_DATABASE_URL: connectionString,
+      [upgradeArtifactKey]: undefined };
+    if (connectionString === "invalid-outage") fixtureEnv.PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS = "invalid";
+    if (connectionString === "invalid-manifest") fixtureEnv[upgradeArtifactKey] = manifest;
+    let output = "", exitCode: unknown;
+    try {
+      await execute(process.execPath, runnerE2ETypeScriptProcessArgs(repositoryRoot,
+        path.join(import.meta.dirname, "controller-replacement-smoke.ts")), {
+        cwd: repositoryRoot, timeout: 20_000,
+        env: fixtureEnv,
+      });
+    } catch (error) {
+      const failure = error as Error & { stdout: string; code: unknown };
+      output = failure.stdout; exitCode = failure.code;
+    }
+    await rm(configRoot, { recursive: true, force: true });
+    const started = output.split("\n").filter(line => line.startsWith("{")).map(line => JSON.parse(line))
+      .find(value => value.started === true);
+    expect(started?.evidence).toEqual(expect.any(String));
+    const temporaryRoot = path.dirname(started.evidence);
+    expect(path.basename(temporaryRoot)).toMatch(/^paperclip-runner-e2e-/);
+    try {
+      expect(exitCode).toBe(1);
+      const report = JSON.parse(await readFile(started.evidence, "utf8"));
+      expect(report).toMatchObject({ schema: "paperclip.e2e.controller-replacement-smoke/v1",
+        passed: false, providerTurns: 0, activeAgentContinuityQualified: false,
+        databaseCreated: false, wrapperStarted: false, cleanupConfirmed: true,
+        error: expect.any(String) });
+      if (connectionString === "invalid-outage") expect(report.error).toContain("Invalid PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS");
+      if (connectionString === "invalid-manifest") expect(report.error).toMatch(/JSON/);
+      expect(report.replacements).toBeUndefined();
+      expect(output).toContain('"passed":false');
+    } finally { await rm(temporaryRoot, { recursive: true, force: true }); }
+  });

tests/runner-e2e/controller-replacement-smoke.ts

@@ -0,0 +1,171 @@
+import { readUpgradeArtifacts, upgradeArtifactKey } from "./upgrade-artifacts.js";
+/** Credential-free main-server replacement. This proves the wrapper/startup/DB
+ * boundary; only the hosted workflow can qualify an active Daytona agent.
+ * Run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/controller-replacement-smoke.ts
+ */
+import assert from "node:assert/strict";
+import { execFileSync, spawn } from "node:child_process";
+import { randomUUID } from "node:crypto";
+import { createWriteStream } from "node:fs";
+import { mkdir, readFile, rename, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { setTimeout as delay } from "node:timers/promises";
+import { createDb, startEmbeddedPostgresTestDatabase } from "../../packages/db/src/index.js";
+import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js";
+import { configuredControllerOutage, validControllerOutageReceipt, type ControllerOutage } from "./controller-outage.js";
+import { runnerE2EServerControlPaths } from "./harness-env.js";
+import { reserveRunnerE2EServerPort } from "./ports.js";
+import { createRunnerE2ETemporaryRoot, prepareRunnerE2EServerConfig } from "./server-config.js";
+import { createRunnerE2EServerStopper } from "./server-stop.js";
+import { cleanupControllerSmokeDatabase } from "./controller-smoke-cleanup.js";
+import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js";
+
+let outage: ControllerOutage;
+let upgrades: Awaited<ReturnType<typeof readUpgradeArtifacts>> = null;
+const repositoryRoot = path.resolve(import.meta.dirname, "../..");
+const temporaryRoot = await createRunnerE2ETemporaryRoot();
+const home = path.join(temporaryRoot, "home");
+const configPath = path.join(home, "instances", "default", "config.json");
+const control = runnerE2EServerControlPaths(temporaryRoot);
+const reportPath = path.join(temporaryRoot, "replacement-evidence.json");
+console.info(JSON.stringify({ started: true, evidence: reportPath }));
+const report: Record<string, unknown> = { schema: "paperclip.e2e.controller-replacement-smoke/v1",
+  providerTurns: 0, activeAgentContinuityQualified: false, cleanupConfirmed: false,
+  databaseCreated: false, wrapperStarted: false };
+// Register resources only after creation so setup failures still have evidence
+// and cleanup never assumes a database or client exists.
+let database: { connectionString: string; cleanup(): Promise<void> } | undefined;
+let db: ReturnType<typeof createDb> | undefined;
+const log = createWriteStream(path.join(temporaryRoot, "wrapper.log"), { mode: 0o600 });
+const stopper = createRunnerE2EServerStopper({ gracefulTimeoutMs: 35_000, forcedTimeoutMs: 5_000,
+  hasSpawnError: () => false, markExpectedStop: () => {}, log: message => log.write(message) });
+let wrapper: ReturnType<typeof spawn> | undefined;
+let failure: unknown;
+let ownedPostgresProcessId: number | null | undefined = process.env.PAPERCLIP_TEST_DATABASE_URL ? null : undefined;
+report.databaseCleanupScope = ownedPostgresProcessId === null ? "owned_database" : "owned_cluster_process";
+try {
+  report.sourceRevision = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8" }).trim();
+  report.sourceDirty = execFileSync("git", ["status", "--porcelain"], { cwd: repositoryRoot, encoding: "utf8" }).trim().length > 0;
+  outage = configuredControllerOutage(process.env);
+  report.outage = outage;
+  upgrades = await readUpgradeArtifacts(process.env[upgradeArtifactKey]);
+  const port = await reserveRunnerE2EServerPort();
+  // The database belongs to this driver, never to either server process tree.
+  const external = process.env.PAPERCLIP_TEST_DATABASE_URL;
+  if (!external) {
+    if (upgrades) throw new Error("Artifact migration proof requires PAPERCLIP_TEST_DATABASE_URL; the embedded test helper pre-migrates its database");
+    database = await startEmbeddedPostgresTestDatabase("controller-replacement-smoke-");
+  } else {
+    // Create an empty database. Artifact A applies its own migrations before B.
+    // Record ownership before closing the admin client, which can itself fail.
+    const name = `upgrade_${randomUUID().replaceAll("-", "")}`;
+    const url = new URL(external); url.pathname = `/${name}`;
+    const admin = createDb(external);
+    try {
+      await admin.$client.unsafe(`create database "${name}"`);
+      database = { connectionString: url.toString(), cleanup: async () => {
+        const cleanup = createDb(external);
+        try { await cleanup.$client.unsafe(`drop database "${name}"`); }
+        finally { await cleanup.$client.end(); }
+      } };
+      report.databaseCreated = true;
+    } finally { await admin.$client.end(); }
+  }
+  report.databaseCreated = true;
+  db = createDb(database.connectionString);
+  if (ownedPostgresProcessId !== null) {
+    const [owner] = await db.$client`select split_part(pg_read_file('postmaster.pid', 0, 64), chr(10), 1)::int as pid`;
+    const pid = Number(owner?.pid);
+    if (!Number.isSafeInteger(pid) || pid <= 0) throw new Error("owned_postgres_identity_unavailable");
+    ownedPostgresProcessId = pid;
+  }
+  const reservation = await prepareRunnerE2EServerConfig({ temporaryRoot, configPath, serverPort: port });
+  await reservation?.close();
+  const config = paperclipConfigSchema.parse(JSON.parse(await readFile(configPath, "utf8")));
+  config.database = { ...config.database, mode: "postgres", connectionString: database.connectionString };
+  await writeFile(configPath, JSON.stringify(config), { mode: 0o600 });
+  const source = { PATH: process.env.PATH, HOME: home, TMPDIR: process.env.TMPDIR,
+    ...(upgrades ? { ...upgrades.artifacts.A.pins, [upgradeArtifactKey]: process.env[upgradeArtifactKey] } : {}),
+    NODE_ENV: "test", PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: configPath, PAPERCLIP_INSTANCE_ID: "default",
+    PAPERCLIP_AGENT_JWT_SECRET: randomUUID(), PAPERCLIP_DECISION_SIGNING_SECRET: randomUUID(),
+    PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: randomUUID(), BETTER_AUTH_SECRET: randomUUID(),
+    PAPERCLIP_RUNNER_E2E_SERVER_LOG: path.join(temporaryRoot, "server.log"),
+    PAPERCLIP_RUNNER_E2E_TEMP_ROOT: temporaryRoot, PAPERCLIP_RUNNER_E2E_PORT: String(port),
+    PAPERCLIP_RUNNER_E2E_EXECUTION_IDS: JSON.stringify(["everyday-workflows.runner-codex.daytona.recover-controller"]) };
+  wrapper = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(repositoryRoot, path.join(import.meta.dirname, "server.ts")),
+    { cwd: repositoryRoot, env: source, detached: process.platform !== "win32", stdio: ["ignore", "pipe", "pipe"] });
+  wrapper.once("spawn", () => { report.wrapperStarted = true; });
+  wrapper.stdout!.pipe(log, { end: false }); wrapper.stderr!.pipe(log, { end: false });
+  const stopped = new Promise<void>((resolve, reject) => { wrapper!.once("error", reject); wrapper!.once("close", () => resolve()); });
+  void stopped.catch(() => {});
+  await stopper.watch(wrapper);
+  async function poll<T>(load: () => Promise<T>, accept: (value: T) => boolean, timeoutMs: number): Promise<T> {
+    const deadline = Date.now() + timeoutMs;
+    while (Date.now() < deadline) {
+      if (wrapper!.exitCode !== null || wrapper!.signalCode !== null) throw new Error("Server supervisor exited; inspect private logs");
+      const value = await load();
+      if (accept(value)) return value;
+      await delay(200);
+    }
+    throw new Error("Main-server replacement smoke timed out; inspect private logs");
+  }
+  const base = `http://127.0.0.1:${port}`;
+  async function api(route: string, body?: object) {
+    const response = await fetch(`${base}/api${route}`, { method: body ? "POST" : "GET",
+      headers: { "content-type": "application/json", origin: base }, body: body ? JSON.stringify(body) : undefined,
+      signal: AbortSignal.timeout(5_000) });
+    if (!response.ok) throw new Error(`Smoke API ${route} returned ${response.status}`);
+    return response.json() as Promise<Record<string, unknown>>;
+  }
+  await poll(() => api("/health").catch(() => null), health => health?.status === "ok", 180_000);
+  const company = await api("/companies", { name: `Restart smoke ${randomUUID()}` });
+  const [before] = await db.$client<{ started: string }[]>`select pg_postmaster_start_time()::text as started`;
+  const receipts: unknown[] = [];
+  for (let index = 1; index < (upgrades?.sequence.length ?? 2); index += 1) {
+    // Empty fixture cache exercises retirement without pretending a runner exists.
+    const cache = path.join(home, "instances", "default", "runtime", "paperclip-runner");
+    await mkdir(cache, { recursive: true });
+    const requestId = `smoke-${index}-${randomUUID()}`;
+    await writeFile(`${control.restartRequestPath}.tmp`, JSON.stringify({ requestId, discardControllerCache: true, controllerOutage: outage }), { mode: 0o600 });
+    await rename(`${control.restartRequestPath}.tmp`, control.restartRequestPath);
+    const ack = await poll(async () => {
+      const value = await readFile(control.restartAcknowledgementPath, "utf8").then(text => JSON.parse(text) as Record<string, unknown>)
+        .catch(error => { if (error.code === "ENOENT") return null; throw error; });
+      if (value?.requestId === requestId && value.status === "failed") throw new Error(String(value.message));
+      return value;
+    }, value => value?.requestId === requestId && value.status === "ready", outage.holdMs + 180_000);
+    assert(validControllerOutageReceipt(ack!.controllerOutage, outage, requestId, upgrades ? {
+      previous: upgrades.artifacts[upgrades.sequence[index - 1]!],
+      replacement: upgrades.artifacts[upgrades.sequence[index]!],
+    } : undefined), "Invalid replacement fault or requested artifact receipt");
+    receipts.push(ack!.controllerOutage);
+    report.replacements = receipts;
+    await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, { mode: 0o600 });
+    assert.equal((await api(`/companies/${company.id}`)).id, company.id);
+    const after: { started: string } = (await db.$client<{ started: string }[]>`select pg_postmaster_start_time()::text as started`)[0]!;
+    assert.equal(after.started, before.started, "Postgres must remain alive across the web-server replacement");
+    await api("/companies", { name: `New admission after restart ${randomUUID()}` });
+  }
+  Object.assign(report, { postgresStayedAlive: true, existingDataReadable: true, newMutationAccepted: true, passed: true });
+  await stopper.stopAll();
+  await stopped;
+} catch (error) {
+  failure = error;
+  Object.assign(report, { passed: false, error: error instanceof Error ? error.message : String(error) });
+} finally {
+  const cleanupErrors: string[] = [];
+  const cleanups: Array<() => Promise<unknown>> = [];
+  if (wrapper) cleanups.push(() => stopper.stopAll());
+  if (db) { const client = db.$client; cleanups.push(() => client.end()); }
+  if (database) { const owned = database; cleanups.push(() => cleanupControllerSmokeDatabase(owned, ownedPostgresProcessId)); }
+  for (const cleanup of cleanups) {
+    try { await cleanup(); }
+    catch (error) { failure ??= error; cleanupErrors.push(error instanceof Error ? error.message : String(error)); }
+  }
+  report.cleanupConfirmed = cleanupErrors.length === 0;
+  if (cleanupErrors.length) { report.cleanupErrors = cleanupErrors; report.passed = false; }
+  await new Promise<void>(resolve => log.end(resolve));
+  await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, { mode: 0o600 });
+  console.info(JSON.stringify({ passed: report.passed, evidence: reportPath, cleanupConfirmed: report.cleanupConfirmed }));
+}
+if (failure) throw failure;

tests/runner-e2e/controller-smoke-cleanup.test.ts

@@ -0,0 +1,34 @@
+import { spawn } from "node:child_process";
+import { once } from "node:events";
+import { expect, it, vi } from "vitest";
+import { cleanupControllerSmokeDatabase } from "./controller-smoke-cleanup.js";
+
+it("does not certify cleanup while the owned database process is still alive", async () => {
+  const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" });
+  await once(child, "spawn");
+  const cleanup = vi.fn(async () => {});
+  try {
+    await expect(cleanupControllerSmokeDatabase({ cleanup }, child.pid!, 30)).rejects.toThrow("owned_postgres_exit_unconfirmed");
+    expect(cleanup).toHaveBeenCalledTimes(1);
+    expect(() => process.kill(child.pid!, 0)).not.toThrow();
+    const exited = once(child, "exit"); child.kill("SIGTERM"); await exited;
+    await expect(cleanupControllerSmokeDatabase({ cleanup }, child.pid!, 30)).resolves.toBeUndefined();
+  } finally {
+    if (child.exitCode === null && child.signalCode === null) {
+      const exited = once(child, "exit"); child.kill("SIGTERM"); await exited;
+    }
+  }
+});
+
+it("preserves cleanup failures and refuses an unknown embedded process identity", async () => {
+  await expect(cleanupControllerSmokeDatabase({ cleanup: async () => { throw new Error("drop_failed"); } }, null))
+    .rejects.toThrow("drop_failed");
+  await expect(cleanupControllerSmokeDatabase({ cleanup: async () => {} }, undefined))
+    .rejects.toThrow("owned_postgres_identity_unavailable");
+});
+
+it("accepts confirmed external database deletion without stopping its shared cluster", async () => {
+  const cleanup = vi.fn(async () => {});
+  await expect(cleanupControllerSmokeDatabase({ cleanup }, null)).resolves.toBeUndefined();
+  expect(cleanup).toHaveBeenCalledTimes(1);
+});

tests/runner-e2e/controller-smoke-cleanup.ts

@@ -0,0 +1,25 @@
+import { setTimeout as delay } from "node:timers/promises";
+
+export async function cleanupControllerSmokeDatabase(
+  database: { cleanup(): Promise<void> },
+  ownedProcessId: number | null | undefined,
+  timeoutMs = 10_000,
+): Promise<void> {
+  await database.cleanup();
+  // An external fixture owns a database, not its shared Postgres process. Its
+  // cleanup awaits DROP DATABASE. Embedded cleanup can return before exit.
+  if (ownedProcessId === null) return;
+  if (typeof ownedProcessId !== "number" || !Number.isSafeInteger(ownedProcessId) || ownedProcessId <= 0) {
+    throw new Error("owned_postgres_identity_unavailable");
+  }
+  const deadline = performance.now() + timeoutMs;
+  while (true) {
+    try { process.kill(ownedProcessId, 0); }
+    catch (error) {
+      if ((error as NodeJS.ErrnoException).code === "ESRCH") return;
+      throw error;
+    }
+    if (performance.now() >= deadline) throw new Error("owned_postgres_exit_unconfirmed");
+    await delay(Math.min(25, deadline - performance.now()));
+  }
+}

tests/runner-e2e/daytona-image-content.ts

@@ -63,11 +63,15 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [
   "packages/paperclip-runner/test/fixtures/copilot-profile-v52-identity.json",
   "packages/paperclip-runner/test/fixtures/copilot-profile-v54-identity.json",
   "packages/paperclip-runner/test/fixtures/copilot-profile-v55-identity.json",
+  "packages/paperclip-runner/test/fixtures/copilot-profile-v56-identity.json",
   "packages/paperclip-runner/test/fixtures/copilot-profile-v57-identity.json",
   "packages/paperclip-runner/test/fixtures/copilot-profile-v58-identity.json",
+  "packages/paperclip-runner/test/fixtures/copilot-profile-v59-identity.json",
   "packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json",
   "packages/paperclip-runner/test/fixtures/copilot-profile-v65-identity.json",
   "packages/paperclip-runner/test/fixtures/copilot-profile-v66-identity.json",
+  "packages/paperclip-runner/test/fixtures/copilot-profile-v67-identity.json",
+  "packages/paperclip-runner/test/fixtures/copilot-profile-v68-identity.json",
   "packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs",
   "packages/paperclip-runner/scripts/cursor-runtime-patch.mjs",
   "packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs",

tests/runner-e2e/everyday-flow.ts

@@ -21,6 +21,7 @@ import { waitForTaskChatRendered, waitForTaskIdentity } from "./continuation-scr
 import { hasPersistedSource, isSavedSourceCheckpoint } from "./everyday-interruption.js";
 import { collectRunEvents } from "./run-observations.js";
 import { gradeRemoteRestart, observeRemoteRestartBoundary, projectRemoteRestartEvents, projectRemoteRestartRun, waitForRemoteRestartBoundary, type RemoteRestartEvidence } from "./everyday-remote-restart.js";
+import { controllerOutageForExecution, type ControllerOutageReceipt } from "./controller-outage.js";
 import { setupAggregatorFixture } from "./aggregator-fixture.js";
 import { gradeProviderChoice, gradeProviderOutcome, requireProviderAccessCard } from "./connection-routing-evidence.js";
 import { setupConnectionReview } from "./connection-reviews.js";
@@ -89,7 +90,8 @@ interface Input {
   workspacePath: string;
   privateDir: string;
   deadlineAt: number;
-  restart(controllerDowntimeMs?: number): Promise<void>;
+  restartCount?: number;
+  restart(binding?: import("./controller-recovery-observer.js").RecoveryBinding, controllerDowntimeMs?: number): Promise<ControllerOutageReceipt | void>;
   observe(issue: StoryIssue, runs: StoryRun[]): void;
   capture(id: string, label: string, file: string): Promise<void>;
   evidence(name: string, value: unknown): Promise<void>;
@@ -168,9 +170,7 @@ export async function runEverydayFlow(input: Input) {
   let review: Awaited<ReturnType<typeof setupConnectionReview>> | undefined;
   let project = fixtures.project;
   const caseId = execution.task.id;
-  const requireRemoteRestart = caseId === "recover-controller" && execution.environment.id === "daytona"
-    && execution.profile.generation === "native" && (execution.profile.provider === "codex"
-      || execution.profile.provider === "acpx");
+  const requireRemoteRestart = controllerOutageForExecution(execution, process.env) !== undefined;
   const providerChoice = caseId === "provider-decline" || caseId === "provider-second";
   const nativeProviderCase = caseId === "provider-native";
   let aggregatorFixture: Awaited<ReturnType<typeof setupAggregatorFixture>> | undefined;
@@ -550,9 +550,9 @@ export async function runEverydayFlow(input: Input) {
   async function sourceReady() {
     await recordSource("source-saved-before-interruption", "source-before-interruption.json", true);
   }
-  async function remoteRestartBoundary(runId?: string) {
+  async function remoteRestartBoundary(runId?: string, recovered = false) {
     await refresh();
-    const active = ev.runs.filter(run => run.status === "running" && run.nativeIssueId === parent!.id
+    const active = ev.runs.filter(run => (run.status === "running" || (recovered && run.status === "succeeded")) && run.nativeIssueId === parent!.id
       && (!runId || run.id === runId));
     if (active.length !== 1) throw new Error("Remote restart boundary unexercised: the original run is not active");
     const run = active[0]!;
@@ -561,7 +561,7 @@ export async function runEverydayFlow(input: Input) {
       api.get<Row[]>(`/api/environments/${fixtures.environment.id}/leases`),
     ]);
     return observeRemoteRestartBoundary(run, rows, leases, { companyId: fixtures.company.id,
-      agentId: fixtures.agent.id, issueId: parent!.id, environmentId: fixtures.environment.id });
+      agentId: fixtures.agent.id, issueId: parent!.id, environmentId: fixtures.environment.id }, recovered);
   }
   async function prepareStopBoundary() {
     // Providers can finish a short first turn before the browser can click
@@ -596,6 +596,11 @@ export async function runEverydayFlow(input: Input) {
       "everyday-delivery.ts",
       "everyday-observations.ts",
       "everyday-remote-restart.ts",
+      "controller-outage.ts",
+      "controller-recovery-observer.ts",
+      "upgrade-artifacts.ts",
+      "server.ts",
+      "server-stop.ts",
       "native-event-envelope.ts",
       "run-observations.ts",
       "everyday-artifact.py",
@@ -937,27 +942,34 @@ export async function runEverydayFlow(input: Input) {
       } else {
         await reply(LATE_REQUIREMENT);
         note("followup-submitted-before-interruption");
-        if (requireRemoteRestart) {
-          const before = await waitForRemoteRestartBoundary({
-            deadlineAt: input.deadlineAt, load: async () => {
-              await refresh();
-              const original = ev.runs.find(run => run.id === active.id);
-              if (!original || original.status !== "running") return { run: original };
-              const rows = await collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${active.id}/events?afterSeq=${afterSeq}&limit=${limit}`));
-              if (!rows.some(row => row.payload?.prpEvent?.eventType === "tool.execution.completed" && row.payload.prpEvent.payload?.status === "completed")) return { run: original };
-              return { run: original, boundary: await remoteRestartBoundary(active.id) };
-            } });
-          ev.remoteRestart = { before };
-          await input.evidence("remote-restart-before.json", before);
-        } else ev.allowedInterruptedRuns.push(active.id);
-        await input.restart(requireRemoteRestart ? 125_000 : undefined);
-        note("controller-restarted");
-        await openParent();
-        if (requireRemoteRestart) {
-          const after = await remoteRestartBoundary(active.id);
-          ev.remoteRestart!.after = after;
-          await input.evidence("remote-restart-after.json", after);
-          if (!gradeRemoteRestart(ev.remoteRestart)[0]!.passed) throw new Error("Remote restart changed the original native process, lease or receipt prefix");
+        for (let restartIndex = 0; restartIndex < (requireRemoteRestart ? input.restartCount ?? 1 : 1); restartIndex += 1) {
+          let currentBefore: RemoteRestartEvidence["before"];
+          if (requireRemoteRestart) {
+            const before = await waitForRemoteRestartBoundary({
+              deadlineAt: input.deadlineAt, load: async () => {
+                await refresh();
+                const original = ev.runs.find(run => run.id === active.id);
+                if (!original || original.status !== "running") return { run: original };
+                const rows = await collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${active.id}/events?afterSeq=${afterSeq}&limit=${limit}`));
+                if (!rows.some(row => row.payload?.prpEvent?.eventType === "tool.execution.completed" && row.payload.prpEvent.payload?.status === "completed")) return { run: original };
+                return { run: original, boundary: await remoteRestartBoundary(active.id) };
+              } });
+            currentBefore = before;
+            ev.remoteRestart ??= { before, expectedReplacements: input.restartCount ?? 1, replacements: [] };
+            await input.evidence(`remote-restart-${restartIndex + 1}-before.json`, before);
+          } else ev.allowedInterruptedRuns.push(active.id);
+          const outage = await input.restart(requireRemoteRestart ? { companyId: fixtures.company.id, issueId: parent!.id, runId: active.id } : undefined, requireRemoteRestart ? 125_000 : undefined);
+          if (requireRemoteRestart) ev.remoteRestart!.outage = outage || undefined;
+          note("controller-restarted");
+          await openParent();
+          if (requireRemoteRestart) {
+            const after = await remoteRestartBoundary(active.id, true);
+            ev.remoteRestart!.after = after;
+            if (!outage || !currentBefore) throw new Error("Missing replacement fault evidence");
+            ev.remoteRestart!.replacements!.push({ before: currentBefore, after, outage });
+            await input.evidence(`remote-restart-${restartIndex + 1}-after.json`, { after, outage });
+            if (!gradeRemoteRestart(ev.remoteRestart)[0]!.passed) throw new Error("Remote restart changed the original native process, lease or receipt prefix");
+          }
         }
       }
     }
@@ -1636,7 +1648,7 @@ export async function runEverydayFlow(input: Input) {
             api.get<StoryIssue>(`/api/issues/${parent!.id}`),
           ]);
           if (!Array.isArray(operationRows) || !Array.isArray(recovery.actions) || !("active" in recovery)) throw new Error("Remote restart finalization evidence is incomplete");
-          const final = { run: projectRemoteRestartRun(run), events: projectRemoteRestartEvents(run, rows),
+          const final = { observedAt: new Date().toISOString(), run: projectRemoteRestartRun(run), events: projectRemoteRestartEvents(run, rows),
             operations: operationRows.map(operation => ({ companyId: operation.companyId, heartbeatRunId: operation.heartbeatRunId,
               issueId: operation.issueId, executionWorkspaceId: operation.executionWorkspaceId, phase: operation.phase, status: operation.status })),
             scheduledRetry: issue.scheduledRetry,

tests/runner-e2e/everyday-remote-restart.test.ts

@@ -1,4 +1,5 @@
 import { afterEach, describe, expect, it, vi } from "vitest";
+import type { ControllerOutageReceipt } from "./controller-outage.js";
 import { storyLifecycleChecks, type StoryRun } from "./everyday-observations.js";
 import { gradeRemoteRestart, observeRemoteRestartBoundary, projectRemoteRestartEvents, projectRemoteRestartRun, waitForRemoteRestartBoundary } from "./everyday-remote-restart.js";
 
@@ -31,10 +32,20 @@ function fixture() {
   } };
   const finalRows = [...afterRows, event(6, "run.result.proposed"), event(7, "turn.completed"),
     event(8, "run.result.accepted", {}, "control_plane", 1), event(9, "run.terminal", {}, "control_plane", 2)];
-  const final = { run: finished, events: projectRemoteRestartEvents(finished, finalRows),
+  const final = { observedAt: "2026-10-22T00:02:00.000Z", run: finished, events: projectRemoteRestartEvents(finished, finalRows),
     operations: [{ companyId: "company", heartbeatRunId: "run", issueId: "task", executionWorkspaceId: "workspace",
       phase: "workspace_finalize", status: "succeeded" }], scheduledRetry: null as unknown, recoveryActions: [] as Record<string, unknown>[] };
-  return { active, scope, leases, event, rows, afterRows, finalRows, evidence: { before, after, final } };
+  const outage: ControllerOutageReceipt = { schema: "paperclip.e2e.controller-outage/v1", requestId: "story-test",
+    signal: "SIGKILL", holdMs: 1000, previousPid: 100, replacementPid: 200,
+    startedAt: "2026-10-22T00:00:00.000Z", stoppedAfterMs: 100, cacheDiscarded: true,
+    replacementStartedAfterMs: 1100, readyAfterMs: 4000,
+    recovery: { binding: { companyId: "company", issueId: "task", runId: "run" }, runnerInstanceId: "runner", nativeSessionId: "session",
+      previousGeneration: 1, generation: 2, previousBootId: "old", bootId: "new", previousLeaseExpiresAt: "2026-10-22T00:01:00.000Z",
+      ownershipAcquiredAt: "2026-10-22T00:01:00.001Z", ownershipObservedAfterMs: 60_010,
+      authenticatedAttachmentObservedAfterMs: 61_000, resumedProcessingObservedAfterMs: 61_000,
+      runnerSourceSeqBefore: 4, runnerSourceSeqAfter: 5, previousAuthenticatedConnectionCount: 1,
+      authenticatedConnectionCount: 2, attachmentEvidence: "runner_checkpoint_authenticated_connection" } };
+  return { active, scope, leases, event, rows, afterRows, finalRows, evidence: { before, after, final, outage } };
 }
 
 describe("remote restart boundary polling", () => {
@@ -66,6 +77,64 @@ describe("remote restart boundary polling", () => {
 });
 
 describe("remote restart lifecycle evidence", () => {
+  it("accepts a plugin task with no controller PID and permits completion during the outage", () => {
+    const f = fixture();
+    const task = { runnerId: "runner", leaseId: "lease", runId: "run", sessionId: "session", turnId: "turn", itemId: "item" };
+    const lease = { ...f.leases[0]!, provider: "plugin", metadata: { runnerTaskDispatched: true, runnerTaskIdentity: task } };
+    const run = { ...f.active, processPid: null, processStartedAt: null };
+    const before = observeRemoteRestartBoundary(run, f.rows, [lease], f.scope);
+    const completedLease = { ...lease, status: "released", metadata: {
+      ...lease.metadata, runnerTaskDispatched: false, runnerTaskExecutionStopped: true,
+    } };
+    const after = observeRemoteRestartBoundary({ ...run, status: "succeeded" }, f.finalRows, [completedLease], f.scope, true);
+    expect(before.identity).toEqual(after.identity);
+    expect(before.identity.processPid).toBeNull();
+    expect(before.identity.taskIdentitySha256).toMatch(/^[a-f0-9]{64}$/);
+    expect(() => observeRemoteRestartBoundary({ ...run, processPid: 123 }, f.rows, [lease], f.scope)).toThrow("not the controller PID");
+    for (const runnerTaskExecutionStopped of [undefined, false]) {
+      const unconfirmed = { ...completedLease, metadata: { ...completedLease.metadata, runnerTaskExecutionStopped } };
+      expect(() => observeRemoteRestartBoundary({ ...run, status: "succeeded" }, f.finalRows, [unconfirmed], f.scope, true))
+        .toThrow("admitted plugin task identity");
+    }
+    const stoppedActiveLease = { ...lease, metadata: completedLease.metadata };
+    expect(() => observeRemoteRestartBoundary(run, f.rows, [stoppedActiveLease], f.scope, true))
+      .toThrow("admitted plugin task identity");
+    const foreignTask = { ...completedLease, metadata: { ...completedLease.metadata, runnerTaskIdentity: { ...task, runId: "replacement" } } };
+    expect(() => observeRemoteRestartBoundary({ ...run, status: "succeeded" }, f.finalRows, [foreignTask], f.scope, true))
+      .toThrow("admitted plugin task identity");
+  });
+  it("does not confuse unchanged metadata or HTTP readiness with authenticated recovery", () => {
+    const { evidence } = fixture();
+    delete evidence.outage.recovery;
+    expect(gradeRemoteRestart(evidence).find(check => check.id === "remote-restart-authenticated-recovery")?.passed).toBe(false);
+  });
+  it("requires every requested immutable replacement instead of counting one restart as rollback", () => {
+    const { evidence } = fixture();
+    const A = { revision: "a".repeat(40), treeSha256: "a".repeat(64), version: "1.0.0", evidence: "development-build" as const };
+    const B = { ...A, revision: "b".repeat(40), treeSha256: "b".repeat(64) };
+    const first = { before: evidence.before, after: evidence.after, outage: { ...evidence.outage, previousArtifact: A, replacementArtifact: B } };
+    const second = { before: evidence.after, after: evidence.after, outage: { ...evidence.outage, previousArtifact: B, replacementArtifact: A } };
+    const grade = (replacements: typeof first[]) => gradeRemoteRestart({ ...evidence, expectedReplacements: 2, replacements })
+      .find(check => check.id === "remote-restart-all-replacements")!.passed;
+    expect(grade([first])).toBe(false);
+    expect(grade([first, second])).toBe(true);
+    for (const field of ["runnerInstanceId", "nativeSessionId"] as const) {
+      expect(grade([{ ...first, outage: { ...first.outage, recovery: { ...first.outage.recovery!, [field]: "another" } } }, second])).toBe(false);
+    }
+    for (const field of ["companyId", "issueId"] as const) {
+      expect(grade([{ ...first, outage: { ...first.outage, recovery: { ...first.outage.recovery!,
+        binding: { ...first.outage.recovery!.binding, [field]: "another" } } } }, second])).toBe(false);
+    }
+    expect(grade([first, { ...second, outage: { ...second.outage, recovery: undefined } }])).toBe(false);
+    expect(grade([first, { ...second, outage: { ...second.outage, previousArtifact: A } }])).toBe(false);
+  });
+  it("rejects otherwise successful recovery without a complete actual-server fault receipt", () => {
+    const { evidence } = fixture();
+    for (const outage of [undefined, { ...evidence.outage, replacementPid: 100 },
+      { ...evidence.outage, replacementStartedAfterMs: 1099 }, { ...evidence.outage, cacheDiscarded: false }]) {
+      expect(gradeRemoteRestart({ ...evidence, outage }).every(check => check.passed)).toBe(false);
+    }
+  });
   it("does not accept a successful replacement as proof that the original remote runner recovered", () => {
     const { evidence } = fixture();
     const replacement = { ...evidence.final.run, id: "replacement", runnerInstanceId: "replacement-runner" };

tests/runner-e2e/everyday-remote-restart.ts

@@ -3,6 +3,8 @@ import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
 import { isValidNativePrpEnvelope } from "./native-event-envelope.js";
 import type { StoryCheck, StoryRun } from "./everyday-observations.js";
 import { pollUntil } from "./api.js";
+import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js";
+import { validControllerRecoveryReceipt } from "./controller-recovery-observer.js";
 
 type Row = Record<string, unknown>;
 const record = (value: unknown): Row => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Row : {};
@@ -17,15 +19,19 @@ export interface RemoteRestartEvent {
 }
 export interface RemoteRestartBoundary {
   identity: { runId: string; companyId: string; agentId: string; issueId: string;
-    runnerInstanceId: string; nativeSessionId: string; processPid: number; processStartedAt: string };
+    runnerInstanceId: string; nativeSessionId: string; processPid: number | null; processStartedAt: string | null;
+    taskIdentitySha256?: string };
   lease: { id: string; environmentId: string; executionWorkspaceId: string | null; providerLeaseSha256: string };
   events: RemoteRestartEvent[];
 }
 export interface RemoteRestartEvidence {
+  expectedReplacements?: number;
+  replacements?: Array<{ before: RemoteRestartBoundary; after: RemoteRestartBoundary; outage: ControllerOutageReceipt }>;
+  outage?: ControllerOutageReceipt;
   before?: RemoteRestartBoundary;
   after?: RemoteRestartBoundary;
   final?: { run: StoryRun; events: RemoteRestartEvent[]; operations: Row[];
-    scheduledRetry: unknown; recoveryActions: Row[] };
+    scheduledRetry: unknown; recoveryActions: Row[]; observedAt?: string };
 }
 
 export async function waitForRemoteRestartBoundary(input: {
@@ -87,30 +93,49 @@ export function projectRemoteRestartEvents(run: StoryRun, rows: readonly Row[]):
 
 export function observeRemoteRestartBoundary(run: StoryRun, rows: readonly Row[], leases: readonly Row[], scope: {
   companyId: string; agentId: string; issueId: string; environmentId: string;
-}): RemoteRestartBoundary {
+}, recovered = false): RemoteRestartBoundary {
   if (run.companyId !== scope.companyId || run.agentId !== scope.agentId || run.nativeIssueId !== scope.issueId
-    || run.status !== "running" || run.runtimeMode !== "native" || !id(run.id) || !id(run.runnerInstanceId)
-    || !id(run.nativeSessionId) || !Number.isSafeInteger(run.processPid) || Number(run.processPid) <= 0
-    || !id(run.processStartedAt) || !Number.isFinite(Date.parse(run.processStartedAt))) {
+    || !(run.status === "running" || (recovered && run.status === "succeeded"))
+    || run.runtimeMode !== "native" || !id(run.id) || !id(run.runnerInstanceId) || !id(run.nativeSessionId)) {
     throw new Error("Remote restart boundary requires the original live native process");
   }
   const owned = leases.filter(lease => lease.companyId === scope.companyId && lease.environmentId === scope.environmentId
-    && lease.issueId === scope.issueId && lease.heartbeatRunId === run.id && lease.provider === "daytona"
-    && lease.status === "active" && lease.releasedAt == null && id(lease.id) && id(lease.providerLeaseId));
+    && lease.issueId === scope.issueId && lease.heartbeatRunId === run.id
+    && (lease.provider === "daytona" || lease.provider === "plugin" || String(lease.provider).startsWith("plugin:"))
+    && ((lease.status === "active" && lease.releasedAt == null) || (recovered && run.status === "succeeded"))
+    && id(lease.id) && id(lease.providerLeaseId));
   if (owned.length !== 1) throw new Error("Remote restart boundary requires one exact active sandbox lease");
   const lease = owned[0]!;
+  const metadata = record(lease.metadata), task = record(metadata.runnerTaskIdentity);
+  const plugin = lease.provider === "plugin" || String(lease.provider).startsWith("plugin:");
+  if (plugin) {
+    // The normal task authority clears dispatch only after provider status confirms
+    // execution stopped. Completion during the outage must retain that proof.
+    const settled = recovered && run.status === "succeeded"
+      && metadata.runnerTaskDispatched === false && metadata.runnerTaskExecutionStopped === true;
+    if (run.processPid != null || run.processStartedAt != null || (!settled && metadata.runnerTaskDispatched !== true)
+      || task.runnerId !== run.runnerInstanceId || task.runId !== run.id || task.sessionId !== run.nativeSessionId
+      || !id(task.leaseId) || !id(task.turnId) || !id(task.itemId)) {
+      throw new Error("Remote restart requires the admitted plugin task identity, not the controller PID");
+    }
+  } else if (!Number.isSafeInteger(run.processPid) || Number(run.processPid) <= 0
+    || !id(run.processStartedAt) || !Number.isFinite(Date.parse(run.processStartedAt))) {
+    throw new Error("Remote restart boundary requires the original live native process");
+  }
   const events = projectRemoteRestartEvents(run, rows);
   const starts = events.filter(event => event.eventType === "turn.started" && event.sourceKind === "runner");
   const completed = events.filter(event => event.toolCompleted && event.sourceKind === "runner" && event.toolExecutionId);
   if (starts.length !== 1 || !id(starts[0]!.turnId) || completed.length === 0
     || completed.some(event => event.turnId !== starts[0]!.turnId)
-    || events.some(event => ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted", "run.terminal"].includes(event.eventType))) {
+    || (!recovered && events.some(event => ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted", "run.terminal"].includes(event.eventType)))) {
     throw new Error("Remote restart boundary has no completed tool in an active original provider turn");
   }
   return {
     identity: { runId: run.id, companyId: scope.companyId, agentId: scope.agentId, issueId: scope.issueId,
       runnerInstanceId: run.runnerInstanceId, nativeSessionId: run.nativeSessionId,
-      processPid: Number(run.processPid), processStartedAt: run.processStartedAt },
+      processPid: plugin ? null : Number(run.processPid), processStartedAt: plugin ? null : run.processStartedAt!,
+      ...(plugin ? { taskIdentitySha256: digest({ runnerId: task.runnerId, leaseId: task.leaseId,
+        runId: task.runId, sessionId: task.sessionId, turnId: task.turnId, itemId: task.itemId }) } : {}) },
     lease: { id: String(lease.id), environmentId: scope.environmentId,
       executionWorkspaceId: typeof lease.executionWorkspaceId === "string" ? lease.executionWorkspaceId : null,
       providerLeaseSha256: digest(lease.providerLeaseId) },
@@ -161,9 +186,40 @@ export function gradeRemoteRestart(evidence: RemoteRestartEvidence | undefined):
       && operation.executionWorkspaceId === before?.lease.executionWorkspaceId)
     && !operations.some(operation => ["pending", "queued", "running"].includes(String(operation.status)))
     && final?.scheduledRetry == null && !final?.recoveryActions.some(action => ["active", "pending"].includes(String(action.status)));
+  const expectedReplacements = evidence?.expectedReplacements ?? 1;
+  const replacements = evidence?.replacements;
+  const allReplacements = replacements === undefined ? expectedReplacements === 1
+    : replacements.length === expectedReplacements && replacements.every((step, index, all) => {
+      const previous = all[index - 1];
+      const preservesIdentity = digest(step.before.identity) === digest(step.after.identity)
+        && digest(step.before.lease) === digest(step.after.lease)
+        && retainsPrefix(step.before.events, step.after.events);
+      const authenticated = validControllerOutageReceipt(step.outage) && validControllerRecoveryReceipt(step.outage)
+        && step.outage.recovery?.binding.runId === step.before.identity.runId
+        && step.outage.recovery.binding.companyId === step.before.identity.companyId
+        && step.outage.recovery.binding.issueId === step.before.identity.issueId
+        && step.outage.recovery.runnerInstanceId === step.before.identity.runnerInstanceId
+        && step.outage.recovery.nativeSessionId === step.before.identity.nativeSessionId;
+      const artifactsPresent = expectedReplacements <= 1 || Boolean(step.outage.previousArtifact && step.outage.replacementArtifact);
+      const chained = !previous || (retainsPrefix(previous.after.events, step.before.events)
+        && (!step.outage.previousArtifact || Boolean(previous.outage.replacementArtifact
+          && digest(previous.outage.replacementArtifact) === digest(step.outage.previousArtifact))));
+      return preservesIdentity && authenticated && artifactsPresent && chained;
+    });
   return [
-    check("remote-restart-same-process-and-lease", sameIdentity, "The same run, native session, Runner process and sandbox lease remain live after restart."),
+    check("remote-restart-same-execution-and-lease", sameIdentity, "The original run, native session, admitted runner identity and sandbox lease survive replacement."),
     check("remote-restart-native-completion-once", nativeCompletion, "The original provider turn completes once and retains its native receipt prefix."),
     check("remote-restart-files-finalized", Boolean(saved), "The original run commits its workspace receipt and leaves no active operation or recovery."),
+    check("remote-restart-fault-exercised", validControllerOutageReceipt(evidence?.outage), "The main server exited, its cache was retired, and a distinct replacement became ready after the requested outage."),
+    check("remote-restart-authenticated-recovery", validControllerRecoveryReceipt(evidence?.outage)
+      && evidence?.outage?.recovery?.binding.runId === identity?.runId
+      && evidence?.outage?.recovery?.runnerInstanceId === identity?.runnerInstanceId
+      && evidence?.outage?.recovery?.nativeSessionId === identity?.nativeSessionId,
+    "Natural ownership acquisition and new authenticated runner receipts were observed within the original 420-second grace."),
+    check("remote-restart-backlog-settled", Boolean(saved && nativeCompletion && final?.observedAt
+      && Date.parse(final.observedAt) >= Date.parse(evidence?.outage?.startedAt ?? "") + (evidence?.outage?.recovery?.resumedProcessingObservedAfterMs ?? Infinity)),
+    "Canonical terminal and workspace receipts settle the original backlog after reattachment."),
+    check("remote-restart-all-replacements", allReplacements,
+      "Every requested upgrade and rollback exercised the original running execution and retained its receipts."),
   ];
 }

tests/runner-e2e/harness-env.ts

@@ -135,6 +135,10 @@ export function buildRunnerE2EProcessEnvironment(
   const installed = usesInstalledCli(source);
   if (installed) assertInstalledCliSelection(source, executions);
   const result = { ...source };
+  // The supervisor's read-only recovery observer must use the same disposable
+  // instance key as the serving child, never a launching shell's vault key.
+  delete result.PAPERCLIP_SECRETS_MASTER_KEY;
+  delete result.PAPERCLIP_SECRETS_MASTER_KEY_FILE;
   delete result.PAPERCLIP_E2E_GATEWAY_PHASE_PATH;
   delete result.PAPERCLIP_E2E_OTLP_RELAY_URL;
   delete result.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN;
@@ -211,7 +215,7 @@ export function buildPaperclipServerEnvironment(
   ]) {
     delete result[key];
   }
-  for (const key of [...GENERATED_SERVER_SECRET_KEYS, ...installedCliKeys, ...installedDaytonaPluginKeys]) delete result[key];
+  for (const key of [...GENERATED_SERVER_SECRET_KEYS, ...installedCliKeys, ...installedDaytonaPluginKeys, "PAPERCLIP_RUNNER_E2E_UPGRADE_ARTIFACTS"]) delete result[key];
   Object.assign(result, overrides);
   return result;
 }

tests/runner-e2e/installed-cli.test.ts

@@ -32,7 +32,7 @@ it("admits all26 current Pi cases through pinned installed public CLI/server wit
 });
 it("admits existing Codex onboarding, useful-file revision and restart journeys without broadening other cells", async () => {
   const { env, server } = await fixture();
-  for (const id of ["first-task.runner-codex.local.task-card-accept", "everyday-workflows.runner-codex.local.build-revise", "agent-chat.runner-codex.local.continuity-restart"]) {
+  for (const id of ["first-task.runner-codex.local.task-card-accept", "everyday-workflows.runner-codex.local.build-revise", "agent-chat.runner-codex.local.continuity-restart", "everyday-workflows.runner-codex.daytona.recover-controller"]) {
     const cell = runnerMatrix.find(e => e.id === id)!;
     expect(cell).toBeDefined();
     expect(await verifyInstalledCli(env, [cell])).toMatchObject({ serverRoot: server, defaultRuntimeResolution: true, qualificationOverride: false });

tests/runner-e2e/installed-cli.ts

@@ -16,6 +16,7 @@ const installedCodexCells = new Set([
   "first-task.runner-codex.local.task-card-accept",
   "everyday-workflows.runner-codex.local.build-revise",
   "agent-chat.runner-codex.local.continuity-restart",
+  "everyday-workflows.runner-codex.daytona.recover-controller",
 ]);
 const installedCopilotCells = new Set([
   "copilot-protection.runner-acpx-copilot.daytona.attached-async-settlement",
@@ -46,7 +47,7 @@ export function assertInstalledCliSelection(env: NodeJS.ProcessEnv, executions:
   if (!usesInstalledCli(env)) return;
   if (!installedCliKeys.every(key => Boolean(env[key])) || executions.length === 0 || executions.some(e => !(
     (e.profile.id === "runner-acpx-pi" && e.profile.qualificationCandidate === "pi" && e.suite.manualOnly && suites.has(e.suite.id)) ||
-    (e.profile.id === "runner-codex" && e.profile.provider === "codex" && e.environment.id === "local" && installedCodexCells.has(e.id)) ||
+    (e.profile.id === "runner-codex" && e.profile.provider === "codex" && installedCodexCells.has(e.id)) ||
     (e.profile.id === "runner-acpx-copilot" && e.profile.qualificationCandidate === "copilot" && e.suite.manualOnly && installedCopilotCells.has(e.id))
   ))) throw new Error("Installed CLI proof requires complete pins and explicit supported Pi, Codex, or Copilot cases");
   for (const key of overrideKeys) if (env[key] !== undefined) throw new Error(`Installed CLI proof forbids ${key}`);

tests/runner-e2e/instance-isolation.test.ts

@@ -0,0 +1,40 @@
+import path from "node:path";
+import { mkdir, readFile, rm, symlink, writeFile } from "node:fs/promises";
+import { expect, it } from "vitest";
+import { assertEmbeddedDatabaseIsolation } from "./instance-isolation.js";
+import { createRunnerE2ETemporaryRoot, prepareRunnerE2EServerConfig } from "./server-config.js";
+
+it.each(["owned", "missing-receipt", "foreign-connection", "wrong-port", "foreign-owner", "escaped-database"])(
+  "checks the supervisor-owned database after the serving process exits: %s", async variant => {
+    const root = await createRunnerE2ETemporaryRoot();
+    const configPath = path.join(root, "instance", "config.json");
+    const outside = await createRunnerE2ETemporaryRoot();
+    try {
+      await (await prepareRunnerE2EServerConfig({ temporaryRoot: root, configPath, serverPort: 3100 }))!.close();
+      const config = JSON.parse(await readFile(configPath, "utf8"));
+      const directory = config.database.embeddedPostgresDataDir;
+      await mkdir(directory, { recursive: true });
+      await writeFile(path.join(directory, "PG_VERSION"), "18\n");
+      await mkdir(path.dirname(config.secrets.localEncrypted.keyFilePath), { recursive: true });
+      await writeFile(config.secrets.localEncrypted.keyFilePath, "fixture-key");
+      config.database.mode = "postgres";
+      config.database.connectionString = `postgres://paperclip:paperclip@127.0.0.1:${config.database.embeddedPostgresPort}/paperclip`;
+      const receipt = { version: 1, configPath, temporaryRoot: root, dataDirectory: directory,
+        connectionString: config.database.connectionString };
+      if (variant === "foreign-owner") receipt.temporaryRoot = outside;
+      if (variant !== "missing-receipt") await writeFile(path.join(directory, ".runner-e2e-controller-owner.json"), JSON.stringify(receipt));
+      if (variant === "foreign-connection") config.database.connectionString = "postgres://paperclip:paperclip@db.example.invalid:5432/paperclip";
+      if (variant === "wrong-port") config.database.embeddedPostgresPort += 1;
+      if (variant === "escaped-database") {
+        await rm(directory, { recursive: true });
+        await symlink(outside, directory, "dir");
+      }
+      await writeFile(configPath, JSON.stringify(config));
+      if (variant === "owned") await expect(assertEmbeddedDatabaseIsolation(configPath, root)).resolves.toBeUndefined();
+      else await expect(assertEmbeddedDatabaseIsolation(configPath, root)).rejects.toThrow(/fixture-owned|escaped|embedded database/);
+    } finally {
+      await rm(root, { recursive: true, force: true });
+      await rm(outside, { recursive: true, force: true });
+    }
+  },
+);

tests/runner-e2e/instance-isolation.ts

@@ -1,5 +1,22 @@
 import path from "node:path";
-import { readFile, stat } from "node:fs/promises";
+import { readFile, realpath, stat, writeFile } from "node:fs/promises";
+
+const CONTROLLER_DATABASE_OWNER_FILE = ".runner-e2e-controller-owner.json";
+
+/** Written only after the supervisor has started this fixture's Postgres. */
+export async function recordControllerDatabaseOwnership(input: {
+  configPath: string; temporaryRoot: string; dataDirectory: string; connectionString: string;
+}) {
+  const temporaryRoot = await realpath(input.temporaryRoot);
+  const dataDirectory = await realpath(input.dataDirectory);
+  if (!dataDirectory.startsWith(`${temporaryRoot}${path.sep}`)) {
+    throw new Error("Controller database path escaped its fixture-owned root");
+  }
+  await writeFile(path.join(dataDirectory, CONTROLLER_DATABASE_OWNER_FILE), JSON.stringify({
+    version: 1, configPath: await realpath(input.configPath), temporaryRoot,
+    dataDirectory, connectionString: input.connectionString,
+  }), { mode: 0o600 });
+}
 
 export async function assertEmbeddedDatabaseIsolation(
   configPath: string,
@@ -11,6 +28,7 @@ export async function assertEmbeddedDatabaseIsolation(
       mode?: string;
       connectionString?: string;
       embeddedPostgresDataDir?: string;
+      embeddedPostgresPort?: number;
       backup?: { dir?: string };
     };
     logging?: { logDir?: string };
@@ -24,13 +42,10 @@ export async function assertEmbeddedDatabaseIsolation(
       localEncrypted?: { keyFilePath?: string };
     };
   };
-  if (
-    config.database?.mode !== "embedded-postgres" ||
-    config.database.connectionString
-  ) {
-    throw new Error(
-      "Runner E2E Paperclip instance did not use its embedded database",
-    );
+  const supervised = config.database?.mode === "postgres";
+  if (!config.database || (!supervised &&
+    (config.database.mode !== "embedded-postgres" || config.database.connectionString))) {
+    throw new Error("Runner E2E Paperclip instance did not use its fixture-owned database");
   }
   if (
     config.storage?.provider !== "local_disk" ||
@@ -60,6 +75,25 @@ export async function assertEmbeddedDatabaseIsolation(
     }
   }
   const databasePath = path.resolve(isolatedPaths.database!);
+  if (supervised) {
+    try {
+      const root = await realpath(temporaryRoot);
+      const directory = await realpath(databasePath);
+      if (!directory.startsWith(`${root}${path.sep}`)) throw new Error("Database escaped fixture");
+      const owner = JSON.parse(await readFile(path.join(directory, CONTROLLER_DATABASE_OWNER_FILE), "utf8"));
+      const port = config.database.embeddedPostgresPort;
+      const expectedConnection = `postgres://paperclip:paperclip@127.0.0.1:${port}/paperclip`;
+      if (!Number.isInteger(port) || Number(port) < 1 || Number(port) > 65535
+        || config.database.connectionString !== expectedConnection
+        || owner.version !== 1 || owner.temporaryRoot !== root || owner.dataDirectory !== directory
+        || owner.configPath !== await realpath(configPath) || owner.connectionString !== expectedConnection
+        || !(await stat(path.join(directory, "PG_VERSION"))).isFile()) {
+        throw new Error("Supervisor database ownership mismatch");
+      }
+    } catch (cause) {
+      throw new Error("Runner E2E Paperclip instance did not use its fixture-owned database", { cause });
+    }
+  }
   const databaseStat = await stat(databasePath);
   if (!databaseStat.isDirectory())
     throw new Error("Embedded database path is not a directory");

tests/runner-e2e/launch.ts

@@ -1,3 +1,4 @@
+import { readUpgradeArtifacts, verifyUpgradeArtifact, upgradeArtifactKey } from "./upgrade-artifacts.js";
 import { recoverWorkerTiming } from "./worker-timing.js";
 import { openEvalOperations, type EvalOperations } from "./operations.js";
 import { readRecoveredFeedback } from "./feedback-recovery.js";
@@ -32,6 +33,7 @@ import {
 import { isImmutableDaytonaImage, runnerMatrix } from "./catalog.js";
 import { renderRunnerE2EDashboard } from "./dashboard.js";
 import { packageEvidence } from "./evidence.js";
+import { copySharedEvidence } from "./shared-evidence.js";
 import { classifyFailure } from "./failure-classifier.js";
 import { mustPreserveRecoveryState, shouldKeepFailedDiagnostics } from "./cleanup-verification.js";
 import { effectiveAutomaticRetryLimit, executeWithAutomaticRetry } from "./automatic-retry.js";
@@ -451,24 +453,6 @@ async function readResult(resultPath: string, fallback: RunnerE2EResult) {
   }
 }
 
-async function copySharedEvidence(privateRoot: string, casePrivateDir: string) {
-  for (const relative of [
-    "server.log",
-    "playwright.log",
-    "junit.xml",
-    "html-report",
-    "blob-report",
-    "playwright-output",
-  ]) {
-    await cp(
-      path.join(privateRoot, relative),
-      path.join(casePrivateDir, relative),
-      { recursive: true, force: true },
-    ).catch((error: NodeJS.ErrnoException) => {
-      if (error.code !== "ENOENT") throw error;
-    });
-  }
-}
 
 async function runAttempt(input: {
   executions: readonly MatrixExecution[];
@@ -494,6 +478,15 @@ async function runAttempt(input: {
       "One isolated runner E2E harness cannot mix profiles or environments",
     );
   }
+  const upgrades = await readUpgradeArtifacts(process.env[upgradeArtifactKey]);
+  if (upgrades) {
+    // All attempts and the browser use A's installed fixture admission. The
+    // supervisor changes only the server artifact; the original sandbox lives on.
+    Object.assign(process.env, upgrades.artifacts.A.pins);
+    for (const artifact of [upgrades.artifacts.A, upgrades.artifacts.B]) {
+      await verifyUpgradeArtifact(artifact, process.env, executions);
+    }
+  }
   assertInstalledCliSelection(process.env, executions);
   const installedCli = await verifyInstalledCli(process.env, executions);
   const installedPlugin = await verifyInstalledDaytonaPlugin(process.env, executions);
@@ -816,7 +809,7 @@ async function runAttempt(input: {
         `attempt-${attempt}`,
       );
       await mkdir(casePrivateDir, { recursive: true });
-      await copySharedEvidence(privateDir, casePrivateDir);
+      await copySharedEvidence(privateDir, casePrivateDir, temporaryRoot, credentials);
       if (persistedStateError && persistenceCheckedExecutions.includes(candidate)) {
         const snapshots = path.join(casePrivateDir, "snapshots");
         await mkdir(snapshots, { recursive: true });

tests/runner-e2e/prepare-report-lock.test.ts

@@ -14,10 +14,22 @@ it.each(['valid', 'unknown-lock', 'changed-patch', 'wrong-output'])(
     try {
       await mkdir(relative, { recursive: true });
       for (const file of ['report-lock.json', 'report-lock.patch']) await copyFile(path.join(here, file), path.join(relative, file));
-      const input = execFileSync('git', ['show', 'HEAD:pnpm-lock.yaml'], { cwd: path.resolve(here, '../..') });
+      const committed = execFileSync('git', ['show', 'HEAD:pnpm-lock.yaml'], { cwd: path.resolve(here, '../..') });
       const policy = JSON.parse(await readFile(path.join(relative, 'report-lock.json'), 'utf8'));
-      expect(createHash('sha256').update(input).digest('hex')).toBe(policy.inputSha256);
+      const committedHash = createHash('sha256').update(committed).digest('hex');
+      expect([policy.inputSha256, policy.outputSha256]).toContain(committedHash);
       const lock = path.join(root, 'pnpm-lock.yaml');
+      await writeFile(lock, committed);
+      if (variant === 'valid') {
+        expect(await prepareReportLock(root)).toBe(committedHash === policy.outputSha256 ? 'already-reviewed' : 'reviewed-correction');
+        expect(createHash('sha256').update(await readFile(lock)).digest('hex')).toBe(policy.outputSha256);
+        await writeFile(lock, committed);
+      }
+      if (committedHash === policy.outputSha256) {
+        execFileSync('git', ['apply', '--reverse', path.join(relative, 'report-lock.patch')], { cwd: root });
+      }
+      const input = await readFile(lock);
+      expect(createHash('sha256').update(input).digest('hex')).toBe(policy.inputSha256);
       await writeFile(lock, variant === 'unknown-lock' ? 'unknown\n' : input);
       if (variant === 'changed-patch') await writeFile(path.join(relative, 'report-lock.patch'), 'changed');
       if (variant === 'wrong-output') { policy.outputSha256 = '0'.repeat(64); await writeFile(path.join(relative, 'report-lock.json'), JSON.stringify(policy)); }

tests/runner-e2e/runner.spec.ts

@@ -2,6 +2,7 @@ import { createWorkerTiming } from "./worker-timing.js";
 import { FEEDBACK_PERSISTENCE_SUITE, gradeFeedbackPersistence } from "./feedback-persistence.js";
 import { EvalAttemptTrace, evalBraintrustConfig, EVAL_OBSERVABILITY_SECRET_NAMES } from "./braintrust.js";
 import { captureFollowupStopFailureApiState } from "./followup-stop-evidence.js";
+import { controllerOutageForExecution, validControllerOutageReceipt, type ControllerOutage, type ControllerOutageReceipt } from "./controller-outage.js";
 import { runFollowupStopResumeFlow } from "./followup-stop-resume-flow.js";
 import { runFinalizationRecoveryFlow } from "./finalization-recovery-flow.js";
 import type { FinalizationCrashBinding } from "./finalization-crash-boundary.js";
@@ -263,8 +264,12 @@ async function restartIsolatedPaperclipServer(input: {
   requestId: string;
   deadlineAt: number;
   preserveRunner?: RestartRunnerIdentity;
+  discardControllerCache?: boolean;
   crashAfterResult?: FinalizationCrashBinding;
   controllerDowntimeMs?: number;
+  controllerOutage?: ControllerOutage;
+  recoveryBinding?: import("./controller-recovery-observer.js").RecoveryBinding;
+  onControllerOutage?: (receipt: ControllerOutageReceipt) => Promise<void>;
 }): Promise<void> {
   const {
     controlDirectory,
@@ -275,14 +280,16 @@ async function restartIsolatedPaperclipServer(input: {
   const temporaryRequestPath = `${requestPath}.${process.pid}.${input.requestId}.tmp`;
   await writeFile(
     temporaryRequestPath,
-    JSON.stringify({ requestId: input.requestId, ...(input.controllerDowntimeMs !== undefined ? { controllerDowntimeMs: input.controllerDowntimeMs } : {}),
+    JSON.stringify({ requestId: input.requestId, ...(input.controllerDowntimeMs !== undefined && !input.controllerOutage ? { controllerDowntimeMs: input.controllerDowntimeMs } : {}), discardControllerCache: input.discardControllerCache === true,
+      ...(input.controllerOutage ? { controllerOutage: input.controllerOutage } : {}),
+      ...(input.recoveryBinding ? { recoveryBinding: input.recoveryBinding } : {}),
       ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}),
       ...(input.crashAfterResult ? { crashAfterResult: input.crashAfterResult } : {}) }),
     { encoding: "utf8", mode: 0o600 },
   );
   await rename(temporaryRequestPath, requestPath);
 
-  await pollUntil({
+  const acknowledgement = await pollUntil({
     label: `isolated server restart ${input.requestId}`,
     timeoutFailureClass: "transient_infrastructure",
     deadlineAt: input.deadlineAt,
@@ -298,7 +305,9 @@ async function restartIsolatedPaperclipServer(input: {
     },
     accept: (acknowledgement) =>
       acknowledgement.requestId === input.requestId &&
-      acknowledgement.status === "ready",
+      acknowledgement.status === "ready" &&
+      (!input.discardControllerCache || acknowledgement.controllerCacheDiscarded === true) &&
+      (!input.controllerOutage || validControllerOutageReceipt(acknowledgement.controllerOutage, input.controllerOutage, input.requestId)),
     reject: (acknowledgement) =>
       acknowledgement.requestId === input.requestId &&
       acknowledgement.status === "failed"
@@ -313,6 +322,7 @@ async function restartIsolatedPaperclipServer(input: {
     load: () => input.api.get<Record<string, unknown>>("/api/health"),
     accept: (health) => Boolean(health),
   });
+  if (input.controllerOutage) await input.onControllerOutage?.(acknowledgement.controllerOutage as ControllerOutageReceipt);
 }
 
 const executionIds = (() => {
@@ -580,6 +590,7 @@ async function expectPlanStageVisible(
 }
 
 for (const execution of executions) {
+  const controllerOutage = controllerOutageForExecution(execution, process.env);
   const privateDir = path.join(privateRoot, "cases", execution.task.id);
   const resultPath = path.join(privateDir, "result.json");
   const snapshotsDir = path.join(privateDir, "snapshots");
@@ -1239,10 +1250,25 @@ for (const execution of executions) {
         });
         issue = continuation.issue as IssueRecord; selectedRuns = continuation.runs as RunRecord[];
       } else if (execution.task.flow === "everyday_workflow") {
+        const { readUpgradeArtifacts, upgradeArtifactKey } = await import("./upgrade-artifacts.js");
+        const upgradeManifest = controllerOutage ? await readUpgradeArtifacts(process.env[upgradeArtifactKey]) : null;
+        let restartIndex = 0;
         const story = await runEverydayFlow({
           page, api, fixtures, execution, nonce, workspacePath, privateDir,
           deadlineAt: startedAtMs + deadlineMs,
-          restart: controllerDowntimeMs => restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}`, deadlineAt: startedAtMs + deadlineMs, controllerDowntimeMs }),
+          restartCount: upgradeManifest ? upgradeManifest.sequence.length - 1 : 1,
+          restart: async (recoveryBinding, controllerDowntimeMs) => {
+            let receipt: ControllerOutageReceipt | undefined;
+            restartIndex += 1;
+            await restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}-${restartIndex}`, deadlineAt: startedAtMs + deadlineMs,
+              discardControllerCache: controllerOutage !== undefined, controllerOutage, controllerDowntimeMs,
+              ...(controllerOutage ? { recoveryBinding } : {}),
+              onControllerOutage: async value => {
+                receipt = value;
+                await writeSanitizedJson(snapshotsDir, `controller-outage-${restartIndex}.json`, value, secrets);
+              } });
+            return receipt;
+          },
           observe: (storyIssue, storyRuns) => { issue = storyIssue; selectedRuns = storyRuns; },
           capture: captureScreenshot,
           evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),

tests/runner-e2e/server-restart-request.test.ts

@@ -0,0 +1,60 @@
+import { mkdtemp, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { afterEach, beforeEach, expect, it } from "vitest";
+import { runnerMatrix } from "./catalog.js";
+import { controllerOutageForExecution } from "./controller-outage.js";
+import { readRestartRequest } from "./server-restart-request.js";
+
+let root: string;
+let requestPath: string;
+beforeEach(async () => {
+  root = await mkdtemp(join(tmpdir(), "paperclip-supervisor-request-"));
+  requestPath = join(root, "restart.json");
+});
+afterEach(async () => { await rm(root, { recursive: true, force: true }); });
+
+const cells = ["everyday-workflows", "daytona-restart-continuity"].flatMap(suite =>
+  ["codex", "acpx"].map(provider => {
+    const execution = runnerMatrix.find(row => row.suite.id === suite && row.task.id === "recover-controller"
+      && row.environment.id === "daytona" && row.profile.generation === "native" && row.profile.provider === provider);
+    if (!execution) throw new Error(`Missing real ${suite}/${provider} continuity cell`);
+    return execution;
+  }));
+
+it.each(cells)("accepts the real supervisor request file for $id", async execution => {
+  const outage = controllerOutageForExecution(execution, {})!;
+  await writeFile(requestPath, JSON.stringify({ requestId: "actual-restart-request",
+    discardControllerCache: true, controllerOutage: outage }));
+  await expect(readRestartRequest(requestPath, [execution.id])).resolves.toMatchObject({
+    requestId: "actual-restart-request", discardControllerCache: true, controllerOutage: outage,
+  });
+});
+
+it("rejects cache replacement for unselected, local and legacy cells", async () => {
+  await writeFile(requestPath, JSON.stringify({ requestId: "unsupported-restart", discardControllerCache: true }));
+  const local = runnerMatrix.find(row => row.environment.id === "local")!;
+  const legacy = runnerMatrix.find(row => row.profile.generation === "legacy")!;
+  expect(local).toBeDefined(); expect(legacy).toBeDefined();
+  for (const ids of [[], ["not-a-cell"], [local.id], [legacy.id]]) {
+    await expect(readRestartRequest(requestPath, ids)).rejects.toThrow(/cache replacement requires/i);
+  }
+});
+
+it.each([
+  { controllerOutage: { signal: "SIGKILL", holdMs: 125_000 }, discardControllerCache: false },
+  { controllerOutage: { signal: "SIGKILL", holdMs: 300_001 }, discardControllerCache: true },
+  { controllerOutage: { signal: "SIGKILL", holdMs: 125_000 }, discardControllerCache: true, preserveRunner: {} },
+  { controllerOutage: { signal: "SIGKILL", holdMs: 125_000 }, discardControllerCache: true, crashAfterResult: {} },
+  { recoveryBinding: {} },
+  { controllerDowntimeMs: -1 },
+])("keeps invalid and conflicting fault boundaries rejected: %j", async fault => {
+  await writeFile(requestPath, JSON.stringify({ requestId: "conflicting-restart", ...fault }));
+  await expect(readRestartRequest(requestPath, [cells[0].id])).rejects.toThrow();
+});
+
+it("ignores absent and incomplete atomic request files", async () => {
+  await expect(readRestartRequest(requestPath, [cells[0].id])).resolves.toBeNull();
+  await writeFile(requestPath, "{");
+  await expect(readRestartRequest(requestPath, [cells[0].id])).resolves.toBeNull();
+});

tests/runner-e2e/server-restart-request.ts

@@ -0,0 +1,68 @@
+import { readFile } from "node:fs/promises";
+import { runnerMatrix } from "./catalog.js";
+import { controllerOutageForExecution, parseControllerOutage, type ControllerOutage } from "./controller-outage.js";
+import { parseRecoveryBinding, type RecoveryBinding } from "./controller-recovery-observer.js";
+import { parseFinalizationCrashBinding, type FinalizationCrashBinding } from "./finalization-crash-boundary.js";
+import { parseRestartRunnerIdentity, type RestartRunnerIdentity } from "./process-tree-owner.js";
+
+export interface RestartRequest {
+  requestId: string;
+  controllerDowntimeMs?: number;
+  preserveRunner?: RestartRunnerIdentity;
+  discardControllerCache?: boolean;
+  crashAfterResult?: FinalizationCrashBinding;
+  controllerOutage?: ControllerOutage;
+  recoveryBinding?: RecoveryBinding;
+}
+
+export async function readRestartRequest(restartRequestPath: string, executionIds: readonly string[]): Promise<RestartRequest | null> {
+  let encoded: string;
+  try {
+    encoded = await readFile(restartRequestPath, "utf8");
+  } catch (error) {
+    if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
+    throw error;
+  }
+  let value: unknown;
+  try {
+    value = JSON.parse(encoded);
+  } catch {
+    // The writer may not have completed its atomic replacement yet.
+    return null;
+  }
+  if (!value || typeof value !== "object" || Array.isArray(value)) return null;
+  const requestId = (value as { requestId?: unknown }).requestId;
+  if (
+    typeof requestId !== "string" ||
+    !/^[A-Za-z0-9._:-]{1,200}$/.test(requestId)
+  ) {
+    return null;
+  }
+  const preserveRunner = (value as { preserveRunner?: unknown }).preserveRunner;
+  const controllerDowntimeMs = (value as { controllerDowntimeMs?: unknown }).controllerDowntimeMs;
+  if (controllerDowntimeMs !== undefined && (!Number.isSafeInteger(controllerDowntimeMs)
+    || Number(controllerDowntimeMs) < 0 || Number(controllerDowntimeMs) > 300_000)) {
+    throw new Error("Controller downtime must be an integer between zero and five minutes");
+  }
+  const discardControllerCache = (value as { discardControllerCache?: unknown }).discardControllerCache === true;
+  if (discardControllerCache && !executionIds.some(id => {
+    const execution = runnerMatrix.find(row => row.id === id);
+    return execution !== undefined && controllerOutageForExecution(execution, {}) !== undefined;
+  })) throw new Error("Controller cache replacement requires an explicit remote controller-outage cell");
+  const crashAfterResult = (value as { crashAfterResult?: unknown }).crashAfterResult;
+  const controllerOutage = (value as { controllerOutage?: unknown }).controllerOutage;
+  const recoveryBinding = (value as { recoveryBinding?: unknown }).recoveryBinding;
+  if (recoveryBinding !== undefined && controllerOutage === undefined) throw new Error("Recovery observation requires a controller outage");
+  if (controllerOutage !== undefined && (!discardControllerCache || preserveRunner !== undefined || crashAfterResult !== undefined)) {
+    throw new Error("Controller outage requires the explicit remote cache-replacement cell without another crash boundary");
+  }
+  if (crashAfterResult !== undefined && (preserveRunner !== undefined || discardControllerCache)) {
+    throw new Error("A finalization crash cannot preserve a running provider or discard its controller cache");
+  }
+  return { requestId, discardControllerCache, ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }),
+    ...(recoveryBinding === undefined ? {} : { recoveryBinding: parseRecoveryBinding(recoveryBinding) }),
+    ...(controllerOutage === undefined ? {} : { controllerOutage: parseControllerOutage(controllerOutage) }),
+    ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }),
+    ...(crashAfterResult === undefined ? {} : { crashAfterResult: parseFinalizationCrashBinding(crashAfterResult) }) };
+
+}

tests/runner-e2e/server.ts

@@ -1,5 +1,9 @@
-import { parseFinalizationCrashBinding, readFinalizationCrashBoundary, type FinalizationCrashBinding } from "./finalization-crash-boundary.js";
-import { parseRestartRunnerIdentity, type RestartRunnerIdentity } from "./process-tree-owner.js";
+import { readRestartRequest, type RestartRequest } from "./server-restart-request.js";
+import { prepareControllerDatabase } from "./controller-database.js";
+import { readUpgradeArtifacts, verifyUpgradeArtifact, assertServingArtifact, upgradeArtifactKey, type ServingArtifact } from "./upgrade-artifacts.js";
+import { createControllerRecoveryObserver } from "./controller-recovery-observer.js";
+import { readFinalizationCrashBoundary } from "./finalization-crash-boundary.js";
+import { replaceController, retireControllerCache, type ControllerOutageReceipt } from "./controller-outage.js";
 import { recoverInterruptedFeedback } from "./feedback-recovery.js";
 import { normalizedSecrets } from "./redaction.js";
 import { CREDENTIAL_NAMES } from "./types.js";
@@ -36,7 +40,9 @@ const port = required("PAPERCLIP_RUNNER_E2E_PORT");
 const repositoryRoot = path.resolve(import.meta.dirname, "../..");
 const paperclipCli = path.join(repositoryRoot, "tests/runner-e2e/server-entry.ts");
 const executionIds: string[] = JSON.parse(process.env.PAPERCLIP_RUNNER_E2E_EXECUTION_IDS ?? "[]");
-const selectedExecutions = usesInstalledCli(process.env) ? executionIds.map(id => {
+const upgradeArtifacts = await readUpgradeArtifacts(process.env[upgradeArtifactKey]);
+let artifactIndex = 0;
+const selectedExecutions = usesInstalledCli(process.env) || upgradeArtifacts ? executionIds.map(id => {
   const execution = runnerMatrix.find(row => row.id === id);
   if (!execution) throw new Error("Unknown installed CLI proof execution");
   return execution;
@@ -48,8 +54,9 @@ const {
 } = runnerE2EServerControlPaths(temporaryRoot);
 const restartTimeoutMs = 180_000;
 const gracefulStopTimeoutMs = 30_000;
-const installedRelease = process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI
-  ? installedReleaseLaunch(process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI) : null;
+const initialCli = upgradeArtifacts?.artifacts.A.pins.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI ?? process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI;
+const installedRelease = initialCli ? installedReleaseLaunch(initialCli) : null;
+const servingArtifacts = new WeakMap<ChildProcess, ServingArtifact>();
 const serverEnvironment = buildPaperclipServerEnvironment(installedRelease
   ? installedReleaseEnvironment(process.env, repositoryRoot, path.join(temporaryRoot, "provider-bin"))
   : process.env, {
@@ -104,7 +111,7 @@ const definedServerEnvironment = Object.fromEntries(
   ),
 );
 
-await prepareSourceServerDependencies(repositoryRoot, definedServerEnvironment, usesInstalledCli(process.env));
+await prepareSourceServerDependencies(repositoryRoot, definedServerEnvironment, Boolean(installedRelease));
 
 await Promise.all([
   mkdir(path.dirname(logPath), { recursive: true }),
@@ -122,6 +129,7 @@ let unexpectedChildFailure: Error | null = null;
 let shutdownSignal: NodeJS.Signals | null = null;
 let activeRestartRequestId: string | null = null;
 let feedbackRecovery: Promise<void> | undefined;
+let controllerDatabase: Awaited<ReturnType<typeof prepareControllerDatabase>> | undefined;
 
 function recoverFeedbackBeforeShutdown() {
   return feedbackRecovery ??= (async () => {
@@ -183,13 +191,24 @@ async function startServer() {
     throw new Error("Refusing to start Paperclip after wrapper shutdown");
   }
   // Recheck bytes and dependency resolution on every controller restart.
-  const installed = await verifyInstalledCli(process.env, selectedExecutions);
+  const selected = upgradeArtifacts?.sequence[artifactIndex];
+  if (upgradeArtifacts && !selected) throw new Error("Upgrade artifact sequence exhausted");
+  const verified = selected ? await verifyUpgradeArtifact(upgradeArtifacts!.artifacts[selected], process.env, selectedExecutions) : undefined;
+  const installed = verified?.installed ?? await verifyInstalledCli(process.env, selectedExecutions);
+  const launch = installed ? installedReleaseLaunch(installed.entry) : null;
+  // The serving process must report its baked-in revision, never a test-supplied SHA.
+  const environment = { ...definedServerEnvironment };
+  if (upgradeArtifacts) {
+    delete environment.PAPERCLIP_BUILD_COMMIT;
+    delete environment[upgradeArtifactKey];
+    environment.GIT_CEILING_DIRECTORIES = upgradeArtifacts.artifacts[selected!].root;
+  }
   const candidate = spawn(
     process.execPath,
     installed ? [installed.entry, "onboard", "--yes", "--run"] : runnerE2ETypeScriptProcessArgs(repositoryRoot, paperclipCli, ["onboard", "--yes", "--run"]),
     {
-      cwd: installedRelease?.cwd ?? repositoryRoot,
-      env: definedServerEnvironment,
+      cwd: launch?.cwd ?? repositoryRoot,
+      env: environment,
       stdio: ["ignore", "pipe", "pipe"],
       // Playwright signals the wrapper's group. Keep that signal from bypassing
       // our single graceful stop; the launcher still tracks descendant groups.
@@ -197,6 +216,7 @@ async function startServer() {
     },
   );
   child = candidate;
+  if (verified) servingArtifacts.set(candidate, verified.receipt);
 
   candidate.stdout?.on("data", (chunk) => {
     process.stdout.write(chunk);
@@ -261,8 +281,13 @@ async function waitForHealth(candidate: ChildProcess) {
       const response = await fetch(healthUrl, {
         signal: AbortSignal.timeout(1_000),
       });
-      if (response.ok) return;
-    } catch {
+      if (response.ok) {
+        const artifact = servingArtifacts.get(candidate);
+        if (artifact) assertServingArtifact(await response.json(), artifact);
+        return;
+      }
+    } catch (error) {
+      if (error instanceof Error && error.message.includes("serving server revision")) throw error;
       // The replacement process may still be booting.
     }
     await delay(250);
@@ -291,54 +316,14 @@ async function waitForHealthToStop() {
   );
 }
 
-interface RestartRequest {
-  requestId: string;
-  controllerDowntimeMs?: number;
-  preserveRunner?: RestartRunnerIdentity;
-  crashAfterResult?: FinalizationCrashBinding;
-}
-
-async function readRestartRequest(): Promise<RestartRequest | null> {
-  let encoded: string;
-  try {
-    encoded = await readFile(restartRequestPath, "utf8");
-  } catch (error) {
-    if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
-    throw error;
-  }
-  let value: unknown;
-  try {
-    value = JSON.parse(encoded);
-  } catch {
-    // The writer may not have completed its atomic replacement yet.
-    return null;
-  }
-  if (!value || typeof value !== "object" || Array.isArray(value)) return null;
-  const requestId = (value as { requestId?: unknown }).requestId;
-  if (
-    typeof requestId !== "string" ||
-    !/^[A-Za-z0-9._:-]{1,200}$/.test(requestId)
-  ) {
-    return null;
-  }
-  const preserveRunner = (value as { preserveRunner?: unknown }).preserveRunner;
-  const controllerDowntimeMs = (value as { controllerDowntimeMs?: unknown }).controllerDowntimeMs;
-  if (controllerDowntimeMs !== undefined && (!Number.isSafeInteger(controllerDowntimeMs)
-    || Number(controllerDowntimeMs) < 0 || Number(controllerDowntimeMs) > 300_000)) {
-    throw new Error("Controller downtime must be an integer between zero and five minutes");
-  }
-  const crashAfterResult = (value as { crashAfterResult?: unknown }).crashAfterResult;
-  if (crashAfterResult !== undefined && preserveRunner !== undefined) throw new Error("A finalization crash cannot preserve a running provider");
-  return { requestId, ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }),
-    ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }),
-    ...(crashAfterResult === undefined ? {} : { crashAfterResult: parseFinalizationCrashBinding(crashAfterResult) }) };
-}
 
 async function writeRestartAck(
   requestId: string,
   status: "ready" | "failed",
   message?: string,
+  controllerCacheDiscarded = false,
   crashBoundary?: Awaited<ReturnType<typeof readFinalizationCrashBoundary>>,
+  controllerOutage?: ControllerOutageReceipt,
 ) {
   const temporaryAckPath = `${restartAckPath}.${process.pid}.tmp`;
   await writeFile(
@@ -346,20 +331,55 @@ async function writeRestartAck(
     `${JSON.stringify({
       requestId,
       status,
+      controllerCacheDiscarded,
       completedAt: new Date().toISOString(),
       ...(message ? { message } : {}),
       ...(crashBoundary ? { crashBoundary } : {}),
+      ...(controllerOutage ? { controllerOutage } : {}),
     })}\n`,
     { encoding: "utf8", mode: 0o600 },
   );
   await rename(temporaryAckPath, restartAckPath);
 }
 
-async function restartServer({ requestId, preserveRunner, crashAfterResult, controllerDowntimeMs = 0 }: RestartRequest) {
+async function restartServer({ requestId, preserveRunner, discardControllerCache, crashAfterResult, controllerOutage, recoveryBinding, controllerDowntimeMs = 0 }: RestartRequest) {
   activeRestartRequestId = requestId;
   appendLog(`\nRestart request ${requestId}: stopping Paperclip\n`);
   const previous = child;
   if (!previous) throw new Error("No Paperclip server is available to restart");
+  const discardCache = async () => {
+    const cache = path.join(paperclipHome, "instances", required("PAPERCLIP_INSTANCE_ID"), "runtime", "paperclip-runner");
+    await retireControllerCache(cache, path.join(controlDirectory, `retired-controller-cache-${requestId}`));
+    appendLog(`Restart request ${requestId}: old controller cache made unavailable\n`);
+  };
+  if (controllerOutage) {
+    if (upgradeArtifacts) artifactIndex += 1;
+    const previousArtifact = servingArtifacts.get(previous);
+    const observer = recoveryBinding ? await createControllerRecoveryObserver(configPath, recoveryBinding) : undefined;
+    try {
+      const receipt = await replaceController({ requestId, previousPid: previous.pid!, outage: controllerOutage,
+        stop: async signal => {
+          if (signal === "SIGKILL") await stopServer.crashForRestart(previous);
+          else await stopServer(previous);
+          if (child === previous) child = null;
+        },
+        waitForUnhealthy: waitForHealthToStop, discardCache, start: startServer, waitForReady: waitForHealth,
+        ...(previousArtifact ? { previousArtifact, replacementArtifact: () => child && servingArtifacts.get(child) || undefined } : {}),
+        cancelled: shutdownRequested,
+        afterStopped: observer?.stopped,
+        afterReady: observer ? (receipt, record) => observer.recovered(receipt, record, shutdownRequested) : undefined,
+        record: async receipt => {
+          const target = path.join(controlDirectory, `controller-outage-${requestId}.json`);
+          await writeFile(`${target}.tmp`, `${JSON.stringify(receipt)}\n`, { mode: 0o600 });
+          await rename(`${target}.tmp`, target);
+        },
+      });
+      await writeRestartAck(requestId, "ready", undefined, true, undefined, receipt);
+      appendLog(`Restart request ${requestId}: Paperclip is healthy after ${Math.round(receipt.readyAfterMs!)}ms\n`);
+      activeRestartRequestId = null;
+      return;
+    } finally { await observer?.close(); }
+  }
   const crashBoundary = crashAfterResult ? await readFinalizationCrashBoundary(async route => {
     const response = await fetch(`http://127.0.0.1:${port}${route}`, { signal: AbortSignal.timeout(5_000), cache: "no-store" });
     if (!response.ok) throw new Error("Finalization crash boundary could not be read");
@@ -377,6 +397,9 @@ async function restartServer({ requestId, preserveRunner, crashAfterResult, cont
     if (shutdownRequested()) throw new Error("Wrapper shutdown interrupted controller downtime");
     await delay(Math.min(250, downtimeDeadline - Date.now()));
   }
+  if (discardControllerCache) {
+    await discardCache();
+  }
   if (shutdownRequested()) {
     throw new Error("Wrapper shutdown interrupted the Paperclip restart");
   }
@@ -387,7 +410,7 @@ async function restartServer({ requestId, preserveRunner, crashAfterResult, cont
   if (shutdownRequested()) {
     throw new Error("Wrapper shutdown interrupted the Paperclip restart");
   }
-  await writeRestartAck(requestId, "ready", undefined, crashBoundary);
+  await writeRestartAck(requestId, "ready", undefined, discardControllerCache, crashBoundary);
   appendLog(`Restart request ${requestId}: Paperclip is healthy\n`);
   activeRestartRequestId = null;
 }
@@ -413,11 +436,17 @@ async function supervise() {
   });
   // Postgres needs the socket itself; release immediately before child spawn.
   await databaseReservation?.close();
-  await startServer();
+  if (upgradeArtifacts || executionIds.some(id => id.endsWith(".recover-controller"))) {
+    // The wrapper owns this sibling process. Killing the serving child must
+    // leave Postgres available to the stopped-controller observer and successor.
+    controllerDatabase = await prepareControllerDatabase(configPath, temporaryRoot);
+  }
+  const initial = await startServer();
+  if (upgradeArtifacts) await waitForHealth(initial);
   let lastRestartRequestId: string | null = null;
   while (!shutdownRequested()) {
     if (unexpectedChildFailure) throw unexpectedChildFailure;
-    const request = await readRestartRequest();
+    const request = await readRestartRequest(restartRequestPath, executionIds);
     if (request && request.requestId !== lastRestartRequestId) {
       lastRestartRequestId = request.requestId;
       await restartServer(request);
@@ -453,5 +482,10 @@ try {
   }
 }
 
+try { await controllerDatabase?.cleanup(); }
+catch (error) {
+  exitCode = 1;
+  appendLog(`Qualification database cleanup failed: ${error instanceof Error ? error.message : String(error)}\n`);
+}
 await new Promise<void>((resolve) => log.end(resolve));
 process.exitCode = exitCode;

tests/runner-e2e/shared-evidence.test.ts

@@ -0,0 +1,96 @@
+import { mkdtemp, mkdir, readFile, readdir, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { expect, it } from "vitest";
+import { replaceController, validControllerOutageReceipt } from "./controller-outage.js";
+import { packageEvidence } from "./evidence.js";
+import { runnerE2EServerControlPaths } from "./harness-env.js";
+import { copySharedEvidence } from "./shared-evidence.js";
+
+it.each(["start", "readiness", "recovery"])("publishes the last outage phase after %s fails and temporary state is deleted", async phase => {
+  const root = await mkdtemp(path.join(tmpdir(), "outage-evidence-"));
+  const temporaryRoot = path.join(root, "temporary");
+  const privateRoot = path.join(temporaryRoot, "private");
+  const caseDir = path.join(privateRoot, "cases", "recover-controller");
+  const uploadDir = path.join(root, "published");
+  const { controlDirectory } = runnerE2EServerControlPaths(temporaryRoot);
+  const secret = "synthetic-evidence-credential";
+  try {
+    await mkdir(controlDirectory, { recursive: true });
+    await mkdir(caseDir, { recursive: true });
+    await writeFile(path.join(privateRoot, "server.log"), "server stopped\n");
+    await writeFile(path.join(controlDirectory, "restart-request.json"), JSON.stringify({ token: secret }));
+    await expect(replaceController({
+      requestId: "attempt-1", previousPid: 100, outage: { signal: "SIGKILL", holdMs: 0 },
+      stop: async () => {}, waitForUnhealthy: async () => {}, discardCache: async () => {},
+      start: async () => { if (phase === "start") throw new Error(phase); return { pid: 200 }; },
+      waitForReady: async () => { if (phase === "readiness") throw new Error(phase); },
+      afterReady: async (receipt, record) => {
+        receipt.recoveryProgress = { bootId: secret };
+        await record();
+        throw new Error(phase);
+      },
+      cancelled: () => false,
+      record: async receipt => {
+        await writeFile(path.join(controlDirectory, "controller-outage-attempt-1.json"), JSON.stringify(receipt));
+      },
+    })).rejects.toThrow(phase);
+    await copySharedEvidence(privateRoot, caseDir, temporaryRoot, [secret]);
+    const evidence = await packageEvidence({ privateDir: caseDir, uploadDir, secrets: [secret], expectPassScreenshot: false });
+    await rm(temporaryRoot, { recursive: true, force: true });
+    expect(evidence.leaks).toEqual([]);
+    const relative = "snapshots/controller-outages/controller-outage-attempt-1.json";
+    expect(evidence.files).toContain(relative);
+    const text = await readFile(path.join(uploadDir, relative), "utf8");
+    const receipt = JSON.parse(text);
+    expect(text).not.toContain(secret);
+    expect(receipt).toMatchObject({ requestId: "attempt-1", cacheDiscarded: true });
+    if (phase === "start") expect(receipt).not.toHaveProperty("replacementPid");
+    else expect(receipt.replacementPid).toBe(200);
+    if (phase === "recovery") expect(receipt.recoveryProgress).toEqual({ bootId: "[REDACTED]" });
+    else {
+      expect(receipt).not.toHaveProperty("readyAfterMs");
+      expect(validControllerOutageReceipt(receipt)).toBe(false);
+    }
+    expect(await readdir(uploadDir)).not.toContain("control");
+    expect(evidence.files.some(file => file.includes("restart-request"))).toBe(false);
+    expect(await readFile(path.join(uploadDir, "server.log"), "utf8")).toBe("server stopped\n");
+  } finally { await rm(root, { recursive: true, force: true }); }
+});
+
+it("keeps ordinary evidence collection working when no outage was requested", async () => {
+  const root = await mkdtemp(path.join(tmpdir(), "ordinary-evidence-"));
+  try {
+    await writeFile(path.join(root, "server.log"), "ordinary run\n");
+    const caseDir = path.join(root, "case");
+    await mkdir(caseDir);
+    await copySharedEvidence(root, caseDir, root, []);
+    expect(await readdir(caseDir)).toEqual(["server.log"]);
+  } finally { await rm(root, { recursive: true, force: true }); }
+});
+
+it("retains sanitized corrupt receipt evidence without losing healthy receipts or logs", async () => {
+  const root = await mkdtemp(path.join(tmpdir(), "corrupt-outage-evidence-"));
+  const temporaryRoot = path.join(root, "temporary"), privateRoot = path.join(temporaryRoot, "private");
+  const caseDir = path.join(privateRoot, "case"), uploadDir = path.join(root, "published");
+  const { controlDirectory } = runnerE2EServerControlPaths(temporaryRoot);
+  const secret = "synthetic-corrupt-receipt-secret";
+  try {
+    await mkdir(controlDirectory, { recursive: true }); await mkdir(caseDir, { recursive: true });
+    await writeFile(path.join(privateRoot, "server.log"), "original failure retained\n");
+    await writeFile(path.join(controlDirectory, "controller-outage-a-broken.json"), `{"note":"${secret}"`);
+    await writeFile(path.join(controlDirectory, "controller-outage-z-good.json"), JSON.stringify({ requestId: "good", stoppedAfterMs: 10 }));
+    await copySharedEvidence(privateRoot, caseDir, temporaryRoot, [secret]);
+    const result = await packageEvidence({ privateDir: caseDir, uploadDir, secrets: [secret], expectPassScreenshot: false });
+    await rm(temporaryRoot, { recursive: true, force: true });
+    expect(result.leaks).toEqual([]);
+    const directory = path.join(uploadDir, "snapshots/controller-outages");
+    const broken = JSON.parse(await readFile(path.join(directory, "controller-outage-a-broken.json"), "utf8"));
+    expect(broken).toMatchObject({ error: "invalid_json", source: "controller-outage-a-broken.json" });
+    expect(validControllerOutageReceipt(broken)).toBe(false);
+    const raw = await readFile(path.join(directory, "controller-outage-a-broken.json.invalid.txt"), "utf8");
+    expect(raw).not.toContain(secret); expect(raw).toContain("[REDACTED]");
+    expect(JSON.parse(await readFile(path.join(directory, "controller-outage-z-good.json"), "utf8"))).toMatchObject({ requestId: "good" });
+    expect(await readFile(path.join(uploadDir, "server.log"), "utf8")).toBe("original failure retained\n");
+  } finally { await rm(root, { recursive: true, force: true }); }
+});

tests/runner-e2e/shared-evidence.ts

@@ -0,0 +1,47 @@
+import { cp, mkdir, readFile, readdir, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { createHash } from "node:crypto";
+import { runnerE2EServerControlPaths } from "./harness-env.js";
+import { redactText, sanitizeJson } from "./redaction.js";
+
+export async function copySharedEvidence(privateRoot: string, casePrivateDir: string,
+  temporaryRoot: string, secrets: readonly string[]) {
+  for (const relative of [
+    "server.log",
+    "playwright.log",
+    "junit.xml",
+    "html-report",
+    "blob-report",
+    "playwright-output",
+  ]) {
+    await cp(
+      path.join(privateRoot, relative),
+      path.join(casePrivateDir, relative),
+      { recursive: true, force: true },
+    ).catch((error: NodeJS.ErrnoException) => {
+      if (error.code !== "ENOENT") throw error;
+    });
+  }
+  // The supervisor records each phase before it can fail. Copy its receipts
+  // independently of the successful-restart acknowledgement before cleanup.
+  const { controlDirectory } = runnerE2EServerControlPaths(temporaryRoot);
+  const entries = await readdir(controlDirectory, { withFileTypes: true }).catch((error: NodeJS.ErrnoException) => {
+    if (error.code === "ENOENT") return [];
+    throw error;
+  });
+  for (const entry of entries) {
+    if (!entry.isFile() || !/^controller-outage-[A-Za-z0-9._:-]{1,200}\.json$/.test(entry.name)) continue;
+    const raw = await readFile(path.join(controlDirectory, entry.name), "utf8");
+    const snapshots = path.join(casePrivateDir, "snapshots", "controller-outages");
+    await mkdir(snapshots, { recursive: true });
+    let receipt: unknown;
+    try { receipt = JSON.parse(raw); }
+    catch {
+      // Never include parser excerpts: they can quote private receipt bytes.
+      receipt = { schema: "paperclip.e2e.controller-outage-parse-error/v1", source: entry.name,
+        error: "invalid_json", byteLength: Buffer.byteLength(raw), sha256: createHash("sha256").update(raw).digest("hex") };
+      await writeFile(path.join(snapshots, `${entry.name}.invalid.txt`), redactText(raw, secrets), { mode: 0o600 });
+    }
+    await writeFile(path.join(snapshots, entry.name), `${JSON.stringify(sanitizeJson(receipt, secrets), null, 2)}\n`, { mode: 0o600 });
+  }
+}

tests/runner-e2e/upgrade-artifacts.test.ts

@@ -0,0 +1,61 @@
+import { createHash } from "node:crypto";
+import { mkdir, mkdtemp, readFile, realpath, rm, symlink, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { afterEach, expect, it } from "vitest";
+import { runnerMatrix } from "./catalog.js";
+import { assertServingArtifact, installedTreeSha256, readUpgradeArtifacts, verifyUpgradeArtifact, type UpgradeArtifact } from "./upgrade-artifacts.js";
+
+const roots: string[] = [];
+afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
+const cell = runnerMatrix.find(e => e.id === "everyday-workflows.runner-codex.daytona.recover-controller")!;
+async function fixture(revision = "a".repeat(40)): Promise<UpgradeArtifact> {
+  const root = await realpath(await mkdtemp(join(tmpdir(), "upgrade-artifact-"))); roots.push(root);
+  const cli = join(root, "node_modules/paperclipai");
+  const server = join(root, "node_modules/@paperclipai/server");
+  for (const directory of [cli, server]) await mkdir(join(directory, "dist"), { recursive: true });
+  await writeFile(join(cli, "package.json"), JSON.stringify({ name: "paperclipai", version: "1.2.3", type: "module", bin: { paperclipai: "./dist/index.js" } }));
+  await writeFile(join(server, "package.json"), JSON.stringify({ name: "@paperclipai/server", version: "1.2.3", type: "module", exports: { ".": { import: "./dist/index.js" } } }));
+  await writeFile(join(cli, "dist/index.js"), "// public cli");
+  await writeFile(join(server, "dist/index.js"), `// public server ${revision}`);
+  await writeFile(join(server, "dist/build-info.json"), JSON.stringify({ commit: revision }));
+  const hash = async (path: string) => createHash("sha256").update(await readFile(path)).digest("hex");
+  return { root, revision, treeSha256: await installedTreeSha256(root), evidence: "development-build", pins: {
+    PAPERCLIP_RUNNER_E2E_INSTALLED_CLI: join(cli, "dist/index.js"),
+    PAPERCLIP_RUNNER_E2E_INSTALLED_CLI_SHA256: await hash(join(cli, "dist/index.js")),
+    PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_ROOT: server,
+    PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_SHA256: await hash(join(server, "dist/index.js")),
+  } };
+}
+it("verifies distinct A/B artifacts and a real serving revision at the same package version", async () => {
+  const A = await fixture(); const B = await fixture("b".repeat(40));
+  const manifest = join(tmpdir(), `upgrade-${process.pid}-${Date.now()}.json`);
+  try {
+    await writeFile(manifest, JSON.stringify({ schema: "paperclip.e2e.upgrade-artifacts/v1", artifacts: { A, B }, sequence: ["A", "B", "A"] }));
+    expect((await readUpgradeArtifacts(manifest))!.sequence).toEqual(["A", "B", "A"]);
+    for (const artifact of [A, B, A]) {
+      const verified = await verifyUpgradeArtifact(artifact, {}, [cell]);
+      expect(verified.receipt.evidence).toBe("development-build");
+      expect(() => assertServingArtifact({ status: "ok", commit: artifact.revision }, verified.receipt)).not.toThrow();
+      expect(() => assertServingArtifact({ status: "ok" }, verified.receipt)).toThrow("serving server revision");
+      expect(() => assertServingArtifact({ status: "ok", commit: "c".repeat(40) }, verified.receipt)).toThrow("serving server revision");
+    }
+    await writeFile(manifest, JSON.stringify({ schema: "paperclip.e2e.upgrade-artifacts/v1", artifacts: { A, B: A }, sequence: ["A", "B", "A"] }));
+    await expect(readUpgradeArtifacts(manifest)).rejects.toThrow("genuinely different");
+  } finally { await rm(manifest, { force: true }); }
+});
+it("detects changed transitive files even when CLI and server entrypoint hashes are unchanged", async () => {
+  const artifact = await fixture();
+  await writeFile(join(artifact.root, "node_modules/transitive.js"), "changed migration or runtime");
+  await expect(verifyUpgradeArtifact(artifact, {}, [cell])).rejects.toThrow("frozen tree");
+});
+it("rejects source checkouts, escaping links and a false build stamp", async () => {
+  const artifact = await fixture(); const other = await fixture("b".repeat(40));
+  await symlink(other.root, join(artifact.root, "foreign"));
+  await expect(installedTreeSha256(artifact.root)).rejects.toThrow("escapes");
+  await rm(join(artifact.root, "foreign"));
+  await mkdir(join(artifact.root, ".git"));
+  await expect(installedTreeSha256(artifact.root)).rejects.toThrow("outside a source checkout");
+  await rm(join(artifact.root, ".git"), { recursive: true });
+  await expect(verifyUpgradeArtifact({ ...artifact, revision: other.revision }, {}, [cell])).rejects.toThrow("build stamp");
+});

tests/runner-e2e/upgrade-artifacts.ts

@@ -0,0 +1,99 @@
+/** Qualification-only artifact selection. Production recovery remains in the server. */
+import { createHash } from "node:crypto";
+import { createReadStream } from "node:fs";
+import { lstat, readFile, readdir, readlink, realpath } from "node:fs/promises";
+import { isAbsolute, join, relative, resolve, sep } from "node:path";
+import { verifyInstalledCli, type installedCliKeys } from "./installed-cli.js";
+import type { MatrixExecution } from "./types.js";
+
+export const upgradeArtifactKey = "PAPERCLIP_RUNNER_E2E_UPGRADE_ARTIFACTS";
+export interface UpgradeArtifact {
+  root: string;
+  revision: string;
+  treeSha256: string;
+  evidence: "development-build" | "released";
+  pins: Record<(typeof installedCliKeys)[number], string>;
+}
+export interface UpgradeArtifacts {
+  schema: "paperclip.e2e.upgrade-artifacts/v1";
+  artifacts: { A: UpgradeArtifact; B: UpgradeArtifact };
+  sequence: Array<"A" | "B">;
+}
+export interface ServingArtifact {
+  revision: string;
+  treeSha256: string;
+  version: string;
+  evidence: UpgradeArtifact["evidence"];
+}
+const hex = (value: unknown, length: number): value is string => typeof value === "string" && new RegExp(`^[a-f0-9]{${length}}$`).test(value);
+const inside = (root: string, file: string) => {
+  const part = relative(root, file);
+  return part !== ".." && !part.startsWith(`..${sep}`) && !isAbsolute(part);
+};
+
+/** Hash the complete installed tree, including transitive packages and migrations.
+ * Links are hashed as links and must resolve within the frozen installation.
+ * Stream large binaries; never buffer a whole installation in memory.
+ */
+export async function installedTreeSha256(root: string): Promise<string> {
+  if (!isAbsolute(root) || resolve(root) !== root || await realpath(root) !== root) throw new Error("Upgrade artifact root must be canonical");
+  const hash = createHash("sha256");
+  async function visit(file: string): Promise<void> {
+    const before = await lstat(file, { bigint: true });
+    const name = relative(root, file).split(sep).join("/");
+    if (before.isSymbolicLink()) {
+      if (!inside(root, await realpath(file))) throw new Error("Upgrade artifact link escapes the installation");
+      hash.update(JSON.stringify([name, "link", await readlink(file)]) + "\n");
+    } else if (before.isDirectory()) {
+      hash.update(JSON.stringify([name, "directory"]) + "\n");
+      for (const child of (await readdir(file)).sort()) {
+        if (child === ".git") throw new Error("Upgrade artifacts must be installed builds outside a source checkout");
+        await visit(join(file, child));
+      }
+    } else if (before.isFile()) {
+      hash.update(JSON.stringify([name, "file", String(before.size), Number(before.mode & 0o777n)]) + "\n");
+      for await (const chunk of createReadStream(file)) hash.update(chunk);
+    } else throw new Error("Upgrade artifacts must contain only regular files, directories and contained links");
+    const after = await lstat(file, { bigint: true });
+    if (before.ino !== after.ino || before.dev !== after.dev || before.size !== after.size || before.mtimeNs !== after.mtimeNs || before.ctimeNs !== after.ctimeNs) {
+      throw new Error("Upgrade artifact changed during verification");
+    }
+  }
+  await visit(root);
+  return hash.digest("hex");
+}
+
+export async function readUpgradeArtifacts(filename: string | undefined): Promise<UpgradeArtifacts | null> {
+  if (!filename) return null;
+  const bytes = await readFile(filename);
+  if (bytes.length > 64 * 1024) throw new Error("Upgrade artifact manifest is too large");
+  const value = JSON.parse(bytes.toString()) as UpgradeArtifacts;
+  if (value?.schema !== "paperclip.e2e.upgrade-artifacts/v1" || !Array.isArray(value.sequence)
+    || value.sequence.length < 3 || value.sequence.length > 9 || value.sequence[0] !== "A"
+    || value.sequence[1] !== "B" || value.sequence[2] !== "A"
+    || value.sequence.some((id, index) => (id !== "A" && id !== "B") || id === value.sequence[index - 1])) throw new Error("Upgrade qualification requires an explicit alternating A, B, A sequence");
+  for (const id of ["A", "B"] as const) {
+    const a = value.artifacts?.[id];
+    if (!a || typeof a.root !== "string" || !hex(a.revision, 40) || !hex(a.treeSha256, 64)
+      || !["released", "development-build"].includes(a.evidence) || !a.pins || typeof a.pins !== "object") throw new Error("Invalid pinned upgrade artifact");
+  }
+  if (value.artifacts.A.root === value.artifacts.B.root || value.artifacts.A.revision === value.artifacts.B.revision
+    || value.artifacts.A.treeSha256 === value.artifacts.B.treeSha256) throw new Error("A and B must be genuinely different immutable artifacts");
+  return value;
+}
+
+export async function verifyUpgradeArtifact(artifact: UpgradeArtifact, env: NodeJS.ProcessEnv, executions: readonly MatrixExecution[]) {
+  const installed = await verifyInstalledCli({ ...env, ...artifact.pins }, executions);
+  if (!installed || !inside(artifact.root, installed.cliRoot) || !inside(artifact.root, installed.serverRoot)) throw new Error("Upgrade packages must belong to their pinned installation");
+  if (await installedTreeSha256(artifact.root) !== artifact.treeSha256) throw new Error("Upgrade installation differs from its frozen tree");
+  const build = JSON.parse(await readFile(join(installed.serverRoot, "dist/build-info.json"), "utf8"));
+  if (build.commit !== artifact.revision) throw new Error("Upgrade build stamp differs from its reviewed revision");
+  return { installed, receipt: { revision: artifact.revision, treeSha256: artifact.treeSha256,
+    version: installed.version, evidence: artifact.evidence } satisfies ServingArtifact };
+}
+
+export function assertServingArtifact(health: unknown, artifact: ServingArtifact): void {
+  if (!health || typeof health !== "object" || (health as { commit?: unknown }).commit !== artifact.revision) {
+    throw new Error("The serving server revision does not match the verified upgrade artifact");
+  }
+}