Qualify rolling web deployments and preserve Stop through handoff (#18526)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The web server owns request handling and remote execution
coordination.
> - Deployment must start a ready replacement while the old server still
serves.
> - An outage test does not prove that two live servers can overlap
safely.
> - This pull request adds a separate Product E2E cell for that
boundary.
> - The existing run, file, recovery, and retained-session checks still
apply.

## Linked Issues or Issue Description

Refs #18518. The runtime prerequisite is merged; this PR qualifies its
ephemeral server and deployment contracts.

## What Changed

- Add the explicit `rolling-web-deployment` Codex Daytona workflow.
- Start independent A/B/C homes and compiled builds behind a test
ingress.
- Wait for readiness before routing, then drain and delete the old home.
- Observe active authority before handoff and use the existing read-only
recovery oracle.
- Require ordinary reads and saves to succeed throughout replacement.
Read back every accepted save exactly once.
- Preserve the five-minute outage suites and their existing validators.
- Add a model-free supervisor smoke and an unready replacement control.
- Require graceful exit before deleting an old home; forced cleanup
cannot satisfy the oracle.
- Preserve a durable Stop through lease release/expiry until the
recovered execution owner acknowledges it. Permit only fenced
reattachment of the original execution for Stop cleanup; submit no new
turn.
- Exclude events saved by A during shutdown from evidence of B making
progress.

## Verification

- E2E TypeScript check passed.
- Affected harness checks: 143 passed. Catalog, fixture and report
checks: 120 passed.
- Actual supervisor smoke passed two replacements: 436 reads, 45 saves,
zero request failures, and confirmed owned resource cleanup.
- B became ready in 18.7 seconds; C became ready in 109.2 seconds. A
prior C timeout remains failed evidence.
- Fresh browser proof: 11 checks passed, 246 reads, 50 acknowledged
saves, zero failures, 1,058ms reconnect, both old homes deleted and
cleanup confirmed.
- Stop lease-gap regression: three PostgreSQL delivery tests passed with
a simulated provider edge. Current server TypeScript build/vendoring
passed.
- Full harness: 5,814 passed, 17 failed, four skipped. Two shutdown
races passed the focused file; two report timeouts passed an isolated
42-test rerun. Onboarding/workflow-security/Pi/date-expired fixtures are
unchanged on inspected master; the host embedded PostgreSQL fixture
remains a setup limitation. This is not a green full local suite.
- CI at `172509b8be` passed all gates. Greptile is 5/5 with every review
thread resolved. The earlier access-service race failure passed its
isolated regression and the subsequent full CI.
- Final review corrections: six focused executor cases, three PostgreSQL
command-delivery cases, 19 harness cases and server/E2E typechecks
passed.
- Three live attempts failed before agent creation: two harness setup
gaps were fixed, then provider authentication failed. The saved OpenAI
key returns 401; a refresh is pending. The Daytona probe reported
cleanup uncertain, and no sandbox with that attempt label was found
through the authenticated staging manager. No live model execution is
claimed.
- Live Runner overlap, released-container qualification and hosted
deployment proof remain pending. Merging this opt-in qualification lane
does not claim those results.

## Risks

- The source lane shares workspace dependencies and does not establish
container packaging correctness.
- The live workflow needs an explicitly owned S3 bucket, PostgreSQL,
pinned remote image and provider credentials.
- Qualification cleanup stops all owned processes after failure. A
failed replacement must not be reported as a pass.

## Model Used

OpenAI Codex, GPT-6. Tool use, code execution and repository review. The
runtime does not expose a more specific model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/plans/2026-10-31-zero-downtime-web-deploys.md

@@ -1,7 +1,7 @@
 # Zero-downtime web deployments
 
-Date: 2026-10-31. Status: implementation in progress; local HTTP/storage overlap,
-failure recovery and graceful drain passed. Cloud shape implementation underway.
+Date: 2026-10-31. Status: runtime and Cloud overlap policy merged; qualification
+and existing-volume enrollment remain in progress. No hosted acceptance claim.
 
 ## Outcome and endpoint
 
@@ -250,10 +250,110 @@ the normal build vendoring step corrected it. All three attempts cleaned up.
 
 Staging read-only discovery and authentication work with AWS `paperclip-dev`
 (account `078455283791`); production `paperclip-prod` is also authenticated.
-The existing shared `canary-staging` stack is `stack-pool-8fabb27531db77af`.
+The existing shared `canary-staging` stack is `stack-pool-8fabb27531db`.
 Do not assume it is exclusively owned: select a dedicated zero-downtime test
 stack or acquire the existing stack operation lock before any hosted mutation.
 No hosted mutation or paid model call has occurred for this task yet.
 
 Remaining: live overlap proof; released artifacts; migration/enrollment and
 compatible rollback; ten staging deployments; internal production canary.
+
+The active-overlap harness is a separate stacked PR [#18526](https://github.com/paperclipai/paperclip/pull/18526)
+on runtime PR #18518 to preserve the repository's review-size boundary. Its first
+review found that the checkpoint observer lacked the disposable encryption key
+and that forced process cleanup could pass graceful qualification. Both are
+corrected; 22 focused tests passed, including actual child shutdown and shared
+key decryption/restoration. The earlier model-free receipt remains evidence for
+HTTP availability and cleanup, not active checkpoint recovery.
+
+The full harness run at `0f122611a1` completed with 5,814 passes, 17 failures and
+four skips. Preserve `/private/tmp/paperclip-zdt-e2e-unit-all-rolling.log`:
+four onboarding mock expectations, four unchanged workflow security expectations,
+one Pi profile version expectation, three date-expired accounting fixtures,
+two report-command timeouts, two process-shutdown races and one host embedded
+Postgres startup failure. The complete shutdown file passed in the focused run;
+other failures still need base-revision classification. No blanket green suite
+or live availability claim is made.
+
+### Current delivery state, 2026-11-01 08:00 UTC
+
+Runtime PR #18518 merged as `006d5691a6245d5a710e25ba3df6d609754e163a` after
+all CI passed and a 5/5 review. The opt-in remains disabled on hosted stacks.
+Qualification PR #18526 remains open; Cloud #1096 remains open. No hosted
+mutation or paid model test has occurred yet.
+
+Fresh browser receipt `paperclip-runner-e2e-tg3nel` passed all 11 checks:
+246 reads, 50 acknowledged/read-back saves, zero failures, 1,058ms browser
+reconnect and zero page errors. Both old homes were deleted; C restored saved
+files with legacy adoption disabled. Unready B left A serving. The admitted
+save and both handoffs now require a graceful exit, not successful forced cleanup.
+Cleanup is confirmed. The receipt uses compiled local server/UI and workspace
+dependencies at `59e6169af6` plus the smoke's strict-exit patch; it is not a
+released-container or live-agent claim.
+
+Qualification exposed an expired/released-lease Stop race: absent a local handle,
+a new server could acknowledge a no-op before recovery attached the execution.
+Stop now remains pending while an unsettled native execution exists, including
+between leases. Three real-PostgreSQL delivery cases pass; the provider edge and
+recovery claim are simulated. Server TypeScript compilation and vendoring pass.
+
+The full harness's two report timeouts passed an isolated 42-test file rerun.
+Onboarding, workflow-security, Pi and accounting files are unchanged by this
+branch versus the inspected master; this is static classification, not a fresh
+full baseline run. The qualification PR's CI has one access-service race failure
+and is being checked in isolation. All other checks passed on `59e6169af6`.
+
+Cloud now gates overlap on identical, provenance-bound package identities.
+Changed or unknown packages use maintenance. Its 376 campaign/compatibility
+cases passed; review identified that failed-deploy rollback also needs the same
+pause. That correction is in verification. Existing-volume enrollment and the
+hosted acceptance sequence remain required work, not deferred scope.
+
+
+### Current delivery state, 2026-11-01 08:40 UTC
+
+Runtime #18518 is merged at `006d5691a6245d5a710e25ba3df6d609754e163a`;
+Cloud #1096 is merged at `23fdc849029e541355fa4e11234a3b5706b01e84`.
+Both passed CI and final 5/5 reviews with comments resolved. Core's Docker and
+readiness artifacts have published; its release workflow remains in progress.
+No explicit hosted enrollment or stack deployment has been performed by this
+chat; automatic release pipelines are separate and need deployed-revision checks.
+
+Qualification #18526 at `f2bcb4c73b` passed all CI checks. Three review findings
+are fixed in the next commit: explicit mock-call isolation; pending Stop permits
+only fenced reattachment of the original remote execution for cleanup; and the
+recovery observer advances its event floor after A is confirmed dead while
+preserving the original owner/authentication baseline. Six focused executor cases,
+three PostgreSQL delivery cases (provider edge simulated), 19 harness cases and
+server/E2E typechecks pass. The fixture initially returned a mock from beforeEach,
+which Vitest invoked as teardown; the corrected hook returns nothing.
+
+Three live attempts failed before agent creation: missing built Daytona plugin,
+missing development TypeScript loader in isolated controller copies, then invalid
+provider credentials. The first two harness setup gaps are fixed. The third
+Daytona preflight reported cleanup uncertain; the saved key returns 401, and the
+staging manager found no sandbox with that exact attempt label. This does not
+prove absence in an unknown old-key organization. The OpenAI key independently
+returns 401. A user request to refresh the E2E credential is pending; do not
+retry a paid attempt until authentication succeeds. No model execution is claimed.
+Private receipts are under `/private/tmp/paperclip-zdt-live-evidence`.
+
+A Linux Runner binary built from `0f122611a1` has SHA256
+`681f9709f21abcc90f6e87fd1d0b263c158896b8fb3d30df6786d89723ddfdad`;
+its source inputs are unchanged at the qualification head. The pinned Daytona
+base image and dynamic library checks pass. The next launcher preflights OpenAI,
+uses the authenticated staging manager to mint one attempt-specific Daytona key,
+and revokes it during cleanup. No attempt key has yet been minted.
+
+Enrollment implementation is isolated in Core
+`/private/tmp/paperclip-zdt-enrollment-20261101` and the Cloud worktree, both on
+`codex/zero-downtime-enrollment`. Core's dedicated maintenance process is being
+verified: ordinary app writers never start; existing owner migrations verify
+shared bytes; the original decision key exports only through a per-attempt
+credential after full eligibility. Cloud's locked, audited provider transition
+must stop all writers, run this process on the retained volume, preserve its key,
+then detach only after a matching receipt and confirmed process exit.
+
+Remaining acceptance is unchanged: live Stop/answer/follow-up and continuation,
+packaged fresh/migrated overlap, ten staging deployments including rollback,
+repair/sleep/wake and idle suspension, then one internal production canary.

server/src/__tests__/native-cancellation-delivery.test.ts

@@ -1,11 +1,11 @@
 import { randomUUID } from "node:crypto";
 import { eq } from "drizzle-orm";
-import { afterAll, beforeAll, afterEach, expect, it, vi } from "vitest";
+import { afterAll, beforeAll, beforeEach, afterEach, expect, it, vi } from "vitest";
 import { agents, companies, issues, heartbeatRuns, nativeRunFinalizations, createDb } from "@paperclipai/db";
 import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
 import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
 import { deploymentNotifications } from "../services/deployment-notifications.js";
-import { nativeCancellationHasRemoteOwner, notifyNativeCancellation, registerNativeCancellationDelivery, waitForNativeCancellation } from "../services/native-runtime/native-cancellation-delivery.js";
+import { nativeCancellationHasUnsettledOwner, notifyNativeCancellation, registerNativeCancellationDelivery, waitForNativeCancellation } from "../services/native-runtime/native-cancellation-delivery.js";
 import { currentNativeControllerIdentity } from "../services/native-runtime/native-restart-recovery.js";
 import { setNativeExecutionActive } from "../services/native-runtime/native-active-runs.js";
 const provider = vi.hoisted(() => ({ stop: vi.fn() }));
@@ -46,6 +46,7 @@ beforeAll(async () => {
   });
   worker = registerNativeCancellationDelivery(a, coordinatorA); await worker.ready;
 }, 90_000);
+beforeEach(() => { provider.stop.mockClear(); });
 afterEach(() => vi.unstubAllEnvs());
 afterAll(async () => {
   setNativeExecutionActive(runId, false); await worker?.stop();
@@ -68,11 +69,28 @@ it("fails closed on a different intent and never accepts a no-op acknowledgement
     acknowledgementAuditId: "not-dispatched" } } }).where(eq(heartbeatRuns.id, runId));
   await expect(waitForNativeCancellation(b, { companyId, runId, intentId }, 50)).rejects.toThrow("pending_recovery");
 });
-it("distinguishes the current controller, a live replacement owner, and an expired owner", async () => {
+it("keeps Stop pending across lease release and delivers after normal ownership resumes", async () => {
   vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
-  expect(await nativeCancellationHasRemoteOwner(b, runId)).toBe(false);
+  expect(await nativeCancellationHasUnsettledOwner(b, runId)).toBe(true);
   await b.update(nativeRunFinalizations).set({ controllerBootId: "another-controller" }).where(eq(nativeRunFinalizations.runId, runId));
-  expect(await nativeCancellationHasRemoteOwner(b, runId)).toBe(true);
+  expect(await nativeCancellationHasUnsettledOwner(b, runId)).toBe(true);
   await b.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(0) }).where(eq(nativeRunFinalizations.runId, runId));
-  expect(await nativeCancellationHasRemoteOwner(b, runId)).toBe(false);
+  expect(await nativeCancellationHasUnsettledOwner(b, runId)).toBe(true);
+  await b.update(heartbeatRuns).set({ resultJson: { nativeCancellation: { schema: "paperclip.native-cancellation.v1", intentId,
+    scope: "run", dispatchState: "pending" } } }).where(eq(heartbeatRuns.id, runId));
+  const acknowledged = waitForNativeCancellation(b, { companyId, runId, intentId });
+  const identity = await currentNativeControllerIdentity();
+  // Simulate only the normal recovery owner's claim; no test-owned dispatch or
+  // acknowledgement. Production delivery must consume the durable pending Stop.
+  await a.transaction(async tx => {
+    await notifyNativeCancellation(tx);
+    await tx.update(nativeRunFinalizations).set({ controllerBootId: identity.bootId, leaseExpiresAt: new Date(Date.now() + 60_000) })
+      .where(eq(nativeRunFinalizations.runId, runId));
+  });
+  await expect(acknowledged).resolves.toMatchObject({ dispatched: true, auditId: "fixture-audit" });
+  expect(provider.stop).toHaveBeenCalledTimes(1);
+  await a.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, runId));
+  expect(await nativeCancellationHasUnsettledOwner(b, runId)).toBe(false);
+  vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "false");
+  expect(await nativeCancellationHasUnsettledOwner(b, runId)).toBe(false);
 });

server/src/services/native-runtime/native-cancellation-delivery.ts

@@ -1,4 +1,4 @@
-import { and, eq, gt, inArray, sql } from "drizzle-orm";
+import { and, eq, gt, inArray, isNull, sql } from "drizzle-orm";
 import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
 import { subscribeDatabaseWork } from "@paperclipai/db/work-signals";
 import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
@@ -9,12 +9,17 @@ import { currentNativeControllerIdentity } from "./native-restart-recovery.js";
 const object = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
 export const notifyNativeCancellation = (tx: object) => notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeCancellation);
 
-export async function nativeCancellationHasRemoteOwner(db: Db, runId: string): Promise<boolean> {
+/** Called only when this process has neither an active handle nor a startup.
+ * Expiry/release authorizes recovery, not a no-op acknowledgement: the remote
+ * execution can still be alive while its next controller is attaching. */
+export async function nativeCancellationHasUnsettledOwner(db: Db, runId: string): Promise<boolean> {
   if (process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true") return false;
-  const identity = await currentNativeControllerIdentity();
-  const [owner] = await db.select({ bootId: nativeRunFinalizations.controllerBootId }).from(nativeRunFinalizations)
-    .where(and(eq(nativeRunFinalizations.runId, runId), gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`))).limit(1);
-  return !!owner?.bootId && owner.bootId !== identity.bootId;
+  const [owner] = await db.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
+    .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
+      eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId)))
+    .where(and(eq(nativeRunFinalizations.runId, runId), isNull(nativeRunFinalizations.resultId),
+      inArray(heartbeatRuns.status, ["running", "queued"]))).limit(1);
+  return Boolean(owner);
 }
 
 /** The accepting HTTP server waits for the real owner's audited ACK. Subscribe

server/src/services/native-runtime/native-session-executor.test.ts

@@ -6603,6 +6603,30 @@ function cancellationDb(options?: {
 }
 
 describe("native startup cancellation fence", () => {
+  it.each(["current", "stale"] as const)("reattaches a %s remote recovery claim only to deliver an existing Stop", async mode => {
+    const { currentNativeControllerIdentity } = await import("./native-restart-recovery.js");
+    const controller = await currentNativeControllerIdentity();
+    const resultJson = { nativeCancellation: { schema: "paperclip.native-cancellation.v1", scope: "run",
+      companyId: execution.binding.companyId, runId: execution.binding.runId, issueId: execution.binding.issueId,
+      dispatchState: "pending", intentId: "native-cancellation:recovered-stop", effects: ["release_run_resources"] } };
+    const cancel = vi.fn(() => ({ cleanup: Promise.resolve() })), submit = vi.fn();
+    state.execute.mockReset().mockImplementationOnce(async options => {
+      try { await options.onSession({ cancel }); submit(); }
+      finally { await options.onSession(null); }
+      throw new Error("a stopped recovery must not submit a turn");
+    });
+    const db = leaseDb(execution, { leaseOwner: "stop-recovery", controllerBootId: controller.bootId,
+      controllerPid: controller.pid, controllerGeneration: 2, attempt: 1 }, resultJson);
+    await expect(executePaperclipNativeSession({ db, execution, runnerInstanceId: "recovered-stop",
+      leaseOwner: "stop-recovery", restartRecovery: { kind: "reattach_remote_runner", runId: execution.binding.runId,
+        leaseOwner: "stop-recovery", controllerGeneration: mode === "current" ? 2 : 1, providerAttempt: 1,
+        restartKind: "graceful", recoveryRequestId: null, remote: { providerLeaseId: "fixture", remoteCwd: "/workspace" } },
+    })).rejects.toThrow(mode === "current" ? "native_cancellation_pending_recovery" : "native_restart_recovery_claim_changed");
+    expect(submit).not.toHaveBeenCalled();
+    expect(cancel).toHaveBeenCalledTimes(mode === "current" ? 1 : 0);
+    expect(state.execute).toHaveBeenCalledTimes(mode === "current" ? 1 : 0);
+  });
+
   it.each(["startupCancellation", "nativeCancellation"])("does not submit a turn when %s arrives during session opening", async (marker) => {
     const resultJson: Record<string, unknown> = {};
     const cancel = vi.fn(() => ({ cleanup: Promise.resolve() }));

server/src/services/native-runtime/native-session-executor.ts

@@ -1,4 +1,4 @@
-import { nativeCancellationHasRemoteOwner, notifyNativeCancellation, waitForNativeCancellation } from "./native-cancellation-delivery.js";
+import { nativeCancellationHasUnsettledOwner, notifyNativeCancellation, waitForNativeCancellation } from "./native-cancellation-delivery.js";
 import { publishNativeHarnessBackup } from "./native-harness-backup-publication.js";
 import { rehydratePinnedSkillAssets } from "./runtime-context.js";
 import { createNativeUsageDelivery } from "./native-usage-delivery.js";
@@ -7710,7 +7710,7 @@ async function performNativeCancellation(
     recoveringCancellationIntent = intent.existing;
     priorCoordinatorDecisionIdAtIntent = intent.priorCoordinatorDecisionId;
     if (!activeNativeSessions.has(runId) && !nativeSessionStartups.has(runId)
-      && await nativeCancellationHasRemoteOwner(options.db, runId)) {
+      && await nativeCancellationHasUnsettledOwner(options.db, runId)) {
       const acknowledged = await waitForNativeCancellation(options.db, {
         companyId: cancellationContext.companyId, runId, intentId: intent.intentId,
       });
@@ -8551,9 +8551,15 @@ async function executePaperclipNativeSessionWithinScope(
               cancellationIntent.issueId === input.execution.binding.issueId;
             if (!intentMatchesBinding)
               throw new Error("native_cancellation_intent_conflict");
-            // Run cancellation remains a claim fence after dispatch ack until
-            // the heartbeat cancellation path terminalizes the run.
-            throw new NativeCancellationPendingRecoveryError();
+            // A pending Stop must be able to reattach its already-admitted
+            // remote execution for cleanup. The recovery claim is fenced below;
+            // onSession dispatches this durable Stop before any turn submission.
+            // Fresh execution, dead-runner resume and acknowledged Stops remain
+            // fenced: none authorize launching another provider turn.
+            if (input.restartRecovery?.kind !== "reattach_remote_runner"
+              || cancellationIntent.dispatchState !== "pending") {
+              throw new NativeCancellationPendingRecoveryError();
+            }
           }
           if (["committed", "applied"].includes(coordinator.phase))
             throw new Error("native_run_already_committed");

tests/runner-e2e/README.md

@@ -96,6 +96,35 @@ PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_REGION=us-east-1 \
 pnpm test:e2e:runner -- --id remote-workspace-storage.runner-codex.daytona.recover-controller --max-automatic-retries 0
 ```
 
+The separate explicit-only `rolling-web-deployment` suite uses the same synthetic
+projectless workflow, four-admission budget, thirty-minute limit, and independent
+run/file oracles. Its supervisor starts an independent empty-home server, waits
+for `/api/ready`, moves ingress, sends SIGTERM to the predecessor, and deletes
+that predecessor's whole home and build. Continuous ordinary task reads and
+comment saves must have zero failures, and every accepted save must be read back
+exactly once. Forced process cleanup fails graceful qualification even if the
+successor recovers the execution. The read-only observer shares the controllers'
+disposable encryption key. It repeats replacement after completion before the retained-session
+follow-up. The five-minute outage suites are unchanged.
+
+```sh
+pnpm test:e2e:runner -- --id rolling-web-deployment.runner-codex.daytona.recover-controller --max-automatic-retries 0
+```
+
+This source lane requires a complete server/UI build and disposable S3 and
+Postgres. Each controller has an independent compiled build and cache; shared
+workspace dependencies mean it is not released-container or Railway evidence.
+The read-only recovery observer captures the live predecessor's identity just
+before handoff, then requires a new controller generation, authenticated Runner
+attachment, and new canonical events for the original execution. No fixture
+modifies leases, checkpoints, provider identity, or completed work.
+
+Before paid execution, set `PAPERCLIP_ROLLING_TEST_STORAGE_CONFIG` and the
+loopback `PAPERCLIP_TEST_DATABASE_URL`, then run
+`tests/runner-e2e/controller-replacement-smoke.ts` through the existing tsx loader.
+It exercises the actual supervisor and two replacements without an agent and
+explicitly records `activeAgentContinuityQualified: false`.
+
 Optional `PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_ENDPOINT` and
 `PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_FORCE_PATH_STYLE=true` select a test S3
 endpoint. Each attempt has a distinct `runner-e2e/<instance-id>/` prefix. The

tests/runner-e2e/SECURITY.md

@@ -7,7 +7,8 @@ reference. Provider credentials and agent homes remain outside that folder.
 The ordinary settings read must report remote storage enabled by default; the
 workflow does not opt itself in. Its five-minute active outage and later completed-session
 replacement retain the existing native identity and no-replay checks.
-The server's S3 override is admitted only for this suite and only from explicit
+The server's S3 override is admitted only for this suite and the explicit
+`rolling-web-deployment` suite, and only from explicit
 `PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_*` configuration, with a unique instance
 prefix. Ambient storage settings remain stripped and rejected for other suites.
 Snapshot objects, databases, raw provider IDs, and execution workspaces remain
@@ -618,3 +619,9 @@ Slack qualification, and they do not publish trusted archives or credentials.
 Report identities preserve legacy chat tasks and exact `area.case` chat catalog
 tasks. Additional task segments, path separators and dotted nonchat tasks remain
 invalid. This parsing does not qualify a result or grant a new execution.
+
+The explicit rolling web deployment cell shares only its disposable PostgreSQL,
+S3 prefix, and generated secrets between controllers. It removes the entire old
+home after graceful handoff. Its read-only observer does not create recovery
+authority. Provider credentials still enter through encrypted fixture secrets.
+Local rolling smoke receipts explicitly exclude active-agent and hosted proof.

tests/runner-e2e/catalog.test.ts

@@ -278,10 +278,10 @@ describe("runner E2E catalog", () => {
     expect(localIntegrityTasks).toHaveLength(2);
     expect(openRouterBreadthTasks).toHaveLength(3);
     expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
-      1, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2,
+      1, 1, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2,
     ]);
-    expect(validateRunnerCatalog()).toHaveLength(796);
-    expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(796);
+    expect(validateRunnerCatalog()).toHaveLength(797);
+    expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(797);
     expect(
       runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
     ).toHaveLength(48);

tests/runner-e2e/catalog.ts

@@ -1,3 +1,4 @@
+import { ROLLING_WEB_DEPLOYMENT_SUITE } from "./controller-rolling-deployment.js";
 import { FEEDBACK_PERSISTENCE_SUITE, feedbackPersistenceTask, feedbackPersistenceDigest } from "./feedback-persistence.js";
 import { followupStopTask, followupStopDigest } from "./followup-stop-resume.js";
 import { finalizationRecoveryTask, finalizationRecoveryDigest } from "./finalization-recovery.js";
@@ -1169,8 +1170,7 @@ const daytonaControllerOutageEnvironment: EnvironmentFixture = {
   },
 };
 
-export const runnerSuites: readonly RunnerSuiteFixture[] = [
-  {
+const remoteWorkspaceStorageSuite: RunnerSuiteFixture = {
     id: REMOTE_WORKSPACE_STORAGE_SUITE, label: "Remote workspace storage and controller replacement", manualOnly: true,
     description: "Keep synthetic files and local Git work in the sandbox, commit saved versions to object storage, and preserve the original active execution through controller replacement.",
     groups: [], profiles: [(() => {
@@ -1196,6 +1196,16 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
       postControllerReplyFormat: "one-plain-text-line-marker-space-actual-command-output",
       grading: "same-active-execution-committed-snapshots-browser-files-and-post-controller-native-continuation",
       source: "projectless-synthetic-files-and-model-created-git", authentication: "api-key", stagingQualification: false },
+  };
+
+export const runnerSuites: readonly RunnerSuiteFixture[] = [
+  remoteWorkspaceStorageSuite,
+  { ...remoteWorkspaceStorageSuite, id: ROLLING_WEB_DEPLOYMENT_SUITE, label: "Rolling control-plane web deployment",
+    description: "Overlap independent ephemeral web servers while a remote execution stays active; keep ordinary reads, saves, files and a later native continuation working.",
+    definitionMetadata: { ...remoteWorkspaceStorageSuite.definitionMetadata, version: 1, controllerDowntimeMs: 0,
+      replacement: "ready-before-traffic-switch-sigterm-delete-entire-home",
+      upgradeReadiness: "active-remote-workspace-recoverable-before-handoff",
+      grading: "same-active-execution-ordinary-reads-saves-and-second-empty-home-continuation" },
   },
   {
     id: "daytona-restart-continuity", label: "Daytona warm controller outage continuity", manualOnly: true,

tests/runner-e2e/controller-outage.ts

@@ -40,6 +40,7 @@ export function configuredControllerOutage(env: NodeJS.ProcessEnv, defaultHoldMs
 }
 
 export function controllerOutageForExecution(execution: MatrixExecution, env: NodeJS.ProcessEnv): ControllerOutage | undefined {
+  if (execution.suite.id === "rolling-web-deployment") return undefined;
   if (execution.task.flow !== "everyday_workflow" || execution.task.id !== "recover-controller"
     || execution.environment.id !== "daytona" || execution.profile.generation !== "native"
     || (execution.profile.provider !== "codex" && execution.profile.provider !== "acpx")) return undefined;

tests/runner-e2e/controller-recovery-observer.test.ts

@@ -1,6 +1,6 @@
 import type { Db } from "../../packages/db/src/client.js";
 import { describe, expect, it } from "vitest";
-import { parseRecoveryBinding, readControllerRecoverySnapshot, recoveryObservation, validControllerRecoveryReceipt, type RecoverySnapshot } from "./controller-recovery-observer.js";
+import { advanceRecoveryCursor, parseRecoveryBinding, readControllerRecoverySnapshot, recoveryObservation, validControllerRecoveryReceipt, type RecoverySnapshot } from "./controller-recovery-observer.js";
 import type { ControllerOutageReceipt } from "./controller-outage.js";
 
 const binding = { companyId: "10000000-0000-0000-0000-000000000001", issueId: "10000000-0000-0000-0000-000000000002", runId: "10000000-0000-0000-0000-000000000003" };
@@ -13,6 +13,16 @@ const fault: ControllerOutageReceipt = { schema: "paperclip.e2e.controller-outag
 const after = (): RecoverySnapshot => ({ ...before, bootId: "new", generation: 2, sourceSeq: 47, authenticatedConnectionCount: 2,
   history: [{ controllerBootId: "new", controllerGeneration: 2, disposition: "reattach_remote_runner", at: "2026-10-24T00:01:00.001Z" }] });
 describe("read-only controller recovery evidence", () => {
+  it("does not count A's shutdown tail as B's processing even if B already claimed", () => {
+    const stopped = { ...after(), checkpointSourceSeq: 48 };
+    const baseline = advanceRecoveryCursor(before, stopped);
+    expect(baseline).toMatchObject({ bootId: "old", generation: 1, authenticatedConnectionCount: 1, sourceSeq: 48 });
+    expect(recoveryObservation(baseline, { ...stopped, sourceSeq: 48 }, fault, binding, 62_000)).toBeNull();
+    expect(recoveryObservation(baseline, { ...stopped, sourceSeq: 49 }, fault, binding, 62_001)).toMatchObject({
+      previousBootId: "old", runnerSourceSeqBefore: 48, runnerSourceSeqAfter: 49,
+    });
+    expect(() => advanceRecoveryCursor(before, { ...stopped, runnerInstanceId: "replacement" })).toThrow("changed the admitted execution");
+  });
   it("requires natural ownership and new original-run receipts after the stopped-process cursor", () => {
     expect(recoveryObservation(before, before, fault, binding, 2500)).toBeNull();
     expect(recoveryObservation(before, { ...after(), sourceSeq: 42 }, fault, binding, 61_000)).toBeNull();

tests/runner-e2e/controller-recovery-observer.ts

@@ -8,7 +8,7 @@ const { sql }: typeof import("../../packages/db/node_modules/drizzle-orm/index.j
 import { createDb, type Db } from "../../packages/db/src/index.js";
 import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js";
 import { loadNativeRunnerCheckpoint } from "../../server/src/services/native-runtime/native-runner-checkpoint.js";
-import type { ControllerOutageReceipt } from "./controller-outage.js";
+import type { ControllerReplacementReceipt } from "./controller-replacement.js";
 
 export interface RecoveryBinding { companyId: string; issueId: string; runId: string }
 export interface RecoverySnapshot {
@@ -47,7 +47,7 @@ export function parseRecoveryBinding(value: unknown): RecoveryBinding {
   return row as unknown as RecoveryBinding;
 }
 
-export function recoveryObservation(before: RecoverySnapshot, after: RecoverySnapshot, fault: ControllerOutageReceipt,
+export function recoveryObservation(before: RecoverySnapshot, after: RecoverySnapshot, fault: ControllerReplacementReceipt,
   binding: RecoveryBinding, observedAfterMs: number): ControllerRecoveryReceipt | null {
   if (before.runId !== binding.runId || after.runId !== before.runId || after.runnerInstanceId !== before.runnerInstanceId
     || after.nativeSessionId !== before.nativeSessionId || after.attempt !== before.attempt) {
@@ -77,7 +77,17 @@ export function recoveryObservation(before: RecoverySnapshot, after: RecoverySna
     authenticatedConnectionCount: after.authenticatedConnectionCount, attachmentEvidence: "runner_checkpoint_authenticated_connection" };
 }
 
-export function validControllerRecoveryReceipt(fault: ControllerOutageReceipt | undefined): boolean {
+/** B can claim before A exits. Preserve A's original owner/authentication
+ * identity while excluding every event committed through A's final exit. */
+export function advanceRecoveryCursor(before: RecoverySnapshot, stopped: RecoverySnapshot): RecoverySnapshot {
+  if (before.runId !== stopped.runId || before.runnerInstanceId !== stopped.runnerInstanceId
+    || before.nativeSessionId !== stopped.nativeSessionId || before.attempt !== stopped.attempt) {
+    throw new Error("Controller replacement changed the admitted execution");
+  }
+  return { ...before, sourceSeq: Math.max(before.sourceSeq, stopped.sourceSeq, stopped.checkpointSourceSeq) };
+}
+
+export function validControllerRecoveryReceipt(fault: ControllerReplacementReceipt | undefined): boolean {
   const r = fault?.recovery;
   return Boolean(r && r.bootId !== r.previousBootId && r.generation > r.previousGeneration
     && r.runnerSourceSeqAfter > r.runnerSourceSeqBefore && r.authenticatedConnectionCount > r.previousAuthenticatedConnectionCount
@@ -120,19 +130,26 @@ export async function createControllerRecoveryObserver(configPath: string, bindi
   const db = createDb(url);
   let before: RecoverySnapshot | undefined;
   const read = () => readControllerRecoverySnapshot(db, binding);
-  return {
-    // Called only after the old server is confirmed dead. This also excludes
-    // receipts committed while it was stopping from replacement evidence.
-    stopped: async (fault: ControllerOutageReceipt) => {
+  const capture = async (fault: ControllerReplacementReceipt) => {
       before = await read();
       before.sourceSeq = Math.max(before.sourceSeq, before.checkpointSourceSeq);
       fault.recoveryProgress = { binding, previousBootId: before.bootId, previousGeneration: before.generation,
         previousLeaseExpiresAt: before.leaseExpiresAt, runnerSourceSeqBefore: before.sourceSeq,
         previousAuthenticatedConnectionCount: before.authenticatedConnectionCount,
         runnerInstanceId: before.runnerInstanceId, nativeSessionId: before.nativeSessionId };
+  };
+  return {
+    beforeHandoff: capture,
+    // Called only after the old server is confirmed dead. This also excludes
+    // receipts committed while it was stopping from replacement evidence.
+    stopped: capture,
+    stoppedAfterHandoff: async (fault: ControllerReplacementReceipt) => {
+      if (!before) throw new Error("Missing pre-handoff controller observation");
+      before = advanceRecoveryCursor(before, await read());
+      fault.recoveryProgress = { ...fault.recoveryProgress, runnerSourceSeqBefore: before.sourceSeq };
     },
-    recovered: async (fault: ControllerOutageReceipt, record: () => Promise<void>, cancelled: () => boolean) => {
-      if (!before) throw new Error("Missing stopped-controller observation");
+    recovered: async (fault: ControllerReplacementReceipt, record: () => Promise<void>, cancelled: () => boolean) => {
+      if (!before) throw new Error("Missing controller boundary observation");
       const deadline = Date.parse(fault.startedAt) + 420_000;
       let ownershipObservedAfterMs: number | undefined;
       let authenticatedObservedAfterMs: number | undefined;

tests/runner-e2e/controller-replacement-smoke.ts

@@ -1,3 +1,5 @@
+import { validControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js";
+import { createRequire } from "node:module";
 import { readUpgradeArtifacts, upgradeArtifactKey } from "./upgrade-artifacts.js";
 /** Credential-free main-server replacement. This proves the wrapper/startup/DB
  * boundary; only the hosted workflow can qualify an active Daytona agent.
@@ -20,6 +22,12 @@ import { createRunnerE2EServerStopper } from "./server-stop.js";
 import { cleanupControllerSmokeDatabase } from "./controller-smoke-cleanup.js";
 import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js";
 
+const rolling = Boolean(process.env.PAPERCLIP_ROLLING_TEST_STORAGE_CONFIG);
+const { S3Client, CreateBucketCommand, ListObjectsV2Command, DeleteObjectsCommand, DeleteBucketCommand } = createRequire(new URL("../../server/package.json", import.meta.url))("@aws-sdk/client-s3");
+let storageClient: InstanceType<typeof S3Client> | undefined;
+const storageBucket = `rolling-supervisor-${randomUUID()}`;
+let storageCreated = false;
+let storageEnvironment: NodeJS.ProcessEnv = {};
 let outage: ControllerOutage;
 let upgrades: Awaited<ReturnType<typeof readUpgradeArtifacts>> = null;
 const repositoryRoot = path.resolve(import.meta.dirname, "../..");
@@ -37,7 +45,7 @@ const report: Record<string, unknown> = { schema: "paperclip.e2e.controller-repl
 let database: { connectionString: string; cleanup(): Promise<void> } | undefined;
 let db: ReturnType<typeof createDb> | undefined;
 const log = createWriteStream(path.join(temporaryRoot, "wrapper.log"), { mode: 0o600 });
-const stopper = createRunnerE2EServerStopper({ gracefulTimeoutMs: 35_000, forcedTimeoutMs: 5_000,
+const stopper = createRunnerE2EServerStopper({ gracefulTimeoutMs: rolling ? 100_000 : 35_000, forcedTimeoutMs: 5_000,
   hasSpawnError: () => false, markExpectedStop: () => {}, log: message => log.write(message) });
 let wrapper: ReturnType<typeof spawn> | undefined;
 let failure: unknown;
@@ -84,14 +92,27 @@ try {
   const config = paperclipConfigSchema.parse(JSON.parse(await readFile(configPath, "utf8")));
   config.database = { ...config.database, mode: "postgres", connectionString: database.connectionString };
   await writeFile(configPath, JSON.stringify(config), { mode: 0o600 });
-  const source = { PATH: process.env.PATH, HOME: home, TMPDIR: process.env.TMPDIR,
+  if (rolling) {
+    if (upgrades) throw new Error("Rolling supervisor smoke does not qualify upgrade artifacts");
+    const storage = JSON.parse(await readFile(process.env.PAPERCLIP_ROLLING_TEST_STORAGE_CONFIG!, "utf8"));
+    const endpoint = new URL(storage.endpoint);
+    if (endpoint.protocol !== "http:" || !["127.0.0.1", "localhost", "[::1]"].includes(endpoint.hostname)
+      || endpoint.username || endpoint.password) throw new Error("Supervisor smoke requires disposable loopback S3");
+    storageClient = new S3Client({ region: "us-east-1", endpoint: storage.endpoint, forcePathStyle: true,
+      credentials: { accessKeyId: storage.accessKeyId, secretAccessKey: storage.secretAccessKey } });
+    await storageClient.send(new CreateBucketCommand({ Bucket: storageBucket })); storageCreated = true;
+    storageEnvironment = { PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_BUCKET: storageBucket,
+      PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_ENDPOINT: storage.endpoint, PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_FORCE_PATH_STYLE: "true",
+      AWS_ACCESS_KEY_ID: storage.accessKeyId, AWS_SECRET_ACCESS_KEY: storage.secretAccessKey, AWS_EC2_METADATA_DISABLED: "true" };
+  }
+  const source = { ...storageEnvironment, PATH: process.env.PATH, HOME: home, TMPDIR: process.env.TMPDIR,
     ...(upgrades ? { ...upgrades.artifacts.A.pins, [upgradeArtifactKey]: process.env[upgradeArtifactKey] } : {}),
     NODE_ENV: "test", PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: configPath, PAPERCLIP_INSTANCE_ID: "default",
     PAPERCLIP_AGENT_JWT_SECRET: randomUUID(), PAPERCLIP_DECISION_SIGNING_SECRET: randomUUID(),
     PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: randomUUID(), BETTER_AUTH_SECRET: randomUUID(),
     PAPERCLIP_RUNNER_E2E_SERVER_LOG: path.join(temporaryRoot, "server.log"),
     PAPERCLIP_RUNNER_E2E_TEMP_ROOT: temporaryRoot, PAPERCLIP_RUNNER_E2E_PORT: String(port),
-    PAPERCLIP_RUNNER_E2E_EXECUTION_IDS: JSON.stringify(["everyday-workflows.runner-codex.daytona.recover-controller"]) };
+    PAPERCLIP_RUNNER_E2E_EXECUTION_IDS: JSON.stringify([`${rolling ? "rolling-web-deployment" : "everyday-workflows"}.runner-codex.daytona.recover-controller`]) };
   wrapper = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(repositoryRoot, path.join(import.meta.dirname, "server.ts")),
     { cwd: repositoryRoot, env: source, detached: process.platform !== "win32", stdio: ["ignore", "pipe", "pipe"] });
   wrapper.once("spawn", () => { report.wrapperStarted = true; });
@@ -121,12 +142,12 @@ try {
   const company = await api("/companies", { name: `Restart smoke ${randomUUID()}` });
   const [before] = await db.$client<{ started: string }[]>`select pg_postmaster_start_time()::text as started`;
   const receipts: unknown[] = [];
-  for (let index = 1; index < (upgrades?.sequence.length ?? 2); index += 1) {
+  for (let index = 1; index < (upgrades?.sequence.length ?? (rolling ? 3 : 2)); index += 1) {
     // Empty fixture cache exercises retirement without pretending a runner exists.
     const cache = path.join(home, "instances", "default", "runtime", "paperclip-runner");
     await mkdir(cache, { recursive: true });
     const requestId = `smoke-${index}-${randomUUID()}`;
-    await writeFile(`${control.restartRequestPath}.tmp`, JSON.stringify({ requestId, discardControllerCache: true, controllerOutage: outage }), { mode: 0o600 });
+    await writeFile(`${control.restartRequestPath}.tmp`, JSON.stringify({ requestId, discardControllerCache: true, ...(rolling ? { rollingDeployment: true } : { controllerOutage: outage }) }), { mode: 0o600 });
     await rename(`${control.restartRequestPath}.tmp`, control.restartRequestPath);
     const ack = await poll(async () => {
       const value = await readFile(control.restartAcknowledgementPath, "utf8").then(text => JSON.parse(text) as Record<string, unknown>)
@@ -134,11 +155,11 @@ try {
       if (value?.requestId === requestId && value.status === "failed") throw new Error(String(value.message));
       return value;
     }, value => value?.requestId === requestId && value.status === "ready", outage.holdMs + 180_000);
-    assert(validControllerOutageReceipt(ack!.controllerOutage, outage, requestId, upgrades ? {
+    assert(rolling ? validControllerRollingDeploymentReceipt(ack!.rollingDeployment, requestId) : validControllerOutageReceipt(ack!.controllerOutage, outage, requestId, upgrades ? {
       previous: upgrades.artifacts[upgrades.sequence[index - 1]!],
       replacement: upgrades.artifacts[upgrades.sequence[index]!],
     } : undefined), "Invalid replacement fault or requested artifact receipt");
-    receipts.push(ack!.controllerOutage);
+    receipts.push(rolling ? ack!.rollingDeployment : ack!.controllerOutage);
     report.replacements = receipts;
     await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, { mode: 0o600 });
     assert.equal((await api(`/companies/${company.id}`)).id, company.id);
@@ -158,6 +179,16 @@ try {
   if (wrapper) cleanups.push(() => stopper.stopAll());
   if (db) { const client = db.$client; cleanups.push(() => client.end()); }
   if (database) { const owned = database; cleanups.push(() => cleanupControllerSmokeDatabase(owned, ownedPostgresProcessId)); }
+  if (storageClient && storageCreated) cleanups.push(async () => {
+    let continuationToken: string | undefined;
+    do {
+      const listed = await storageClient!.send(new ListObjectsV2Command({ Bucket: storageBucket, ContinuationToken: continuationToken }));
+      if (listed.Contents?.length) await storageClient!.send(new DeleteObjectsCommand({ Bucket: storageBucket,
+        Delete: { Objects: listed.Contents.map((object: { Key: string }) => ({ Key: object.Key })) } }));
+      continuationToken = listed.NextContinuationToken;
+    } while (continuationToken);
+    await storageClient!.send(new DeleteBucketCommand({ Bucket: storageBucket })); storageClient!.destroy();
+  });
   for (const cleanup of cleanups) {
     try { await cleanup(); }
     catch (error) { failure ??= error; cleanupErrors.push(error instanceof Error ? error.message : String(error)); }

tests/runner-e2e/controller-replacement.ts

@@ -0,0 +1,7 @@
+import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js";
+import { validControllerRollingDeploymentReceipt, type ControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js";
+
+export type ControllerReplacementReceipt = ControllerOutageReceipt | ControllerRollingDeploymentReceipt;
+export function validControllerReplacementReceipt(value: unknown): value is ControllerReplacementReceipt {
+  return validControllerOutageReceipt(value) || validControllerRollingDeploymentReceipt(value);
+}

tests/runner-e2e/controller-rolling-deployment.test.ts

@@ -0,0 +1,58 @@
+import { describe, expect, it } from "vitest";
+import { rollController, validControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js";
+import { validControllerReplacementReceipt } from "./controller-replacement.js";
+import { controllerOutageForExecution, validControllerOutageReceipt } from "./controller-outage.js";
+import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
+
+function fixture() {
+  const order: string[] = [];
+  return { order, input: { requestId: "rolling-1", previousPid: 100,
+    start: async () => { order.push("start-B"); return { pid: 200 }; },
+    waitForReady: async () => { order.push("B-ready"); },
+    beforeHandoff: async () => { order.push("observe-A"); },
+    switchTraffic: () => { order.push("route-B"); },
+    stopPrevious: async () => { order.push("stop-A"); },
+    afterPreviousStopped: async () => { order.push("observe-A-stopped"); },
+    deletePreviousHome: async () => { order.push("delete-home-A"); },
+    afterHandoff: async () => { order.push("observe-recovery"); },
+    cancelled: () => false, record: async () => {},
+  } };
+}
+
+describe("rolling controller qualification", () => {
+  it("moves traffic only after readiness, then removes all predecessor state", async () => {
+    const f = fixture();
+    const receipt = await rollController(f.input);
+    expect(f.order).toEqual(["start-B", "B-ready", "observe-A", "route-B", "stop-A", "observe-A-stopped", "delete-home-A", "observe-recovery"]);
+    expect(validControllerRollingDeploymentReceipt(receipt)).toBe(false); // A process receipt alone is not availability proof.
+    receipt.probes = { reads: 10, writes: 3, verifiedWrites: 3, failures: [] };
+    expect(validControllerReplacementReceipt(receipt)).toBe(true);
+    expect(validControllerOutageReceipt(receipt)).toBe(false);
+    for (const changed of [
+      { previousHomeRemoved: false }, { cacheDiscarded: false }, { replacementPid: 100 }, { gracefulShutdown: false },
+      { trafficSwitchedAfterMs: -1 }, { readyAfterMs: receipt.stoppedAfterMs! + 1 },
+      { probes: { reads: 10, writes: 3, verifiedWrites: 2, failures: [] } },
+      { probes: { reads: 10, writes: 3, verifiedWrites: 3, failures: ["502"] } },
+    ]) expect(validControllerReplacementReceipt({ ...receipt, ...changed })).toBe(false);
+    expect(validControllerRollingDeploymentReceipt(receipt, "other-request")).toBe(false);
+  });
+  it("keeps A serving when B never becomes ready", async () => {
+    const f = fixture(); f.input.waitForReady = async () => { throw new Error("B unready"); };
+    await expect(rollController(f.input)).rejects.toThrow("B unready");
+    expect(f.order).toEqual(["start-B"]);
+  });
+  it("rejects forced predecessor cleanup before deleting its home or observing recovery", async () => {
+    const f = fixture(); f.input.stopPrevious = async () => { throw new Error("forced shutdown"); };
+    await expect(rollController(f.input)).rejects.toThrow("forced shutdown");
+    expect(f.order).toEqual(["start-B", "B-ready", "observe-A", "route-B"]);
+  });
+  it("keeps the original outage contract separate from the explicit rolling cell", () => {
+    const [rolling] = selectRunnerExecutions(parseRunnerSelectors(["--suite", "rolling-web-deployment"]));
+    expect(rolling).toBeDefined();
+    expect(rolling!.suite.manualOnly).toBe(true);
+    expect(rolling!.suite.definitionMetadata?.retainedControllerContinuation).toBe(true);
+    expect(controllerOutageForExecution(rolling!, {})).toBeUndefined();
+    const [outage] = selectRunnerExecutions(parseRunnerSelectors(["--suite", "remote-workspace-storage"]));
+    expect(controllerOutageForExecution(outage!, {})).toEqual({ signal: "SIGKILL", holdMs: 300_000 });
+  });
+});

tests/runner-e2e/controller-rolling-deployment.ts

@@ -0,0 +1,66 @@
+/** Test-owned routing/process fault. Recovery stays entirely in production code. */
+import type { MatrixExecution } from "./types.js";
+import type { ControllerOutageReceipt } from "./controller-outage.js";
+
+export const ROLLING_WEB_DEPLOYMENT_SUITE = "rolling-web-deployment";
+export const isRollingWebDeployment = (execution: Pick<MatrixExecution, "suite">) => execution.suite.id === ROLLING_WEB_DEPLOYMENT_SUITE;
+
+export interface ControllerRollingDeploymentReceipt extends Omit<ControllerOutageReceipt, "schema" | "holdMs" | "signal"> {
+  schema: "paperclip.e2e.controller-rolling-deployment/v1";
+  signal: "SIGTERM";
+  gracefulShutdown?: boolean;
+  trafficSwitchedAfterMs?: number;
+  homeDeletedAfterMs?: number;
+  previousHomeRemoved?: boolean;
+  probes?: { reads: number; writes: number; verifiedWrites: number; failures: string[] };
+}
+
+export function validControllerRollingDeploymentReceipt(value: unknown, requestId?: string): value is ControllerRollingDeploymentReceipt {
+  if (!value || typeof value !== "object") return false;
+  const r = value as ControllerRollingDeploymentReceipt;
+  const times = [r.replacementStartedAfterMs, r.readyAfterMs, r.trafficSwitchedAfterMs, r.stoppedAfterMs, r.homeDeletedAfterMs];
+  return r.schema === "paperclip.e2e.controller-rolling-deployment/v1" && r.signal === "SIGTERM"
+    && /^[A-Za-z0-9._:-]{1,200}$/.test(r.requestId) && (requestId === undefined || r.requestId === requestId)
+    && Number.isFinite(Date.parse(r.startedAt))
+    && Number.isSafeInteger(r.previousPid) && r.previousPid > 0
+    && Number.isSafeInteger(r.replacementPid) && r.replacementPid! > 0 && r.previousPid !== r.replacementPid
+    && r.cacheDiscarded === true && r.previousHomeRemoved === true && r.gracefulShutdown === true
+    && times.every(ms => typeof ms === "number" && Number.isFinite(ms) && ms >= 0)
+    && r.replacementStartedAfterMs! <= r.readyAfterMs! && r.readyAfterMs! <= r.trafficSwitchedAfterMs!
+    && r.trafficSwitchedAfterMs! <= r.stoppedAfterMs! && r.stoppedAfterMs! <= r.homeDeletedAfterMs!
+    && r.homeDeletedAfterMs! < 420_000
+    && Boolean(r.probes && Number.isSafeInteger(r.probes.reads) && r.probes.reads > 0
+      && Number.isSafeInteger(r.probes.writes) && r.probes.writes > 0 && r.probes.verifiedWrites === r.probes.writes
+      && Array.isArray(r.probes.failures) && r.probes.failures.length === 0);
+}
+
+export async function rollController<T extends { pid?: number }>(input: {
+  requestId: string; previousPid: number; start: () => Promise<T>;
+  waitForReady: (replacement: T) => Promise<void>; switchTraffic: (replacement: T) => void;
+  stopPrevious: () => Promise<void>; deletePreviousHome: () => Promise<void>;
+  beforeHandoff?: (receipt: ControllerRollingDeploymentReceipt) => Promise<void>;
+  afterPreviousStopped?: (receipt: ControllerRollingDeploymentReceipt) => Promise<void>;
+  afterHandoff?: (receipt: ControllerRollingDeploymentReceipt, record: () => Promise<void>) => Promise<void>;
+  cancelled: () => boolean; record: (receipt: ControllerRollingDeploymentReceipt) => Promise<void>;
+}): Promise<ControllerRollingDeploymentReceipt> {
+  const started = performance.now();
+  const receipt: ControllerRollingDeploymentReceipt = { schema: "paperclip.e2e.controller-rolling-deployment/v1",
+    requestId: input.requestId, previousPid: input.previousPid, startedAt: new Date().toISOString(), signal: "SIGTERM" };
+  const elapsed = () => performance.now() - started;
+  const check = () => { if (input.cancelled()) throw new Error("Wrapper shutdown interrupted the rolling deployment"); };
+  const record = () => input.record({ ...receipt });
+  check(); await record();
+  receipt.replacementStartedAfterMs = elapsed();
+  const replacement = await input.start(); receipt.replacementPid = replacement.pid; await record();
+  await input.waitForReady(replacement); check(); receipt.readyAfterMs = elapsed(); await record();
+  // Capture while A still owns its live lease. B may adopt immediately after
+  // SIGTERM, so an observation taken after A exits would miss that boundary.
+  await input.beforeHandoff?.(receipt); check();
+  input.switchTraffic(replacement); receipt.trafficSwitchedAfterMs = elapsed(); await record();
+  await input.stopPrevious(); receipt.gracefulShutdown = true; receipt.stoppedAfterMs = elapsed(); await record();
+  await input.afterPreviousStopped?.(receipt); check(); await record();
+  await input.deletePreviousHome(); receipt.homeDeletedAfterMs = elapsed();
+  receipt.previousHomeRemoved = true; receipt.cacheDiscarded = true; await record();
+  await input.afterHandoff?.(receipt, record);
+  return receipt;
+}

tests/runner-e2e/everyday-flow.ts

@@ -25,7 +25,9 @@ import { waitForTaskChatRendered, waitForTaskIdentity } from "./continuation-scr
 import { hasPersistedSource, isSavedSourceCheckpoint } from "./everyday-interruption.js";
 import { collectRunEvents } from "./run-observations.js";
 import { gradeRemoteRestart, observeRemoteRestartBoundary, projectRemoteRestartEvents, projectRemoteRestartRun, waitForRemoteRestartBoundary, type RemoteRestartEvidence } from "./everyday-remote-restart.js";
-import { controllerOutageForExecution, type ControllerOutageReceipt } from "./controller-outage.js";
+import { controllerOutageForExecution } from "./controller-outage.js";
+import type { ControllerReplacementReceipt } from "./controller-replacement.js";
+import { isRollingWebDeployment } from "./controller-rolling-deployment.js";
 import { gradeRemoteWorkspaceStorage, isRemoteWorkspaceStorageExecution, readRemoteWorkspaceStorageCheckpoint,
   REMOTE_WORKSPACE_PROOF_FILE, type RemoteWorkspaceStorageEvidence } from "./remote-workspace-storage.js";
 import { gradeRetainedControllerContinuation, retainedControllerRun, retainedControllerFollowupTransport, submitRetainedControllerComment, onePersistedRetainedControllerComment } from "./retained-controller-continuation.js";
@@ -102,7 +104,7 @@ interface Input {
   privateDir: string;
   deadlineAt: number;
   restartCount?: number;
-  restart(binding?: import("./controller-recovery-observer.js").RecoveryBinding, controllerDowntimeMs?: number): Promise<ControllerOutageReceipt | void>;
+  restart(binding?: import("./controller-recovery-observer.js").RecoveryBinding, controllerDowntimeMs?: number): Promise<ControllerReplacementReceipt | void>;
   observe(issue: StoryIssue, runs: StoryRun[]): void;
   capture(id: string, label: string, file: string): Promise<void>;
   evidence(name: string, value: unknown): Promise<void>;
@@ -182,7 +184,7 @@ export async function runEverydayFlow(input: Input) {
   let project = fixtures.project;
   const caseId = execution.task.id;
   const controllerOutage = controllerOutageForExecution(execution, process.env);
-  const requireRemoteRestart = controllerOutage !== undefined;
+  const requireRemoteRestart = controllerOutage !== undefined || isRollingWebDeployment(execution);
   const remoteWorkspaceStorage = isRemoteWorkspaceStorageExecution(execution);
   if (remoteWorkspaceStorage) ev.remoteWorkspaceStorage = { browserReadPhases: [] };
   const providerChoice = caseId === "provider-decline" || caseId === "provider-second";
@@ -610,6 +612,9 @@ export async function runEverydayFlow(input: Input) {
       "everyday-remote-restart.ts",
       "retained-controller-continuation.ts",
       "controller-outage.ts",
+      "controller-rolling-deployment.ts",
+      "controller-replacement.ts",
+      "rolling-deployment-supervisor.ts",
       "controller-recovery-observer.ts",
       "upgrade-artifacts.ts",
       "server.ts",

tests/runner-e2e/everyday-remote-restart.ts

@@ -3,7 +3,7 @@ import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
 import { isValidNativePrpEnvelope } from "./native-event-envelope.js";
 import type { StoryCheck, StoryRun } from "./everyday-observations.js";
 import { pollUntil } from "./api.js";
-import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js";
+import { validControllerReplacementReceipt, type ControllerReplacementReceipt } from "./controller-replacement.js";
 import { validControllerRecoveryReceipt } from "./controller-recovery-observer.js";
 
 type Row = Record<string, unknown>;
@@ -26,8 +26,8 @@ export interface RemoteRestartBoundary {
 }
 export interface RemoteRestartEvidence {
   expectedReplacements?: number;
-  replacements?: Array<{ before: RemoteRestartBoundary; after: RemoteRestartBoundary; outage: ControllerOutageReceipt }>;
-  outage?: ControllerOutageReceipt;
+  replacements?: Array<{ before: RemoteRestartBoundary; after: RemoteRestartBoundary; outage: ControllerReplacementReceipt }>;
+  outage?: ControllerReplacementReceipt;
   before?: RemoteRestartBoundary;
   after?: RemoteRestartBoundary;
   final?: { run: StoryRun; events: RemoteRestartEvent[]; operations: Row[];
@@ -194,7 +194,7 @@ export function gradeRemoteRestart(evidence: RemoteRestartEvidence | undefined):
       const preservesIdentity = digest(step.before.identity) === digest(step.after.identity)
         && digest(step.before.lease) === digest(step.after.lease)
         && retainsPrefix(step.before.events, step.after.events);
-      const authenticated = validControllerOutageReceipt(step.outage) && validControllerRecoveryReceipt(step.outage)
+      const authenticated = validControllerReplacementReceipt(step.outage) && validControllerRecoveryReceipt(step.outage)
         && step.outage.recovery?.binding.runId === step.before.identity.runId
         && step.outage.recovery.binding.companyId === step.before.identity.companyId
         && step.outage.recovery.binding.issueId === step.before.identity.issueId
@@ -210,7 +210,7 @@ export function gradeRemoteRestart(evidence: RemoteRestartEvidence | undefined):
     check("remote-restart-same-execution-and-lease", sameIdentity, "The original run, native session, admitted runner identity and sandbox lease survive replacement."),
     check("remote-restart-native-completion-once", nativeCompletion, "The original provider turn completes once and retains its native receipt prefix."),
     check("remote-restart-files-finalized", Boolean(saved), "The original run commits its workspace receipt and leaves no active operation or recovery."),
-    check("remote-restart-fault-exercised", validControllerOutageReceipt(evidence?.outage), "The main server exited, its cache was retired, and a distinct replacement became ready after the requested outage."),
+    check("remote-restart-fault-exercised", validControllerReplacementReceipt(evidence?.outage), "The requested outage or ready-before-handoff replacement occurred, and the departing server state is unavailable."),
     check("remote-restart-authenticated-recovery", validControllerRecoveryReceipt(evidence?.outage)
       && evidence?.outage?.recovery?.binding.runId === identity?.runId
       && evidence?.outage?.recovery?.runnerInstanceId === identity?.runnerInstanceId

tests/runner-e2e/live-fixtures.ts

@@ -218,7 +218,7 @@ export async function setupLiveFixtures(input: {
           : ["native-completion", "native-instruction-consolidation", "native-connection-guidance"].includes(execution.suite.id)
           || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id)) ? NATIVE_COMPLETION_BUDGET_CENTS
           : execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS
-          : ["stock-harness", "task-workspaces", "case-handoffs", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage"].includes(execution.suite.id) ? 1_000 : 0,
+          : ["stock-harness", "task-workspaces", "case-handoffs", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage", "rolling-web-deployment"].includes(execution.suite.id) ? 1_000 : 0,
       });
       await input.onCompanyCreated?.(company.id);
       return company;
@@ -413,7 +413,7 @@ export async function setupLiveFixtures(input: {
           setupProbe.defaultRunner = selected.defaultRunner;
         }
       }
-      if (["stock-harness", "native-connection-guidance", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage"].includes(execution.suite.id)
+      if (["stock-harness", "native-connection-guidance", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage", "rolling-web-deployment"].includes(execution.suite.id)
         || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id))) {
         agent.budgetMonthlyCents = 1_000;
       }

tests/runner-e2e/playwright.config.ts

@@ -72,7 +72,9 @@ export default defineConfig({
     // blob reports. The wrapper inherits the test process and strips provider
     // keys before spawning the real Paperclip process.
     command: runnerE2EWebServerCommand(repositoryRoot),
-    gracefulShutdown: runnerE2EWebServerGracefulShutdown,
+    gracefulShutdown: { ...runnerE2EWebServerGracefulShutdown,
+      ...(JSON.parse(process.env.PAPERCLIP_RUNNER_E2E_EXECUTION_IDS ?? "[]").some((id: string) => id.startsWith("rolling-web-deployment.")) ? { timeout: 105_000 } : {}),
+    },
     url: `${baseURL}/api/health`,
     reuseExistingServer: false,
     // Fresh databases apply the complete migration history before health is ready.

tests/runner-e2e/remote-workspace-storage.ts

@@ -1,3 +1,4 @@
+import { isRollingWebDeployment } from "./controller-rolling-deployment.js";
 import { createHash } from "node:crypto";
 import { remoteWorkspaceAuthoritySchema, remoteWorkspaceRunReferenceSchema } from "../../packages/shared/src/remote-workspace.js";
 import type { ExecutionWorkspace, WorkspaceFileContent } from "../../packages/shared/src/index.js";
@@ -32,7 +33,7 @@ export interface RemoteWorkspaceStorageEvidence {
 }
 
 export function isRemoteWorkspaceStorageExecution(execution: MatrixExecution): boolean {
-  return execution.suite.id === REMOTE_WORKSPACE_STORAGE_SUITE;
+  return execution.suite.id === REMOTE_WORKSPACE_STORAGE_SUITE || isRollingWebDeployment(execution);
 }
 
 export function remoteWorkspaceStoragePrompt(base: string): string {

tests/runner-e2e/retained-controller-continuation.ts

@@ -2,11 +2,11 @@ import type { RunnerApi } from "./api.js";
 import type { MatrixExecution } from "./types.js";
 import { createHash } from "node:crypto";
 import type { StoryCheck, StoryRun } from "./everyday-observations.js";
-import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js";
+import { validControllerReplacementReceipt, type ControllerReplacementReceipt } from "./controller-replacement.js";
 
 /** Only these explicit recovery definitions change the post-replacement user entry point. */
 export function retainedControllerFollowupTransport(execution: Pick<MatrixExecution, "suite" | "task">): "browser" | "task_comment_api" {
-  return ["daytona-restart-continuity", "remote-workspace-storage"].includes(execution.suite.id) && execution.task.id === "recover-controller"
+  return ["daytona-restart-continuity", "remote-workspace-storage", "rolling-web-deployment"].includes(execution.suite.id) && execution.task.id === "recover-controller"
     && execution.suite.definitionMetadata?.retainedControllerContinuation === true
     && execution.suite.definitionMetadata.retainedControllerFollowupTransport === "task_comment_api"
     ? "task_comment_api" : "browser";
@@ -37,13 +37,13 @@ export function retainedControllerRun(run: StoryRun) {
 export function gradeRetainedControllerContinuation(input: {
   before: ReturnType<typeof retainedControllerRun>;
   after: ReturnType<typeof retainedControllerRun>;
-  outage: ControllerOutageReceipt | undefined;
+  outage: ControllerReplacementReceipt | undefined;
   requestedAt: string;
   completedTool: boolean;
   usefulReply: boolean;
 }): StoryCheck[] {
   const { before, after, outage } = input;
-  const passed = validControllerOutageReceipt(outage)
+  const passed = validControllerReplacementReceipt(outage)
     && before.status === "succeeded" && after.status === "succeeded"
     && before.runtimeMode === "native" && after.runtimeMode === "native"
     && before.runId !== after.runId && before.companyId === after.companyId

tests/runner-e2e/rolling-deployment-process.ts

@@ -45,6 +45,9 @@ export async function startRollingDeploymentProcess(input: {
     await cp(path.join(input.repositoryRoot, "server/dist"), path.join(appRoot, "dist"), { recursive: true });
     await cp(path.join(input.repositoryRoot, "ui/dist"), path.join(appRoot, "ui-dist"), { recursive: true });
     await symlink(path.join(input.repositoryRoot, "server/node_modules"), path.join(appRoot, "node_modules"));
+    // This lane uses workspace dependencies, whose exports include TypeScript.
+    // Preserve the normal development plugin loader beside each compiled copy.
+    await symlink(path.join(input.repositoryRoot, "cli"), path.join(home, "app/cli"));
     await writeFile(path.join(appRoot, "package.json"), '{"type":"module"}');
     for (const omittedAsset of input.browser.omitAssets ?? []) {
       if (!/^[A-Za-z0-9_.-]+\.js$/.test(omittedAsset)) throw new Error("Unsafe omitted asset fixture");
@@ -55,12 +58,12 @@ export async function startRollingDeploymentProcess(input: {
   const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(input.repositoryRoot, entry), {
     cwd: input.repositoryRoot,
     env: {
-      PATH: process.env.PATH, HOME: home, TMPDIR: temporary,
+      ...input.sharedEnvironment, PATH: input.sharedEnvironment.PATH ?? process.env.PATH, HOME: home, TMPDIR: temporary, XDG_CACHE_HOME: path.join(home, "cache"),
       NODE_ENV: "test", PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: configPath,
-      PAPERCLIP_INSTANCE_ID: "default", DATABASE_URL: input.databaseUrl,
+      PORT: String(port), PAPERCLIP_INSTANCE_ID: "default", DATABASE_URL: input.databaseUrl,
       PAPERCLIP_API_URL: input.ingressOrigin, PAPERCLIP_TELEMETRY_ENABLED: "false",
       PAPERCLIP_DISABLE_TELEMETRY: "true", PAPERCLIP_LOG_LEVEL: "warn",
-      HEARTBEAT_SCHEDULER_ENABLED: "false", ...input.sharedEnvironment,
+      HEARTBEAT_SCHEDULER_ENABLED: input.sharedEnvironment.HEARTBEAT_SCHEDULER_ENABLED ?? "false",
     },
     detached: process.platform !== "win32", stdio: ["ignore", "pipe", "pipe"],
   });
@@ -70,7 +73,7 @@ export async function startRollingDeploymentProcess(input: {
   const origin = `http://127.0.0.1:${port}`;
   let stopped = false;
   return {
-    child, home, origin,
+    child, home, configPath, origin,
     async waitForReady() {
       const started = performance.now();
       while (performance.now() - started < 180_000) {
@@ -84,11 +87,13 @@ export async function startRollingDeploymentProcess(input: {
       }
       throw new Error(`Server ${input.name} did not become ready; inspect ${input.name}.log`);
     },
-    async stop() {
-      if (stopped) return;
-      await stopper.stopAll();
-      await new Promise<void>(resolve => log.end(resolve));
-      stopped = true;
+    async stop(options: { requireGraceful?: boolean } = {}) {
+      if (!stopped) {
+        await stopper.stopAll();
+        await new Promise<void>(resolve => log.end(resolve));
+        stopped = true;
+      }
+      if (options.requireGraceful) stopper.assertGracefulExit(child);
     },
   };
 }

tests/runner-e2e/rolling-deployment-proxy.ts

@@ -4,7 +4,7 @@ import type { Socket } from "node:net";
 
 /** Test-owned ingress. Each request keeps the backend selected at admission;
  * switching traffic never retries a request or terminates its existing stream. */
-export async function startRollingDeploymentProxy() {
+export async function startRollingDeploymentProxy(port = 0) {
   let target: URL | undefined;
   const agent = new Agent({ keepAlive: true });
   const sockets = new Set<Socket>();
@@ -56,7 +56,7 @@ export async function startRollingDeploymentProxy() {
     upstream.on("error", () => socket.destroy());
     upstream.end();
   });
-  server.listen(0, "127.0.0.1");
+  server.listen(port, "127.0.0.1");
   await once(server, "listening");
   const address = server.address();
   if (!address || typeof address === "string") throw new Error("Proxy address unavailable");

tests/runner-e2e/rolling-deployment-smoke.ts

@@ -150,6 +150,15 @@ try {
   })();
   if (browserEnabled) browser = await openRollingDeploymentBrowser({ origin: proxy.origin, root, taskTitle: task.title,
     taskPath: `/issues/${task.id}`, omittedAssets });
+  await check("Unready replacement leaves the healthy server serving", async () => {
+    const unready = await startRollingDeploymentProcess({ root, name: "unready", repositoryRoot,
+      databaseUrl: database!.connectionString, ingressOrigin: proxy!.origin,
+      sharedEnvironment: { ...sharedEnvironment, PAPERCLIP_EPHEMERAL_SERVER: "true", PAPERCLIP_DECISION_SIGNING_SECRET: "" } });
+    processes.push(unready);
+    await assert.rejects(unready.waitForReady(), /exited before readiness/);
+    assert.equal((await api(proxy!.origin, `/issues/${task.id}`)).id, task.id);
+    assert.equal((await api(a.origin, `/issues/${task.id}`)).id, task.id);
+  });
   const b = await start("B");
   await check("B reads acknowledged task while A is alive", async () => {
     assert.equal((await api(b.origin, `/issues/${task.id}`)).title, task.title);
@@ -185,7 +194,7 @@ try {
       while (storageProxy!.heldUploads < 1 && Date.now() < deadline) await delay(25);
       assert.equal(storageProxy!.heldUploads, 1);
       proxy!.routeTo(b.origin);
-      stopping = a.stop();
+      stopping = a.stop({ requireGraceful: true });
       await delay(100);
       assert.equal(acknowledged, false);
       storageProxy!.resumeUploads();
@@ -198,7 +207,7 @@ try {
   });
   proxy.routeTo(b.origin);
   await delay(500);
-  await a.stop();
+  await a.stop({ requireGraceful: true });
   await rm(a.home, { recursive: true, force: true });
   await delay(1_000);
   report.handoffElapsedMs = Math.round(performance.now() - switchedAt);
@@ -261,7 +270,7 @@ try {
       if (fixture.seeded) assert.equal((await api(b.origin, `${route}?path=AGENTS.md`)).content, "Durable original");
     }
   });
-  await b.stop(); await rm(b.home, { recursive: true, force: true });
+  await b.stop({ requireGraceful: true }); await rm(b.home, { recursive: true, force: true });
   const c = await start("C", false); proxy.routeTo(c.origin);
   await check("Fresh C restores files after both homes are deleted, even with adoption opt-in disabled", async () => {
     assert.equal((await api(c.origin, `${fileRoute}?path=reference.txt`)).content, content);

tests/runner-e2e/rolling-deployment-supervisor.test.ts

@@ -0,0 +1,42 @@
+import { afterEach, expect, it, vi } from "vitest";
+import { createLocalEncryptedCipher } from "../../server/src/secrets/local-encrypted-provider.js";
+
+const mocks = vi.hoisted(() => ({
+  start: vi.fn(), observer: vi.fn(), stop: vi.fn(), close: vi.fn(), writeFile: vi.fn(),
+}));
+vi.mock("node:fs/promises", () => ({ mkdir: async () => {}, writeFile: mocks.writeFile,
+  readFile: async () => JSON.stringify({ database: { mode: "postgres", connectionString: "postgres://fixture" },
+    secrets: { localEncrypted: { keyFilePath: "/fixture/secrets/master.key" } } }) }));
+vi.mock("../../packages/shared/src/config-schema.js", () => ({ paperclipConfigSchema: { parse: (value: unknown) => value } }));
+vi.mock("./rolling-deployment-process.js", () => ({ startRollingDeploymentProcess: mocks.start }));
+vi.mock("./rolling-deployment-proxy.js", () => ({ startRollingDeploymentProxy: async () => ({ origin: "http://127.0.0.1:1", routeTo() {}, close: mocks.close }) }));
+vi.mock("./controller-recovery-observer.js", () => ({ createControllerRecoveryObserver: mocks.observer }));
+vi.mock("./server-restart-request.js", () => ({ readRestartRequest: async () => ({ requestId: "key-check", rollingDeployment: true, recoveryBinding: {} }) }));
+import { superviseRollingDeployment } from "./rolling-deployment-supervisor.js";
+
+afterEach(() => { vi.unstubAllEnvs(); vi.clearAllMocks(); });
+
+it.each([undefined, "prior-disposable-key"])("shares the server key with checkpoint observation and restores %s on failure", async prior => {
+  vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", prior);
+  let material: ReturnType<ReturnType<typeof createLocalEncryptedCipher>["encrypt"]>;
+  mocks.start.mockImplementation(async ({ sharedEnvironment }) => {
+    expect(process.env.PAPERCLIP_SECRETS_MASTER_KEY).toBe(sharedEnvironment.PAPERCLIP_SECRETS_MASTER_KEY);
+    expect(mocks.writeFile).toHaveBeenCalledWith("/fixture/secrets/master.key", sharedEnvironment.PAPERCLIP_SECRETS_MASTER_KEY,
+      { mode: 0o600, flag: "wx" });
+    material = createLocalEncryptedCipher("write").encrypt("synthetic checkpoint");
+    return { child: { exitCode: null, signalCode: null }, configPath: "/fixture/config.json",
+      origin: "http://127.0.0.1:2", waitForReady: async () => {}, stop: mocks.stop };
+  });
+  mocks.observer.mockImplementation(async () => {
+    expect(createLocalEncryptedCipher("read").decrypt(material)).toBe("synthetic checkpoint");
+    throw new Error("synthetic observer failure");
+  });
+  await expect(superviseRollingDeployment({ temporaryRoot: "/fixture", repositoryRoot: "/fixture",
+    configPath: "/fixture/config.json", port: 1, executionIds: ["rolling"], cancelled: () => false,
+    environment: { PAPERCLIP_STORAGE_PROVIDER: "s3" },
+  })).rejects.toThrow("synthetic observer failure");
+  expect(mocks.observer).toHaveBeenCalledOnce();
+  expect(mocks.stop).toHaveBeenCalledOnce();
+  expect(mocks.close).toHaveBeenCalledOnce();
+  expect(process.env.PAPERCLIP_SECRETS_MASTER_KEY).toBe(prior);
+});

tests/runner-e2e/rolling-deployment-supervisor.ts

@@ -0,0 +1,135 @@
+import { randomBytes, randomUUID } from "node:crypto";
+import { mkdir, readFile, rename, rm, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { setTimeout as delay } from "node:timers/promises";
+import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js";
+import { createControllerRecoveryObserver } from "./controller-recovery-observer.js";
+import { rollController, validControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js";
+import { readRestartRequest } from "./server-restart-request.js";
+import { startRollingDeploymentProcess } from "./rolling-deployment-process.js";
+import { startRollingDeploymentProxy } from "./rolling-deployment-proxy.js";
+import { runnerE2EServerControlPaths } from "./harness-env.js";
+
+type Server = Awaited<ReturnType<typeof startRollingDeploymentProcess>>;
+
+/** Isolated Product E2E supervisor. Shares only Postgres, S3 and generated
+ * secrets. It never patches leases, checkpoints, execution or recovery state. */
+export async function superviseRollingDeployment(input: {
+  temporaryRoot: string; repositoryRoot: string; configPath: string; port: number;
+  environment: NodeJS.ProcessEnv; executionIds: string[]; cancelled: () => boolean;
+}) {
+  const config = paperclipConfigSchema.parse(JSON.parse(await readFile(input.configPath, "utf8")));
+  if (config.database.mode !== "postgres" || !config.database.connectionString) throw new Error("Rolling qualification requires independently owned Postgres");
+  if (input.environment.PAPERCLIP_STORAGE_PROVIDER !== "s3") throw new Error("Rolling qualification requires disposable S3 storage");
+  const databaseUrl = config.database.connectionString;
+  const controls = runnerE2EServerControlPaths(input.temporaryRoot);
+  const proxy = await startRollingDeploymentProxy(input.port);
+  const servers: Server[] = [];
+  const environment = { ...input.environment,
+    PAPERCLIP_EPHEMERAL_SERVER: "true", PAPERCLIP_DURABLE_AGENT_FILES: "true", PAPERCLIP_SHARED_UI_ASSETS: "true",
+    PAPERCLIP_DATABASE_LISTEN_URL: databaseUrl, PAPERCLIP_SECRETS_MASTER_KEY: randomBytes(32).toString("base64"),
+    HEARTBEAT_SCHEDULER_ENABLED: "true", RUN_LOG_S3_BUCKET: input.environment.PAPERCLIP_STORAGE_S3_BUCKET,
+    RUN_LOG_S3_ENDPOINT: input.environment.PAPERCLIP_STORAGE_S3_ENDPOINT,
+    RUN_LOG_S3_REGION: input.environment.PAPERCLIP_STORAGE_S3_REGION,
+    RUN_LOG_S3_FORCE_PATH_STYLE: input.environment.PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE,
+  };
+  const start = async () => {
+    const server = await startRollingDeploymentProcess({ root: input.temporaryRoot, name: `controller-${servers.length + 1}`,
+      repositoryRoot: input.repositoryRoot, databaseUrl, ingressOrigin: proxy.origin, sharedEnvironment: environment, browser: {} });
+    servers.push(server);
+    return server;
+  };
+  const json = async (filename: string, value: unknown) => {
+    await writeFile(`${filename}.tmp`, JSON.stringify(value), { mode: 0o600 }); await rename(`${filename}.tmp`, filename);
+  };
+  const api = async <T>(route: string, method = "GET", body?: object): Promise<T> => {
+    const response = await fetch(`${proxy.origin}/api${route}`, { method,
+      headers: { "content-type": "application/json", origin: proxy.origin },
+      body: body ? JSON.stringify(body) : undefined, signal: AbortSignal.timeout(10_000) });
+    if (!response.ok) throw new Error(`${method} ${route} returned ${response.status}`);
+    return await response.json() as T;
+  };
+  let current: Server | undefined, lastRequest: string | undefined, probeIssueId: string | undefined;
+  // The read-only observer uses the production checkpoint loader in this
+  // isolated supervisor process. Give it the same disposable key as A/B/C.
+  const previousMasterKey = process.env.PAPERCLIP_SECRETS_MASTER_KEY;
+  process.env.PAPERCLIP_SECRETS_MASTER_KEY = environment.PAPERCLIP_SECRETS_MASTER_KEY;
+  try {
+    // Keep the fixture's isolation evidence for its one generated stable secret.
+    // It is outside A/B/C homes, like the independent database and object store;
+    // each server receives this exact value through its environment.
+    const keyPath = path.resolve(config.secrets.localEncrypted.keyFilePath);
+    if (!keyPath.startsWith(`${path.resolve(input.temporaryRoot)}${path.sep}`)) throw new Error("Rolling fixture key escaped its owned root");
+    await mkdir(path.dirname(keyPath), { recursive: true, mode: 0o700 });
+    await writeFile(keyPath, environment.PAPERCLIP_SECRETS_MASTER_KEY, { mode: 0o600, flag: "wx" });
+    current = await start(); await current.waitForReady(); proxy.routeTo(current.origin);
+    while (!input.cancelled()) {
+      if (current.child.exitCode !== null || current.child.signalCode !== null) throw new Error("Serving rolling controller exited unexpectedly");
+      const request = await readRestartRequest(controls.restartRequestPath, input.executionIds);
+      if (!request || request.requestId === lastRequest) { await delay(100); continue; }
+      lastRequest = request.requestId;
+      if (!request.rollingDeployment) throw new Error("Rolling supervisor requires an explicit rolling deployment request");
+      const previous = current;
+      const observer = request.recoveryBinding ? await createControllerRecoveryObserver(previous.configPath, request.recoveryBinding) : undefined;
+      let probing = false, probes: Promise<void> | undefined;
+      const stats = { reads: 0, writes: 0, verifiedWrites: 0, failures: [] as string[] };
+      const writes: Array<{ id: string; body: string }> = [];
+      try {
+        if (!probeIssueId) {
+          const companyId = request.recoveryBinding?.companyId ?? (await api<{ id: string }>("/companies", "POST", { name: "Synthetic deployment probes" })).id;
+          const issue = await api<{ id: string }>(`/companies/${companyId}/issues`, "POST", {
+            title: "Synthetic deployment read/save probe", description: "Unassigned fixture for ordinary user saves during deployment.", status: "todo",
+          });
+          probeIssueId = issue.id;
+        }
+        const issueRoute = `/issues/${probeIssueId}`;
+        probing = true;
+        probes = (async () => {
+          while (probing) {
+            try { await api(issueRoute); stats.reads++; } catch (error) { stats.failures.push(String(error)); }
+            if (stats.reads % 10 === 0 || stats.writes === 0) {
+              const body = `Synthetic deployment save ${randomUUID()}`;
+              try {
+                const saved = await api<{ id: string }>(`${issueRoute}/comments`, "POST", { body, clientRequestId: randomUUID() });
+                writes.push({ id: saved.id, body }); stats.writes++;
+              } catch (error) { stats.failures.push(String(error)); }
+            }
+            await delay(100);
+          }
+        })();
+        const receiptPath = path.join(controls.controlDirectory, `controller-rolling-${request.requestId}.json`);
+        const receipt = await rollController({ requestId: request.requestId, previousPid: previous.child.pid!,
+          start: async () => { const server = await start(); return { pid: server.child.pid, server }; },
+          waitForReady: async replacement => { await replacement.server.waitForReady(); },
+          switchTraffic: replacement => { current = replacement.server; proxy.routeTo(current.origin); },
+          stopPrevious: () => previous.stop({ requireGraceful: true }),
+          deletePreviousHome: () => rm(previous.home, { recursive: true }),
+          beforeHandoff: observer?.beforeHandoff,
+          afterPreviousStopped: observer?.stoppedAfterHandoff,
+          afterHandoff: observer ? (receipt, record) => observer.recovered(receipt, record, input.cancelled) : undefined,
+          cancelled: input.cancelled, record: value => json(receiptPath, value),
+        });
+        probing = false; await probes;
+        const saved = await api<Array<{ id: string; body: string }>>(`${issueRoute}/comments`);
+        stats.verifiedWrites = writes.filter(write => saved.filter(comment => comment.id === write.id && comment.body === write.body).length === 1).length;
+        receipt.probes = stats; await json(receiptPath, receipt);
+        if (!validControllerRollingDeploymentReceipt(receipt, request.requestId)) throw new Error("Rolling boundary failed overlap, empty-home, or ordinary read/save qualification");
+        await json(controls.restartAcknowledgementPath, { requestId: request.requestId, status: "ready", controllerCacheDiscarded: true,
+          completedAt: new Date().toISOString(), rollingDeployment: receipt });
+      } catch (error) {
+        await json(controls.restartAcknowledgementPath, { requestId: request.requestId, status: "failed", message: String(error) });
+        throw error;
+      } finally { probing = false; await probes; await observer?.close(); }
+    }
+  } finally {
+    // Every started server remains owned, including an unready successor.
+    // End-of-attempt cleanup also runs after a retained failing receipt.
+    try {
+      try { await Promise.all(servers.map(server => server.stop())); }
+      finally { await proxy.close(); }
+    } finally {
+      if (previousMasterKey === undefined) delete process.env.PAPERCLIP_SECRETS_MASTER_KEY;
+      else process.env.PAPERCLIP_SECRETS_MASTER_KEY = previousMasterKey;
+    }
+  }
+}

tests/runner-e2e/runner.spec.ts

@@ -1,3 +1,5 @@
+import { isRollingWebDeployment, validControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js";
+import type { ControllerReplacementReceipt } from "./controller-replacement.js";
 import { kimiCandidateAdmissionForExecution } from "./server-restart-request.js";
 import { runPlanRevisionAcceptance, isPendingPlanConfirmation, normalizePlanMarkdown, expectPlanStageVisible } from "./plan-revision-flow.js";
 import { createWorkerTiming } from "./worker-timing.js";
@@ -273,8 +275,9 @@ async function restartIsolatedPaperclipServer(input: {
   crashAfterResult?: FinalizationCrashBinding;
   controllerDowntimeMs?: number;
   controllerOutage?: ControllerOutage;
+  rollingDeployment?: true;
   recoveryBinding?: import("./controller-recovery-observer.js").RecoveryBinding;
-  onControllerOutage?: (receipt: ControllerOutageReceipt) => Promise<void>;
+  onControllerOutage?: (receipt: ControllerReplacementReceipt) => Promise<void>;
 }): Promise<void> {
   const {
     controlDirectory,
@@ -287,6 +290,7 @@ async function restartIsolatedPaperclipServer(input: {
     temporaryRequestPath,
     JSON.stringify({ requestId: input.requestId, ...(input.kimiCandidateAdmission ? { kimiCandidateAdmission: input.kimiCandidateAdmission } : {}), ...(input.controllerDowntimeMs !== undefined && !input.controllerOutage ? { controllerDowntimeMs: input.controllerDowntimeMs } : {}), discardControllerCache: input.discardControllerCache === true,
       ...(input.controllerOutage ? { controllerOutage: input.controllerOutage } : {}),
+      ...(input.rollingDeployment ? { rollingDeployment: true } : {}),
       ...(input.recoveryBinding ? { recoveryBinding: input.recoveryBinding } : {}),
       ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}),
       ...(input.crashAfterResult ? { crashAfterResult: input.crashAfterResult } : {}) }),
@@ -312,6 +316,7 @@ async function restartIsolatedPaperclipServer(input: {
       acknowledgement.requestId === input.requestId &&
       acknowledgement.status === "ready" &&
       (!input.discardControllerCache || acknowledgement.controllerCacheDiscarded === true) &&
+      (!input.rollingDeployment || validControllerRollingDeploymentReceipt(acknowledgement.rollingDeployment, input.requestId)) &&
       (!input.controllerOutage || validControllerOutageReceipt(acknowledgement.controllerOutage, input.controllerOutage, input.requestId)),
     reject: (acknowledgement) =>
       acknowledgement.requestId === input.requestId &&
@@ -327,7 +332,7 @@ async function restartIsolatedPaperclipServer(input: {
     load: () => input.api.get<Record<string, unknown>>("/api/health"),
     accept: (health) => Boolean(health),
   });
-  if (input.controllerOutage) await input.onControllerOutage?.(acknowledgement.controllerOutage as ControllerOutageReceipt);
+  if (input.controllerOutage || input.rollingDeployment) await input.onControllerOutage?.((input.rollingDeployment ? acknowledgement.rollingDeployment : acknowledgement.controllerOutage) as ControllerReplacementReceipt);
 }
 
 const executionIds = (() => {
@@ -1256,13 +1261,14 @@ for (const execution of executions) {
           restartCount: upgradeManifest ? upgradeManifest.sequence.length -
             (execution.suite.definitionMetadata?.retainedControllerContinuation === true ? 2 : 1) : 1,
           restart: async (recoveryBinding, controllerDowntimeMs) => {
-            let receipt: ControllerOutageReceipt | undefined;
+            let receipt: ControllerReplacementReceipt | undefined;
             restartIndex += 1;
             await restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}-${restartIndex}`, deadlineAt: startedAtMs + deadlineMs,
-              discardControllerCache: controllerOutage !== undefined,
+              discardControllerCache: controllerOutage !== undefined || isRollingWebDeployment(execution),
+              ...(isRollingWebDeployment(execution) ? { rollingDeployment: true as const } : {}),
               controllerOutage: controllerOutage && !recoveryBinding ? { ...controllerOutage, holdMs: 1000 } : controllerOutage,
-              controllerDowntimeMs,
-              ...(controllerOutage ? { recoveryBinding } : {}),
+              ...(!isRollingWebDeployment(execution) ? { controllerDowntimeMs } : {}),
+              ...((controllerOutage || isRollingWebDeployment(execution)) ? { recoveryBinding } : {}),
               onControllerOutage: async value => {
                 receipt = value;
                 await writeSanitizedJson(snapshotsDir, `controller-outage-${restartIndex}.json`, value, secrets);

tests/runner-e2e/server-restart-request.ts

@@ -1,3 +1,4 @@
+import { isRollingWebDeployment } from "./controller-rolling-deployment.js";
 import { readFile } from "node:fs/promises";
 import { runnerMatrix } from "./catalog.js";
 import { parseAcpxCandidateAdmission, type AcpxCandidateAdmission } from "../../packages/paperclip-runner/src/drivers/acpx/candidate-admission.js";
@@ -17,6 +18,7 @@ export interface RestartRequest {
   discardControllerCache?: boolean;
   crashAfterResult?: FinalizationCrashBinding;
   controllerOutage?: ControllerOutage;
+  rollingDeployment?: true;
   recoveryBinding?: RecoveryBinding;
 }
 
@@ -87,22 +89,29 @@ export async function readRestartRequest(restartRequestPath: string, executionId
     || Number(controllerDowntimeMs) < 0 || Number(controllerDowntimeMs) > 300_000)) {
     throw new Error("Controller downtime must be an integer between zero and five minutes");
   }
+  const rollingDeployment = (value as { rollingDeployment?: unknown }).rollingDeployment;
+  if (rollingDeployment !== undefined && (rollingDeployment !== true || executionIds.length !== 1
+    || !executionIds.every(id => { const execution = runnerMatrix.find(row => row.id === id); return execution && isRollingWebDeployment(execution); }))) {
+    throw new Error("Rolling deployment requires its explicit isolated cell");
+  }
   const discardControllerCache = (value as { discardControllerCache?: unknown }).discardControllerCache === true;
   if (discardControllerCache && !executionIds.some(id => {
     const execution = runnerMatrix.find(row => row.id === id);
-    return execution !== undefined && controllerOutageForExecution(execution, {}) !== undefined;
+    return execution !== undefined && (controllerOutageForExecution(execution, {}) !== undefined || isRollingWebDeployment(execution));
   })) throw new Error("Controller cache replacement requires an explicit remote controller-outage cell");
   const crashAfterResult = (value as { crashAfterResult?: unknown }).crashAfterResult;
   const controllerOutage = (value as { controllerOutage?: unknown }).controllerOutage;
   const recoveryBinding = (value as { recoveryBinding?: unknown }).recoveryBinding;
-  if (recoveryBinding !== undefined && controllerOutage === undefined) throw new Error("Recovery observation requires a controller outage");
+  if (recoveryBinding !== undefined && controllerOutage === undefined && !rollingDeployment) throw new Error("Recovery observation requires a controller outage");
+  if (rollingDeployment && (!discardControllerCache || controllerOutage !== undefined || controllerDowntimeMs !== undefined
+    || preserveRunner !== undefined || crashAfterResult !== undefined)) throw new Error("Rolling deployment cannot combine an outage or another fault");
   if (controllerOutage !== undefined && (!discardControllerCache || preserveRunner !== undefined || crashAfterResult !== undefined)) {
     throw new Error("Controller outage requires the explicit remote cache-replacement cell without another crash boundary");
   }
   if (crashAfterResult !== undefined && (preserveRunner !== undefined || discardControllerCache)) {
     throw new Error("A finalization crash cannot preserve a running provider or discard its controller cache");
   }
-  return { requestId, discardControllerCache, ...(kimiCandidateAdmission ? { kimiCandidateAdmission } : {}), ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }),
+  return { requestId, discardControllerCache, ...(rollingDeployment ? { rollingDeployment: true as const } : {}), ...(kimiCandidateAdmission ? { kimiCandidateAdmission } : {}), ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }),
     ...(recoveryBinding === undefined ? {} : { recoveryBinding: parseRecoveryBinding(recoveryBinding) }),
     ...(controllerOutage === undefined ? {} : { controllerOutage: parseControllerOutage(controllerOutage) }),
     ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }),

tests/runner-e2e/server-stop.test.ts

@@ -89,6 +89,7 @@ it.skipIf(process.platform === "win32")("joins signal, final cleanup and repeate
   expect(stop(child)).toBe(first);
   await stop(child);
   expect(logs).toEqual([]);
+  expect(() => stop.assertGracefulExit(child)).not.toThrow();
 });
 
 it.skipIf(process.platform === "win32")("bounds an unresponsive child and reaps it with one escalation", async () => {
@@ -98,6 +99,7 @@ it.skipIf(process.platform === "win32")("bounds an unresponsive child and reaps
   const { stop, logs } = stopper(50);
   await Promise.all([stop(child), stop(child), stop(child)]);
   expect(child.signalCode).toBe("SIGKILL");
+  expect(() => stop.assertGracefulExit(child)).toThrow("forced cleanup cannot qualify");
   expect(logs).toHaveLength(1);
   expect(() => process.kill(child.pid!, 0)).toThrow();
 });

tests/runner-e2e/server-stop.ts

@@ -105,6 +105,12 @@ export function createRunnerE2EServerStopper(options: {
     return state.promise;
   };
   return Object.assign(stop, {
+    assertGracefulExit: (child: ChildProcess): void => {
+      const state = states.get(child);
+      if (!state?.finalComplete || state.escalationLogged || child.exitCode !== 0 || child.signalCode !== null) {
+        throw new Error("Controller did not exit gracefully; forced cleanup cannot qualify a rolling deployment");
+      }
+    },
     watch: (child: ChildProcess) => watch(child).owner.observe(),
     crashForRestart: (child: ChildProcess): Promise<void> => {
       if (process.platform === "win32") return Promise.reject(new Error("Owned crash restart requires process-group inspection"));

tests/runner-e2e/server.ts

@@ -1,3 +1,5 @@
+import { isRollingWebDeployment } from "./controller-rolling-deployment.js";
+import { superviseRollingDeployment } from "./rolling-deployment-supervisor.js";
 import { readRestartRequest, type RestartRequest } from "./server-restart-request.js";
 import { prepareControllerDatabase } from "./controller-database.js";
 import { readUpgradeArtifacts, verifyUpgradeArtifact, assertServingArtifact, upgradeArtifactKey, type ServingArtifact } from "./upgrade-artifacts.js";
@@ -451,6 +453,12 @@ async function supervise() {
     // leave Postgres available to the stopped-controller observer and successor.
     controllerDatabase = await prepareControllerDatabase(configPath, temporaryRoot);
   }
+  if (executionIds.some(id => { const execution = runnerMatrix.find(row => row.id === id); return execution && isRollingWebDeployment(execution); })) {
+    if (upgradeArtifacts || installedRelease) throw new Error("Rolling source qualification cannot claim installed-artifact evidence");
+    await superviseRollingDeployment({ temporaryRoot, repositoryRoot, configPath, port: Number(port),
+      environment: definedServerEnvironment, executionIds, cancelled: shutdownRequested });
+    return;
+  }
   const initial = await startServer();
   if (upgradeArtifacts) await waitForHealth(initial);
   let lastRestartRequestId: string | null = null;