From 16787e4167fa94bd968419cf6e6e9e8ebb2748c4 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 1 Nov 2026 02:53:27 -0600 Subject: [PATCH] Qualify rolling web deployments and preserve Stop through handoff (#18526) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server owns request handling and remote execution coordination. > - Deployment must start a ready replacement while the old server still serves. > - An outage test does not prove that two live servers can overlap safely. > - This pull request adds a separate Product E2E cell for that boundary. > - The existing run, file, recovery, and retained-session checks still apply. ## Linked Issues or Issue Description Refs #18518. The runtime prerequisite is merged; this PR qualifies its ephemeral server and deployment contracts. ## What Changed - Add the explicit `rolling-web-deployment` Codex Daytona workflow. - Start independent A/B/C homes and compiled builds behind a test ingress. - Wait for readiness before routing, then drain and delete the old home. - Observe active authority before handoff and use the existing read-only recovery oracle. - Require ordinary reads and saves to succeed throughout replacement. Read back every accepted save exactly once. - Preserve the five-minute outage suites and their existing validators. - Add a model-free supervisor smoke and an unready replacement control. - Require graceful exit before deleting an old home; forced cleanup cannot satisfy the oracle. - Preserve a durable Stop through lease release/expiry until the recovered execution owner acknowledges it. Permit only fenced reattachment of the original execution for Stop cleanup; submit no new turn. - Exclude events saved by A during shutdown from evidence of B making progress. ## Verification - E2E TypeScript check passed. - Affected harness checks: 143 passed. Catalog, fixture and report checks: 120 passed. - Actual supervisor smoke passed two replacements: 436 reads, 45 saves, zero request failures, and confirmed owned resource cleanup. - B became ready in 18.7 seconds; C became ready in 109.2 seconds. A prior C timeout remains failed evidence. - Fresh browser proof: 11 checks passed, 246 reads, 50 acknowledged saves, zero failures, 1,058ms reconnect, both old homes deleted and cleanup confirmed. - Stop lease-gap regression: three PostgreSQL delivery tests passed with a simulated provider edge. Current server TypeScript build/vendoring passed. - Full harness: 5,814 passed, 17 failed, four skipped. Two shutdown races passed the focused file; two report timeouts passed an isolated 42-test rerun. Onboarding/workflow-security/Pi/date-expired fixtures are unchanged on inspected master; the host embedded PostgreSQL fixture remains a setup limitation. This is not a green full local suite. - CI at `172509b8be` passed all gates. Greptile is 5/5 with every review thread resolved. The earlier access-service race failure passed its isolated regression and the subsequent full CI. - Final review corrections: six focused executor cases, three PostgreSQL command-delivery cases, 19 harness cases and server/E2E typechecks passed. - Three live attempts failed before agent creation: two harness setup gaps were fixed, then provider authentication failed. The saved OpenAI key returns 401; a refresh is pending. The Daytona probe reported cleanup uncertain, and no sandbox with that attempt label was found through the authenticated staging manager. No live model execution is claimed. - Live Runner overlap, released-container qualification and hosted deployment proof remain pending. Merging this opt-in qualification lane does not claim those results. ## Risks - The source lane shares workspace dependencies and does not establish container packaging correctness. - The live workflow needs an explicitly owned S3 bucket, PostgreSQL, pinned remote image and provider credentials. - Qualification cleanup stops all owned processes after failure. A failed replacement must not be reported as a pass. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../2026-10-31-zero-downtime-web-deploys.md | 106 +++++++++++++- .../native-cancellation-delivery.test.ts | 30 +++- .../native-cancellation-delivery.ts | 17 ++- .../native-session-executor.test.ts | 24 ++++ .../native-runtime/native-session-executor.ts | 16 ++- tests/runner-e2e/README.md | 29 ++++ tests/runner-e2e/SECURITY.md | 9 +- tests/runner-e2e/catalog.test.ts | 6 +- tests/runner-e2e/catalog.ts | 14 +- tests/runner-e2e/controller-outage.ts | 1 + .../controller-recovery-observer.test.ts | 12 +- .../controller-recovery-observer.ts | 35 +++-- .../controller-replacement-smoke.ts | 45 +++++- tests/runner-e2e/controller-replacement.ts | 7 + .../controller-rolling-deployment.test.ts | 58 ++++++++ .../controller-rolling-deployment.ts | 66 +++++++++ tests/runner-e2e/everyday-flow.ts | 11 +- tests/runner-e2e/everyday-remote-restart.ts | 10 +- tests/runner-e2e/live-fixtures.ts | 4 +- tests/runner-e2e/playwright.config.ts | 4 +- tests/runner-e2e/remote-workspace-storage.ts | 3 +- .../retained-controller-continuation.ts | 8 +- .../runner-e2e/rolling-deployment-process.ts | 23 +-- tests/runner-e2e/rolling-deployment-proxy.ts | 4 +- tests/runner-e2e/rolling-deployment-smoke.ts | 15 +- .../rolling-deployment-supervisor.test.ts | 42 ++++++ .../rolling-deployment-supervisor.ts | 135 ++++++++++++++++++ tests/runner-e2e/runner.spec.ts | 18 ++- tests/runner-e2e/server-restart-request.ts | 15 +- tests/runner-e2e/server-stop.test.ts | 2 + tests/runner-e2e/server-stop.ts | 6 + tests/runner-e2e/server.ts | 8 ++ 32 files changed, 701 insertions(+), 82 deletions(-) create mode 100644 tests/runner-e2e/controller-replacement.ts create mode 100644 tests/runner-e2e/controller-rolling-deployment.test.ts create mode 100644 tests/runner-e2e/controller-rolling-deployment.ts create mode 100644 tests/runner-e2e/rolling-deployment-supervisor.test.ts create mode 100644 tests/runner-e2e/rolling-deployment-supervisor.ts diff --git a/doc/plans/2026-10-31-zero-downtime-web-deploys.md b/doc/plans/2026-10-31-zero-downtime-web-deploys.md index 88336ec4224..878a1c05d94 100644 --- a/doc/plans/2026-10-31-zero-downtime-web-deploys.md +++ b/doc/plans/2026-10-31-zero-downtime-web-deploys.md @@ -1,7 +1,7 @@ # Zero-downtime web deployments -Date: 2026-10-31. Status: implementation in progress; local HTTP/storage overlap, -failure recovery and graceful drain passed. Cloud shape implementation underway. +Date: 2026-10-31. Status: runtime and Cloud overlap policy merged; qualification +and existing-volume enrollment remain in progress. No hosted acceptance claim. ## Outcome and endpoint @@ -250,10 +250,110 @@ the normal build vendoring step corrected it. All three attempts cleaned up. Staging read-only discovery and authentication work with AWS `paperclip-dev` (account `078455283791`); production `paperclip-prod` is also authenticated. -The existing shared `canary-staging` stack is `stack-pool-8fabb27531db77af`. +The existing shared `canary-staging` stack is `stack-pool-8fabb27531db`. Do not assume it is exclusively owned: select a dedicated zero-downtime test stack or acquire the existing stack operation lock before any hosted mutation. No hosted mutation or paid model call has occurred for this task yet. Remaining: live overlap proof; released artifacts; migration/enrollment and compatible rollback; ten staging deployments; internal production canary. + +The active-overlap harness is a separate stacked PR [#18526](https://github.com/paperclipai/paperclip/pull/18526) +on runtime PR #18518 to preserve the repository's review-size boundary. Its first +review found that the checkpoint observer lacked the disposable encryption key +and that forced process cleanup could pass graceful qualification. Both are +corrected; 22 focused tests passed, including actual child shutdown and shared +key decryption/restoration. The earlier model-free receipt remains evidence for +HTTP availability and cleanup, not active checkpoint recovery. + +The full harness run at `0f122611a1` completed with 5,814 passes, 17 failures and +four skips. Preserve `/private/tmp/paperclip-zdt-e2e-unit-all-rolling.log`: +four onboarding mock expectations, four unchanged workflow security expectations, +one Pi profile version expectation, three date-expired accounting fixtures, +two report-command timeouts, two process-shutdown races and one host embedded +Postgres startup failure. The complete shutdown file passed in the focused run; +other failures still need base-revision classification. No blanket green suite +or live availability claim is made. + +### Current delivery state, 2026-11-01 08:00 UTC + +Runtime PR #18518 merged as `006d5691a6245d5a710e25ba3df6d609754e163a` after +all CI passed and a 5/5 review. The opt-in remains disabled on hosted stacks. +Qualification PR #18526 remains open; Cloud #1096 remains open. No hosted +mutation or paid model test has occurred yet. + +Fresh browser receipt `paperclip-runner-e2e-tg3nel` passed all 11 checks: +246 reads, 50 acknowledged/read-back saves, zero failures, 1,058ms browser +reconnect and zero page errors. Both old homes were deleted; C restored saved +files with legacy adoption disabled. Unready B left A serving. The admitted +save and both handoffs now require a graceful exit, not successful forced cleanup. +Cleanup is confirmed. The receipt uses compiled local server/UI and workspace +dependencies at `59e6169af6` plus the smoke's strict-exit patch; it is not a +released-container or live-agent claim. + +Qualification exposed an expired/released-lease Stop race: absent a local handle, +a new server could acknowledge a no-op before recovery attached the execution. +Stop now remains pending while an unsettled native execution exists, including +between leases. Three real-PostgreSQL delivery cases pass; the provider edge and +recovery claim are simulated. Server TypeScript compilation and vendoring pass. + +The full harness's two report timeouts passed an isolated 42-test file rerun. +Onboarding, workflow-security, Pi and accounting files are unchanged by this +branch versus the inspected master; this is static classification, not a fresh +full baseline run. The qualification PR's CI has one access-service race failure +and is being checked in isolation. All other checks passed on `59e6169af6`. + +Cloud now gates overlap on identical, provenance-bound package identities. +Changed or unknown packages use maintenance. Its 376 campaign/compatibility +cases passed; review identified that failed-deploy rollback also needs the same +pause. That correction is in verification. Existing-volume enrollment and the +hosted acceptance sequence remain required work, not deferred scope. + + +### Current delivery state, 2026-11-01 08:40 UTC + +Runtime #18518 is merged at `006d5691a6245d5a710e25ba3df6d609754e163a`; +Cloud #1096 is merged at `23fdc849029e541355fa4e11234a3b5706b01e84`. +Both passed CI and final 5/5 reviews with comments resolved. Core's Docker and +readiness artifacts have published; its release workflow remains in progress. +No explicit hosted enrollment or stack deployment has been performed by this +chat; automatic release pipelines are separate and need deployed-revision checks. + +Qualification #18526 at `f2bcb4c73b` passed all CI checks. Three review findings +are fixed in the next commit: explicit mock-call isolation; pending Stop permits +only fenced reattachment of the original remote execution for cleanup; and the +recovery observer advances its event floor after A is confirmed dead while +preserving the original owner/authentication baseline. Six focused executor cases, +three PostgreSQL delivery cases (provider edge simulated), 19 harness cases and +server/E2E typechecks pass. The fixture initially returned a mock from beforeEach, +which Vitest invoked as teardown; the corrected hook returns nothing. + +Three live attempts failed before agent creation: missing built Daytona plugin, +missing development TypeScript loader in isolated controller copies, then invalid +provider credentials. The first two harness setup gaps are fixed. The third +Daytona preflight reported cleanup uncertain; the saved key returns 401, and the +staging manager found no sandbox with that exact attempt label. This does not +prove absence in an unknown old-key organization. The OpenAI key independently +returns 401. A user request to refresh the E2E credential is pending; do not +retry a paid attempt until authentication succeeds. No model execution is claimed. +Private receipts are under `/private/tmp/paperclip-zdt-live-evidence`. + +A Linux Runner binary built from `0f122611a1` has SHA256 +`681f9709f21abcc90f6e87fd1d0b263c158896b8fb3d30df6786d89723ddfdad`; +its source inputs are unchanged at the qualification head. The pinned Daytona +base image and dynamic library checks pass. The next launcher preflights OpenAI, +uses the authenticated staging manager to mint one attempt-specific Daytona key, +and revokes it during cleanup. No attempt key has yet been minted. + +Enrollment implementation is isolated in Core +`/private/tmp/paperclip-zdt-enrollment-20261101` and the Cloud worktree, both on +`codex/zero-downtime-enrollment`. Core's dedicated maintenance process is being +verified: ordinary app writers never start; existing owner migrations verify +shared bytes; the original decision key exports only through a per-attempt +credential after full eligibility. Cloud's locked, audited provider transition +must stop all writers, run this process on the retained volume, preserve its key, +then detach only after a matching receipt and confirmed process exit. + +Remaining acceptance is unchanged: live Stop/answer/follow-up and continuation, +packaged fresh/migrated overlap, ten staging deployments including rollback, +repair/sleep/wake and idle suspension, then one internal production canary. diff --git a/server/src/__tests__/native-cancellation-delivery.test.ts b/server/src/__tests__/native-cancellation-delivery.test.ts index b5f8261d1bb..1a8b5d5cabb 100644 --- a/server/src/__tests__/native-cancellation-delivery.test.ts +++ b/server/src/__tests__/native-cancellation-delivery.test.ts @@ -1,11 +1,11 @@ import { randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; -import { afterAll, beforeAll, afterEach, expect, it, vi } from "vitest"; +import { afterAll, beforeAll, beforeEach, afterEach, expect, it, vi } from "vitest"; import { agents, companies, issues, heartbeatRuns, nativeRunFinalizations, createDb } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; import { deploymentNotifications } from "../services/deployment-notifications.js"; -import { nativeCancellationHasRemoteOwner, notifyNativeCancellation, registerNativeCancellationDelivery, waitForNativeCancellation } from "../services/native-runtime/native-cancellation-delivery.js"; +import { nativeCancellationHasUnsettledOwner, notifyNativeCancellation, registerNativeCancellationDelivery, waitForNativeCancellation } from "../services/native-runtime/native-cancellation-delivery.js"; import { currentNativeControllerIdentity } from "../services/native-runtime/native-restart-recovery.js"; import { setNativeExecutionActive } from "../services/native-runtime/native-active-runs.js"; const provider = vi.hoisted(() => ({ stop: vi.fn() })); @@ -46,6 +46,7 @@ beforeAll(async () => { }); worker = registerNativeCancellationDelivery(a, coordinatorA); await worker.ready; }, 90_000); +beforeEach(() => { provider.stop.mockClear(); }); afterEach(() => vi.unstubAllEnvs()); afterAll(async () => { setNativeExecutionActive(runId, false); await worker?.stop(); @@ -68,11 +69,28 @@ it("fails closed on a different intent and never accepts a no-op acknowledgement acknowledgementAuditId: "not-dispatched" } } }).where(eq(heartbeatRuns.id, runId)); await expect(waitForNativeCancellation(b, { companyId, runId, intentId }, 50)).rejects.toThrow("pending_recovery"); }); -it("distinguishes the current controller, a live replacement owner, and an expired owner", async () => { +it("keeps Stop pending across lease release and delivers after normal ownership resumes", async () => { vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); - expect(await nativeCancellationHasRemoteOwner(b, runId)).toBe(false); + expect(await nativeCancellationHasUnsettledOwner(b, runId)).toBe(true); await b.update(nativeRunFinalizations).set({ controllerBootId: "another-controller" }).where(eq(nativeRunFinalizations.runId, runId)); - expect(await nativeCancellationHasRemoteOwner(b, runId)).toBe(true); + expect(await nativeCancellationHasUnsettledOwner(b, runId)).toBe(true); await b.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(0) }).where(eq(nativeRunFinalizations.runId, runId)); - expect(await nativeCancellationHasRemoteOwner(b, runId)).toBe(false); + expect(await nativeCancellationHasUnsettledOwner(b, runId)).toBe(true); + await b.update(heartbeatRuns).set({ resultJson: { nativeCancellation: { schema: "paperclip.native-cancellation.v1", intentId, + scope: "run", dispatchState: "pending" } } }).where(eq(heartbeatRuns.id, runId)); + const acknowledged = waitForNativeCancellation(b, { companyId, runId, intentId }); + const identity = await currentNativeControllerIdentity(); + // Simulate only the normal recovery owner's claim; no test-owned dispatch or + // acknowledgement. Production delivery must consume the durable pending Stop. + await a.transaction(async tx => { + await notifyNativeCancellation(tx); + await tx.update(nativeRunFinalizations).set({ controllerBootId: identity.bootId, leaseExpiresAt: new Date(Date.now() + 60_000) }) + .where(eq(nativeRunFinalizations.runId, runId)); + }); + await expect(acknowledged).resolves.toMatchObject({ dispatched: true, auditId: "fixture-audit" }); + expect(provider.stop).toHaveBeenCalledTimes(1); + await a.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, runId)); + expect(await nativeCancellationHasUnsettledOwner(b, runId)).toBe(false); + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "false"); + expect(await nativeCancellationHasUnsettledOwner(b, runId)).toBe(false); }); diff --git a/server/src/services/native-runtime/native-cancellation-delivery.ts b/server/src/services/native-runtime/native-cancellation-delivery.ts index 636a694c696..8195741db8a 100644 --- a/server/src/services/native-runtime/native-cancellation-delivery.ts +++ b/server/src/services/native-runtime/native-cancellation-delivery.ts @@ -1,4 +1,4 @@ -import { and, eq, gt, inArray, sql } from "drizzle-orm"; +import { and, eq, gt, inArray, isNull, sql } from "drizzle-orm"; import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { subscribeDatabaseWork } from "@paperclipai/db/work-signals"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; @@ -9,12 +9,17 @@ import { currentNativeControllerIdentity } from "./native-restart-recovery.js"; const object = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; export const notifyNativeCancellation = (tx: object) => notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeCancellation); -export async function nativeCancellationHasRemoteOwner(db: Db, runId: string): Promise { +/** Called only when this process has neither an active handle nor a startup. + * Expiry/release authorizes recovery, not a no-op acknowledgement: the remote + * execution can still be alive while its next controller is attaching. */ +export async function nativeCancellationHasUnsettledOwner(db: Db, runId: string): Promise { if (process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true") return false; - const identity = await currentNativeControllerIdentity(); - const [owner] = await db.select({ bootId: nativeRunFinalizations.controllerBootId }).from(nativeRunFinalizations) - .where(and(eq(nativeRunFinalizations.runId, runId), gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`))).limit(1); - return !!owner?.bootId && owner.bootId !== identity.bootId; + const [owner] = await db.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations) + .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId), + eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId))) + .where(and(eq(nativeRunFinalizations.runId, runId), isNull(nativeRunFinalizations.resultId), + inArray(heartbeatRuns.status, ["running", "queued"]))).limit(1); + return Boolean(owner); } /** The accepting HTTP server waits for the real owner's audited ACK. Subscribe diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index d663a4801de..07965da0400 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -6603,6 +6603,30 @@ function cancellationDb(options?: { } describe("native startup cancellation fence", () => { + it.each(["current", "stale"] as const)("reattaches a %s remote recovery claim only to deliver an existing Stop", async mode => { + const { currentNativeControllerIdentity } = await import("./native-restart-recovery.js"); + const controller = await currentNativeControllerIdentity(); + const resultJson = { nativeCancellation: { schema: "paperclip.native-cancellation.v1", scope: "run", + companyId: execution.binding.companyId, runId: execution.binding.runId, issueId: execution.binding.issueId, + dispatchState: "pending", intentId: "native-cancellation:recovered-stop", effects: ["release_run_resources"] } }; + const cancel = vi.fn(() => ({ cleanup: Promise.resolve() })), submit = vi.fn(); + state.execute.mockReset().mockImplementationOnce(async options => { + try { await options.onSession({ cancel }); submit(); } + finally { await options.onSession(null); } + throw new Error("a stopped recovery must not submit a turn"); + }); + const db = leaseDb(execution, { leaseOwner: "stop-recovery", controllerBootId: controller.bootId, + controllerPid: controller.pid, controllerGeneration: 2, attempt: 1 }, resultJson); + await expect(executePaperclipNativeSession({ db, execution, runnerInstanceId: "recovered-stop", + leaseOwner: "stop-recovery", restartRecovery: { kind: "reattach_remote_runner", runId: execution.binding.runId, + leaseOwner: "stop-recovery", controllerGeneration: mode === "current" ? 2 : 1, providerAttempt: 1, + restartKind: "graceful", recoveryRequestId: null, remote: { providerLeaseId: "fixture", remoteCwd: "/workspace" } }, + })).rejects.toThrow(mode === "current" ? "native_cancellation_pending_recovery" : "native_restart_recovery_claim_changed"); + expect(submit).not.toHaveBeenCalled(); + expect(cancel).toHaveBeenCalledTimes(mode === "current" ? 1 : 0); + expect(state.execute).toHaveBeenCalledTimes(mode === "current" ? 1 : 0); + }); + it.each(["startupCancellation", "nativeCancellation"])("does not submit a turn when %s arrives during session opening", async (marker) => { const resultJson: Record = {}; const cancel = vi.fn(() => ({ cleanup: Promise.resolve() })); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index ce7507ab28f..b2799be84cb 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,4 +1,4 @@ -import { nativeCancellationHasRemoteOwner, notifyNativeCancellation, waitForNativeCancellation } from "./native-cancellation-delivery.js"; +import { nativeCancellationHasUnsettledOwner, notifyNativeCancellation, waitForNativeCancellation } from "./native-cancellation-delivery.js"; import { publishNativeHarnessBackup } from "./native-harness-backup-publication.js"; import { rehydratePinnedSkillAssets } from "./runtime-context.js"; import { createNativeUsageDelivery } from "./native-usage-delivery.js"; @@ -7710,7 +7710,7 @@ async function performNativeCancellation( recoveringCancellationIntent = intent.existing; priorCoordinatorDecisionIdAtIntent = intent.priorCoordinatorDecisionId; if (!activeNativeSessions.has(runId) && !nativeSessionStartups.has(runId) - && await nativeCancellationHasRemoteOwner(options.db, runId)) { + && await nativeCancellationHasUnsettledOwner(options.db, runId)) { const acknowledged = await waitForNativeCancellation(options.db, { companyId: cancellationContext.companyId, runId, intentId: intent.intentId, }); @@ -8551,9 +8551,15 @@ async function executePaperclipNativeSessionWithinScope( cancellationIntent.issueId === input.execution.binding.issueId; if (!intentMatchesBinding) throw new Error("native_cancellation_intent_conflict"); - // Run cancellation remains a claim fence after dispatch ack until - // the heartbeat cancellation path terminalizes the run. - throw new NativeCancellationPendingRecoveryError(); + // A pending Stop must be able to reattach its already-admitted + // remote execution for cleanup. The recovery claim is fenced below; + // onSession dispatches this durable Stop before any turn submission. + // Fresh execution, dead-runner resume and acknowledged Stops remain + // fenced: none authorize launching another provider turn. + if (input.restartRecovery?.kind !== "reattach_remote_runner" + || cancellationIntent.dispatchState !== "pending") { + throw new NativeCancellationPendingRecoveryError(); + } } if (["committed", "applied"].includes(coordinator.phase)) throw new Error("native_run_already_committed"); diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 3cfe7626586..8d6610f5358 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -96,6 +96,35 @@ PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_REGION=us-east-1 \ pnpm test:e2e:runner -- --id remote-workspace-storage.runner-codex.daytona.recover-controller --max-automatic-retries 0 ``` +The separate explicit-only `rolling-web-deployment` suite uses the same synthetic +projectless workflow, four-admission budget, thirty-minute limit, and independent +run/file oracles. Its supervisor starts an independent empty-home server, waits +for `/api/ready`, moves ingress, sends SIGTERM to the predecessor, and deletes +that predecessor's whole home and build. Continuous ordinary task reads and +comment saves must have zero failures, and every accepted save must be read back +exactly once. Forced process cleanup fails graceful qualification even if the +successor recovers the execution. The read-only observer shares the controllers' +disposable encryption key. It repeats replacement after completion before the retained-session +follow-up. The five-minute outage suites are unchanged. + +```sh +pnpm test:e2e:runner -- --id rolling-web-deployment.runner-codex.daytona.recover-controller --max-automatic-retries 0 +``` + +This source lane requires a complete server/UI build and disposable S3 and +Postgres. Each controller has an independent compiled build and cache; shared +workspace dependencies mean it is not released-container or Railway evidence. +The read-only recovery observer captures the live predecessor's identity just +before handoff, then requires a new controller generation, authenticated Runner +attachment, and new canonical events for the original execution. No fixture +modifies leases, checkpoints, provider identity, or completed work. + +Before paid execution, set `PAPERCLIP_ROLLING_TEST_STORAGE_CONFIG` and the +loopback `PAPERCLIP_TEST_DATABASE_URL`, then run +`tests/runner-e2e/controller-replacement-smoke.ts` through the existing tsx loader. +It exercises the actual supervisor and two replacements without an agent and +explicitly records `activeAgentContinuityQualified: false`. + Optional `PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_ENDPOINT` and `PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_FORCE_PATH_STYLE=true` select a test S3 endpoint. Each attempt has a distinct `runner-e2e//` prefix. The diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index db2b3fa038e..173610b4afc 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -7,7 +7,8 @@ reference. Provider credentials and agent homes remain outside that folder. The ordinary settings read must report remote storage enabled by default; the workflow does not opt itself in. Its five-minute active outage and later completed-session replacement retain the existing native identity and no-replay checks. -The server's S3 override is admitted only for this suite and only from explicit +The server's S3 override is admitted only for this suite and the explicit +`rolling-web-deployment` suite, and only from explicit `PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_*` configuration, with a unique instance prefix. Ambient storage settings remain stripped and rejected for other suites. Snapshot objects, databases, raw provider IDs, and execution workspaces remain @@ -618,3 +619,9 @@ Slack qualification, and they do not publish trusted archives or credentials. Report identities preserve legacy chat tasks and exact `area.case` chat catalog tasks. Additional task segments, path separators and dotted nonchat tasks remain invalid. This parsing does not qualify a result or grant a new execution. + +The explicit rolling web deployment cell shares only its disposable PostgreSQL, +S3 prefix, and generated secrets between controllers. It removes the entire old +home after graceful handoff. Its read-only observer does not create recovery +authority. Provider credentials still enter through encrypted fixture secrets. +Local rolling smoke receipts explicitly exclude active-agent and hosted proof. diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 8ce668acff0..dcfe9840da1 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -278,10 +278,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 1, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, + 1, 1, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(796); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(796); + expect(validateRunnerCatalog()).toHaveLength(797); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(797); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 12c2a1b74d8..49653d7f55c 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1,3 +1,4 @@ +import { ROLLING_WEB_DEPLOYMENT_SUITE } from "./controller-rolling-deployment.js"; import { FEEDBACK_PERSISTENCE_SUITE, feedbackPersistenceTask, feedbackPersistenceDigest } from "./feedback-persistence.js"; import { followupStopTask, followupStopDigest } from "./followup-stop-resume.js"; import { finalizationRecoveryTask, finalizationRecoveryDigest } from "./finalization-recovery.js"; @@ -1169,8 +1170,7 @@ const daytonaControllerOutageEnvironment: EnvironmentFixture = { }, }; -export const runnerSuites: readonly RunnerSuiteFixture[] = [ - { +const remoteWorkspaceStorageSuite: RunnerSuiteFixture = { id: REMOTE_WORKSPACE_STORAGE_SUITE, label: "Remote workspace storage and controller replacement", manualOnly: true, description: "Keep synthetic files and local Git work in the sandbox, commit saved versions to object storage, and preserve the original active execution through controller replacement.", groups: [], profiles: [(() => { @@ -1196,6 +1196,16 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ postControllerReplyFormat: "one-plain-text-line-marker-space-actual-command-output", grading: "same-active-execution-committed-snapshots-browser-files-and-post-controller-native-continuation", source: "projectless-synthetic-files-and-model-created-git", authentication: "api-key", stagingQualification: false }, + }; + +export const runnerSuites: readonly RunnerSuiteFixture[] = [ + remoteWorkspaceStorageSuite, + { ...remoteWorkspaceStorageSuite, id: ROLLING_WEB_DEPLOYMENT_SUITE, label: "Rolling control-plane web deployment", + description: "Overlap independent ephemeral web servers while a remote execution stays active; keep ordinary reads, saves, files and a later native continuation working.", + definitionMetadata: { ...remoteWorkspaceStorageSuite.definitionMetadata, version: 1, controllerDowntimeMs: 0, + replacement: "ready-before-traffic-switch-sigterm-delete-entire-home", + upgradeReadiness: "active-remote-workspace-recoverable-before-handoff", + grading: "same-active-execution-ordinary-reads-saves-and-second-empty-home-continuation" }, }, { id: "daytona-restart-continuity", label: "Daytona warm controller outage continuity", manualOnly: true, diff --git a/tests/runner-e2e/controller-outage.ts b/tests/runner-e2e/controller-outage.ts index 844552db748..32ec972c938 100644 --- a/tests/runner-e2e/controller-outage.ts +++ b/tests/runner-e2e/controller-outage.ts @@ -40,6 +40,7 @@ export function configuredControllerOutage(env: NodeJS.ProcessEnv, defaultHoldMs } export function controllerOutageForExecution(execution: MatrixExecution, env: NodeJS.ProcessEnv): ControllerOutage | undefined { + if (execution.suite.id === "rolling-web-deployment") return undefined; if (execution.task.flow !== "everyday_workflow" || execution.task.id !== "recover-controller" || execution.environment.id !== "daytona" || execution.profile.generation !== "native" || (execution.profile.provider !== "codex" && execution.profile.provider !== "acpx")) return undefined; diff --git a/tests/runner-e2e/controller-recovery-observer.test.ts b/tests/runner-e2e/controller-recovery-observer.test.ts index 19bcdcd9b97..8597ee97a27 100644 --- a/tests/runner-e2e/controller-recovery-observer.test.ts +++ b/tests/runner-e2e/controller-recovery-observer.test.ts @@ -1,6 +1,6 @@ import type { Db } from "../../packages/db/src/client.js"; import { describe, expect, it } from "vitest"; -import { parseRecoveryBinding, readControllerRecoverySnapshot, recoveryObservation, validControllerRecoveryReceipt, type RecoverySnapshot } from "./controller-recovery-observer.js"; +import { advanceRecoveryCursor, parseRecoveryBinding, readControllerRecoverySnapshot, recoveryObservation, validControllerRecoveryReceipt, type RecoverySnapshot } from "./controller-recovery-observer.js"; import type { ControllerOutageReceipt } from "./controller-outage.js"; const binding = { companyId: "10000000-0000-0000-0000-000000000001", issueId: "10000000-0000-0000-0000-000000000002", runId: "10000000-0000-0000-0000-000000000003" }; @@ -13,6 +13,16 @@ const fault: ControllerOutageReceipt = { schema: "paperclip.e2e.controller-outag const after = (): RecoverySnapshot => ({ ...before, bootId: "new", generation: 2, sourceSeq: 47, authenticatedConnectionCount: 2, history: [{ controllerBootId: "new", controllerGeneration: 2, disposition: "reattach_remote_runner", at: "2026-10-24T00:01:00.001Z" }] }); describe("read-only controller recovery evidence", () => { + it("does not count A's shutdown tail as B's processing even if B already claimed", () => { + const stopped = { ...after(), checkpointSourceSeq: 48 }; + const baseline = advanceRecoveryCursor(before, stopped); + expect(baseline).toMatchObject({ bootId: "old", generation: 1, authenticatedConnectionCount: 1, sourceSeq: 48 }); + expect(recoveryObservation(baseline, { ...stopped, sourceSeq: 48 }, fault, binding, 62_000)).toBeNull(); + expect(recoveryObservation(baseline, { ...stopped, sourceSeq: 49 }, fault, binding, 62_001)).toMatchObject({ + previousBootId: "old", runnerSourceSeqBefore: 48, runnerSourceSeqAfter: 49, + }); + expect(() => advanceRecoveryCursor(before, { ...stopped, runnerInstanceId: "replacement" })).toThrow("changed the admitted execution"); + }); it("requires natural ownership and new original-run receipts after the stopped-process cursor", () => { expect(recoveryObservation(before, before, fault, binding, 2500)).toBeNull(); expect(recoveryObservation(before, { ...after(), sourceSeq: 42 }, fault, binding, 61_000)).toBeNull(); diff --git a/tests/runner-e2e/controller-recovery-observer.ts b/tests/runner-e2e/controller-recovery-observer.ts index 0ee66823a66..a236b478c07 100644 --- a/tests/runner-e2e/controller-recovery-observer.ts +++ b/tests/runner-e2e/controller-recovery-observer.ts @@ -8,7 +8,7 @@ const { sql }: typeof import("../../packages/db/node_modules/drizzle-orm/index.j import { createDb, type Db } from "../../packages/db/src/index.js"; import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js"; import { loadNativeRunnerCheckpoint } from "../../server/src/services/native-runtime/native-runner-checkpoint.js"; -import type { ControllerOutageReceipt } from "./controller-outage.js"; +import type { ControllerReplacementReceipt } from "./controller-replacement.js"; export interface RecoveryBinding { companyId: string; issueId: string; runId: string } export interface RecoverySnapshot { @@ -47,7 +47,7 @@ export function parseRecoveryBinding(value: unknown): RecoveryBinding { return row as unknown as RecoveryBinding; } -export function recoveryObservation(before: RecoverySnapshot, after: RecoverySnapshot, fault: ControllerOutageReceipt, +export function recoveryObservation(before: RecoverySnapshot, after: RecoverySnapshot, fault: ControllerReplacementReceipt, binding: RecoveryBinding, observedAfterMs: number): ControllerRecoveryReceipt | null { if (before.runId !== binding.runId || after.runId !== before.runId || after.runnerInstanceId !== before.runnerInstanceId || after.nativeSessionId !== before.nativeSessionId || after.attempt !== before.attempt) { @@ -77,7 +77,17 @@ export function recoveryObservation(before: RecoverySnapshot, after: RecoverySna authenticatedConnectionCount: after.authenticatedConnectionCount, attachmentEvidence: "runner_checkpoint_authenticated_connection" }; } -export function validControllerRecoveryReceipt(fault: ControllerOutageReceipt | undefined): boolean { +/** B can claim before A exits. Preserve A's original owner/authentication + * identity while excluding every event committed through A's final exit. */ +export function advanceRecoveryCursor(before: RecoverySnapshot, stopped: RecoverySnapshot): RecoverySnapshot { + if (before.runId !== stopped.runId || before.runnerInstanceId !== stopped.runnerInstanceId + || before.nativeSessionId !== stopped.nativeSessionId || before.attempt !== stopped.attempt) { + throw new Error("Controller replacement changed the admitted execution"); + } + return { ...before, sourceSeq: Math.max(before.sourceSeq, stopped.sourceSeq, stopped.checkpointSourceSeq) }; +} + +export function validControllerRecoveryReceipt(fault: ControllerReplacementReceipt | undefined): boolean { const r = fault?.recovery; return Boolean(r && r.bootId !== r.previousBootId && r.generation > r.previousGeneration && r.runnerSourceSeqAfter > r.runnerSourceSeqBefore && r.authenticatedConnectionCount > r.previousAuthenticatedConnectionCount @@ -120,19 +130,26 @@ export async function createControllerRecoveryObserver(configPath: string, bindi const db = createDb(url); let before: RecoverySnapshot | undefined; const read = () => readControllerRecoverySnapshot(db, binding); - return { - // Called only after the old server is confirmed dead. This also excludes - // receipts committed while it was stopping from replacement evidence. - stopped: async (fault: ControllerOutageReceipt) => { + const capture = async (fault: ControllerReplacementReceipt) => { before = await read(); before.sourceSeq = Math.max(before.sourceSeq, before.checkpointSourceSeq); fault.recoveryProgress = { binding, previousBootId: before.bootId, previousGeneration: before.generation, previousLeaseExpiresAt: before.leaseExpiresAt, runnerSourceSeqBefore: before.sourceSeq, previousAuthenticatedConnectionCount: before.authenticatedConnectionCount, runnerInstanceId: before.runnerInstanceId, nativeSessionId: before.nativeSessionId }; + }; + return { + beforeHandoff: capture, + // Called only after the old server is confirmed dead. This also excludes + // receipts committed while it was stopping from replacement evidence. + stopped: capture, + stoppedAfterHandoff: async (fault: ControllerReplacementReceipt) => { + if (!before) throw new Error("Missing pre-handoff controller observation"); + before = advanceRecoveryCursor(before, await read()); + fault.recoveryProgress = { ...fault.recoveryProgress, runnerSourceSeqBefore: before.sourceSeq }; }, - recovered: async (fault: ControllerOutageReceipt, record: () => Promise, cancelled: () => boolean) => { - if (!before) throw new Error("Missing stopped-controller observation"); + recovered: async (fault: ControllerReplacementReceipt, record: () => Promise, cancelled: () => boolean) => { + if (!before) throw new Error("Missing controller boundary observation"); const deadline = Date.parse(fault.startedAt) + 420_000; let ownershipObservedAfterMs: number | undefined; let authenticatedObservedAfterMs: number | undefined; diff --git a/tests/runner-e2e/controller-replacement-smoke.ts b/tests/runner-e2e/controller-replacement-smoke.ts index b937dcc3850..6f957cdcc4d 100644 --- a/tests/runner-e2e/controller-replacement-smoke.ts +++ b/tests/runner-e2e/controller-replacement-smoke.ts @@ -1,3 +1,5 @@ +import { validControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js"; +import { createRequire } from "node:module"; import { readUpgradeArtifacts, upgradeArtifactKey } from "./upgrade-artifacts.js"; /** Credential-free main-server replacement. This proves the wrapper/startup/DB * boundary; only the hosted workflow can qualify an active Daytona agent. @@ -20,6 +22,12 @@ import { createRunnerE2EServerStopper } from "./server-stop.js"; import { cleanupControllerSmokeDatabase } from "./controller-smoke-cleanup.js"; import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js"; +const rolling = Boolean(process.env.PAPERCLIP_ROLLING_TEST_STORAGE_CONFIG); +const { S3Client, CreateBucketCommand, ListObjectsV2Command, DeleteObjectsCommand, DeleteBucketCommand } = createRequire(new URL("../../server/package.json", import.meta.url))("@aws-sdk/client-s3"); +let storageClient: InstanceType | undefined; +const storageBucket = `rolling-supervisor-${randomUUID()}`; +let storageCreated = false; +let storageEnvironment: NodeJS.ProcessEnv = {}; let outage: ControllerOutage; let upgrades: Awaited> = null; const repositoryRoot = path.resolve(import.meta.dirname, "../.."); @@ -37,7 +45,7 @@ const report: Record = { schema: "paperclip.e2e.controller-repl let database: { connectionString: string; cleanup(): Promise } | undefined; let db: ReturnType | undefined; const log = createWriteStream(path.join(temporaryRoot, "wrapper.log"), { mode: 0o600 }); -const stopper = createRunnerE2EServerStopper({ gracefulTimeoutMs: 35_000, forcedTimeoutMs: 5_000, +const stopper = createRunnerE2EServerStopper({ gracefulTimeoutMs: rolling ? 100_000 : 35_000, forcedTimeoutMs: 5_000, hasSpawnError: () => false, markExpectedStop: () => {}, log: message => log.write(message) }); let wrapper: ReturnType | undefined; let failure: unknown; @@ -84,14 +92,27 @@ try { const config = paperclipConfigSchema.parse(JSON.parse(await readFile(configPath, "utf8"))); config.database = { ...config.database, mode: "postgres", connectionString: database.connectionString }; await writeFile(configPath, JSON.stringify(config), { mode: 0o600 }); - const source = { PATH: process.env.PATH, HOME: home, TMPDIR: process.env.TMPDIR, + if (rolling) { + if (upgrades) throw new Error("Rolling supervisor smoke does not qualify upgrade artifacts"); + const storage = JSON.parse(await readFile(process.env.PAPERCLIP_ROLLING_TEST_STORAGE_CONFIG!, "utf8")); + const endpoint = new URL(storage.endpoint); + if (endpoint.protocol !== "http:" || !["127.0.0.1", "localhost", "[::1]"].includes(endpoint.hostname) + || endpoint.username || endpoint.password) throw new Error("Supervisor smoke requires disposable loopback S3"); + storageClient = new S3Client({ region: "us-east-1", endpoint: storage.endpoint, forcePathStyle: true, + credentials: { accessKeyId: storage.accessKeyId, secretAccessKey: storage.secretAccessKey } }); + await storageClient.send(new CreateBucketCommand({ Bucket: storageBucket })); storageCreated = true; + storageEnvironment = { PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_BUCKET: storageBucket, + PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_ENDPOINT: storage.endpoint, PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_FORCE_PATH_STYLE: "true", + AWS_ACCESS_KEY_ID: storage.accessKeyId, AWS_SECRET_ACCESS_KEY: storage.secretAccessKey, AWS_EC2_METADATA_DISABLED: "true" }; + } + const source = { ...storageEnvironment, PATH: process.env.PATH, HOME: home, TMPDIR: process.env.TMPDIR, ...(upgrades ? { ...upgrades.artifacts.A.pins, [upgradeArtifactKey]: process.env[upgradeArtifactKey] } : {}), NODE_ENV: "test", PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: configPath, PAPERCLIP_INSTANCE_ID: "default", PAPERCLIP_AGENT_JWT_SECRET: randomUUID(), PAPERCLIP_DECISION_SIGNING_SECRET: randomUUID(), PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: randomUUID(), BETTER_AUTH_SECRET: randomUUID(), PAPERCLIP_RUNNER_E2E_SERVER_LOG: path.join(temporaryRoot, "server.log"), PAPERCLIP_RUNNER_E2E_TEMP_ROOT: temporaryRoot, PAPERCLIP_RUNNER_E2E_PORT: String(port), - PAPERCLIP_RUNNER_E2E_EXECUTION_IDS: JSON.stringify(["everyday-workflows.runner-codex.daytona.recover-controller"]) }; + PAPERCLIP_RUNNER_E2E_EXECUTION_IDS: JSON.stringify([`${rolling ? "rolling-web-deployment" : "everyday-workflows"}.runner-codex.daytona.recover-controller`]) }; wrapper = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(repositoryRoot, path.join(import.meta.dirname, "server.ts")), { cwd: repositoryRoot, env: source, detached: process.platform !== "win32", stdio: ["ignore", "pipe", "pipe"] }); wrapper.once("spawn", () => { report.wrapperStarted = true; }); @@ -121,12 +142,12 @@ try { const company = await api("/companies", { name: `Restart smoke ${randomUUID()}` }); const [before] = await db.$client<{ started: string }[]>`select pg_postmaster_start_time()::text as started`; const receipts: unknown[] = []; - for (let index = 1; index < (upgrades?.sequence.length ?? 2); index += 1) { + for (let index = 1; index < (upgrades?.sequence.length ?? (rolling ? 3 : 2)); index += 1) { // Empty fixture cache exercises retirement without pretending a runner exists. const cache = path.join(home, "instances", "default", "runtime", "paperclip-runner"); await mkdir(cache, { recursive: true }); const requestId = `smoke-${index}-${randomUUID()}`; - await writeFile(`${control.restartRequestPath}.tmp`, JSON.stringify({ requestId, discardControllerCache: true, controllerOutage: outage }), { mode: 0o600 }); + await writeFile(`${control.restartRequestPath}.tmp`, JSON.stringify({ requestId, discardControllerCache: true, ...(rolling ? { rollingDeployment: true } : { controllerOutage: outage }) }), { mode: 0o600 }); await rename(`${control.restartRequestPath}.tmp`, control.restartRequestPath); const ack = await poll(async () => { const value = await readFile(control.restartAcknowledgementPath, "utf8").then(text => JSON.parse(text) as Record) @@ -134,11 +155,11 @@ try { if (value?.requestId === requestId && value.status === "failed") throw new Error(String(value.message)); return value; }, value => value?.requestId === requestId && value.status === "ready", outage.holdMs + 180_000); - assert(validControllerOutageReceipt(ack!.controllerOutage, outage, requestId, upgrades ? { + assert(rolling ? validControllerRollingDeploymentReceipt(ack!.rollingDeployment, requestId) : validControllerOutageReceipt(ack!.controllerOutage, outage, requestId, upgrades ? { previous: upgrades.artifacts[upgrades.sequence[index - 1]!], replacement: upgrades.artifacts[upgrades.sequence[index]!], } : undefined), "Invalid replacement fault or requested artifact receipt"); - receipts.push(ack!.controllerOutage); + receipts.push(rolling ? ack!.rollingDeployment : ack!.controllerOutage); report.replacements = receipts; await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, { mode: 0o600 }); assert.equal((await api(`/companies/${company.id}`)).id, company.id); @@ -158,6 +179,16 @@ try { if (wrapper) cleanups.push(() => stopper.stopAll()); if (db) { const client = db.$client; cleanups.push(() => client.end()); } if (database) { const owned = database; cleanups.push(() => cleanupControllerSmokeDatabase(owned, ownedPostgresProcessId)); } + if (storageClient && storageCreated) cleanups.push(async () => { + let continuationToken: string | undefined; + do { + const listed = await storageClient!.send(new ListObjectsV2Command({ Bucket: storageBucket, ContinuationToken: continuationToken })); + if (listed.Contents?.length) await storageClient!.send(new DeleteObjectsCommand({ Bucket: storageBucket, + Delete: { Objects: listed.Contents.map((object: { Key: string }) => ({ Key: object.Key })) } })); + continuationToken = listed.NextContinuationToken; + } while (continuationToken); + await storageClient!.send(new DeleteBucketCommand({ Bucket: storageBucket })); storageClient!.destroy(); + }); for (const cleanup of cleanups) { try { await cleanup(); } catch (error) { failure ??= error; cleanupErrors.push(error instanceof Error ? error.message : String(error)); } diff --git a/tests/runner-e2e/controller-replacement.ts b/tests/runner-e2e/controller-replacement.ts new file mode 100644 index 00000000000..e97aa40b736 --- /dev/null +++ b/tests/runner-e2e/controller-replacement.ts @@ -0,0 +1,7 @@ +import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js"; +import { validControllerRollingDeploymentReceipt, type ControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js"; + +export type ControllerReplacementReceipt = ControllerOutageReceipt | ControllerRollingDeploymentReceipt; +export function validControllerReplacementReceipt(value: unknown): value is ControllerReplacementReceipt { + return validControllerOutageReceipt(value) || validControllerRollingDeploymentReceipt(value); +} diff --git a/tests/runner-e2e/controller-rolling-deployment.test.ts b/tests/runner-e2e/controller-rolling-deployment.test.ts new file mode 100644 index 00000000000..2dfa052eefb --- /dev/null +++ b/tests/runner-e2e/controller-rolling-deployment.test.ts @@ -0,0 +1,58 @@ +import { describe, expect, it } from "vitest"; +import { rollController, validControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js"; +import { validControllerReplacementReceipt } from "./controller-replacement.js"; +import { controllerOutageForExecution, validControllerOutageReceipt } from "./controller-outage.js"; +import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; + +function fixture() { + const order: string[] = []; + return { order, input: { requestId: "rolling-1", previousPid: 100, + start: async () => { order.push("start-B"); return { pid: 200 }; }, + waitForReady: async () => { order.push("B-ready"); }, + beforeHandoff: async () => { order.push("observe-A"); }, + switchTraffic: () => { order.push("route-B"); }, + stopPrevious: async () => { order.push("stop-A"); }, + afterPreviousStopped: async () => { order.push("observe-A-stopped"); }, + deletePreviousHome: async () => { order.push("delete-home-A"); }, + afterHandoff: async () => { order.push("observe-recovery"); }, + cancelled: () => false, record: async () => {}, + } }; +} + +describe("rolling controller qualification", () => { + it("moves traffic only after readiness, then removes all predecessor state", async () => { + const f = fixture(); + const receipt = await rollController(f.input); + expect(f.order).toEqual(["start-B", "B-ready", "observe-A", "route-B", "stop-A", "observe-A-stopped", "delete-home-A", "observe-recovery"]); + expect(validControllerRollingDeploymentReceipt(receipt)).toBe(false); // A process receipt alone is not availability proof. + receipt.probes = { reads: 10, writes: 3, verifiedWrites: 3, failures: [] }; + expect(validControllerReplacementReceipt(receipt)).toBe(true); + expect(validControllerOutageReceipt(receipt)).toBe(false); + for (const changed of [ + { previousHomeRemoved: false }, { cacheDiscarded: false }, { replacementPid: 100 }, { gracefulShutdown: false }, + { trafficSwitchedAfterMs: -1 }, { readyAfterMs: receipt.stoppedAfterMs! + 1 }, + { probes: { reads: 10, writes: 3, verifiedWrites: 2, failures: [] } }, + { probes: { reads: 10, writes: 3, verifiedWrites: 3, failures: ["502"] } }, + ]) expect(validControllerReplacementReceipt({ ...receipt, ...changed })).toBe(false); + expect(validControllerRollingDeploymentReceipt(receipt, "other-request")).toBe(false); + }); + it("keeps A serving when B never becomes ready", async () => { + const f = fixture(); f.input.waitForReady = async () => { throw new Error("B unready"); }; + await expect(rollController(f.input)).rejects.toThrow("B unready"); + expect(f.order).toEqual(["start-B"]); + }); + it("rejects forced predecessor cleanup before deleting its home or observing recovery", async () => { + const f = fixture(); f.input.stopPrevious = async () => { throw new Error("forced shutdown"); }; + await expect(rollController(f.input)).rejects.toThrow("forced shutdown"); + expect(f.order).toEqual(["start-B", "B-ready", "observe-A", "route-B"]); + }); + it("keeps the original outage contract separate from the explicit rolling cell", () => { + const [rolling] = selectRunnerExecutions(parseRunnerSelectors(["--suite", "rolling-web-deployment"])); + expect(rolling).toBeDefined(); + expect(rolling!.suite.manualOnly).toBe(true); + expect(rolling!.suite.definitionMetadata?.retainedControllerContinuation).toBe(true); + expect(controllerOutageForExecution(rolling!, {})).toBeUndefined(); + const [outage] = selectRunnerExecutions(parseRunnerSelectors(["--suite", "remote-workspace-storage"])); + expect(controllerOutageForExecution(outage!, {})).toEqual({ signal: "SIGKILL", holdMs: 300_000 }); + }); +}); diff --git a/tests/runner-e2e/controller-rolling-deployment.ts b/tests/runner-e2e/controller-rolling-deployment.ts new file mode 100644 index 00000000000..75ace959092 --- /dev/null +++ b/tests/runner-e2e/controller-rolling-deployment.ts @@ -0,0 +1,66 @@ +/** Test-owned routing/process fault. Recovery stays entirely in production code. */ +import type { MatrixExecution } from "./types.js"; +import type { ControllerOutageReceipt } from "./controller-outage.js"; + +export const ROLLING_WEB_DEPLOYMENT_SUITE = "rolling-web-deployment"; +export const isRollingWebDeployment = (execution: Pick) => execution.suite.id === ROLLING_WEB_DEPLOYMENT_SUITE; + +export interface ControllerRollingDeploymentReceipt extends Omit { + schema: "paperclip.e2e.controller-rolling-deployment/v1"; + signal: "SIGTERM"; + gracefulShutdown?: boolean; + trafficSwitchedAfterMs?: number; + homeDeletedAfterMs?: number; + previousHomeRemoved?: boolean; + probes?: { reads: number; writes: number; verifiedWrites: number; failures: string[] }; +} + +export function validControllerRollingDeploymentReceipt(value: unknown, requestId?: string): value is ControllerRollingDeploymentReceipt { + if (!value || typeof value !== "object") return false; + const r = value as ControllerRollingDeploymentReceipt; + const times = [r.replacementStartedAfterMs, r.readyAfterMs, r.trafficSwitchedAfterMs, r.stoppedAfterMs, r.homeDeletedAfterMs]; + return r.schema === "paperclip.e2e.controller-rolling-deployment/v1" && r.signal === "SIGTERM" + && /^[A-Za-z0-9._:-]{1,200}$/.test(r.requestId) && (requestId === undefined || r.requestId === requestId) + && Number.isFinite(Date.parse(r.startedAt)) + && Number.isSafeInteger(r.previousPid) && r.previousPid > 0 + && Number.isSafeInteger(r.replacementPid) && r.replacementPid! > 0 && r.previousPid !== r.replacementPid + && r.cacheDiscarded === true && r.previousHomeRemoved === true && r.gracefulShutdown === true + && times.every(ms => typeof ms === "number" && Number.isFinite(ms) && ms >= 0) + && r.replacementStartedAfterMs! <= r.readyAfterMs! && r.readyAfterMs! <= r.trafficSwitchedAfterMs! + && r.trafficSwitchedAfterMs! <= r.stoppedAfterMs! && r.stoppedAfterMs! <= r.homeDeletedAfterMs! + && r.homeDeletedAfterMs! < 420_000 + && Boolean(r.probes && Number.isSafeInteger(r.probes.reads) && r.probes.reads > 0 + && Number.isSafeInteger(r.probes.writes) && r.probes.writes > 0 && r.probes.verifiedWrites === r.probes.writes + && Array.isArray(r.probes.failures) && r.probes.failures.length === 0); +} + +export async function rollController(input: { + requestId: string; previousPid: number; start: () => Promise; + waitForReady: (replacement: T) => Promise; switchTraffic: (replacement: T) => void; + stopPrevious: () => Promise; deletePreviousHome: () => Promise; + beforeHandoff?: (receipt: ControllerRollingDeploymentReceipt) => Promise; + afterPreviousStopped?: (receipt: ControllerRollingDeploymentReceipt) => Promise; + afterHandoff?: (receipt: ControllerRollingDeploymentReceipt, record: () => Promise) => Promise; + cancelled: () => boolean; record: (receipt: ControllerRollingDeploymentReceipt) => Promise; +}): Promise { + const started = performance.now(); + const receipt: ControllerRollingDeploymentReceipt = { schema: "paperclip.e2e.controller-rolling-deployment/v1", + requestId: input.requestId, previousPid: input.previousPid, startedAt: new Date().toISOString(), signal: "SIGTERM" }; + const elapsed = () => performance.now() - started; + const check = () => { if (input.cancelled()) throw new Error("Wrapper shutdown interrupted the rolling deployment"); }; + const record = () => input.record({ ...receipt }); + check(); await record(); + receipt.replacementStartedAfterMs = elapsed(); + const replacement = await input.start(); receipt.replacementPid = replacement.pid; await record(); + await input.waitForReady(replacement); check(); receipt.readyAfterMs = elapsed(); await record(); + // Capture while A still owns its live lease. B may adopt immediately after + // SIGTERM, so an observation taken after A exits would miss that boundary. + await input.beforeHandoff?.(receipt); check(); + input.switchTraffic(replacement); receipt.trafficSwitchedAfterMs = elapsed(); await record(); + await input.stopPrevious(); receipt.gracefulShutdown = true; receipt.stoppedAfterMs = elapsed(); await record(); + await input.afterPreviousStopped?.(receipt); check(); await record(); + await input.deletePreviousHome(); receipt.homeDeletedAfterMs = elapsed(); + receipt.previousHomeRemoved = true; receipt.cacheDiscarded = true; await record(); + await input.afterHandoff?.(receipt, record); + return receipt; +} diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index d37a9d8f264..a699646cbee 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -25,7 +25,9 @@ import { waitForTaskChatRendered, waitForTaskIdentity } from "./continuation-scr import { hasPersistedSource, isSavedSourceCheckpoint } from "./everyday-interruption.js"; import { collectRunEvents } from "./run-observations.js"; import { gradeRemoteRestart, observeRemoteRestartBoundary, projectRemoteRestartEvents, projectRemoteRestartRun, waitForRemoteRestartBoundary, type RemoteRestartEvidence } from "./everyday-remote-restart.js"; -import { controllerOutageForExecution, type ControllerOutageReceipt } from "./controller-outage.js"; +import { controllerOutageForExecution } from "./controller-outage.js"; +import type { ControllerReplacementReceipt } from "./controller-replacement.js"; +import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; import { gradeRemoteWorkspaceStorage, isRemoteWorkspaceStorageExecution, readRemoteWorkspaceStorageCheckpoint, REMOTE_WORKSPACE_PROOF_FILE, type RemoteWorkspaceStorageEvidence } from "./remote-workspace-storage.js"; import { gradeRetainedControllerContinuation, retainedControllerRun, retainedControllerFollowupTransport, submitRetainedControllerComment, onePersistedRetainedControllerComment } from "./retained-controller-continuation.js"; @@ -102,7 +104,7 @@ interface Input { privateDir: string; deadlineAt: number; restartCount?: number; - restart(binding?: import("./controller-recovery-observer.js").RecoveryBinding, controllerDowntimeMs?: number): Promise; + restart(binding?: import("./controller-recovery-observer.js").RecoveryBinding, controllerDowntimeMs?: number): Promise; observe(issue: StoryIssue, runs: StoryRun[]): void; capture(id: string, label: string, file: string): Promise; evidence(name: string, value: unknown): Promise; @@ -182,7 +184,7 @@ export async function runEverydayFlow(input: Input) { let project = fixtures.project; const caseId = execution.task.id; const controllerOutage = controllerOutageForExecution(execution, process.env); - const requireRemoteRestart = controllerOutage !== undefined; + const requireRemoteRestart = controllerOutage !== undefined || isRollingWebDeployment(execution); const remoteWorkspaceStorage = isRemoteWorkspaceStorageExecution(execution); if (remoteWorkspaceStorage) ev.remoteWorkspaceStorage = { browserReadPhases: [] }; const providerChoice = caseId === "provider-decline" || caseId === "provider-second"; @@ -610,6 +612,9 @@ export async function runEverydayFlow(input: Input) { "everyday-remote-restart.ts", "retained-controller-continuation.ts", "controller-outage.ts", + "controller-rolling-deployment.ts", + "controller-replacement.ts", + "rolling-deployment-supervisor.ts", "controller-recovery-observer.ts", "upgrade-artifacts.ts", "server.ts", diff --git a/tests/runner-e2e/everyday-remote-restart.ts b/tests/runner-e2e/everyday-remote-restart.ts index cbc4a87548c..60b12c19de3 100644 --- a/tests/runner-e2e/everyday-remote-restart.ts +++ b/tests/runner-e2e/everyday-remote-restart.ts @@ -3,7 +3,7 @@ import { canonicalJson } from "../../packages/shared/src/portability-hash.js"; import { isValidNativePrpEnvelope } from "./native-event-envelope.js"; import type { StoryCheck, StoryRun } from "./everyday-observations.js"; import { pollUntil } from "./api.js"; -import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js"; +import { validControllerReplacementReceipt, type ControllerReplacementReceipt } from "./controller-replacement.js"; import { validControllerRecoveryReceipt } from "./controller-recovery-observer.js"; type Row = Record; @@ -26,8 +26,8 @@ export interface RemoteRestartBoundary { } export interface RemoteRestartEvidence { expectedReplacements?: number; - replacements?: Array<{ before: RemoteRestartBoundary; after: RemoteRestartBoundary; outage: ControllerOutageReceipt }>; - outage?: ControllerOutageReceipt; + replacements?: Array<{ before: RemoteRestartBoundary; after: RemoteRestartBoundary; outage: ControllerReplacementReceipt }>; + outage?: ControllerReplacementReceipt; before?: RemoteRestartBoundary; after?: RemoteRestartBoundary; final?: { run: StoryRun; events: RemoteRestartEvent[]; operations: Row[]; @@ -194,7 +194,7 @@ export function gradeRemoteRestart(evidence: RemoteRestartEvidence | undefined): const preservesIdentity = digest(step.before.identity) === digest(step.after.identity) && digest(step.before.lease) === digest(step.after.lease) && retainsPrefix(step.before.events, step.after.events); - const authenticated = validControllerOutageReceipt(step.outage) && validControllerRecoveryReceipt(step.outage) + const authenticated = validControllerReplacementReceipt(step.outage) && validControllerRecoveryReceipt(step.outage) && step.outage.recovery?.binding.runId === step.before.identity.runId && step.outage.recovery.binding.companyId === step.before.identity.companyId && step.outage.recovery.binding.issueId === step.before.identity.issueId @@ -210,7 +210,7 @@ export function gradeRemoteRestart(evidence: RemoteRestartEvidence | undefined): check("remote-restart-same-execution-and-lease", sameIdentity, "The original run, native session, admitted runner identity and sandbox lease survive replacement."), check("remote-restart-native-completion-once", nativeCompletion, "The original provider turn completes once and retains its native receipt prefix."), check("remote-restart-files-finalized", Boolean(saved), "The original run commits its workspace receipt and leaves no active operation or recovery."), - check("remote-restart-fault-exercised", validControllerOutageReceipt(evidence?.outage), "The main server exited, its cache was retired, and a distinct replacement became ready after the requested outage."), + check("remote-restart-fault-exercised", validControllerReplacementReceipt(evidence?.outage), "The requested outage or ready-before-handoff replacement occurred, and the departing server state is unavailable."), check("remote-restart-authenticated-recovery", validControllerRecoveryReceipt(evidence?.outage) && evidence?.outage?.recovery?.binding.runId === identity?.runId && evidence?.outage?.recovery?.runnerInstanceId === identity?.runnerInstanceId diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index ec3bd321ab4..f6726c67f75 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -218,7 +218,7 @@ export async function setupLiveFixtures(input: { : ["native-completion", "native-instruction-consolidation", "native-connection-guidance"].includes(execution.suite.id) || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id)) ? NATIVE_COMPLETION_BUDGET_CENTS : execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS - : ["stock-harness", "task-workspaces", "case-handoffs", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage"].includes(execution.suite.id) ? 1_000 : 0, + : ["stock-harness", "task-workspaces", "case-handoffs", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage", "rolling-web-deployment"].includes(execution.suite.id) ? 1_000 : 0, }); await input.onCompanyCreated?.(company.id); return company; @@ -413,7 +413,7 @@ export async function setupLiveFixtures(input: { setupProbe.defaultRunner = selected.defaultRunner; } } - if (["stock-harness", "native-connection-guidance", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage"].includes(execution.suite.id) + if (["stock-harness", "native-connection-guidance", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage", "rolling-web-deployment"].includes(execution.suite.id) || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id))) { agent.budgetMonthlyCents = 1_000; } diff --git a/tests/runner-e2e/playwright.config.ts b/tests/runner-e2e/playwright.config.ts index 4cdf143167c..7acdf6083ff 100644 --- a/tests/runner-e2e/playwright.config.ts +++ b/tests/runner-e2e/playwright.config.ts @@ -72,7 +72,9 @@ export default defineConfig({ // blob reports. The wrapper inherits the test process and strips provider // keys before spawning the real Paperclip process. command: runnerE2EWebServerCommand(repositoryRoot), - gracefulShutdown: runnerE2EWebServerGracefulShutdown, + gracefulShutdown: { ...runnerE2EWebServerGracefulShutdown, + ...(JSON.parse(process.env.PAPERCLIP_RUNNER_E2E_EXECUTION_IDS ?? "[]").some((id: string) => id.startsWith("rolling-web-deployment.")) ? { timeout: 105_000 } : {}), + }, url: `${baseURL}/api/health`, reuseExistingServer: false, // Fresh databases apply the complete migration history before health is ready. diff --git a/tests/runner-e2e/remote-workspace-storage.ts b/tests/runner-e2e/remote-workspace-storage.ts index 4487b2120b7..7a820c60bd9 100644 --- a/tests/runner-e2e/remote-workspace-storage.ts +++ b/tests/runner-e2e/remote-workspace-storage.ts @@ -1,3 +1,4 @@ +import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; import { createHash } from "node:crypto"; import { remoteWorkspaceAuthoritySchema, remoteWorkspaceRunReferenceSchema } from "../../packages/shared/src/remote-workspace.js"; import type { ExecutionWorkspace, WorkspaceFileContent } from "../../packages/shared/src/index.js"; @@ -32,7 +33,7 @@ export interface RemoteWorkspaceStorageEvidence { } export function isRemoteWorkspaceStorageExecution(execution: MatrixExecution): boolean { - return execution.suite.id === REMOTE_WORKSPACE_STORAGE_SUITE; + return execution.suite.id === REMOTE_WORKSPACE_STORAGE_SUITE || isRollingWebDeployment(execution); } export function remoteWorkspaceStoragePrompt(base: string): string { diff --git a/tests/runner-e2e/retained-controller-continuation.ts b/tests/runner-e2e/retained-controller-continuation.ts index 12254ca5e1c..80ceedc314a 100644 --- a/tests/runner-e2e/retained-controller-continuation.ts +++ b/tests/runner-e2e/retained-controller-continuation.ts @@ -2,11 +2,11 @@ import type { RunnerApi } from "./api.js"; import type { MatrixExecution } from "./types.js"; import { createHash } from "node:crypto"; import type { StoryCheck, StoryRun } from "./everyday-observations.js"; -import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js"; +import { validControllerReplacementReceipt, type ControllerReplacementReceipt } from "./controller-replacement.js"; /** Only these explicit recovery definitions change the post-replacement user entry point. */ export function retainedControllerFollowupTransport(execution: Pick): "browser" | "task_comment_api" { - return ["daytona-restart-continuity", "remote-workspace-storage"].includes(execution.suite.id) && execution.task.id === "recover-controller" + return ["daytona-restart-continuity", "remote-workspace-storage", "rolling-web-deployment"].includes(execution.suite.id) && execution.task.id === "recover-controller" && execution.suite.definitionMetadata?.retainedControllerContinuation === true && execution.suite.definitionMetadata.retainedControllerFollowupTransport === "task_comment_api" ? "task_comment_api" : "browser"; @@ -37,13 +37,13 @@ export function retainedControllerRun(run: StoryRun) { export function gradeRetainedControllerContinuation(input: { before: ReturnType; after: ReturnType; - outage: ControllerOutageReceipt | undefined; + outage: ControllerReplacementReceipt | undefined; requestedAt: string; completedTool: boolean; usefulReply: boolean; }): StoryCheck[] { const { before, after, outage } = input; - const passed = validControllerOutageReceipt(outage) + const passed = validControllerReplacementReceipt(outage) && before.status === "succeeded" && after.status === "succeeded" && before.runtimeMode === "native" && after.runtimeMode === "native" && before.runId !== after.runId && before.companyId === after.companyId diff --git a/tests/runner-e2e/rolling-deployment-process.ts b/tests/runner-e2e/rolling-deployment-process.ts index 2951b236452..00377990759 100644 --- a/tests/runner-e2e/rolling-deployment-process.ts +++ b/tests/runner-e2e/rolling-deployment-process.ts @@ -45,6 +45,9 @@ export async function startRollingDeploymentProcess(input: { await cp(path.join(input.repositoryRoot, "server/dist"), path.join(appRoot, "dist"), { recursive: true }); await cp(path.join(input.repositoryRoot, "ui/dist"), path.join(appRoot, "ui-dist"), { recursive: true }); await symlink(path.join(input.repositoryRoot, "server/node_modules"), path.join(appRoot, "node_modules")); + // This lane uses workspace dependencies, whose exports include TypeScript. + // Preserve the normal development plugin loader beside each compiled copy. + await symlink(path.join(input.repositoryRoot, "cli"), path.join(home, "app/cli")); await writeFile(path.join(appRoot, "package.json"), '{"type":"module"}'); for (const omittedAsset of input.browser.omitAssets ?? []) { if (!/^[A-Za-z0-9_.-]+\.js$/.test(omittedAsset)) throw new Error("Unsafe omitted asset fixture"); @@ -55,12 +58,12 @@ export async function startRollingDeploymentProcess(input: { const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(input.repositoryRoot, entry), { cwd: input.repositoryRoot, env: { - PATH: process.env.PATH, HOME: home, TMPDIR: temporary, + ...input.sharedEnvironment, PATH: input.sharedEnvironment.PATH ?? process.env.PATH, HOME: home, TMPDIR: temporary, XDG_CACHE_HOME: path.join(home, "cache"), NODE_ENV: "test", PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: configPath, - PAPERCLIP_INSTANCE_ID: "default", DATABASE_URL: input.databaseUrl, + PORT: String(port), PAPERCLIP_INSTANCE_ID: "default", DATABASE_URL: input.databaseUrl, PAPERCLIP_API_URL: input.ingressOrigin, PAPERCLIP_TELEMETRY_ENABLED: "false", PAPERCLIP_DISABLE_TELEMETRY: "true", PAPERCLIP_LOG_LEVEL: "warn", - HEARTBEAT_SCHEDULER_ENABLED: "false", ...input.sharedEnvironment, + HEARTBEAT_SCHEDULER_ENABLED: input.sharedEnvironment.HEARTBEAT_SCHEDULER_ENABLED ?? "false", }, detached: process.platform !== "win32", stdio: ["ignore", "pipe", "pipe"], }); @@ -70,7 +73,7 @@ export async function startRollingDeploymentProcess(input: { const origin = `http://127.0.0.1:${port}`; let stopped = false; return { - child, home, origin, + child, home, configPath, origin, async waitForReady() { const started = performance.now(); while (performance.now() - started < 180_000) { @@ -84,11 +87,13 @@ export async function startRollingDeploymentProcess(input: { } throw new Error(`Server ${input.name} did not become ready; inspect ${input.name}.log`); }, - async stop() { - if (stopped) return; - await stopper.stopAll(); - await new Promise(resolve => log.end(resolve)); - stopped = true; + async stop(options: { requireGraceful?: boolean } = {}) { + if (!stopped) { + await stopper.stopAll(); + await new Promise(resolve => log.end(resolve)); + stopped = true; + } + if (options.requireGraceful) stopper.assertGracefulExit(child); }, }; } diff --git a/tests/runner-e2e/rolling-deployment-proxy.ts b/tests/runner-e2e/rolling-deployment-proxy.ts index dc910f61d15..3cc15286907 100644 --- a/tests/runner-e2e/rolling-deployment-proxy.ts +++ b/tests/runner-e2e/rolling-deployment-proxy.ts @@ -4,7 +4,7 @@ import type { Socket } from "node:net"; /** Test-owned ingress. Each request keeps the backend selected at admission; * switching traffic never retries a request or terminates its existing stream. */ -export async function startRollingDeploymentProxy() { +export async function startRollingDeploymentProxy(port = 0) { let target: URL | undefined; const agent = new Agent({ keepAlive: true }); const sockets = new Set(); @@ -56,7 +56,7 @@ export async function startRollingDeploymentProxy() { upstream.on("error", () => socket.destroy()); upstream.end(); }); - server.listen(0, "127.0.0.1"); + server.listen(port, "127.0.0.1"); await once(server, "listening"); const address = server.address(); if (!address || typeof address === "string") throw new Error("Proxy address unavailable"); diff --git a/tests/runner-e2e/rolling-deployment-smoke.ts b/tests/runner-e2e/rolling-deployment-smoke.ts index 1485814d2b6..6154d39b9d2 100644 --- a/tests/runner-e2e/rolling-deployment-smoke.ts +++ b/tests/runner-e2e/rolling-deployment-smoke.ts @@ -150,6 +150,15 @@ try { })(); if (browserEnabled) browser = await openRollingDeploymentBrowser({ origin: proxy.origin, root, taskTitle: task.title, taskPath: `/issues/${task.id}`, omittedAssets }); + await check("Unready replacement leaves the healthy server serving", async () => { + const unready = await startRollingDeploymentProcess({ root, name: "unready", repositoryRoot, + databaseUrl: database!.connectionString, ingressOrigin: proxy!.origin, + sharedEnvironment: { ...sharedEnvironment, PAPERCLIP_EPHEMERAL_SERVER: "true", PAPERCLIP_DECISION_SIGNING_SECRET: "" } }); + processes.push(unready); + await assert.rejects(unready.waitForReady(), /exited before readiness/); + assert.equal((await api(proxy!.origin, `/issues/${task.id}`)).id, task.id); + assert.equal((await api(a.origin, `/issues/${task.id}`)).id, task.id); + }); const b = await start("B"); await check("B reads acknowledged task while A is alive", async () => { assert.equal((await api(b.origin, `/issues/${task.id}`)).title, task.title); @@ -185,7 +194,7 @@ try { while (storageProxy!.heldUploads < 1 && Date.now() < deadline) await delay(25); assert.equal(storageProxy!.heldUploads, 1); proxy!.routeTo(b.origin); - stopping = a.stop(); + stopping = a.stop({ requireGraceful: true }); await delay(100); assert.equal(acknowledged, false); storageProxy!.resumeUploads(); @@ -198,7 +207,7 @@ try { }); proxy.routeTo(b.origin); await delay(500); - await a.stop(); + await a.stop({ requireGraceful: true }); await rm(a.home, { recursive: true, force: true }); await delay(1_000); report.handoffElapsedMs = Math.round(performance.now() - switchedAt); @@ -261,7 +270,7 @@ try { if (fixture.seeded) assert.equal((await api(b.origin, `${route}?path=AGENTS.md`)).content, "Durable original"); } }); - await b.stop(); await rm(b.home, { recursive: true, force: true }); + await b.stop({ requireGraceful: true }); await rm(b.home, { recursive: true, force: true }); const c = await start("C", false); proxy.routeTo(c.origin); await check("Fresh C restores files after both homes are deleted, even with adoption opt-in disabled", async () => { assert.equal((await api(c.origin, `${fileRoute}?path=reference.txt`)).content, content); diff --git a/tests/runner-e2e/rolling-deployment-supervisor.test.ts b/tests/runner-e2e/rolling-deployment-supervisor.test.ts new file mode 100644 index 00000000000..13b7f9320a0 --- /dev/null +++ b/tests/runner-e2e/rolling-deployment-supervisor.test.ts @@ -0,0 +1,42 @@ +import { afterEach, expect, it, vi } from "vitest"; +import { createLocalEncryptedCipher } from "../../server/src/secrets/local-encrypted-provider.js"; + +const mocks = vi.hoisted(() => ({ + start: vi.fn(), observer: vi.fn(), stop: vi.fn(), close: vi.fn(), writeFile: vi.fn(), +})); +vi.mock("node:fs/promises", () => ({ mkdir: async () => {}, writeFile: mocks.writeFile, + readFile: async () => JSON.stringify({ database: { mode: "postgres", connectionString: "postgres://fixture" }, + secrets: { localEncrypted: { keyFilePath: "/fixture/secrets/master.key" } } }) })); +vi.mock("../../packages/shared/src/config-schema.js", () => ({ paperclipConfigSchema: { parse: (value: unknown) => value } })); +vi.mock("./rolling-deployment-process.js", () => ({ startRollingDeploymentProcess: mocks.start })); +vi.mock("./rolling-deployment-proxy.js", () => ({ startRollingDeploymentProxy: async () => ({ origin: "http://127.0.0.1:1", routeTo() {}, close: mocks.close }) })); +vi.mock("./controller-recovery-observer.js", () => ({ createControllerRecoveryObserver: mocks.observer })); +vi.mock("./server-restart-request.js", () => ({ readRestartRequest: async () => ({ requestId: "key-check", rollingDeployment: true, recoveryBinding: {} }) })); +import { superviseRollingDeployment } from "./rolling-deployment-supervisor.js"; + +afterEach(() => { vi.unstubAllEnvs(); vi.clearAllMocks(); }); + +it.each([undefined, "prior-disposable-key"])("shares the server key with checkpoint observation and restores %s on failure", async prior => { + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", prior); + let material: ReturnType["encrypt"]>; + mocks.start.mockImplementation(async ({ sharedEnvironment }) => { + expect(process.env.PAPERCLIP_SECRETS_MASTER_KEY).toBe(sharedEnvironment.PAPERCLIP_SECRETS_MASTER_KEY); + expect(mocks.writeFile).toHaveBeenCalledWith("/fixture/secrets/master.key", sharedEnvironment.PAPERCLIP_SECRETS_MASTER_KEY, + { mode: 0o600, flag: "wx" }); + material = createLocalEncryptedCipher("write").encrypt("synthetic checkpoint"); + return { child: { exitCode: null, signalCode: null }, configPath: "/fixture/config.json", + origin: "http://127.0.0.1:2", waitForReady: async () => {}, stop: mocks.stop }; + }); + mocks.observer.mockImplementation(async () => { + expect(createLocalEncryptedCipher("read").decrypt(material)).toBe("synthetic checkpoint"); + throw new Error("synthetic observer failure"); + }); + await expect(superviseRollingDeployment({ temporaryRoot: "/fixture", repositoryRoot: "/fixture", + configPath: "/fixture/config.json", port: 1, executionIds: ["rolling"], cancelled: () => false, + environment: { PAPERCLIP_STORAGE_PROVIDER: "s3" }, + })).rejects.toThrow("synthetic observer failure"); + expect(mocks.observer).toHaveBeenCalledOnce(); + expect(mocks.stop).toHaveBeenCalledOnce(); + expect(mocks.close).toHaveBeenCalledOnce(); + expect(process.env.PAPERCLIP_SECRETS_MASTER_KEY).toBe(prior); +}); diff --git a/tests/runner-e2e/rolling-deployment-supervisor.ts b/tests/runner-e2e/rolling-deployment-supervisor.ts new file mode 100644 index 00000000000..0f3a9814f2c --- /dev/null +++ b/tests/runner-e2e/rolling-deployment-supervisor.ts @@ -0,0 +1,135 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { mkdir, readFile, rename, rm, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js"; +import { createControllerRecoveryObserver } from "./controller-recovery-observer.js"; +import { rollController, validControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js"; +import { readRestartRequest } from "./server-restart-request.js"; +import { startRollingDeploymentProcess } from "./rolling-deployment-process.js"; +import { startRollingDeploymentProxy } from "./rolling-deployment-proxy.js"; +import { runnerE2EServerControlPaths } from "./harness-env.js"; + +type Server = Awaited>; + +/** Isolated Product E2E supervisor. Shares only Postgres, S3 and generated + * secrets. It never patches leases, checkpoints, execution or recovery state. */ +export async function superviseRollingDeployment(input: { + temporaryRoot: string; repositoryRoot: string; configPath: string; port: number; + environment: NodeJS.ProcessEnv; executionIds: string[]; cancelled: () => boolean; +}) { + const config = paperclipConfigSchema.parse(JSON.parse(await readFile(input.configPath, "utf8"))); + if (config.database.mode !== "postgres" || !config.database.connectionString) throw new Error("Rolling qualification requires independently owned Postgres"); + if (input.environment.PAPERCLIP_STORAGE_PROVIDER !== "s3") throw new Error("Rolling qualification requires disposable S3 storage"); + const databaseUrl = config.database.connectionString; + const controls = runnerE2EServerControlPaths(input.temporaryRoot); + const proxy = await startRollingDeploymentProxy(input.port); + const servers: Server[] = []; + const environment = { ...input.environment, + PAPERCLIP_EPHEMERAL_SERVER: "true", PAPERCLIP_DURABLE_AGENT_FILES: "true", PAPERCLIP_SHARED_UI_ASSETS: "true", + PAPERCLIP_DATABASE_LISTEN_URL: databaseUrl, PAPERCLIP_SECRETS_MASTER_KEY: randomBytes(32).toString("base64"), + HEARTBEAT_SCHEDULER_ENABLED: "true", RUN_LOG_S3_BUCKET: input.environment.PAPERCLIP_STORAGE_S3_BUCKET, + RUN_LOG_S3_ENDPOINT: input.environment.PAPERCLIP_STORAGE_S3_ENDPOINT, + RUN_LOG_S3_REGION: input.environment.PAPERCLIP_STORAGE_S3_REGION, + RUN_LOG_S3_FORCE_PATH_STYLE: input.environment.PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE, + }; + const start = async () => { + const server = await startRollingDeploymentProcess({ root: input.temporaryRoot, name: `controller-${servers.length + 1}`, + repositoryRoot: input.repositoryRoot, databaseUrl, ingressOrigin: proxy.origin, sharedEnvironment: environment, browser: {} }); + servers.push(server); + return server; + }; + const json = async (filename: string, value: unknown) => { + await writeFile(`${filename}.tmp`, JSON.stringify(value), { mode: 0o600 }); await rename(`${filename}.tmp`, filename); + }; + const api = async (route: string, method = "GET", body?: object): Promise => { + const response = await fetch(`${proxy.origin}/api${route}`, { method, + headers: { "content-type": "application/json", origin: proxy.origin }, + body: body ? JSON.stringify(body) : undefined, signal: AbortSignal.timeout(10_000) }); + if (!response.ok) throw new Error(`${method} ${route} returned ${response.status}`); + return await response.json() as T; + }; + let current: Server | undefined, lastRequest: string | undefined, probeIssueId: string | undefined; + // The read-only observer uses the production checkpoint loader in this + // isolated supervisor process. Give it the same disposable key as A/B/C. + const previousMasterKey = process.env.PAPERCLIP_SECRETS_MASTER_KEY; + process.env.PAPERCLIP_SECRETS_MASTER_KEY = environment.PAPERCLIP_SECRETS_MASTER_KEY; + try { + // Keep the fixture's isolation evidence for its one generated stable secret. + // It is outside A/B/C homes, like the independent database and object store; + // each server receives this exact value through its environment. + const keyPath = path.resolve(config.secrets.localEncrypted.keyFilePath); + if (!keyPath.startsWith(`${path.resolve(input.temporaryRoot)}${path.sep}`)) throw new Error("Rolling fixture key escaped its owned root"); + await mkdir(path.dirname(keyPath), { recursive: true, mode: 0o700 }); + await writeFile(keyPath, environment.PAPERCLIP_SECRETS_MASTER_KEY, { mode: 0o600, flag: "wx" }); + current = await start(); await current.waitForReady(); proxy.routeTo(current.origin); + while (!input.cancelled()) { + if (current.child.exitCode !== null || current.child.signalCode !== null) throw new Error("Serving rolling controller exited unexpectedly"); + const request = await readRestartRequest(controls.restartRequestPath, input.executionIds); + if (!request || request.requestId === lastRequest) { await delay(100); continue; } + lastRequest = request.requestId; + if (!request.rollingDeployment) throw new Error("Rolling supervisor requires an explicit rolling deployment request"); + const previous = current; + const observer = request.recoveryBinding ? await createControllerRecoveryObserver(previous.configPath, request.recoveryBinding) : undefined; + let probing = false, probes: Promise | undefined; + const stats = { reads: 0, writes: 0, verifiedWrites: 0, failures: [] as string[] }; + const writes: Array<{ id: string; body: string }> = []; + try { + if (!probeIssueId) { + const companyId = request.recoveryBinding?.companyId ?? (await api<{ id: string }>("/companies", "POST", { name: "Synthetic deployment probes" })).id; + const issue = await api<{ id: string }>(`/companies/${companyId}/issues`, "POST", { + title: "Synthetic deployment read/save probe", description: "Unassigned fixture for ordinary user saves during deployment.", status: "todo", + }); + probeIssueId = issue.id; + } + const issueRoute = `/issues/${probeIssueId}`; + probing = true; + probes = (async () => { + while (probing) { + try { await api(issueRoute); stats.reads++; } catch (error) { stats.failures.push(String(error)); } + if (stats.reads % 10 === 0 || stats.writes === 0) { + const body = `Synthetic deployment save ${randomUUID()}`; + try { + const saved = await api<{ id: string }>(`${issueRoute}/comments`, "POST", { body, clientRequestId: randomUUID() }); + writes.push({ id: saved.id, body }); stats.writes++; + } catch (error) { stats.failures.push(String(error)); } + } + await delay(100); + } + })(); + const receiptPath = path.join(controls.controlDirectory, `controller-rolling-${request.requestId}.json`); + const receipt = await rollController({ requestId: request.requestId, previousPid: previous.child.pid!, + start: async () => { const server = await start(); return { pid: server.child.pid, server }; }, + waitForReady: async replacement => { await replacement.server.waitForReady(); }, + switchTraffic: replacement => { current = replacement.server; proxy.routeTo(current.origin); }, + stopPrevious: () => previous.stop({ requireGraceful: true }), + deletePreviousHome: () => rm(previous.home, { recursive: true }), + beforeHandoff: observer?.beforeHandoff, + afterPreviousStopped: observer?.stoppedAfterHandoff, + afterHandoff: observer ? (receipt, record) => observer.recovered(receipt, record, input.cancelled) : undefined, + cancelled: input.cancelled, record: value => json(receiptPath, value), + }); + probing = false; await probes; + const saved = await api>(`${issueRoute}/comments`); + stats.verifiedWrites = writes.filter(write => saved.filter(comment => comment.id === write.id && comment.body === write.body).length === 1).length; + receipt.probes = stats; await json(receiptPath, receipt); + if (!validControllerRollingDeploymentReceipt(receipt, request.requestId)) throw new Error("Rolling boundary failed overlap, empty-home, or ordinary read/save qualification"); + await json(controls.restartAcknowledgementPath, { requestId: request.requestId, status: "ready", controllerCacheDiscarded: true, + completedAt: new Date().toISOString(), rollingDeployment: receipt }); + } catch (error) { + await json(controls.restartAcknowledgementPath, { requestId: request.requestId, status: "failed", message: String(error) }); + throw error; + } finally { probing = false; await probes; await observer?.close(); } + } + } finally { + // Every started server remains owned, including an unready successor. + // End-of-attempt cleanup also runs after a retained failing receipt. + try { + try { await Promise.all(servers.map(server => server.stop())); } + finally { await proxy.close(); } + } finally { + if (previousMasterKey === undefined) delete process.env.PAPERCLIP_SECRETS_MASTER_KEY; + else process.env.PAPERCLIP_SECRETS_MASTER_KEY = previousMasterKey; + } + } +} diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 988ce11cb53..d479f882fd3 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1,3 +1,5 @@ +import { isRollingWebDeployment, validControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js"; +import type { ControllerReplacementReceipt } from "./controller-replacement.js"; import { kimiCandidateAdmissionForExecution } from "./server-restart-request.js"; import { runPlanRevisionAcceptance, isPendingPlanConfirmation, normalizePlanMarkdown, expectPlanStageVisible } from "./plan-revision-flow.js"; import { createWorkerTiming } from "./worker-timing.js"; @@ -273,8 +275,9 @@ async function restartIsolatedPaperclipServer(input: { crashAfterResult?: FinalizationCrashBinding; controllerDowntimeMs?: number; controllerOutage?: ControllerOutage; + rollingDeployment?: true; recoveryBinding?: import("./controller-recovery-observer.js").RecoveryBinding; - onControllerOutage?: (receipt: ControllerOutageReceipt) => Promise; + onControllerOutage?: (receipt: ControllerReplacementReceipt) => Promise; }): Promise { const { controlDirectory, @@ -287,6 +290,7 @@ async function restartIsolatedPaperclipServer(input: { temporaryRequestPath, JSON.stringify({ requestId: input.requestId, ...(input.kimiCandidateAdmission ? { kimiCandidateAdmission: input.kimiCandidateAdmission } : {}), ...(input.controllerDowntimeMs !== undefined && !input.controllerOutage ? { controllerDowntimeMs: input.controllerDowntimeMs } : {}), discardControllerCache: input.discardControllerCache === true, ...(input.controllerOutage ? { controllerOutage: input.controllerOutage } : {}), + ...(input.rollingDeployment ? { rollingDeployment: true } : {}), ...(input.recoveryBinding ? { recoveryBinding: input.recoveryBinding } : {}), ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}), ...(input.crashAfterResult ? { crashAfterResult: input.crashAfterResult } : {}) }), @@ -312,6 +316,7 @@ async function restartIsolatedPaperclipServer(input: { acknowledgement.requestId === input.requestId && acknowledgement.status === "ready" && (!input.discardControllerCache || acknowledgement.controllerCacheDiscarded === true) && + (!input.rollingDeployment || validControllerRollingDeploymentReceipt(acknowledgement.rollingDeployment, input.requestId)) && (!input.controllerOutage || validControllerOutageReceipt(acknowledgement.controllerOutage, input.controllerOutage, input.requestId)), reject: (acknowledgement) => acknowledgement.requestId === input.requestId && @@ -327,7 +332,7 @@ async function restartIsolatedPaperclipServer(input: { load: () => input.api.get>("/api/health"), accept: (health) => Boolean(health), }); - if (input.controllerOutage) await input.onControllerOutage?.(acknowledgement.controllerOutage as ControllerOutageReceipt); + if (input.controllerOutage || input.rollingDeployment) await input.onControllerOutage?.((input.rollingDeployment ? acknowledgement.rollingDeployment : acknowledgement.controllerOutage) as ControllerReplacementReceipt); } const executionIds = (() => { @@ -1256,13 +1261,14 @@ for (const execution of executions) { restartCount: upgradeManifest ? upgradeManifest.sequence.length - (execution.suite.definitionMetadata?.retainedControllerContinuation === true ? 2 : 1) : 1, restart: async (recoveryBinding, controllerDowntimeMs) => { - let receipt: ControllerOutageReceipt | undefined; + let receipt: ControllerReplacementReceipt | undefined; restartIndex += 1; await restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}-${restartIndex}`, deadlineAt: startedAtMs + deadlineMs, - discardControllerCache: controllerOutage !== undefined, + discardControllerCache: controllerOutage !== undefined || isRollingWebDeployment(execution), + ...(isRollingWebDeployment(execution) ? { rollingDeployment: true as const } : {}), controllerOutage: controllerOutage && !recoveryBinding ? { ...controllerOutage, holdMs: 1000 } : controllerOutage, - controllerDowntimeMs, - ...(controllerOutage ? { recoveryBinding } : {}), + ...(!isRollingWebDeployment(execution) ? { controllerDowntimeMs } : {}), + ...((controllerOutage || isRollingWebDeployment(execution)) ? { recoveryBinding } : {}), onControllerOutage: async value => { receipt = value; await writeSanitizedJson(snapshotsDir, `controller-outage-${restartIndex}.json`, value, secrets); diff --git a/tests/runner-e2e/server-restart-request.ts b/tests/runner-e2e/server-restart-request.ts index fa246ee2ab1..dc335c83c3b 100644 --- a/tests/runner-e2e/server-restart-request.ts +++ b/tests/runner-e2e/server-restart-request.ts @@ -1,3 +1,4 @@ +import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; import { readFile } from "node:fs/promises"; import { runnerMatrix } from "./catalog.js"; import { parseAcpxCandidateAdmission, type AcpxCandidateAdmission } from "../../packages/paperclip-runner/src/drivers/acpx/candidate-admission.js"; @@ -17,6 +18,7 @@ export interface RestartRequest { discardControllerCache?: boolean; crashAfterResult?: FinalizationCrashBinding; controllerOutage?: ControllerOutage; + rollingDeployment?: true; recoveryBinding?: RecoveryBinding; } @@ -87,22 +89,29 @@ export async function readRestartRequest(restartRequestPath: string, executionId || Number(controllerDowntimeMs) < 0 || Number(controllerDowntimeMs) > 300_000)) { throw new Error("Controller downtime must be an integer between zero and five minutes"); } + const rollingDeployment = (value as { rollingDeployment?: unknown }).rollingDeployment; + if (rollingDeployment !== undefined && (rollingDeployment !== true || executionIds.length !== 1 + || !executionIds.every(id => { const execution = runnerMatrix.find(row => row.id === id); return execution && isRollingWebDeployment(execution); }))) { + throw new Error("Rolling deployment requires its explicit isolated cell"); + } const discardControllerCache = (value as { discardControllerCache?: unknown }).discardControllerCache === true; if (discardControllerCache && !executionIds.some(id => { const execution = runnerMatrix.find(row => row.id === id); - return execution !== undefined && controllerOutageForExecution(execution, {}) !== undefined; + return execution !== undefined && (controllerOutageForExecution(execution, {}) !== undefined || isRollingWebDeployment(execution)); })) throw new Error("Controller cache replacement requires an explicit remote controller-outage cell"); const crashAfterResult = (value as { crashAfterResult?: unknown }).crashAfterResult; const controllerOutage = (value as { controllerOutage?: unknown }).controllerOutage; const recoveryBinding = (value as { recoveryBinding?: unknown }).recoveryBinding; - if (recoveryBinding !== undefined && controllerOutage === undefined) throw new Error("Recovery observation requires a controller outage"); + if (recoveryBinding !== undefined && controllerOutage === undefined && !rollingDeployment) throw new Error("Recovery observation requires a controller outage"); + if (rollingDeployment && (!discardControllerCache || controllerOutage !== undefined || controllerDowntimeMs !== undefined + || preserveRunner !== undefined || crashAfterResult !== undefined)) throw new Error("Rolling deployment cannot combine an outage or another fault"); if (controllerOutage !== undefined && (!discardControllerCache || preserveRunner !== undefined || crashAfterResult !== undefined)) { throw new Error("Controller outage requires the explicit remote cache-replacement cell without another crash boundary"); } if (crashAfterResult !== undefined && (preserveRunner !== undefined || discardControllerCache)) { throw new Error("A finalization crash cannot preserve a running provider or discard its controller cache"); } - return { requestId, discardControllerCache, ...(kimiCandidateAdmission ? { kimiCandidateAdmission } : {}), ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }), + return { requestId, discardControllerCache, ...(rollingDeployment ? { rollingDeployment: true as const } : {}), ...(kimiCandidateAdmission ? { kimiCandidateAdmission } : {}), ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }), ...(recoveryBinding === undefined ? {} : { recoveryBinding: parseRecoveryBinding(recoveryBinding) }), ...(controllerOutage === undefined ? {} : { controllerOutage: parseControllerOutage(controllerOutage) }), ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }), diff --git a/tests/runner-e2e/server-stop.test.ts b/tests/runner-e2e/server-stop.test.ts index 2747ccf01ad..1655146b30c 100644 --- a/tests/runner-e2e/server-stop.test.ts +++ b/tests/runner-e2e/server-stop.test.ts @@ -89,6 +89,7 @@ it.skipIf(process.platform === "win32")("joins signal, final cleanup and repeate expect(stop(child)).toBe(first); await stop(child); expect(logs).toEqual([]); + expect(() => stop.assertGracefulExit(child)).not.toThrow(); }); it.skipIf(process.platform === "win32")("bounds an unresponsive child and reaps it with one escalation", async () => { @@ -98,6 +99,7 @@ it.skipIf(process.platform === "win32")("bounds an unresponsive child and reaps const { stop, logs } = stopper(50); await Promise.all([stop(child), stop(child), stop(child)]); expect(child.signalCode).toBe("SIGKILL"); + expect(() => stop.assertGracefulExit(child)).toThrow("forced cleanup cannot qualify"); expect(logs).toHaveLength(1); expect(() => process.kill(child.pid!, 0)).toThrow(); }); diff --git a/tests/runner-e2e/server-stop.ts b/tests/runner-e2e/server-stop.ts index c1199460541..a74cff18614 100644 --- a/tests/runner-e2e/server-stop.ts +++ b/tests/runner-e2e/server-stop.ts @@ -105,6 +105,12 @@ export function createRunnerE2EServerStopper(options: { return state.promise; }; return Object.assign(stop, { + assertGracefulExit: (child: ChildProcess): void => { + const state = states.get(child); + if (!state?.finalComplete || state.escalationLogged || child.exitCode !== 0 || child.signalCode !== null) { + throw new Error("Controller did not exit gracefully; forced cleanup cannot qualify a rolling deployment"); + } + }, watch: (child: ChildProcess) => watch(child).owner.observe(), crashForRestart: (child: ChildProcess): Promise => { if (process.platform === "win32") return Promise.reject(new Error("Owned crash restart requires process-group inspection")); diff --git a/tests/runner-e2e/server.ts b/tests/runner-e2e/server.ts index 3b792973f62..a68737020a3 100644 --- a/tests/runner-e2e/server.ts +++ b/tests/runner-e2e/server.ts @@ -1,3 +1,5 @@ +import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; +import { superviseRollingDeployment } from "./rolling-deployment-supervisor.js"; import { readRestartRequest, type RestartRequest } from "./server-restart-request.js"; import { prepareControllerDatabase } from "./controller-database.js"; import { readUpgradeArtifacts, verifyUpgradeArtifact, assertServingArtifact, upgradeArtifactKey, type ServingArtifact } from "./upgrade-artifacts.js"; @@ -451,6 +453,12 @@ async function supervise() { // leave Postgres available to the stopped-controller observer and successor. controllerDatabase = await prepareControllerDatabase(configPath, temporaryRoot); } + if (executionIds.some(id => { const execution = runnerMatrix.find(row => row.id === id); return execution && isRollingWebDeployment(execution); })) { + if (upgradeArtifacts || installedRelease) throw new Error("Rolling source qualification cannot claim installed-artifact evidence"); + await superviseRollingDeployment({ temporaryRoot, repositoryRoot, configPath, port: Number(port), + environment: definedServerEnvironment, executionIds, cancelled: shutdownRequested }); + return; + } const initial = await startServer(); if (upgradeArtifacts) await waitForHealth(initial); let lastRestartRequestId: string | null = null;