Files
PaperClipAI/tests/runner-e2e/controller-rolling-deployment.test.ts
T
DottaandPaperclip 16787e4167 Qualify rolling web deployments and preserve Stop through handoff (#18526)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The web server owns request handling and remote execution
coordination.
> - Deployment must start a ready replacement while the old server still
serves.
> - An outage test does not prove that two live servers can overlap
safely.
> - This pull request adds a separate Product E2E cell for that
boundary.
> - The existing run, file, recovery, and retained-session checks still
apply.

## Linked Issues or Issue Description

Refs #18518. The runtime prerequisite is merged; this PR qualifies its
ephemeral server and deployment contracts.

## What Changed

- Add the explicit `rolling-web-deployment` Codex Daytona workflow.
- Start independent A/B/C homes and compiled builds behind a test
ingress.
- Wait for readiness before routing, then drain and delete the old home.
- Observe active authority before handoff and use the existing read-only
recovery oracle.
- Require ordinary reads and saves to succeed throughout replacement.
Read back every accepted save exactly once.
- Preserve the five-minute outage suites and their existing validators.
- Add a model-free supervisor smoke and an unready replacement control.
- Require graceful exit before deleting an old home; forced cleanup
cannot satisfy the oracle.
- Preserve a durable Stop through lease release/expiry until the
recovered execution owner acknowledges it. Permit only fenced
reattachment of the original execution for Stop cleanup; submit no new
turn.
- Exclude events saved by A during shutdown from evidence of B making
progress.

## Verification

- E2E TypeScript check passed.
- Affected harness checks: 143 passed. Catalog, fixture and report
checks: 120 passed.
- Actual supervisor smoke passed two replacements: 436 reads, 45 saves,
zero request failures, and confirmed owned resource cleanup.
- B became ready in 18.7 seconds; C became ready in 109.2 seconds. A
prior C timeout remains failed evidence.
- Fresh browser proof: 11 checks passed, 246 reads, 50 acknowledged
saves, zero failures, 1,058ms reconnect, both old homes deleted and
cleanup confirmed.
- Stop lease-gap regression: three PostgreSQL delivery tests passed with
a simulated provider edge. Current server TypeScript build/vendoring
passed.
- Full harness: 5,814 passed, 17 failed, four skipped. Two shutdown
races passed the focused file; two report timeouts passed an isolated
42-test rerun. Onboarding/workflow-security/Pi/date-expired fixtures are
unchanged on inspected master; the host embedded PostgreSQL fixture
remains a setup limitation. This is not a green full local suite.
- CI at `172509b8be` passed all gates. Greptile is 5/5 with every review
thread resolved. The earlier access-service race failure passed its
isolated regression and the subsequent full CI.
- Final review corrections: six focused executor cases, three PostgreSQL
command-delivery cases, 19 harness cases and server/E2E typechecks
passed.
- Three live attempts failed before agent creation: two harness setup
gaps were fixed, then provider authentication failed. The saved OpenAI
key returns 401; a refresh is pending. The Daytona probe reported
cleanup uncertain, and no sandbox with that attempt label was found
through the authenticated staging manager. No live model execution is
claimed.
- Live Runner overlap, released-container qualification and hosted
deployment proof remain pending. Merging this opt-in qualification lane
does not claim those results.

## Risks

- The source lane shares workspace dependencies and does not establish
container packaging correctness.
- The live workflow needs an explicitly owned S3 bucket, PostgreSQL,
pinned remote image and provider credentials.
- Qualification cleanup stops all owned processes after failure. A
failed replacement must not be reported as a pass.

## Model Used

OpenAI Codex, GPT-6. Tool use, code execution and repository review. The
runtime does not expose a more specific model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-01 02:53:27 -06:00

59 lines
3.6 KiB
TypeScript

import { describe, expect, it } from "vitest";
import { rollController, validControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js";
import { validControllerReplacementReceipt } from "./controller-replacement.js";
import { controllerOutageForExecution, validControllerOutageReceipt } from "./controller-outage.js";
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
function fixture() {
const order: string[] = [];
return { order, input: { requestId: "rolling-1", previousPid: 100,
start: async () => { order.push("start-B"); return { pid: 200 }; },
waitForReady: async () => { order.push("B-ready"); },
beforeHandoff: async () => { order.push("observe-A"); },
switchTraffic: () => { order.push("route-B"); },
stopPrevious: async () => { order.push("stop-A"); },
afterPreviousStopped: async () => { order.push("observe-A-stopped"); },
deletePreviousHome: async () => { order.push("delete-home-A"); },
afterHandoff: async () => { order.push("observe-recovery"); },
cancelled: () => false, record: async () => {},
} };
}
describe("rolling controller qualification", () => {
it("moves traffic only after readiness, then removes all predecessor state", async () => {
const f = fixture();
const receipt = await rollController(f.input);
expect(f.order).toEqual(["start-B", "B-ready", "observe-A", "route-B", "stop-A", "observe-A-stopped", "delete-home-A", "observe-recovery"]);
expect(validControllerRollingDeploymentReceipt(receipt)).toBe(false); // A process receipt alone is not availability proof.
receipt.probes = { reads: 10, writes: 3, verifiedWrites: 3, failures: [] };
expect(validControllerReplacementReceipt(receipt)).toBe(true);
expect(validControllerOutageReceipt(receipt)).toBe(false);
for (const changed of [
{ previousHomeRemoved: false }, { cacheDiscarded: false }, { replacementPid: 100 }, { gracefulShutdown: false },
{ trafficSwitchedAfterMs: -1 }, { readyAfterMs: receipt.stoppedAfterMs! + 1 },
{ probes: { reads: 10, writes: 3, verifiedWrites: 2, failures: [] } },
{ probes: { reads: 10, writes: 3, verifiedWrites: 3, failures: ["502"] } },
]) expect(validControllerReplacementReceipt({ ...receipt, ...changed })).toBe(false);
expect(validControllerRollingDeploymentReceipt(receipt, "other-request")).toBe(false);
});
it("keeps A serving when B never becomes ready", async () => {
const f = fixture(); f.input.waitForReady = async () => { throw new Error("B unready"); };
await expect(rollController(f.input)).rejects.toThrow("B unready");
expect(f.order).toEqual(["start-B"]);
});
it("rejects forced predecessor cleanup before deleting its home or observing recovery", async () => {
const f = fixture(); f.input.stopPrevious = async () => { throw new Error("forced shutdown"); };
await expect(rollController(f.input)).rejects.toThrow("forced shutdown");
expect(f.order).toEqual(["start-B", "B-ready", "observe-A", "route-B"]);
});
it("keeps the original outage contract separate from the explicit rolling cell", () => {
const [rolling] = selectRunnerExecutions(parseRunnerSelectors(["--suite", "rolling-web-deployment"]));
expect(rolling).toBeDefined();
expect(rolling!.suite.manualOnly).toBe(true);
expect(rolling!.suite.definitionMetadata?.retainedControllerContinuation).toBe(true);
expect(controllerOutageForExecution(rolling!, {})).toBeUndefined();
const [outage] = selectRunnerExecutions(parseRunnerSelectors(["--suite", "remote-workspace-storage"]));
expect(controllerOutageForExecution(outage!, {})).toEqual({ signal: "SIGKILL", holdMs: 300_000 });
});
});