diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index e1a3e703eb5..46b11781627 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -879,6 +879,59 @@ not click Retry and no other fault fires until the last replacement is ready. A task restarts again after the same gaps. A replacement that exits on its own during a gap fails the fault as a harness error, with the stops so far recorded. +## Two servers, one database (MS1–MS3) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ + --adapters codex --workloads B5,Q1 --faults MS1,MS2 --concurrency 1 --tasks 3 --seed 301 +``` + +Every other fault runs one server per wave, so no wave can reach a race between two servers. #20508 fixed +one such race with a database test only: server B's orphan sweep failed a run whose owner, server A, had +taken its lease back. + +**Why this shape.** Hosted Cloud deploys by rolling overlap on one Neon database. The new server B starts +while the old server A still serves, then A drains. The defaults are a 5 s overlap and up to 90 s of +drain (`doc/plans/2026-10-31-zero-downtime-web-deploys.md`). B's startup recovery and periodic sweeps +therefore run beside A's live runs. The plan's K12 names the same overlap +(`doc/plans/2026-11-06-control-plane-restart-chaos.md`). This repository has no replica count or deploy +strategy of its own (`docker/ecs-task-definition.json` describes one task), and the self-hosting docs do +not describe several servers on one database. + +| Fault | What happens | Seeded hold | +| --- | --- | --- | +| MS1 | B boots on the same database and stays up beside a live A for the hold, then gets SIGTERM. A is not touched. | 60–150 s of overlap | +| MS2 | B boots. A and its process tree are frozen (SIGSTOP, TM2's freeze without aging) past the 90 s controller lease, then thawed. B stays up 30 s more, while A takes its leases back, then stops. | 100–150 s frozen | +| MS3 | Rolling-upgrade shape. B boots, then A gets K2 (SIGTERM) and a replacement on A's port. B is the only live server until the replacement is ready, then stops after the hold. | 20–45 s tail | + +B shares A's database, instance directory, local storage and secrets. It listens on its own loopback port +and writes `/server-b.log`. The harness and the agents keep using A's port. While a fault is in +progress the observers hold board actions (Stop, answers, follow-ups) until it ends, as for every fault. + +Limits: + +- B runs on the same host and PID namespace as A. It can check A's PIDs directly, which a Railway + replica cannot, so these faults are a lower bound on cross-server exposure. +- The native executor lease is 20 minutes. MS2 outlasts the 90 s controller lease, not that one. +- Local only: a remote runner is under neither server. No storm, crash point, K14, TM2 or TM3 in the same wave (aging would move the database times the invariant compares with real process windows). +- B must live through its whole window. If it exits on its own, the fault fails (its tree is still cleaned up). + +**Cross-server invariant** (`multi-server.ts`, `/cross-server.json`). It is checked once per wave +that fired an MS fault, over the wave database and the process windows the supervisor recorded (each +server's pid, start, exit and freezes). A violation fails the wave. + +- `MS_foreign_failure`: a run failed with `process_lost` or `native_execution_ownership_unverified` while + its owning server was alive and not frozen. The owner comes from the run's controller boot id and + `controller_instances.pid`. The time is the run's `finished_at`. A freeze before the write does not + excuse it: that is #20508's shape. A run still `running` with + that code (how #20508's marker is stored) counts at its `updated_at`. The owner's own + `controller_lease_lost` counts only when no freeze past the lease came before it. +- `MS_double_executor`: a run recorded a second local runner identity + (`native.process_identity_recorded`) before the first was shown stopped (`native.local_process_stopped`). +- Ownership failures whose owner is not a wave server (or is unknown) are listed as `unattributed`. + +The usual R1–R6, U1–U4 and T checks grade every task as before. + ## Fault timing (`--triggers`) Each task samples one trigger with its cell. `time` (default) fires at the @@ -923,6 +976,7 @@ re-arms the task's own triggered faults instead of using their offsets. | `acpx-provider-faults.ts`, `provider-shim/acpx-*` | PV1–PV3 for Claude (ACPX): the stand-in runner package, the server hook, the sidecar/provider tap, and `--provider-fault-restart`. | | `question-handoff.ts` | Q1 saved question: the board's answer step and its oracle. | | `time-faults.ts`, `time-faults-config.ts` | TM1–TM3: process freezing, database aging with recorded rows, and the time-corrected API the grader reads through. | +| `multi-server.ts` | MS1–MS3: the second server's steps, the seeded holds, and the cross-server invariant over the wave database. | | `repeated-restarts.ts` | RR1–RR3: seeded restart steps, `--rr-gap-seconds`, and the stop-and-replace loop the supervisor runs. | | `daytona-sweep.ts` | Daytona R6: read-only reconciliation of this wave's labelled sandboxes with leases; cleanup only through Paperclip. | diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts index d8f702f1f00..aab847bef0c 100644 --- a/tests/restart-chaos/config.ts +++ b/tests/restart-chaos/config.ts @@ -4,6 +4,7 @@ import { parseArgs } from "node:util"; import { applyJourneyOptions } from "./journey-matrix.js"; import { takesJourneyTriggers } from "./journeys.js"; import { STORAGE_FAULT_KINDS, type StorageFaultKind } from "./storage-chaos.js"; +import { validateMultiServerOptions } from "./multi-server.js"; import { parseTimeFaultOptions } from "./time-faults-config.js"; import { parseStorageTrigger } from "./storage-trigger.js"; import { parseRepeatedRestartOptions } from "./repeated-restarts.js"; @@ -223,6 +224,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC || values["provider-fault-restart"] !== undefined || values["provider-fault-restart-after-seconds"] !== undefined) { throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults"); } + validateMultiServerOptions(options); Object.assign(options, parseTimeFaultOptions(options, values["time-age-hours"])); Object.assign(options, parseRepeatedRestartOptions(options, values["rr-gap-seconds"], values["rr-stop-on"])); const upgradeFrom = values["upgrade-from"]; diff --git a/tests/restart-chaos/journey-matrix.ts b/tests/restart-chaos/journey-matrix.ts index b88571c51b0..6acb49229ba 100644 --- a/tests/restart-chaos/journey-matrix.ts +++ b/tests/restart-chaos/journey-matrix.ts @@ -14,6 +14,7 @@ import { WORKLOAD_TRIGGERS } from "./journeys.js"; import { withProviderFaultRules } from "./provider-faults.js"; import { createRng, k4HoldRange, TRIGGER_DELAY_RANGE_MS } from "./scheduler.js"; import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js"; +import { isMultiServerFault, multiServerHoldMs } from "./multi-server.js"; import { isRepeatedRestartFault, planRestartSteps } from "./repeated-restarts.js"; import { workloadDefinitions } from "./workloads.js"; import type { AdapterId, TaskFaultKind, TaskPlan, TriggerId, WaveOptions, WavePlan, WorkloadId } from "./types.js"; @@ -57,7 +58,8 @@ export function planMatrixTasks(options: Pick { rules: Object.fromEntries(Object.entries(grade.rules).map(([id, rule]) => [id, rule.verdict])), T: grade.T.verdict, universal: Object.fromEntries(Object.entries(grade.universal ?? {}).map(([id, rule]) => [id, `${rule.verdict}${rule.enforced ? "" : " (warning)"}`])) }); } + // MS1–MS3: the cross-server invariant over the whole wave (multi-server.ts). + if (faults.some(receipt => receipt.multiServer)) { + const crossServer = await readCrossServerEvidence(evidenceDb, lab.companyId) + .then(evidence => checkCrossServer({ ...evidence, windows: supervisor.serverWindows })) + .catch(error => { harnessError ??= new Error(`Cross-server invariant check failed: ${String(error)}`); return null; }); + if (crossServer) { + await writeFile(path.join(waveDir, "cross-server.json"), `${JSON.stringify(crossServer, null, 2)}\n`, { mode: 0o600 }); + timeline.add("cross_server_invariant", { violations: crossServer.violations.map(violation => ({ rule: violation.rule, runId: violation.runId, + issueId: violation.issueId, detail: violation.detail })), unattributed: crossServer.unattributed.length }); + if (crossServer.violations.length) anyFailure = true; + } + } if (storageAuditor) { storageChaos?.stop(); await credentialRotation?.settled(); diff --git a/tests/restart-chaos/multi-server-db.test.ts b/tests/restart-chaos/multi-server-db.test.ts new file mode 100644 index 00000000000..a620f392676 --- /dev/null +++ b/tests/restart-chaos/multi-server-db.test.ts @@ -0,0 +1,58 @@ +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { createDb, type Db } from "../../packages/db/src/index.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../packages/db/src/test-embedded-postgres.js"; +import { checkCrossServer, readCrossServerEvidence, type ServerProcessWindow } from "./multi-server.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const COMPANY = "00000000-0000-4000-8000-000000000001"; +const NOON = Date.parse("2026-11-10T12:00:00.000Z"); +const iso = (ms: number) => new Date(ms).toISOString(); + +describe.skipIf(!support.supported)("reading cross-server evidence from a wave database", () => { + let temporary: Awaited>; + let db: Db; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("chaos-ms-"); + db = createDb(temporary.connectionString, { maxConnections: 1 }); + // Only the columns the reader selects, with the product's names and types, in a schema ahead of the + // migrated one (one connection, so the search path holds for every query). + await db.$client.unsafe(`create schema chaos_ms`); + await db.$client.unsafe(`set search_path to chaos_ms, public`); + await db.$client.unsafe(`create table heartbeat_runs (id uuid primary key, company_id uuid not null, issue_id uuid, status text not null, + error_code text, finished_at timestamptz, updated_at timestamptz not null, controller_boot_id uuid)`); + await db.$client.unsafe(`create table native_run_finalizations (run_id uuid primary key, controller_boot_id text, lease_holder jsonb)`); + await db.$client.unsafe(`create table controller_instances (boot_id text primary key, pid integer not null)`); + await db.$client.unsafe(`create table heartbeat_run_events (run_id uuid not null, event_type text not null, payload jsonb, created_at timestamptz not null)`); + await db.$client.unsafe(`insert into controller_instances values ('boot-a', 100), ('boot-b', 200)`); + // #20508's marker: still running, with the ownership code, written by another server's sweep. + await db.$client.unsafe(`insert into heartbeat_runs values ('00000000-0000-4000-8000-0000000000a1', $1, null, 'running', + 'native_execution_ownership_unverified', null, $2, null)`, [COMPANY, iso(NOON)]); + await db.$client.unsafe(`insert into native_run_finalizations values ('00000000-0000-4000-8000-0000000000a1', 'boot-a', null)`); + // An ordinary success: not read as a failure. + await db.$client.unsafe(`insert into heartbeat_runs values ('00000000-0000-4000-8000-0000000000a2', $1, null, 'succeeded', null, $2, $2, null)`, [COMPANY, iso(NOON)]); + await db.$client.unsafe(`insert into heartbeat_run_events values ('00000000-0000-4000-8000-0000000000a2', 'native.process_identity_recorded', '{"processPid": 11}', $1), + ('00000000-0000-4000-8000-0000000000a2', 'native.process_identity_recorded', '{"processPid": 12}', $2)`, [iso(NOON - 2_000), iso(NOON - 1_000)]); + }, 120_000); + + afterAll(async () => { + await db?.$client.end(); + await temporary?.cleanup(); + }); + + it("reads a running run that carries the ownership marker, at its write time, and charges it to a live owner", async () => { + const evidence = await readCrossServerEvidence(db, COMPANY); + expect(evidence.runs).toEqual([expect.objectContaining({ runId: "00000000-0000-4000-8000-0000000000a1", status: "running", + errorCode: "native_execution_ownership_unverified", endedAt: NOON, ownerBootId: "boot-a", ownerPid: 100 })]); + expect(evidence.events.map(event => event.pid)).toEqual(expect.arrayContaining([11, 12])); + const windows: ServerProcessWindow[] = [ + { pid: 100, role: "primary", spawnedAt: NOON - 60_000, exitedAt: null, frozen: [{ from: NOON - 50_000, to: NOON - 5_000 }] }, + { pid: 200, role: "secondary", spawnedAt: NOON - 55_000, exitedAt: null, frozen: [] }, + ]; + const check = checkCrossServer({ ...evidence, windows }); + expect(check.violations.map(violation => [violation.rule, violation.runId])).toEqual([ + ["MS_foreign_failure", "00000000-0000-4000-8000-0000000000a1"], + ["MS_double_executor", "00000000-0000-4000-8000-0000000000a2"], + ]); + }); +}); diff --git a/tests/restart-chaos/multi-server.test.ts b/tests/restart-chaos/multi-server.test.ts new file mode 100644 index 00000000000..eda735f23c2 --- /dev/null +++ b/tests/restart-chaos/multi-server.test.ts @@ -0,0 +1,148 @@ +import { describe, expect, it } from "vitest"; +import { parseCommand } from "./config.js"; +import { planTasks } from "./scheduler.js"; +import { + CONTROLLER_LEASE_TTL_MS, MS2_THAW_TAIL_MS, MULTI_SERVER_HOLD_RANGE_MS, checkCrossServer, runMultiServerFault, + type CrossServerRun, type MultiServerHooks, type MultiServerKind, type MultiServerRecord, type ServerProcessWindow, +} from "./multi-server.js"; +import { FAULT_SIGNAL } from "./supervisor.js"; + +describe("two-server fault options", () => { + it("accepts MS faults locally and gives each a seeded hold in its range", () => { + const parsed = parseCommand(["--faults", "MS1,MS2,MS3", "--adapters", "codex", "--workloads", "B5,Q1", "--seed", "3"]); + if (parsed.mode !== "wave") throw new Error("expected a wave"); + expect(parsed.options.faults).toEqual(["MS1", "MS2", "MS3"]); + const plan = planTasks({ seed: 3, tasks: 12, workloads: ["B5"], adapters: ["codex"], faults: ["MS1", "MS2", "MS3"] }); + expect(new Set(plan.map(task => task.fault))).toEqual(new Set(["MS1", "MS2", "MS3"])); + for (const task of plan) { + const [min, max] = MULTI_SERVER_HOLD_RANGE_MS[task.fault as MultiServerKind]; + expect(task.holdMs).toBeGreaterThanOrEqual(min); + expect(task.holdMs).toBeLessThanOrEqual(max); + } + // MS2 freezes the primary past the controller lease, so a second server may legitimately fence it. + expect(MULTI_SERVER_HOLD_RANGE_MS.MS2[0]).toBeGreaterThan(CONTROLLER_LEASE_TTL_MS); + expect(FAULT_SIGNAL).toMatchObject({ MS1: "none", MS2: "SIGSTOP", MS3: "SIGTERM" }); + }); + + it("refuses combinations a second local server cannot serve", () => { + expect(() => parseCommand(["--faults", "MS1", "--env", "daytona", "--daytona-image", `r/i@sha256:${"a".repeat(64)}`])).toThrow(/--env local/); + expect(() => parseCommand(["--faults", "MS2", "--storm-minutes", "10"])).toThrow(/storm/); + expect(() => parseCommand(["--faults", "MS3,K14", "--upgrade-from", "/tmp/x"])).toThrow(/K14/); + expect(() => parseCommand(["--faults", "MS1,TM2", "--workloads", "B5"])).toThrow(/TM2 or TM3/); + }); +}); + +function fakeHooks(options: { failOn?: "start" | "freeze" | "restart"; exitAt?: number } = {}) { + let clock = 0; + const calls: string[] = []; + const hooks: MultiServerHooks = { + now: () => clock, + sleep: async ms => { clock += ms; }, + cancelled: () => false, + secondaryFailure: () => (options.exitAt !== undefined && clock >= options.exitAt ? "code=1 signal=null" : null), + startSecondary: async () => { calls.push("start"); if (options.failOn === "start") throw new Error("B did not start"); }, + stopSecondary: async () => { calls.push("stop"); }, + freezePrimary: async holdMs => { calls.push(`freeze:${holdMs}`); clock += holdMs; if (options.failOn === "freeze") throw new Error("A could not be shown stopped"); }, + restartPrimary: async () => { calls.push("restart"); clock += 5_000; if (options.failOn === "restart") throw new Error("A' not ready"); }, + }; + return { hooks, calls, clock: () => clock }; +} + +const record = (kind: MultiServerKind): MultiServerRecord => ({ kind, secondary: null, steps: [] }); + +describe("runMultiServerFault", () => { + it("MS1 keeps B beside a live A for the hold, then stops B", async () => { + const fake = fakeHooks(); + const result = record("MS1"); + await runMultiServerFault("MS1", 90_000, result, fake.hooks); + expect(fake.calls).toEqual(["start", "stop"]); + expect(fake.clock()).toBe(90_000); + expect(result.steps.map(step => step.step)).toEqual(["secondary_starting", "secondary_ready", "secondary_stopping", "secondary_stopped"]); + }); + + it("MS2 freezes A past the lease while B is up, keeps B for the thaw tail, then stops B", async () => { + const fake = fakeHooks(); + const result = record("MS2"); + await runMultiServerFault("MS2", 120_000, result, fake.hooks); + expect(fake.calls).toEqual(["start", "freeze:120000", "stop"]); + expect(fake.clock()).toBe(120_000 + MS2_THAW_TAIL_MS); + expect(result.steps.map(step => step.step)).toContain("primary_thawed"); + }); + + it("MS3 restarts A (K2) while B serves, holds B's tail, then stops B", async () => { + const fake = fakeHooks(); + await runMultiServerFault("MS3", 30_000, record("MS3"), fake.hooks); + expect(fake.calls).toEqual(["start", "restart", "stop"]); + expect(fake.clock()).toBe(35_000); + }); + + it("fails the fault when B exits on its own during the overlap or a freeze, and still stops it", async () => { + const overlap = fakeHooks({ exitAt: 30_000 }); + await expect(runMultiServerFault("MS1", 90_000, record("MS1"), overlap.hooks)).rejects.toThrow(/exited before its planned stop/); + expect(overlap.calls).toEqual(["start", "stop"]); + expect(overlap.clock()).toBeLessThan(90_000); + const frozen = fakeHooks({ exitAt: 50_000 }); + await expect(runMultiServerFault("MS2", 120_000, record("MS2"), frozen.hooks)).rejects.toThrow(/exited before its planned stop/); + expect(frozen.calls.at(-1)).toBe("stop"); + const restarted = fakeHooks({ exitAt: 1_000 }); + await expect(runMultiServerFault("MS3", 0, record("MS3"), restarted.hooks)).rejects.toThrow(/exited before its planned stop/); + }); + + it.each(["start", "freeze", "restart"] as const)("stops B and fails the fault when %s fails", async failOn => { + const fake = fakeHooks({ failOn }); + const kind: MultiServerKind = failOn === "freeze" ? "MS2" : failOn === "restart" ? "MS3" : "MS1"; + await expect(runMultiServerFault(kind, 60_000, record(kind), fake.hooks)).rejects.toThrow(); + expect(fake.calls.at(-1)).toBe("stop"); + }); +}); + +const A: ServerProcessWindow = { pid: 100, role: "primary", spawnedAt: 0, exitedAt: null, frozen: [] }; +const B: ServerProcessWindow = { pid: 200, role: "secondary", spawnedAt: 50_000, exitedAt: 300_000, frozen: [] }; +const run = (overrides: Partial): CrossServerRun => ({ runId: "r1", issueId: "i1", status: "failed", + errorCode: "native_execution_ownership_unverified", endedAt: 60_000, ownerBootId: "boot-a", ownerPid: 100, ...overrides }); + +describe("checkCrossServer", () => { + it("charges an ownership failure written while the owner was alive and running", () => { + const check = checkCrossServer({ runs: [run({})], events: [], windows: [A, B] }); + expect(check.violations).toMatchObject([{ rule: "MS_foreign_failure", runId: "r1", issueId: "i1" }]); + }); + + it("allows an ownership failure while the owner was frozen or after it exited", () => { + const frozen = { ...A, frozen: [{ from: 55_000, to: 200_000 }] }; + expect(checkCrossServer({ runs: [run({})], events: [], windows: [frozen, B] }).violations).toEqual([]); + const exited = { ...A, exitedAt: 59_000 }; + const check = checkCrossServer({ runs: [run({})], events: [], windows: [exited, B] }); + expect(check.violations).toEqual([]); + expect(check.unattributed).toHaveLength(1); + }); + + it("charges #20508's shape: a sweep's write after the frozen owner thawed and took its lease back", () => { + const thawed = { ...A, frozen: [{ from: 10_000, to: 110_000 }] }; + const check = checkCrossServer({ runs: [run({ endedAt: 115_000 })], events: [], windows: [thawed, B] }); + expect(check.violations.map(violation => violation.rule)).toEqual(["MS_foreign_failure"]); + }); + + it("allows the owner's own lease loss after a freeze past the lease, not without one", () => { + const leaseLost = run({ errorCode: "controller_lease_lost", endedAt: 115_000 }); + const pastLease = { ...A, frozen: [{ from: 10_000, to: 10_000 + CONTROLLER_LEASE_TTL_MS + 1 }] }; + expect(checkCrossServer({ runs: [leaseLost], events: [], windows: [pastLease, B] }).violations).toEqual([]); + const shortFreeze = { ...A, frozen: [{ from: 10_000, to: 40_000 }] }; + expect(checkCrossServer({ runs: [leaseLost], events: [], windows: [shortFreeze, B] }).violations).toHaveLength(1); + }); + + it("ignores ordinary failures and reports owners it cannot attribute", () => { + expect(checkCrossServer({ runs: [run({ errorCode: "adapter_failed" })], events: [], windows: [A] })).toMatchObject({ violations: [], unattributed: [] }); + expect(checkCrossServer({ runs: [run({ ownerPid: null })], events: [], windows: [A] }).unattributed).toHaveLength(1); + }); + + it("charges a second runner identity recorded before the first was shown stopped", () => { + const recorded = (pid: number, at: number) => ({ runId: "r1", kind: "recorded" as const, pid, at }); + const stopped = (pid: number, at: number) => ({ runId: "r1", kind: "stopped" as const, pid, at }); + const runs = [run({ errorCode: null, status: "succeeded" })]; + expect(checkCrossServer({ runs, events: [recorded(1, 1), recorded(2, 5)], windows: [A] }).violations) + .toMatchObject([{ rule: "MS_double_executor", evidence: { firstPid: 1, secondPid: 2 } }]); + expect(checkCrossServer({ runs, events: [recorded(1, 1), stopped(1, 4), recorded(2, 5)], windows: [A] }).violations).toEqual([]); + // The same runner recorded again (a re-attach) is one executor. + expect(checkCrossServer({ runs, events: [recorded(1, 1), recorded(1, 5)], windows: [A] }).violations).toEqual([]); + }); +}); diff --git a/tests/restart-chaos/multi-server.ts b/tests/restart-chaos/multi-server.ts new file mode 100644 index 00000000000..aa2dfebce79 --- /dev/null +++ b/tests/restart-chaos/multi-server.ts @@ -0,0 +1,285 @@ +/** + * MS1–MS3: two control planes on one database (README "Two servers, one database"). + * + * Hosted Cloud deploys by rolling overlap: a new server B starts on the same database while the old + * server A still serves, then A drains (doc/plans/2026-10-31-zero-downtime-web-deploys.md: 5 s overlap, + * up to 90 s drain). B's startup recovery and periodic sweeps therefore run beside A's live runs. A wave + * otherwise runs one server, so it cannot reach a cross-server race such as #20508 (B's orphan sweep + * failing a run whose owner had taken its lease back). + * + * MS1 B boots and stays up beside a live A for the hold, then stops (SIGTERM). A is not touched. + * MS2 B boots; A and its process tree are frozen (SIGSTOP) past the 90 s controller lease, then + * thawed. B stays up for a short tail so it sees A take its leases back, then stops. + * MS3 Rolling-upgrade shape: B boots, then A gets K2 (SIGTERM, restart on its own port). B is the + * only live server between A's stop and its replacement's readiness, then stops after a tail. + * + * B shares A's database, instance directory, local storage and secrets, and listens on its own + * loopback port. The harness and agents keep talking to A's port. B runs on the same host and PID + * namespace as A, so it can check A's PIDs directly; a Railway replica cannot. That makes these + * faults a lower bound on cross-server exposure, not a model of separate hosts. + * + * The cross-server invariant (`checkCrossServer`) is graded once per wave over the database: + * MS_foreign_failure a run failed with an ownership code while its owning server was alive and running + * MS_double_executor a run recorded a second local runner identity before the first was shown stopped + */ +import type { Db } from "../../packages/db/src/index.js"; +import type { TimeFaultRecord } from "./time-faults.js"; + +export const MULTI_SERVER_FAULT_KINDS = ["MS1", "MS2", "MS3"] as const; +export type MultiServerKind = (typeof MULTI_SERVER_FAULT_KINDS)[number]; + +export function isMultiServerFault(kind: string): kind is MultiServerKind { + return (MULTI_SERVER_FAULT_KINDS as readonly string[]).includes(kind); +} + +/** `CONTROLLER_LEASE_TTL_MS` in server/src/services/controller-instance-lease.ts. */ +export const CONTROLLER_LEASE_TTL_MS = 90_000; + +/** + * Seeded hold per kind. MS1: B's overlap, long enough for its boot recovery and a periodic sweep. + * MS2: A's freeze, past the controller lease. MS3: B's tail after A's replacement is ready. + */ +export const MULTI_SERVER_HOLD_RANGE_MS: Record = { + MS1: [60_000, 150_000], + MS2: [CONTROLLER_LEASE_TTL_MS + 10_000, 150_000], + MS3: [20_000, 45_000], +}; + +/** MS2: B stays up this long after A is thawed, the window in which #20508's foreign write landed. */ +export const MS2_THAW_TAIL_MS = 30_000; + +export function multiServerHoldMs(kind: MultiServerKind, int: (min: number, max: number) => number): number { + return int(...MULTI_SERVER_HOLD_RANGE_MS[kind]); +} + +/** `--faults MS*` validation; the planner itself is the ordinary per-task planner. */ +export function validateMultiServerOptions(options: { env: string; faults: readonly string[]; stormMinutes?: number; + crashPoint?: string; providerFaults?: readonly string[] }): void { + const kinds = options.faults.filter(isMultiServerFault); + if (!kinds.length) return; + // B shares A's instance directory and local runners; a remote runner is not under either server. + if (options.env !== "local") throw new Error("Two-server faults (MS*) start a second local server; run them with --env local"); + if (options.stormMinutes) throw new Error("Two-server faults (MS*) are planned per task; do not combine them with --storm-minutes"); + if (options.crashPoint) throw new Error("Two-server faults (MS*) do not combine with --crash-point"); + if (options.faults.includes("K14")) throw new Error("Two-server faults (MS*) do not combine with K14; both servers run the current build"); + // Aging moves database times; the cross-server invariant compares them with real process windows. + if (options.faults.some(kind => kind === "TM2" || kind === "TM3")) throw new Error("Two-server faults (MS*) do not combine with TM2 or TM3; age the database in its own wave"); +} + +/** One server process the wave started, for attributing a run's owner at a moment in time. */ +export interface ServerProcessWindow { + pid: number; + role: "primary" | "secondary"; + spawnedAt: number; + exitedAt: number | null; + /** SIGSTOP windows; an open end means still frozen. */ + frozen: Array<{ from: number; to: number | null }>; +} + +export interface MultiServerRecord { + kind: MultiServerKind; + secondary: { pid: number | null; port: number; spawnedAt: number | null; readyAt: number | null; stoppedAt: number | null } | null; + /** MS2: A's freeze (tree SIGSTOP, no aging). */ + primaryFreeze?: TimeFaultRecord; + /** MS3: A's K2 and its replacement. */ + primaryRestart?: { stoppedAt: number | null; restartedAt: number | null; readyAt: number | null; replacementPid: number | null }; + steps: Array<{ step: string; at: number }>; +} + +export interface MultiServerHooks { + now(): number; + sleep(ms: number): Promise; + cancelled(): boolean; + /** Why B stopped on its own (its exit code or signal), or null while it runs as planned. */ + secondaryFailure(): string | null; + /** Spawns B on its own port and waits until it reports ready. */ + startSecondary(record: MultiServerRecord): Promise; + /** SIGTERM B and wait for it to exit; always called once B was spawned. */ + stopSecondary(record: MultiServerRecord): Promise; + /** MS2: freeze A's tree for `holdMs`, thaw it, and replace A if it exited after the thaw. */ + freezePrimary(holdMs: number, record: MultiServerRecord): Promise; + /** MS3: K2 on A, start its replacement on A's port, and wait until it is ready. */ + restartPrimary(record: MultiServerRecord): Promise; +} + +async function hold(hooks: MultiServerHooks, ms: number): Promise { + const until = hooks.now() + ms; + while (hooks.now() < until) { + if (hooks.cancelled()) throw new Error("Harness shutdown interrupted the two-server fault"); + const exited = hooks.secondaryFailure(); + if (exited) throw new Error(`The second server exited before its planned stop: ${exited}`); + await hooks.sleep(Math.min(1_000, until - hooks.now())); + } +} + +/** Runs one MS fault. B is stopped on every path once it was started, so it never outlives its fault. */ +export async function runMultiServerFault(kind: MultiServerKind, holdMs: number, record: MultiServerRecord, hooks: MultiServerHooks): Promise { + const step = (name: string) => record.steps.push({ step: name, at: hooks.now() }); + let started = false; + let failure: unknown = null; + try { + started = true; + step("secondary_starting"); + await hooks.startSecondary(record); + step("secondary_ready"); + if (kind === "MS1") { + await hold(hooks, holdMs); + } else if (kind === "MS2") { + step("primary_freezing"); + await hooks.freezePrimary(holdMs, record); + step("primary_thawed"); + await hold(hooks, MS2_THAW_TAIL_MS); + } else { + step("primary_restarting"); + await hooks.restartPrimary(record); + step("primary_ready"); + await hold(hooks, holdMs); + } + // B must have lived through the whole window, including a freeze or restart without a hold check. + const exited = hooks.secondaryFailure(); + if (exited) throw new Error(`The second server exited before its planned stop: ${exited}`); + } catch (error) { + failure = error; + } + if (started) { + step("secondary_stopping"); + try { await hooks.stopSecondary(record); step("secondary_stopped"); } + catch (error) { failure ??= error; } + } + if (failure) throw failure; +} + +// ---------- cross-server invariant ---------- + +/** A sweep's verdict on another server's run. */ +export const FOREIGN_FAILURE_CODES: ReadonlySet = new Set(["process_lost", "native_execution_ownership_unverified"]); +/** + * The owner's own lease loss: expected after a freeze past the lease, a violation otherwise. Only the legacy + * 60 s run lease persists a run code; a native session lease loss is recorded as `native_session_interrupted`. + */ +export const LEASE_LOSS_CODES: ReadonlySet = new Set(["controller_lease_lost"]); + +export interface CrossServerRun { + runId: string; + issueId: string | null; + status: string; + errorCode: string | null; + /** + * When the run reached its final status (finished_at), else when it was last written (updated_at), ms. + * #20508's marker leaves the run `running` with the ownership code, so its write time is updated_at. + */ + endedAt: number | null; + ownerBootId: string | null; + ownerPid: number | null; +} + +export interface RunIdentityEvent { runId: string; kind: "recorded" | "stopped"; pid: number; at: number } + +export interface CrossServerViolation { + rule: "MS_foreign_failure" | "MS_double_executor"; + runId: string; + issueId: string | null; + detail: string; + evidence: Record; +} + +export interface CrossServerCheck { + violations: CrossServerViolation[]; + /** Ownership failures the harness could not attribute (owner boot or pid unknown, or not a wave server). */ + unattributed: Array<{ runId: string; errorCode: string | null; ownerBootId: string | null; ownerPid: number | null }>; + windows: ServerProcessWindow[]; +} + +const aliveAt = (window: ServerProcessWindow, at: number) => window.spawnedAt <= at && (window.exitedAt === null || at < window.exitedAt); +const frozenAt = (window: ServerProcessWindow, at: number) => window.frozen.some(span => span.from <= at && (span.to === null || at < span.to)); +const frozenPastLeaseBefore = (window: ServerProcessWindow, at: number, leaseTtlMs: number) => + window.frozen.some(span => span.from < at && ((span.to ?? at) - span.from) >= leaseTtlMs); + +/** + * Pure: grade the wave's runs against the server process windows the harness recorded. + * A failure is charged only when the owner is a wave server, alive and running (not frozen) at that moment. + */ +export function checkCrossServer(input: { runs: readonly CrossServerRun[]; events: readonly RunIdentityEvent[]; + windows: readonly ServerProcessWindow[]; leaseTtlMs?: number }): CrossServerCheck { + const leaseTtlMs = input.leaseTtlMs ?? CONTROLLER_LEASE_TTL_MS; + const violations: CrossServerViolation[] = []; + const unattributed: CrossServerCheck["unattributed"] = []; + const issueOf = new Map(input.runs.map(run => [run.runId, run.issueId])); + for (const run of input.runs) { + const code = run.errorCode ?? ""; + const foreign = FOREIGN_FAILURE_CODES.has(code); + const leaseLoss = LEASE_LOSS_CODES.has(code); + if (!foreign && !leaseLoss) continue; + // Several boots can reuse a pid only after the first exited: pick the window alive at the end time. + const at = run.endedAt; + const owner = at === null || run.ownerPid === null ? undefined + : input.windows.find(window => window.pid === run.ownerPid && aliveAt(window, at)); + if (!owner || at === null) { + unattributed.push({ runId: run.runId, errorCode: run.errorCode, ownerBootId: run.ownerBootId, ownerPid: run.ownerPid }); + continue; + } + if (frozenAt(owner, at)) continue; + // After a freeze past the lease the owner may lose its own lease; a sweep's write on a thawed owner is still foreign. + if (leaseLoss && frozenPastLeaseBefore(owner, at, leaseTtlMs)) continue; + violations.push({ rule: "MS_foreign_failure", runId: run.runId, issueId: run.issueId, + detail: `${code} at ${new Date(at).toISOString()} while owner pid ${owner.pid} (${owner.role}) was alive and running`, + evidence: { errorCode: code, endedAt: at, ownerBootId: run.ownerBootId, ownerPid: owner.pid, ownerRole: owner.role, + ownerFrozen: owner.frozen } }); + } + const byRun = new Map(); + for (const event of input.events) { + const list = byRun.get(event.runId) ?? []; + list.push(event); + byRun.set(event.runId, list); + } + for (const [runId, events] of byRun) { + const live = new Set(); + for (const event of [...events].sort((left, right) => left.at - right.at || (left.kind === "stopped" ? -1 : 1))) { + if (event.kind === "stopped") { live.delete(event.pid); continue; } + const other = [...live].find(pid => pid !== event.pid); + if (other !== undefined) { + violations.push({ rule: "MS_double_executor", runId, issueId: issueOf.get(runId) ?? null, + detail: `runner pid ${event.pid} recorded at ${new Date(event.at).toISOString()} while pid ${other} was never shown stopped`, + evidence: { firstPid: other, secondPid: event.pid, at: event.at } }); + live.delete(other); + } + live.add(event.pid); + } + } + return { violations, unattributed, windows: [...input.windows] }; +} + +type Row = Record; +const num = (value: unknown) => value === null || value === undefined || value === "" ? null : Number(value); + +/** Reads the runs and runner identity events `checkCrossServer` needs from the wave database. */ +export async function readCrossServerEvidence(db: Db, companyId: string): Promise<{ runs: CrossServerRun[]; events: RunIdentityEvent[] }> { + const client = db.$client; + const runs = await client.unsafe( + `select r.id::text as "runId", r.issue_id::text as "issueId", r.status, r.error_code as "errorCode", + floor(extract(epoch from coalesce(r.finished_at, r.updated_at)) * 1000) as "endedAt", + coalesce(r.controller_boot_id::text, f.controller_boot_id, f.lease_holder->>'bootId') as "ownerBootId", + ci.pid as "ownerPid" + from heartbeat_runs r + left join native_run_finalizations f on f.run_id = r.id + left join controller_instances ci on ci.boot_id = coalesce(r.controller_boot_id::text, f.controller_boot_id, f.lease_holder->>'bootId') + where r.company_id::text = $1 + and (r.status in ('failed', 'timed_out', 'cancelled') or r.error_code in ('process_lost', 'native_execution_ownership_unverified'))`, + [companyId]) as Row[]; + const events = await client.unsafe( + `select e.run_id::text as "runId", e.event_type as "eventType", e.payload->>'processPid' as pid, + floor(extract(epoch from e.created_at) * 1000) as at + from heartbeat_run_events e join heartbeat_runs r on r.id = e.run_id + where r.company_id::text = $1 and e.event_type in ('native.process_identity_recorded', 'native.local_process_stopped')`, [companyId]) as Row[]; + return { + runs: runs.map(row => ({ runId: String(row.runId), issueId: row.issueId === null ? null : String(row.issueId), status: String(row.status), + errorCode: row.errorCode === null ? null : String(row.errorCode), endedAt: num(row.endedAt), + ownerBootId: row.ownerBootId === null ? null : String(row.ownerBootId), ownerPid: num(row.ownerPid) })), + events: events.flatMap(row => { + const pid = num(row.pid); + const at = num(row.at); + if (pid === null || at === null || !Number.isSafeInteger(pid)) return []; + return [{ runId: String(row.runId), kind: row.eventType === "native.local_process_stopped" ? "stopped" as const : "recorded" as const, pid, at }]; + }), + }; +} diff --git a/tests/restart-chaos/scheduler.ts b/tests/restart-chaos/scheduler.ts index 44f7fa806bf..f3d41b9f3ea 100644 --- a/tests/restart-chaos/scheduler.ts +++ b/tests/restart-chaos/scheduler.ts @@ -2,6 +2,7 @@ import { isProviderFault, withProviderFaultRules } from "./provider-faults.js"; import { takesJourneyTriggers } from "./journeys.js"; import { workloadDefinitions } from "./workloads.js"; import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js"; +import { isMultiServerFault, multiServerHoldMs } from "./multi-server.js"; import { isRepeatedRestartFault, planRestartSteps, type RestartStep } from "./repeated-restarts.js"; import { K4_MIN_HOLD_MS, type AdapterId, type FaultKind, type ScheduledFault, type StormFault, type StormPlan, type TaskFaultKind, type TaskPlan, type TriggerId, type WavePlan, type WaveOptions, type WorkloadId } from "./types.js"; import type { ReplayRef } from "./config.js"; @@ -125,7 +126,8 @@ export function planTasks(options: Pick { @@ -47,8 +48,10 @@ const HEALTH_TIMEOUT_MS = (() => { const READY_TIMEOUT_MS = 300_000; const STOP_TIMEOUT_MS = 30_000; -export const FAULT_SIGNAL: Record = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM", - TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL", RR1: "SIGKILL", RR2: "SIGKILL", RR3: "SIGKILL" }; +export const FAULT_SIGNAL: Record = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM", + TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL", RR1: "SIGKILL", RR2: "SIGKILL", RR3: "SIGKILL", + // MS1 leaves the primary alone; MS2 freezes it; MS3 stops it gracefully (multi-server.ts). + MS1: "none", MS2: "SIGSTOP", MS3: "SIGTERM" }; /** The checkout's commit, for the K14 receipt. */ function checkoutSha(root: string): string | null { @@ -140,6 +143,10 @@ export class ChaosSupervisor { timeAgeMs = DEFAULT_TIME_AGE_MS; /** RR1–RR3 (`--rr-stop-on`): stop each replacement on this `/api/ready` status instead of after its gap. */ rrStopOn: RepeatedRestartStopOn | null = null; + /** Every server process this wave started, with its lifetime and freezes (MS cross-server invariant). */ + readonly serverWindows: ServerProcessWindow[] = []; + /** MS1–MS3: the second server, while one is running. */ + private secondary: { child: ChildProcess; port: number; log: WriteStream; exit: string | null } | null = null; /** Every aging shift so far, in order; the grader's API reads times through them (time-faults.ts). */ readonly agingShifts: AgingShift[] = []; private readonly createdAt = Date.now(); @@ -253,11 +260,13 @@ export class ChaosSupervisor { const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]), { cwd: root, env, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached }); this.child = child; + const window = this.trackWindow(child, "primary"); child.stdout?.on("data", chunk => this.log.write(chunk)); child.stderr?.on("data", chunk => this.log.write(chunk)); child.once("error", error => { this.spawnErrors.set(child, error); }); child.once("exit", (code, signal) => { + if (window) window.exitedAt = Date.now(); this.log.write(`\n[restart-chaos] server pid=${child.pid} exited code=${code} signal=${signal}\n`); if (control && signal === "SIGKILL" && !this.expectedStops.has(child) && !this.shuttingDown) { const hit = readCrashPointReceipt(control.receiptPath, child.pid); @@ -274,26 +283,41 @@ export class ChaosSupervisor { return child; } + private trackWindow(child: ChildProcess, role: ServerProcessWindow["role"]): ServerProcessWindow | null { + if (!child.pid) return null; + const window: ServerProcessWindow = { pid: child.pid, role, spawnedAt: Date.now(), exitedAt: null, frozen: [] }; + this.serverWindows.push(window); + return window; + } + + /** Records a freeze or thaw of a server process for the MS invariant (TM1/TM2 and MS2). */ + private markFrozen(pid: number | undefined, at: number, frozen: boolean): void { + const window = [...this.serverWindows].reverse().find(candidate => candidate.pid === pid && candidate.exitedAt === null); + if (!window) return; + if (frozen) window.frozen.push({ from: at, to: null }); + else { const open = window.frozen.find(span => span.to === null); if (open) open.to = at; } + } + private exited(child: ChildProcess) { return child.exitCode !== null || child.signalCode !== null || this.spawnErrors.has(child); } - private async readPhase(): Promise { + private async readPhase(baseUrl = this.baseUrl): Promise { try { - const response = await fetch(`${this.baseUrl}/api/ready`, { signal: AbortSignal.timeout(1_000), cache: "no-store" }); + const response = await fetch(`${baseUrl}/api/ready`, { signal: AbortSignal.timeout(1_000), cache: "no-store" }); return String(((await response.json()) as { status?: string }).status ?? response.status); } catch { return "unreachable"; } } - private async waitForHealth(child: ChildProcess, phases?: FaultReceipt["phases"]): Promise { + private async waitForHealth(child: ChildProcess, phases?: FaultReceipt["phases"], baseUrl = this.baseUrl): Promise { const deadline = Date.now() + HEALTH_TIMEOUT_MS; while (Date.now() < deadline) { if (this.shuttingDown) throw new Error("Harness shutdown interrupted startup"); if (this.exited(child)) throw new Error(`Paperclip server exited during startup (code=${child.exitCode} signal=${child.signalCode})`); if (phases) { - const phase = await this.readPhase(); + const phase = await this.readPhase(baseUrl); if (phase !== phases[phases.length - 1]?.phase) phases.push({ phase, at: Date.now() }); } try { - const response = await fetch(`${this.baseUrl}/api/health`, { signal: AbortSignal.timeout(1_000) }); + const response = await fetch(`${baseUrl}/api/health`, { signal: AbortSignal.timeout(1_000) }); await response.body?.cancel(); if (response.ok) return Date.now(); } catch { /* still booting */ } @@ -433,6 +457,7 @@ export class ChaosSupervisor { if (!previous?.pid || this.exited(previous)) throw new Error("No live Paperclip server to fault"); let outage: ControllerOutageReceipt | undefined; if (kind === "TM1" || kind === "TM2") await this.freeze(kind, previous, holdMs, receipt); + else if (isMultiServerFault(kind)) await this.multiServer(kind, previous, holdMs, receipt); else outage = await replaceController({ requestId: `chaos-${receipt.faultId}-${requestedBy}`, previousPid: previous.pid, @@ -516,8 +541,8 @@ export class ChaosSupervisor { const times = await runFreezeFault({ kind, rootPid: server.pid!, holdMs, record: timeFault, freezer: new ProcessFreezer(), ...(kind === "TM2" ? { age: (onLockTimeout: () => Promise) => this.age(receipt.faultId, onLockTimeout) } : {}), cancelled: () => this.shuttingDown, - onFrozen: pids => this.timeline.add("fault_server_frozen", { faultId: receipt.faultId, pids, holdMs }), - onThawed: frozenMs => this.timeline.add("fault_server_thawed", { faultId: receipt.faultId, frozenMs }), + onFrozen: pids => { this.markFrozen(server.pid, Date.now(), true); this.timeline.add("fault_server_frozen", { faultId: receipt.faultId, pids, holdMs }); }, + onThawed: frozenMs => { this.markFrozen(server.pid, Date.now(), false); this.timeline.add("fault_server_thawed", { faultId: receipt.faultId, frozenMs }); }, waitHealthy: () => this.waitForHealth(server, receipt.phases), exitStatus: () => this.exited(server) ? { code: server.exitCode, signal: server.signalCode } : null, replaceExited: async () => { @@ -532,6 +557,117 @@ export class ChaosSupervisor { receipt.healthyAt = times.healthyAt; } + /** + * MS1–MS3 (multi-server.ts): a second server B on its own port, with the primary's database, instance + * directory, storage and secrets. The primary keeps its port, so tasks and agents keep using it. + */ + private async multiServer(kind: MultiServerKind, primary: ChildProcess, holdMs: number, receipt: FaultReceipt): Promise { + const record: MultiServerRecord = { kind, secondary: null, steps: [] }; + receipt.multiServer = record; + await runMultiServerFault(kind, holdMs, record, { + now: () => Date.now(), + sleep: ms => delay(ms), + cancelled: () => this.shuttingDown, + secondaryFailure: () => this.secondary?.exit ?? null, + startSecondary: async current => { + const child = await this.spawnSecondary(current); + current.secondary!.readyAt = await this.waitForSecondaryReady(child, current.secondary!.port); + this.timeline.add("multi_server_secondary_ready", { faultId: receipt.faultId, pid: child.pid, port: current.secondary!.port }); + }, + stopSecondary: async current => { + const secondary = this.secondary; + if (!secondary) return; + // An exited B still has its owned tree cleaned up; a live one gets the graceful stop. + try { await (secondary.exit ? this.stopper.crashForRestart(secondary.child) : this.stopper(secondary.child, "SIGTERM")); } + finally { + this.secondary = null; + await new Promise(resolve => secondary.log.end(resolve)); + if (current.secondary) current.secondary.stoppedAt = Date.now(); + this.timeline.add("multi_server_secondary_stopped", { faultId: receipt.faultId, pid: secondary.child.pid }); + } + }, + freezePrimary: async (freezeMs, current) => { + // TM2's tree freeze without aging: the primary and its runners stop; the time passes for real. + const timeFault: TimeFaultRecord = { kind: "TM2" }; + current.primaryFreeze = timeFault; + const times = await runFreezeFault({ kind: "TM2", rootPid: primary.pid!, holdMs: freezeMs, record: timeFault, freezer: new ProcessFreezer(), + cancelled: () => this.shuttingDown, + onFrozen: pids => { this.markFrozen(primary.pid, Date.now(), true); this.timeline.add("fault_server_frozen", { faultId: receipt.faultId, pids, holdMs: freezeMs }); }, + onThawed: frozenMs => { this.markFrozen(primary.pid, Date.now(), false); this.timeline.add("fault_server_thawed", { faultId: receipt.faultId, frozenMs }); }, + waitHealthy: () => this.waitForHealth(primary, receipt.phases), + exitStatus: () => this.exited(primary) ? { code: primary.exitCode, signal: primary.signalCode } : null, + replaceExited: async () => { + this.timeline.add("time_fault_server_exited_after_thaw", { faultId: receipt.faultId, ...timeFault.serverExitedAfterThaw }); + try { await this.stopper.crashForRestart(primary); } finally { if (this.child === primary) this.child = null; } + }, + startReplacement: async () => this.waitForHealth(await this.spawnServer(), receipt.phases), + }); + receipt.stoppedAt = times.stoppedAt; + receipt.restartedAt = times.restartedAt; + receipt.healthyAt = times.healthyAt; + }, + restartPrimary: async current => { + const restart: NonNullable = { stoppedAt: null, restartedAt: null, readyAt: null, replacementPid: null }; + current.primaryRestart = restart; + await this.stopper(primary, "SIGTERM"); + if (this.child === primary) this.child = null; + await this.waitForHealthToStop(); + receipt.stoppedAt = restart.stoppedAt = Date.now(); + this.timeline.add("fault_server_down", { faultId: receipt.faultId }); + receipt.restartedAt = restart.restartedAt = Date.now(); + const replacement = await this.spawnServer(); + restart.replacementPid = replacement.pid ?? null; + receipt.healthyAt = await this.waitForHealth(replacement, receipt.phases); + restart.readyAt = await this.waitForReadyPhase(receipt.phases); + }, + }); + } + + private async spawnSecondary(record: MultiServerRecord): Promise { + if (this.shuttingDown) throw new Error("Refusing to start a second server during harness shutdown"); + if (this.secondary) throw new Error("A second server is already running"); + const port = await reserveRunnerE2EServerPort(); + const log = createWriteStream(path.join(path.dirname(this.paths.serverLogPath), "server-b.log"), { flags: "a", mode: 0o600 }); + const root = this.serverRoot; + const entry = path.join(root, "tests/runner-e2e/server-entry.ts"); + // Same database, instance directory, storage and secrets as the primary; its own port and API URL. + const env = { ...this.environment, PORT: String(port), ...waveServerUrlEnvironment(port) }; + const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]), + { cwd: root, env, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached }); + const window = this.trackWindow(child, "secondary"); + record.secondary = { pid: child.pid ?? null, port, spawnedAt: Date.now(), readyAt: null, stoppedAt: null }; + const secondary = { child, port, log, exit: null as string | null }; + this.secondary = secondary; + child.stdout?.on("data", chunk => log.write(chunk)); + child.stderr?.on("data", chunk => log.write(chunk)); + child.once("error", error => { this.spawnErrors.set(child, error); }); + child.once("exit", (code, signal) => { + if (window) window.exitedAt = Date.now(); + log.write(`\n[restart-chaos] second server pid=${child.pid} exited code=${code} signal=${signal}\n`); + if (!this.expectedStops.has(child) && !this.shuttingDown) { + secondary.exit = `code=${code} signal=${signal}`; + this.timeline.add("multi_server_secondary_exited", { pid: child.pid, code, signal }); + } + }); + await this.stopper.watch(child); + this.timeline.add("multi_server_secondary_spawned", { pid: child.pid, port }); + return child; + } + + /** B is ready when its own /api/ready says so; it never changes the primary's readiness. */ + private async waitForSecondaryReady(child: ChildProcess, port: number): Promise { + const baseUrl = `http://127.0.0.1:${port}`; + await this.waitForHealth(child, undefined, baseUrl); + const deadline = Date.now() + READY_TIMEOUT_MS; + while (Date.now() < deadline) { + if (this.shuttingDown) throw new Error("Harness shutdown interrupted the second server's start"); + if (this.exited(child)) throw new Error(`Second server exited during startup (code=${child.exitCode} signal=${child.signalCode})`); + if (await this.readPhase(baseUrl) === "ready") return Date.now(); + await delay(250); + } + throw new Error(`Second server did not report ready within ${READY_TIMEOUT_MS}ms`); + } + /** * K14: move the server from the older build to this checkout. Only the first K14 switches; a later * one restarts the current build (recorded with `switched: false`). diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts index 8d745327e0e..aeea6588252 100644 --- a/tests/restart-chaos/types.ts +++ b/tests/restart-chaos/types.ts @@ -8,8 +8,9 @@ import type { TimeFaultRecord } from "./time-faults.js"; * K14 (upgrade mid-run): a graceful stop, then the replacement starts from the current build (`--upgrade-from`). * TM1–TM3 (time faults, time-faults.ts): freeze the server, sleep with aged rows, or an aged outage. * RR1–RR3 (repeated restarts, repeated-restarts.ts): a K1, then more restarts while the replacement recovers. + * MS1–MS3 (two servers, one database, multi-server.ts): a second server beside the first. */ -export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3"] as const; +export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3"] as const; export type FaultKind = (typeof FAULT_KINDS)[number]; /** The shortest seeded K4 outage (scheduler.ts K4_HOLD_RANGE_MS), also the K4 hold after a provider fault. */ export const K4_MIN_HOLD_MS = 120_000; @@ -256,7 +257,8 @@ export interface FaultReceipt { kind: FaultKind; /** Task that scheduled the fault; every other running task is also exposed. */ requestedBy: string; - signal: "SIGKILL" | "SIGTERM" | "SIGSTOP"; + /** "none": MS1 sends no signal to the primary server. */ + signal: "SIGKILL" | "SIGTERM" | "SIGSTOP" | "none"; holdMs: number; requestedAt: number; startedAt: number; @@ -277,6 +279,8 @@ export interface FaultReceipt { timeFault?: TimeFaultRecord; /** RR1–RR3: every stop and replacement of the sequence, the first kill included. */ repeatedRestart?: RepeatedRestartRecord; + /** MS1–MS3: the second server and what happened to the first. */ + multiServer?: import("./multi-server.js").MultiServerRecord; serverLogOffsetAtStart: number; serverLogOffsetAtReady: number | null; error: string | null;