mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
test(restart-chaos): two servers on one database (MS1-MS3) and a cross-server invariant (#20518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) tests how runs survive control-plane faults > - Every wave runs one server, so no wave can reach a race between two servers on one database > - Hosted Cloud deploys by rolling overlap: a new server's startup recovery and sweeps run beside the old server's live runs > - #20508 fixed such a race (another server's orphan sweep failing a run whose owner took its lease back) with a database test only > - This pull request adds three two-server faults (MS1–MS3) and a cross-server invariant > - The benefit is that cross-server ownership races can be reproduced and graded in a live wave ## Linked Issues or Issue Description Refs #20508 (the race this harness can now reach live). **What is missing?** The harness starts one control plane per wave. A fault replaces or freezes that one server. Nothing starts a second server on the same database while the first still owns runs, so cross-server ownership bugs are found only by database tests. **How production runs (checked before choosing the scenarios)** - Hosted Cloud runs on Railway with Neon Postgres. A deploy is a rolling A/B overlap: B starts on the same database, A serves until B is ready, then A drains. Defaults are a 5 s overlap and up to 90 s of drain (`doc/plans/2026-10-31-zero-downtime-web-deploys.md`, lines 374–375). The chaos plan names the same shape as K12 (`doc/plans/2026-11-06-control-plane-restart-chaos.md`, line 103). It also records B's startup recovery hitting a lock on a run A still owned (lines 229–230). - This repository has no replica count or deploy strategy of its own. `docker/ecs-task-definition.json` describes one task, and the real Cloud deploy lives in another repository. The self-hosting docs do not describe several servers on one database. - Server-side ownership: per-boot controller lease of 90 s (`server/src/services/controller-instance-lease.ts`), legacy run lease of 60 s, native executor lease of 20 min, and the orphan sweep in `server/src/services/heartbeat/recovery.ts`. So MS1 models B's overlap beside a live A, MS2 models A paused (for example a host sleep) past its controller lease while B is up, and MS3 models the rolling upgrade itself. ## What Changed - New `tests/restart-chaos/multi-server.ts`: - MS1: B boots on the same database, instance directory, storage and secrets, on its own port. It stays up 60–150 s beside a live A, then gets SIGTERM. - MS2: B boots. A and its tree are frozen (TM2's freeze without aging) for 100–150 s, past the 90 s controller lease, then thawed. B stays up 30 s more, then stops. - MS3: B boots, then A gets K2 and a replacement on A's port. B stops 20–45 s after the replacement is ready. - B is always stopped once started, also when a step fails. - `checkCrossServer`, graded once per wave with an MS fault, over the wave database and the server process windows (pid, start, exit, freezes): - `MS_foreign_failure`: a run failed with `process_lost` or `native_execution_ownership_unverified` while its owner was alive and not frozen (#20508's shape included). The owner's own lease loss counts only without a prior freeze past the lease. - `MS_double_executor`: a run recorded a second runner identity before the first was shown stopped. - Unattributable ownership failures are listed, not charged. - `supervisor.ts`: records every server's process window and freezes, spawns and stops B (`<wave>/server-b.log`), and runs the MS steps. TM freezes are recorded too. - `types.ts`, `scheduler.ts`, `journey-matrix.ts`, `config.ts`: MS1–MS3 fault kinds, seeded holds, and validation (local only; no storm, crash point or K14). - `main.ts`: writes `<wave>/cross-server.json` and fails the wave on a violation. - README section "Two servers, one database (MS1–MS3)", with the production evidence and the limits. Test-only. No `package.json` or `.github` changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 files, 473 tests pass. - New `multi-server.test.ts` (14 tests): option validation and seeded holds; MS1/MS2/MS3 step order and timing with fake hooks; B stopped when a step fails; the invariant for a live owner, a frozen or exited owner, #20508's thaw-then-write shape, lease loss with and without a freeze past the lease, unattributed owners, and double executors. - New `multi-server-db.test.ts` (embedded Postgres): a run still `running` with `native_execution_ownership_unverified` (#20508's marker) is read at its write time and charged as `MS_foreign_failure`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave `w-20261110T105448-s301` on head `a9ec5a4502` (master product code + this harness), `--adapters codex --workloads B5,Q1 --faults MS1,MS2 --concurrency 1 --tasks 3`: 3 of 3 pass, all crossed their fault. B booted on the shared database in about 10 s each time. MS2 froze A for 104 s; A stayed up after the thaw. Cross-server invariant: 0 violations, 0 unattributed. ## Risks - Test-harness code only. - B runs on the same host and PID namespace as A, so it can check A's PIDs directly, which a Railway replica cannot. These faults are a lower bound on cross-server exposure. The README says so. - MS2 outlasts the 90 s controller lease, not the 20 min native executor lease. - MS faults refuse TM2/TM3 in the same wave, and fail when B exits before its planned stop. - `MS_foreign_failure` uses the run's `finished_at` (or `updated_at` for a still-running row) and the owner recorded on the run. A run whose owner changed after the failure could be misattributed; the evidence file keeps the owner and windows for review. - `MS_double_executor` reads runner identity events; it cannot see a second executor that never recorded one. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed
1 parent
c0593b69f1
commit
be34a2d9d2
10 files changed
+717
-13
No files matched your search
@@ -879,6 +879,59 @@ not click Retry and no other fault fires until the last replacement is ready. A
|
||||
task restarts again after the same gaps. A replacement that exits on its own during a gap fails the
|
||||
fault as a harness error, with the stops so far recorded.
|
||||
|
||||
## Two servers, one database (MS1–MS3)
|
||||
|
||||
```sh
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \
|
||||
--adapters codex --workloads B5,Q1 --faults MS1,MS2 --concurrency 1 --tasks 3 --seed 301
|
||||
```
|
||||
|
||||
Every other fault runs one server per wave, so no wave can reach a race between two servers. #20508 fixed
|
||||
one such race with a database test only: server B's orphan sweep failed a run whose owner, server A, had
|
||||
taken its lease back.
|
||||
|
||||
**Why this shape.** Hosted Cloud deploys by rolling overlap on one Neon database. The new server B starts
|
||||
while the old server A still serves, then A drains. The defaults are a 5 s overlap and up to 90 s of
|
||||
drain (`doc/plans/2026-10-31-zero-downtime-web-deploys.md`). B's startup recovery and periodic sweeps
|
||||
therefore run beside A's live runs. The plan's K12 names the same overlap
|
||||
(`doc/plans/2026-11-06-control-plane-restart-chaos.md`). This repository has no replica count or deploy
|
||||
strategy of its own (`docker/ecs-task-definition.json` describes one task), and the self-hosting docs do
|
||||
not describe several servers on one database.
|
||||
|
||||
| Fault | What happens | Seeded hold |
|
||||
| --- | --- | --- |
|
||||
| MS1 | B boots on the same database and stays up beside a live A for the hold, then gets SIGTERM. A is not touched. | 60–150 s of overlap |
|
||||
| MS2 | B boots. A and its process tree are frozen (SIGSTOP, TM2's freeze without aging) past the 90 s controller lease, then thawed. B stays up 30 s more, while A takes its leases back, then stops. | 100–150 s frozen |
|
||||
| MS3 | Rolling-upgrade shape. B boots, then A gets K2 (SIGTERM) and a replacement on A's port. B is the only live server until the replacement is ready, then stops after the hold. | 20–45 s tail |
|
||||
|
||||
B shares A's database, instance directory, local storage and secrets. It listens on its own loopback port
|
||||
and writes `<wave>/server-b.log`. The harness and the agents keep using A's port. While a fault is in
|
||||
progress the observers hold board actions (Stop, answers, follow-ups) until it ends, as for every fault.
|
||||
|
||||
Limits:
|
||||
|
||||
- B runs on the same host and PID namespace as A. It can check A's PIDs directly, which a Railway
|
||||
replica cannot, so these faults are a lower bound on cross-server exposure.
|
||||
- The native executor lease is 20 minutes. MS2 outlasts the 90 s controller lease, not that one.
|
||||
- Local only: a remote runner is under neither server. No storm, crash point, K14, TM2 or TM3 in the same wave (aging would move the database times the invariant compares with real process windows).
|
||||
- B must live through its whole window. If it exits on its own, the fault fails (its tree is still cleaned up).
|
||||
|
||||
**Cross-server invariant** (`multi-server.ts`, `<wave>/cross-server.json`). It is checked once per wave
|
||||
that fired an MS fault, over the wave database and the process windows the supervisor recorded (each
|
||||
server's pid, start, exit and freezes). A violation fails the wave.
|
||||
|
||||
- `MS_foreign_failure`: a run failed with `process_lost` or `native_execution_ownership_unverified` while
|
||||
its owning server was alive and not frozen. The owner comes from the run's controller boot id and
|
||||
`controller_instances.pid`. The time is the run's `finished_at`. A freeze before the write does not
|
||||
excuse it: that is #20508's shape. A run still `running` with
|
||||
that code (how #20508's marker is stored) counts at its `updated_at`. The owner's own
|
||||
`controller_lease_lost` counts only when no freeze past the lease came before it.
|
||||
- `MS_double_executor`: a run recorded a second local runner identity
|
||||
(`native.process_identity_recorded`) before the first was shown stopped (`native.local_process_stopped`).
|
||||
- Ownership failures whose owner is not a wave server (or is unknown) are listed as `unattributed`.
|
||||
|
||||
The usual R1–R6, U1–U4 and T checks grade every task as before.
|
||||
|
||||
## Fault timing (`--triggers`)
|
||||
|
||||
Each task samples one trigger with its cell. `time` (default) fires at the
|
||||
@@ -923,6 +976,7 @@ re-arms the task's own triggered faults instead of using their offsets.
|
||||
| `acpx-provider-faults.ts`, `provider-shim/acpx-*` | PV1–PV3 for Claude (ACPX): the stand-in runner package, the server hook, the sidecar/provider tap, and `--provider-fault-restart`. |
|
||||
| `question-handoff.ts` | Q1 saved question: the board's answer step and its oracle. |
|
||||
| `time-faults.ts`, `time-faults-config.ts` | TM1–TM3: process freezing, database aging with recorded rows, and the time-corrected API the grader reads through. |
|
||||
| `multi-server.ts` | MS1–MS3: the second server's steps, the seeded holds, and the cross-server invariant over the wave database. |
|
||||
| `repeated-restarts.ts` | RR1–RR3: seeded restart steps, `--rr-gap-seconds`, and the stop-and-replace loop the supervisor runs. |
|
||||
| `daytona-sweep.ts` | Daytona R6: read-only reconciliation of this wave's labelled sandboxes with leases; cleanup only through Paperclip. |
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ import { parseArgs } from "node:util";
|
||||
import { applyJourneyOptions } from "./journey-matrix.js";
|
||||
import { takesJourneyTriggers } from "./journeys.js";
|
||||
import { STORAGE_FAULT_KINDS, type StorageFaultKind } from "./storage-chaos.js";
|
||||
import { validateMultiServerOptions } from "./multi-server.js";
|
||||
import { parseTimeFaultOptions } from "./time-faults-config.js";
|
||||
import { parseStorageTrigger } from "./storage-trigger.js";
|
||||
import { parseRepeatedRestartOptions } from "./repeated-restarts.js";
|
||||
@@ -223,6 +224,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
|| values["provider-fault-restart"] !== undefined || values["provider-fault-restart-after-seconds"] !== undefined) {
|
||||
throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults");
|
||||
}
|
||||
validateMultiServerOptions(options);
|
||||
Object.assign(options, parseTimeFaultOptions(options, values["time-age-hours"]));
|
||||
Object.assign(options, parseRepeatedRestartOptions(options, values["rr-gap-seconds"], values["rr-stop-on"]));
|
||||
const upgradeFrom = values["upgrade-from"];
|
||||
|
||||
@@ -14,6 +14,7 @@ import { WORKLOAD_TRIGGERS } from "./journeys.js";
|
||||
import { withProviderFaultRules } from "./provider-faults.js";
|
||||
import { createRng, k4HoldRange, TRIGGER_DELAY_RANGE_MS } from "./scheduler.js";
|
||||
import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js";
|
||||
import { isMultiServerFault, multiServerHoldMs } from "./multi-server.js";
|
||||
import { isRepeatedRestartFault, planRestartSteps } from "./repeated-restarts.js";
|
||||
import { workloadDefinitions } from "./workloads.js";
|
||||
import type { AdapterId, TaskFaultKind, TaskPlan, TriggerId, WaveOptions, WavePlan, WorkloadId } from "./types.js";
|
||||
@@ -57,7 +58,8 @@ export function planMatrixTasks(options: Pick<WaveOptions, "seed" | "workloads"
|
||||
...(cell.trigger === "time" ? {} : { trigger: cell.trigger, triggerDelayMs: rng.int(...TRIGGER_DELAY_RANGE_MS[cell.trigger]) }),
|
||||
// `--fault-after-seconds` pins time-fired restarts, as it does outside the matrix.
|
||||
fireAfterMs: cell.fault === "K0" || cell.fault.startsWith("PV") ? 0 : cell.trigger === "time" ? options.faultAfterMs ?? drawn : drawn,
|
||||
holdMs: cell.fault === "K4" ? rng.int(...k4HoldRange(options.k4HoldMaxMs)) : isTimeFault(cell.fault) ? timeFaultHoldMs(cell.fault, rng.int) : 0,
|
||||
holdMs: cell.fault === "K4" ? rng.int(...k4HoldRange(options.k4HoldMaxMs)) : isTimeFault(cell.fault) ? timeFaultHoldMs(cell.fault, rng.int)
|
||||
: isMultiServerFault(cell.fault) ? multiServerHoldMs(cell.fault, rng.int) : 0,
|
||||
...(isRepeatedRestartFault(cell.fault) ? { restartSteps: planRestartSteps(cell.fault, rng.int, options.rrGapMs) } : {}),
|
||||
followUpAfterMs: definition.followUpWindowMs ? rng.int(definition.followUpWindowMs[0], definition.followUpWindowMs[1]) : null,
|
||||
nonce: rng.hex(8),
|
||||
|
||||
@@ -44,6 +44,7 @@ import { TimeShiftedApi } from "./time-faults.js";
|
||||
import type { RestartStep } from "./repeated-restarts.js";
|
||||
import { isBrokenStdoutError, markTimelineEchoBroken, Timeline } from "./timeline.js";
|
||||
import { guardedRunnerOutputs, worktreeFingerprint } from "./wave-guard.js";
|
||||
import { checkCrossServer, readCrossServerEvidence } from "./multi-server.js";
|
||||
import type { FaultKind, FaultReceipt, OracleArtifacts, TriggerId, WavePlan } from "./types.js";
|
||||
|
||||
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
|
||||
@@ -628,6 +629,18 @@ async function main(): Promise<number> {
|
||||
rules: Object.fromEntries(Object.entries(grade.rules).map(([id, rule]) => [id, rule.verdict])), T: grade.T.verdict,
|
||||
universal: Object.fromEntries(Object.entries(grade.universal ?? {}).map(([id, rule]) => [id, `${rule.verdict}${rule.enforced ? "" : " (warning)"}`])) });
|
||||
}
|
||||
// MS1–MS3: the cross-server invariant over the whole wave (multi-server.ts).
|
||||
if (faults.some(receipt => receipt.multiServer)) {
|
||||
const crossServer = await readCrossServerEvidence(evidenceDb, lab.companyId)
|
||||
.then(evidence => checkCrossServer({ ...evidence, windows: supervisor.serverWindows }))
|
||||
.catch(error => { harnessError ??= new Error(`Cross-server invariant check failed: ${String(error)}`); return null; });
|
||||
if (crossServer) {
|
||||
await writeFile(path.join(waveDir, "cross-server.json"), `${JSON.stringify(crossServer, null, 2)}\n`, { mode: 0o600 });
|
||||
timeline.add("cross_server_invariant", { violations: crossServer.violations.map(violation => ({ rule: violation.rule, runId: violation.runId,
|
||||
issueId: violation.issueId, detail: violation.detail })), unattributed: crossServer.unattributed.length });
|
||||
if (crossServer.violations.length) anyFailure = true;
|
||||
}
|
||||
}
|
||||
if (storageAuditor) {
|
||||
storageChaos?.stop();
|
||||
await credentialRotation?.settled();
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { createDb, type Db } from "../../packages/db/src/index.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../packages/db/src/test-embedded-postgres.js";
|
||||
import { checkCrossServer, readCrossServerEvidence, type ServerProcessWindow } from "./multi-server.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const COMPANY = "00000000-0000-4000-8000-000000000001";
|
||||
const NOON = Date.parse("2026-11-10T12:00:00.000Z");
|
||||
const iso = (ms: number) => new Date(ms).toISOString();
|
||||
|
||||
describe.skipIf(!support.supported)("reading cross-server evidence from a wave database", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: Db;
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("chaos-ms-");
|
||||
db = createDb(temporary.connectionString, { maxConnections: 1 });
|
||||
// Only the columns the reader selects, with the product's names and types, in a schema ahead of the
|
||||
// migrated one (one connection, so the search path holds for every query).
|
||||
await db.$client.unsafe(`create schema chaos_ms`);
|
||||
await db.$client.unsafe(`set search_path to chaos_ms, public`);
|
||||
await db.$client.unsafe(`create table heartbeat_runs (id uuid primary key, company_id uuid not null, issue_id uuid, status text not null,
|
||||
error_code text, finished_at timestamptz, updated_at timestamptz not null, controller_boot_id uuid)`);
|
||||
await db.$client.unsafe(`create table native_run_finalizations (run_id uuid primary key, controller_boot_id text, lease_holder jsonb)`);
|
||||
await db.$client.unsafe(`create table controller_instances (boot_id text primary key, pid integer not null)`);
|
||||
await db.$client.unsafe(`create table heartbeat_run_events (run_id uuid not null, event_type text not null, payload jsonb, created_at timestamptz not null)`);
|
||||
await db.$client.unsafe(`insert into controller_instances values ('boot-a', 100), ('boot-b', 200)`);
|
||||
// #20508's marker: still running, with the ownership code, written by another server's sweep.
|
||||
await db.$client.unsafe(`insert into heartbeat_runs values ('00000000-0000-4000-8000-0000000000a1', $1, null, 'running',
|
||||
'native_execution_ownership_unverified', null, $2, null)`, [COMPANY, iso(NOON)]);
|
||||
await db.$client.unsafe(`insert into native_run_finalizations values ('00000000-0000-4000-8000-0000000000a1', 'boot-a', null)`);
|
||||
// An ordinary success: not read as a failure.
|
||||
await db.$client.unsafe(`insert into heartbeat_runs values ('00000000-0000-4000-8000-0000000000a2', $1, null, 'succeeded', null, $2, $2, null)`, [COMPANY, iso(NOON)]);
|
||||
await db.$client.unsafe(`insert into heartbeat_run_events values ('00000000-0000-4000-8000-0000000000a2', 'native.process_identity_recorded', '{"processPid": 11}', $1),
|
||||
('00000000-0000-4000-8000-0000000000a2', 'native.process_identity_recorded', '{"processPid": 12}', $2)`, [iso(NOON - 2_000), iso(NOON - 1_000)]);
|
||||
}, 120_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end();
|
||||
await temporary?.cleanup();
|
||||
});
|
||||
|
||||
it("reads a running run that carries the ownership marker, at its write time, and charges it to a live owner", async () => {
|
||||
const evidence = await readCrossServerEvidence(db, COMPANY);
|
||||
expect(evidence.runs).toEqual([expect.objectContaining({ runId: "00000000-0000-4000-8000-0000000000a1", status: "running",
|
||||
errorCode: "native_execution_ownership_unverified", endedAt: NOON, ownerBootId: "boot-a", ownerPid: 100 })]);
|
||||
expect(evidence.events.map(event => event.pid)).toEqual(expect.arrayContaining([11, 12]));
|
||||
const windows: ServerProcessWindow[] = [
|
||||
{ pid: 100, role: "primary", spawnedAt: NOON - 60_000, exitedAt: null, frozen: [{ from: NOON - 50_000, to: NOON - 5_000 }] },
|
||||
{ pid: 200, role: "secondary", spawnedAt: NOON - 55_000, exitedAt: null, frozen: [] },
|
||||
];
|
||||
const check = checkCrossServer({ ...evidence, windows });
|
||||
expect(check.violations.map(violation => [violation.rule, violation.runId])).toEqual([
|
||||
["MS_foreign_failure", "00000000-0000-4000-8000-0000000000a1"],
|
||||
["MS_double_executor", "00000000-0000-4000-8000-0000000000a2"],
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,148 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { parseCommand } from "./config.js";
|
||||
import { planTasks } from "./scheduler.js";
|
||||
import {
|
||||
CONTROLLER_LEASE_TTL_MS, MS2_THAW_TAIL_MS, MULTI_SERVER_HOLD_RANGE_MS, checkCrossServer, runMultiServerFault,
|
||||
type CrossServerRun, type MultiServerHooks, type MultiServerKind, type MultiServerRecord, type ServerProcessWindow,
|
||||
} from "./multi-server.js";
|
||||
import { FAULT_SIGNAL } from "./supervisor.js";
|
||||
|
||||
describe("two-server fault options", () => {
|
||||
it("accepts MS faults locally and gives each a seeded hold in its range", () => {
|
||||
const parsed = parseCommand(["--faults", "MS1,MS2,MS3", "--adapters", "codex", "--workloads", "B5,Q1", "--seed", "3"]);
|
||||
if (parsed.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(parsed.options.faults).toEqual(["MS1", "MS2", "MS3"]);
|
||||
const plan = planTasks({ seed: 3, tasks: 12, workloads: ["B5"], adapters: ["codex"], faults: ["MS1", "MS2", "MS3"] });
|
||||
expect(new Set(plan.map(task => task.fault))).toEqual(new Set(["MS1", "MS2", "MS3"]));
|
||||
for (const task of plan) {
|
||||
const [min, max] = MULTI_SERVER_HOLD_RANGE_MS[task.fault as MultiServerKind];
|
||||
expect(task.holdMs).toBeGreaterThanOrEqual(min);
|
||||
expect(task.holdMs).toBeLessThanOrEqual(max);
|
||||
}
|
||||
// MS2 freezes the primary past the controller lease, so a second server may legitimately fence it.
|
||||
expect(MULTI_SERVER_HOLD_RANGE_MS.MS2[0]).toBeGreaterThan(CONTROLLER_LEASE_TTL_MS);
|
||||
expect(FAULT_SIGNAL).toMatchObject({ MS1: "none", MS2: "SIGSTOP", MS3: "SIGTERM" });
|
||||
});
|
||||
|
||||
it("refuses combinations a second local server cannot serve", () => {
|
||||
expect(() => parseCommand(["--faults", "MS1", "--env", "daytona", "--daytona-image", `r/i@sha256:${"a".repeat(64)}`])).toThrow(/--env local/);
|
||||
expect(() => parseCommand(["--faults", "MS2", "--storm-minutes", "10"])).toThrow(/storm/);
|
||||
expect(() => parseCommand(["--faults", "MS3,K14", "--upgrade-from", "/tmp/x"])).toThrow(/K14/);
|
||||
expect(() => parseCommand(["--faults", "MS1,TM2", "--workloads", "B5"])).toThrow(/TM2 or TM3/);
|
||||
});
|
||||
});
|
||||
|
||||
function fakeHooks(options: { failOn?: "start" | "freeze" | "restart"; exitAt?: number } = {}) {
|
||||
let clock = 0;
|
||||
const calls: string[] = [];
|
||||
const hooks: MultiServerHooks = {
|
||||
now: () => clock,
|
||||
sleep: async ms => { clock += ms; },
|
||||
cancelled: () => false,
|
||||
secondaryFailure: () => (options.exitAt !== undefined && clock >= options.exitAt ? "code=1 signal=null" : null),
|
||||
startSecondary: async () => { calls.push("start"); if (options.failOn === "start") throw new Error("B did not start"); },
|
||||
stopSecondary: async () => { calls.push("stop"); },
|
||||
freezePrimary: async holdMs => { calls.push(`freeze:${holdMs}`); clock += holdMs; if (options.failOn === "freeze") throw new Error("A could not be shown stopped"); },
|
||||
restartPrimary: async () => { calls.push("restart"); clock += 5_000; if (options.failOn === "restart") throw new Error("A' not ready"); },
|
||||
};
|
||||
return { hooks, calls, clock: () => clock };
|
||||
}
|
||||
|
||||
const record = (kind: MultiServerKind): MultiServerRecord => ({ kind, secondary: null, steps: [] });
|
||||
|
||||
describe("runMultiServerFault", () => {
|
||||
it("MS1 keeps B beside a live A for the hold, then stops B", async () => {
|
||||
const fake = fakeHooks();
|
||||
const result = record("MS1");
|
||||
await runMultiServerFault("MS1", 90_000, result, fake.hooks);
|
||||
expect(fake.calls).toEqual(["start", "stop"]);
|
||||
expect(fake.clock()).toBe(90_000);
|
||||
expect(result.steps.map(step => step.step)).toEqual(["secondary_starting", "secondary_ready", "secondary_stopping", "secondary_stopped"]);
|
||||
});
|
||||
|
||||
it("MS2 freezes A past the lease while B is up, keeps B for the thaw tail, then stops B", async () => {
|
||||
const fake = fakeHooks();
|
||||
const result = record("MS2");
|
||||
await runMultiServerFault("MS2", 120_000, result, fake.hooks);
|
||||
expect(fake.calls).toEqual(["start", "freeze:120000", "stop"]);
|
||||
expect(fake.clock()).toBe(120_000 + MS2_THAW_TAIL_MS);
|
||||
expect(result.steps.map(step => step.step)).toContain("primary_thawed");
|
||||
});
|
||||
|
||||
it("MS3 restarts A (K2) while B serves, holds B's tail, then stops B", async () => {
|
||||
const fake = fakeHooks();
|
||||
await runMultiServerFault("MS3", 30_000, record("MS3"), fake.hooks);
|
||||
expect(fake.calls).toEqual(["start", "restart", "stop"]);
|
||||
expect(fake.clock()).toBe(35_000);
|
||||
});
|
||||
|
||||
it("fails the fault when B exits on its own during the overlap or a freeze, and still stops it", async () => {
|
||||
const overlap = fakeHooks({ exitAt: 30_000 });
|
||||
await expect(runMultiServerFault("MS1", 90_000, record("MS1"), overlap.hooks)).rejects.toThrow(/exited before its planned stop/);
|
||||
expect(overlap.calls).toEqual(["start", "stop"]);
|
||||
expect(overlap.clock()).toBeLessThan(90_000);
|
||||
const frozen = fakeHooks({ exitAt: 50_000 });
|
||||
await expect(runMultiServerFault("MS2", 120_000, record("MS2"), frozen.hooks)).rejects.toThrow(/exited before its planned stop/);
|
||||
expect(frozen.calls.at(-1)).toBe("stop");
|
||||
const restarted = fakeHooks({ exitAt: 1_000 });
|
||||
await expect(runMultiServerFault("MS3", 0, record("MS3"), restarted.hooks)).rejects.toThrow(/exited before its planned stop/);
|
||||
});
|
||||
|
||||
it.each(["start", "freeze", "restart"] as const)("stops B and fails the fault when %s fails", async failOn => {
|
||||
const fake = fakeHooks({ failOn });
|
||||
const kind: MultiServerKind = failOn === "freeze" ? "MS2" : failOn === "restart" ? "MS3" : "MS1";
|
||||
await expect(runMultiServerFault(kind, 60_000, record(kind), fake.hooks)).rejects.toThrow();
|
||||
expect(fake.calls.at(-1)).toBe("stop");
|
||||
});
|
||||
});
|
||||
|
||||
const A: ServerProcessWindow = { pid: 100, role: "primary", spawnedAt: 0, exitedAt: null, frozen: [] };
|
||||
const B: ServerProcessWindow = { pid: 200, role: "secondary", spawnedAt: 50_000, exitedAt: 300_000, frozen: [] };
|
||||
const run = (overrides: Partial<CrossServerRun>): CrossServerRun => ({ runId: "r1", issueId: "i1", status: "failed",
|
||||
errorCode: "native_execution_ownership_unverified", endedAt: 60_000, ownerBootId: "boot-a", ownerPid: 100, ...overrides });
|
||||
|
||||
describe("checkCrossServer", () => {
|
||||
it("charges an ownership failure written while the owner was alive and running", () => {
|
||||
const check = checkCrossServer({ runs: [run({})], events: [], windows: [A, B] });
|
||||
expect(check.violations).toMatchObject([{ rule: "MS_foreign_failure", runId: "r1", issueId: "i1" }]);
|
||||
});
|
||||
|
||||
it("allows an ownership failure while the owner was frozen or after it exited", () => {
|
||||
const frozen = { ...A, frozen: [{ from: 55_000, to: 200_000 }] };
|
||||
expect(checkCrossServer({ runs: [run({})], events: [], windows: [frozen, B] }).violations).toEqual([]);
|
||||
const exited = { ...A, exitedAt: 59_000 };
|
||||
const check = checkCrossServer({ runs: [run({})], events: [], windows: [exited, B] });
|
||||
expect(check.violations).toEqual([]);
|
||||
expect(check.unattributed).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("charges #20508's shape: a sweep's write after the frozen owner thawed and took its lease back", () => {
|
||||
const thawed = { ...A, frozen: [{ from: 10_000, to: 110_000 }] };
|
||||
const check = checkCrossServer({ runs: [run({ endedAt: 115_000 })], events: [], windows: [thawed, B] });
|
||||
expect(check.violations.map(violation => violation.rule)).toEqual(["MS_foreign_failure"]);
|
||||
});
|
||||
|
||||
it("allows the owner's own lease loss after a freeze past the lease, not without one", () => {
|
||||
const leaseLost = run({ errorCode: "controller_lease_lost", endedAt: 115_000 });
|
||||
const pastLease = { ...A, frozen: [{ from: 10_000, to: 10_000 + CONTROLLER_LEASE_TTL_MS + 1 }] };
|
||||
expect(checkCrossServer({ runs: [leaseLost], events: [], windows: [pastLease, B] }).violations).toEqual([]);
|
||||
const shortFreeze = { ...A, frozen: [{ from: 10_000, to: 40_000 }] };
|
||||
expect(checkCrossServer({ runs: [leaseLost], events: [], windows: [shortFreeze, B] }).violations).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("ignores ordinary failures and reports owners it cannot attribute", () => {
|
||||
expect(checkCrossServer({ runs: [run({ errorCode: "adapter_failed" })], events: [], windows: [A] })).toMatchObject({ violations: [], unattributed: [] });
|
||||
expect(checkCrossServer({ runs: [run({ ownerPid: null })], events: [], windows: [A] }).unattributed).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("charges a second runner identity recorded before the first was shown stopped", () => {
|
||||
const recorded = (pid: number, at: number) => ({ runId: "r1", kind: "recorded" as const, pid, at });
|
||||
const stopped = (pid: number, at: number) => ({ runId: "r1", kind: "stopped" as const, pid, at });
|
||||
const runs = [run({ errorCode: null, status: "succeeded" })];
|
||||
expect(checkCrossServer({ runs, events: [recorded(1, 1), recorded(2, 5)], windows: [A] }).violations)
|
||||
.toMatchObject([{ rule: "MS_double_executor", evidence: { firstPid: 1, secondPid: 2 } }]);
|
||||
expect(checkCrossServer({ runs, events: [recorded(1, 1), stopped(1, 4), recorded(2, 5)], windows: [A] }).violations).toEqual([]);
|
||||
// The same runner recorded again (a re-attach) is one executor.
|
||||
expect(checkCrossServer({ runs, events: [recorded(1, 1), recorded(1, 5)], windows: [A] }).violations).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,285 @@
|
||||
/**
|
||||
* MS1–MS3: two control planes on one database (README "Two servers, one database").
|
||||
*
|
||||
* Hosted Cloud deploys by rolling overlap: a new server B starts on the same database while the old
|
||||
* server A still serves, then A drains (doc/plans/2026-10-31-zero-downtime-web-deploys.md: 5 s overlap,
|
||||
* up to 90 s drain). B's startup recovery and periodic sweeps therefore run beside A's live runs. A wave
|
||||
* otherwise runs one server, so it cannot reach a cross-server race such as #20508 (B's orphan sweep
|
||||
* failing a run whose owner had taken its lease back).
|
||||
*
|
||||
* MS1 B boots and stays up beside a live A for the hold, then stops (SIGTERM). A is not touched.
|
||||
* MS2 B boots; A and its process tree are frozen (SIGSTOP) past the 90 s controller lease, then
|
||||
* thawed. B stays up for a short tail so it sees A take its leases back, then stops.
|
||||
* MS3 Rolling-upgrade shape: B boots, then A gets K2 (SIGTERM, restart on its own port). B is the
|
||||
* only live server between A's stop and its replacement's readiness, then stops after a tail.
|
||||
*
|
||||
* B shares A's database, instance directory, local storage and secrets, and listens on its own
|
||||
* loopback port. The harness and agents keep talking to A's port. B runs on the same host and PID
|
||||
* namespace as A, so it can check A's PIDs directly; a Railway replica cannot. That makes these
|
||||
* faults a lower bound on cross-server exposure, not a model of separate hosts.
|
||||
*
|
||||
* The cross-server invariant (`checkCrossServer`) is graded once per wave over the database:
|
||||
* MS_foreign_failure a run failed with an ownership code while its owning server was alive and running
|
||||
* MS_double_executor a run recorded a second local runner identity before the first was shown stopped
|
||||
*/
|
||||
import type { Db } from "../../packages/db/src/index.js";
|
||||
import type { TimeFaultRecord } from "./time-faults.js";
|
||||
|
||||
export const MULTI_SERVER_FAULT_KINDS = ["MS1", "MS2", "MS3"] as const;
|
||||
export type MultiServerKind = (typeof MULTI_SERVER_FAULT_KINDS)[number];
|
||||
|
||||
export function isMultiServerFault(kind: string): kind is MultiServerKind {
|
||||
return (MULTI_SERVER_FAULT_KINDS as readonly string[]).includes(kind);
|
||||
}
|
||||
|
||||
/** `CONTROLLER_LEASE_TTL_MS` in server/src/services/controller-instance-lease.ts. */
|
||||
export const CONTROLLER_LEASE_TTL_MS = 90_000;
|
||||
|
||||
/**
|
||||
* Seeded hold per kind. MS1: B's overlap, long enough for its boot recovery and a periodic sweep.
|
||||
* MS2: A's freeze, past the controller lease. MS3: B's tail after A's replacement is ready.
|
||||
*/
|
||||
export const MULTI_SERVER_HOLD_RANGE_MS: Record<MultiServerKind, readonly [number, number]> = {
|
||||
MS1: [60_000, 150_000],
|
||||
MS2: [CONTROLLER_LEASE_TTL_MS + 10_000, 150_000],
|
||||
MS3: [20_000, 45_000],
|
||||
};
|
||||
|
||||
/** MS2: B stays up this long after A is thawed, the window in which #20508's foreign write landed. */
|
||||
export const MS2_THAW_TAIL_MS = 30_000;
|
||||
|
||||
export function multiServerHoldMs(kind: MultiServerKind, int: (min: number, max: number) => number): number {
|
||||
return int(...MULTI_SERVER_HOLD_RANGE_MS[kind]);
|
||||
}
|
||||
|
||||
/** `--faults MS*` validation; the planner itself is the ordinary per-task planner. */
|
||||
export function validateMultiServerOptions(options: { env: string; faults: readonly string[]; stormMinutes?: number;
|
||||
crashPoint?: string; providerFaults?: readonly string[] }): void {
|
||||
const kinds = options.faults.filter(isMultiServerFault);
|
||||
if (!kinds.length) return;
|
||||
// B shares A's instance directory and local runners; a remote runner is not under either server.
|
||||
if (options.env !== "local") throw new Error("Two-server faults (MS*) start a second local server; run them with --env local");
|
||||
if (options.stormMinutes) throw new Error("Two-server faults (MS*) are planned per task; do not combine them with --storm-minutes");
|
||||
if (options.crashPoint) throw new Error("Two-server faults (MS*) do not combine with --crash-point");
|
||||
if (options.faults.includes("K14")) throw new Error("Two-server faults (MS*) do not combine with K14; both servers run the current build");
|
||||
// Aging moves database times; the cross-server invariant compares them with real process windows.
|
||||
if (options.faults.some(kind => kind === "TM2" || kind === "TM3")) throw new Error("Two-server faults (MS*) do not combine with TM2 or TM3; age the database in its own wave");
|
||||
}
|
||||
|
||||
/** One server process the wave started, for attributing a run's owner at a moment in time. */
|
||||
export interface ServerProcessWindow {
|
||||
pid: number;
|
||||
role: "primary" | "secondary";
|
||||
spawnedAt: number;
|
||||
exitedAt: number | null;
|
||||
/** SIGSTOP windows; an open end means still frozen. */
|
||||
frozen: Array<{ from: number; to: number | null }>;
|
||||
}
|
||||
|
||||
export interface MultiServerRecord {
|
||||
kind: MultiServerKind;
|
||||
secondary: { pid: number | null; port: number; spawnedAt: number | null; readyAt: number | null; stoppedAt: number | null } | null;
|
||||
/** MS2: A's freeze (tree SIGSTOP, no aging). */
|
||||
primaryFreeze?: TimeFaultRecord;
|
||||
/** MS3: A's K2 and its replacement. */
|
||||
primaryRestart?: { stoppedAt: number | null; restartedAt: number | null; readyAt: number | null; replacementPid: number | null };
|
||||
steps: Array<{ step: string; at: number }>;
|
||||
}
|
||||
|
||||
export interface MultiServerHooks {
|
||||
now(): number;
|
||||
sleep(ms: number): Promise<void>;
|
||||
cancelled(): boolean;
|
||||
/** Why B stopped on its own (its exit code or signal), or null while it runs as planned. */
|
||||
secondaryFailure(): string | null;
|
||||
/** Spawns B on its own port and waits until it reports ready. */
|
||||
startSecondary(record: MultiServerRecord): Promise<void>;
|
||||
/** SIGTERM B and wait for it to exit; always called once B was spawned. */
|
||||
stopSecondary(record: MultiServerRecord): Promise<void>;
|
||||
/** MS2: freeze A's tree for `holdMs`, thaw it, and replace A if it exited after the thaw. */
|
||||
freezePrimary(holdMs: number, record: MultiServerRecord): Promise<void>;
|
||||
/** MS3: K2 on A, start its replacement on A's port, and wait until it is ready. */
|
||||
restartPrimary(record: MultiServerRecord): Promise<void>;
|
||||
}
|
||||
|
||||
async function hold(hooks: MultiServerHooks, ms: number): Promise<void> {
|
||||
const until = hooks.now() + ms;
|
||||
while (hooks.now() < until) {
|
||||
if (hooks.cancelled()) throw new Error("Harness shutdown interrupted the two-server fault");
|
||||
const exited = hooks.secondaryFailure();
|
||||
if (exited) throw new Error(`The second server exited before its planned stop: ${exited}`);
|
||||
await hooks.sleep(Math.min(1_000, until - hooks.now()));
|
||||
}
|
||||
}
|
||||
|
||||
/** Runs one MS fault. B is stopped on every path once it was started, so it never outlives its fault. */
|
||||
export async function runMultiServerFault(kind: MultiServerKind, holdMs: number, record: MultiServerRecord, hooks: MultiServerHooks): Promise<void> {
|
||||
const step = (name: string) => record.steps.push({ step: name, at: hooks.now() });
|
||||
let started = false;
|
||||
let failure: unknown = null;
|
||||
try {
|
||||
started = true;
|
||||
step("secondary_starting");
|
||||
await hooks.startSecondary(record);
|
||||
step("secondary_ready");
|
||||
if (kind === "MS1") {
|
||||
await hold(hooks, holdMs);
|
||||
} else if (kind === "MS2") {
|
||||
step("primary_freezing");
|
||||
await hooks.freezePrimary(holdMs, record);
|
||||
step("primary_thawed");
|
||||
await hold(hooks, MS2_THAW_TAIL_MS);
|
||||
} else {
|
||||
step("primary_restarting");
|
||||
await hooks.restartPrimary(record);
|
||||
step("primary_ready");
|
||||
await hold(hooks, holdMs);
|
||||
}
|
||||
// B must have lived through the whole window, including a freeze or restart without a hold check.
|
||||
const exited = hooks.secondaryFailure();
|
||||
if (exited) throw new Error(`The second server exited before its planned stop: ${exited}`);
|
||||
} catch (error) {
|
||||
failure = error;
|
||||
}
|
||||
if (started) {
|
||||
step("secondary_stopping");
|
||||
try { await hooks.stopSecondary(record); step("secondary_stopped"); }
|
||||
catch (error) { failure ??= error; }
|
||||
}
|
||||
if (failure) throw failure;
|
||||
}
|
||||
|
||||
// ---------- cross-server invariant ----------
|
||||
|
||||
/** A sweep's verdict on another server's run. */
|
||||
export const FOREIGN_FAILURE_CODES: ReadonlySet<string> = new Set(["process_lost", "native_execution_ownership_unverified"]);
|
||||
/**
|
||||
* The owner's own lease loss: expected after a freeze past the lease, a violation otherwise. Only the legacy
|
||||
* 60 s run lease persists a run code; a native session lease loss is recorded as `native_session_interrupted`.
|
||||
*/
|
||||
export const LEASE_LOSS_CODES: ReadonlySet<string> = new Set(["controller_lease_lost"]);
|
||||
|
||||
export interface CrossServerRun {
|
||||
runId: string;
|
||||
issueId: string | null;
|
||||
status: string;
|
||||
errorCode: string | null;
|
||||
/**
|
||||
* When the run reached its final status (finished_at), else when it was last written (updated_at), ms.
|
||||
* #20508's marker leaves the run `running` with the ownership code, so its write time is updated_at.
|
||||
*/
|
||||
endedAt: number | null;
|
||||
ownerBootId: string | null;
|
||||
ownerPid: number | null;
|
||||
}
|
||||
|
||||
export interface RunIdentityEvent { runId: string; kind: "recorded" | "stopped"; pid: number; at: number }
|
||||
|
||||
export interface CrossServerViolation {
|
||||
rule: "MS_foreign_failure" | "MS_double_executor";
|
||||
runId: string;
|
||||
issueId: string | null;
|
||||
detail: string;
|
||||
evidence: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface CrossServerCheck {
|
||||
violations: CrossServerViolation[];
|
||||
/** Ownership failures the harness could not attribute (owner boot or pid unknown, or not a wave server). */
|
||||
unattributed: Array<{ runId: string; errorCode: string | null; ownerBootId: string | null; ownerPid: number | null }>;
|
||||
windows: ServerProcessWindow[];
|
||||
}
|
||||
|
||||
const aliveAt = (window: ServerProcessWindow, at: number) => window.spawnedAt <= at && (window.exitedAt === null || at < window.exitedAt);
|
||||
const frozenAt = (window: ServerProcessWindow, at: number) => window.frozen.some(span => span.from <= at && (span.to === null || at < span.to));
|
||||
const frozenPastLeaseBefore = (window: ServerProcessWindow, at: number, leaseTtlMs: number) =>
|
||||
window.frozen.some(span => span.from < at && ((span.to ?? at) - span.from) >= leaseTtlMs);
|
||||
|
||||
/**
|
||||
* Pure: grade the wave's runs against the server process windows the harness recorded.
|
||||
* A failure is charged only when the owner is a wave server, alive and running (not frozen) at that moment.
|
||||
*/
|
||||
export function checkCrossServer(input: { runs: readonly CrossServerRun[]; events: readonly RunIdentityEvent[];
|
||||
windows: readonly ServerProcessWindow[]; leaseTtlMs?: number }): CrossServerCheck {
|
||||
const leaseTtlMs = input.leaseTtlMs ?? CONTROLLER_LEASE_TTL_MS;
|
||||
const violations: CrossServerViolation[] = [];
|
||||
const unattributed: CrossServerCheck["unattributed"] = [];
|
||||
const issueOf = new Map(input.runs.map(run => [run.runId, run.issueId]));
|
||||
for (const run of input.runs) {
|
||||
const code = run.errorCode ?? "";
|
||||
const foreign = FOREIGN_FAILURE_CODES.has(code);
|
||||
const leaseLoss = LEASE_LOSS_CODES.has(code);
|
||||
if (!foreign && !leaseLoss) continue;
|
||||
// Several boots can reuse a pid only after the first exited: pick the window alive at the end time.
|
||||
const at = run.endedAt;
|
||||
const owner = at === null || run.ownerPid === null ? undefined
|
||||
: input.windows.find(window => window.pid === run.ownerPid && aliveAt(window, at));
|
||||
if (!owner || at === null) {
|
||||
unattributed.push({ runId: run.runId, errorCode: run.errorCode, ownerBootId: run.ownerBootId, ownerPid: run.ownerPid });
|
||||
continue;
|
||||
}
|
||||
if (frozenAt(owner, at)) continue;
|
||||
// After a freeze past the lease the owner may lose its own lease; a sweep's write on a thawed owner is still foreign.
|
||||
if (leaseLoss && frozenPastLeaseBefore(owner, at, leaseTtlMs)) continue;
|
||||
violations.push({ rule: "MS_foreign_failure", runId: run.runId, issueId: run.issueId,
|
||||
detail: `${code} at ${new Date(at).toISOString()} while owner pid ${owner.pid} (${owner.role}) was alive and running`,
|
||||
evidence: { errorCode: code, endedAt: at, ownerBootId: run.ownerBootId, ownerPid: owner.pid, ownerRole: owner.role,
|
||||
ownerFrozen: owner.frozen } });
|
||||
}
|
||||
const byRun = new Map<string, RunIdentityEvent[]>();
|
||||
for (const event of input.events) {
|
||||
const list = byRun.get(event.runId) ?? [];
|
||||
list.push(event);
|
||||
byRun.set(event.runId, list);
|
||||
}
|
||||
for (const [runId, events] of byRun) {
|
||||
const live = new Set<number>();
|
||||
for (const event of [...events].sort((left, right) => left.at - right.at || (left.kind === "stopped" ? -1 : 1))) {
|
||||
if (event.kind === "stopped") { live.delete(event.pid); continue; }
|
||||
const other = [...live].find(pid => pid !== event.pid);
|
||||
if (other !== undefined) {
|
||||
violations.push({ rule: "MS_double_executor", runId, issueId: issueOf.get(runId) ?? null,
|
||||
detail: `runner pid ${event.pid} recorded at ${new Date(event.at).toISOString()} while pid ${other} was never shown stopped`,
|
||||
evidence: { firstPid: other, secondPid: event.pid, at: event.at } });
|
||||
live.delete(other);
|
||||
}
|
||||
live.add(event.pid);
|
||||
}
|
||||
}
|
||||
return { violations, unattributed, windows: [...input.windows] };
|
||||
}
|
||||
|
||||
type Row = Record<string, unknown>;
|
||||
const num = (value: unknown) => value === null || value === undefined || value === "" ? null : Number(value);
|
||||
|
||||
/** Reads the runs and runner identity events `checkCrossServer` needs from the wave database. */
|
||||
export async function readCrossServerEvidence(db: Db, companyId: string): Promise<{ runs: CrossServerRun[]; events: RunIdentityEvent[] }> {
|
||||
const client = db.$client;
|
||||
const runs = await client.unsafe(
|
||||
`select r.id::text as "runId", r.issue_id::text as "issueId", r.status, r.error_code as "errorCode",
|
||||
floor(extract(epoch from coalesce(r.finished_at, r.updated_at)) * 1000) as "endedAt",
|
||||
coalesce(r.controller_boot_id::text, f.controller_boot_id, f.lease_holder->>'bootId') as "ownerBootId",
|
||||
ci.pid as "ownerPid"
|
||||
from heartbeat_runs r
|
||||
left join native_run_finalizations f on f.run_id = r.id
|
||||
left join controller_instances ci on ci.boot_id = coalesce(r.controller_boot_id::text, f.controller_boot_id, f.lease_holder->>'bootId')
|
||||
where r.company_id::text = $1
|
||||
and (r.status in ('failed', 'timed_out', 'cancelled') or r.error_code in ('process_lost', 'native_execution_ownership_unverified'))`,
|
||||
[companyId]) as Row[];
|
||||
const events = await client.unsafe(
|
||||
`select e.run_id::text as "runId", e.event_type as "eventType", e.payload->>'processPid' as pid,
|
||||
floor(extract(epoch from e.created_at) * 1000) as at
|
||||
from heartbeat_run_events e join heartbeat_runs r on r.id = e.run_id
|
||||
where r.company_id::text = $1 and e.event_type in ('native.process_identity_recorded', 'native.local_process_stopped')`, [companyId]) as Row[];
|
||||
return {
|
||||
runs: runs.map(row => ({ runId: String(row.runId), issueId: row.issueId === null ? null : String(row.issueId), status: String(row.status),
|
||||
errorCode: row.errorCode === null ? null : String(row.errorCode), endedAt: num(row.endedAt),
|
||||
ownerBootId: row.ownerBootId === null ? null : String(row.ownerBootId), ownerPid: num(row.ownerPid) })),
|
||||
events: events.flatMap(row => {
|
||||
const pid = num(row.pid);
|
||||
const at = num(row.at);
|
||||
if (pid === null || at === null || !Number.isSafeInteger(pid)) return [];
|
||||
return [{ runId: String(row.runId), kind: row.eventType === "native.local_process_stopped" ? "stopped" as const : "recorded" as const, pid, at }];
|
||||
}),
|
||||
};
|
||||
}
|
||||
@@ -2,6 +2,7 @@ import { isProviderFault, withProviderFaultRules } from "./provider-faults.js";
|
||||
import { takesJourneyTriggers } from "./journeys.js";
|
||||
import { workloadDefinitions } from "./workloads.js";
|
||||
import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js";
|
||||
import { isMultiServerFault, multiServerHoldMs } from "./multi-server.js";
|
||||
import { isRepeatedRestartFault, planRestartSteps, type RestartStep } from "./repeated-restarts.js";
|
||||
import { K4_MIN_HOLD_MS, type AdapterId, type FaultKind, type ScheduledFault, type StormFault, type StormPlan, type TaskFaultKind, type TaskPlan, type TriggerId, type WavePlan, type WaveOptions, type WorkloadId } from "./types.js";
|
||||
import type { ReplayRef } from "./config.js";
|
||||
@@ -125,7 +126,8 @@ export function planTasks(options: Pick<WaveOptions, "seed" | "tasks" | "workloa
|
||||
...cellRest,
|
||||
...(trigger === "time" ? {} : { trigger, triggerDelayMs: rng.int(...TRIGGER_DELAY_RANGE_MS[trigger]) }),
|
||||
fireAfterMs: cell.fault === "K13" ? 0 : pinnedFireAfterMs(options.faultAfterMs, rng.int(fireMin, fireMax)),
|
||||
holdMs: cell.fault === "K4" ? rng.int(...k4HoldRange(options.k4HoldMaxMs)) : isTimeFault(cell.fault) ? timeFaultHoldMs(cell.fault, rng.int) : 0,
|
||||
holdMs: cell.fault === "K4" ? rng.int(...k4HoldRange(options.k4HoldMaxMs)) : isTimeFault(cell.fault) ? timeFaultHoldMs(cell.fault, rng.int)
|
||||
: isMultiServerFault(cell.fault) ? multiServerHoldMs(cell.fault, rng.int) : 0,
|
||||
// Drawn only for RR tasks, so the plans of existing seeds do not change.
|
||||
...(isRepeatedRestartFault(cell.fault) ? { restartSteps: planRestartSteps(cell.fault, rng.int, options.rrGapMs) } : {}),
|
||||
followUpAfterMs: definition.followUpWindowMs ? rng.int(definition.followUpWindowMs[0], definition.followUpWindowMs[1]) : null,
|
||||
|
||||
@@ -38,6 +38,7 @@ import type { Timeline } from "./timeline.js";
|
||||
import { isRepeatedRestartFault, newRepeatedRestartRecord, recordFirstRestart, runRepeatedRestarts, stopPhaseWatcher, type RepeatedRestartKind, type RepeatedRestartStopOn,
|
||||
type RepeatedRestartRecord, type RestartStep } from "./repeated-restarts.js";
|
||||
import type { FaultKind, FaultReceipt, TriggerId } from "./types.js";
|
||||
import { isMultiServerFault, runMultiServerFault, type MultiServerKind, type MultiServerRecord, type ServerProcessWindow } from "./multi-server.js";
|
||||
|
||||
/** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */
|
||||
const HEALTH_TIMEOUT_MS = (() => {
|
||||
@@ -47,8 +48,10 @@ const HEALTH_TIMEOUT_MS = (() => {
|
||||
const READY_TIMEOUT_MS = 300_000;
|
||||
const STOP_TIMEOUT_MS = 30_000;
|
||||
|
||||
export const FAULT_SIGNAL: Record<FaultKind, "SIGKILL" | "SIGTERM" | "SIGSTOP"> = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM",
|
||||
TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL", RR1: "SIGKILL", RR2: "SIGKILL", RR3: "SIGKILL" };
|
||||
export const FAULT_SIGNAL: Record<FaultKind, FaultReceipt["signal"]> = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM",
|
||||
TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL", RR1: "SIGKILL", RR2: "SIGKILL", RR3: "SIGKILL",
|
||||
// MS1 leaves the primary alone; MS2 freezes it; MS3 stops it gracefully (multi-server.ts).
|
||||
MS1: "none", MS2: "SIGSTOP", MS3: "SIGTERM" };
|
||||
|
||||
/** The checkout's commit, for the K14 receipt. */
|
||||
function checkoutSha(root: string): string | null {
|
||||
@@ -140,6 +143,10 @@ export class ChaosSupervisor {
|
||||
timeAgeMs = DEFAULT_TIME_AGE_MS;
|
||||
/** RR1–RR3 (`--rr-stop-on`): stop each replacement on this `/api/ready` status instead of after its gap. */
|
||||
rrStopOn: RepeatedRestartStopOn | null = null;
|
||||
/** Every server process this wave started, with its lifetime and freezes (MS cross-server invariant). */
|
||||
readonly serverWindows: ServerProcessWindow[] = [];
|
||||
/** MS1–MS3: the second server, while one is running. */
|
||||
private secondary: { child: ChildProcess; port: number; log: WriteStream; exit: string | null } | null = null;
|
||||
/** Every aging shift so far, in order; the grader's API reads times through them (time-faults.ts). */
|
||||
readonly agingShifts: AgingShift[] = [];
|
||||
private readonly createdAt = Date.now();
|
||||
@@ -253,11 +260,13 @@ export class ChaosSupervisor {
|
||||
const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]),
|
||||
{ cwd: root, env, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached });
|
||||
this.child = child;
|
||||
const window = this.trackWindow(child, "primary");
|
||||
|
||||
child.stdout?.on("data", chunk => this.log.write(chunk));
|
||||
child.stderr?.on("data", chunk => this.log.write(chunk));
|
||||
child.once("error", error => { this.spawnErrors.set(child, error); });
|
||||
child.once("exit", (code, signal) => {
|
||||
if (window) window.exitedAt = Date.now();
|
||||
this.log.write(`\n[restart-chaos] server pid=${child.pid} exited code=${code} signal=${signal}\n`);
|
||||
if (control && signal === "SIGKILL" && !this.expectedStops.has(child) && !this.shuttingDown) {
|
||||
const hit = readCrashPointReceipt(control.receiptPath, child.pid);
|
||||
@@ -274,26 +283,41 @@ export class ChaosSupervisor {
|
||||
return child;
|
||||
}
|
||||
|
||||
private trackWindow(child: ChildProcess, role: ServerProcessWindow["role"]): ServerProcessWindow | null {
|
||||
if (!child.pid) return null;
|
||||
const window: ServerProcessWindow = { pid: child.pid, role, spawnedAt: Date.now(), exitedAt: null, frozen: [] };
|
||||
this.serverWindows.push(window);
|
||||
return window;
|
||||
}
|
||||
|
||||
/** Records a freeze or thaw of a server process for the MS invariant (TM1/TM2 and MS2). */
|
||||
private markFrozen(pid: number | undefined, at: number, frozen: boolean): void {
|
||||
const window = [...this.serverWindows].reverse().find(candidate => candidate.pid === pid && candidate.exitedAt === null);
|
||||
if (!window) return;
|
||||
if (frozen) window.frozen.push({ from: at, to: null });
|
||||
else { const open = window.frozen.find(span => span.to === null); if (open) open.to = at; }
|
||||
}
|
||||
|
||||
private exited(child: ChildProcess) { return child.exitCode !== null || child.signalCode !== null || this.spawnErrors.has(child); }
|
||||
|
||||
private async readPhase(): Promise<string> {
|
||||
private async readPhase(baseUrl = this.baseUrl): Promise<string> {
|
||||
try {
|
||||
const response = await fetch(`${this.baseUrl}/api/ready`, { signal: AbortSignal.timeout(1_000), cache: "no-store" });
|
||||
const response = await fetch(`${baseUrl}/api/ready`, { signal: AbortSignal.timeout(1_000), cache: "no-store" });
|
||||
return String(((await response.json()) as { status?: string }).status ?? response.status);
|
||||
} catch { return "unreachable"; }
|
||||
}
|
||||
|
||||
private async waitForHealth(child: ChildProcess, phases?: FaultReceipt["phases"]): Promise<number> {
|
||||
private async waitForHealth(child: ChildProcess, phases?: FaultReceipt["phases"], baseUrl = this.baseUrl): Promise<number> {
|
||||
const deadline = Date.now() + HEALTH_TIMEOUT_MS;
|
||||
while (Date.now() < deadline) {
|
||||
if (this.shuttingDown) throw new Error("Harness shutdown interrupted startup");
|
||||
if (this.exited(child)) throw new Error(`Paperclip server exited during startup (code=${child.exitCode} signal=${child.signalCode})`);
|
||||
if (phases) {
|
||||
const phase = await this.readPhase();
|
||||
const phase = await this.readPhase(baseUrl);
|
||||
if (phase !== phases[phases.length - 1]?.phase) phases.push({ phase, at: Date.now() });
|
||||
}
|
||||
try {
|
||||
const response = await fetch(`${this.baseUrl}/api/health`, { signal: AbortSignal.timeout(1_000) });
|
||||
const response = await fetch(`${baseUrl}/api/health`, { signal: AbortSignal.timeout(1_000) });
|
||||
await response.body?.cancel();
|
||||
if (response.ok) return Date.now();
|
||||
} catch { /* still booting */ }
|
||||
@@ -433,6 +457,7 @@ export class ChaosSupervisor {
|
||||
if (!previous?.pid || this.exited(previous)) throw new Error("No live Paperclip server to fault");
|
||||
let outage: ControllerOutageReceipt | undefined;
|
||||
if (kind === "TM1" || kind === "TM2") await this.freeze(kind, previous, holdMs, receipt);
|
||||
else if (isMultiServerFault(kind)) await this.multiServer(kind, previous, holdMs, receipt);
|
||||
else outage = await replaceController({
|
||||
requestId: `chaos-${receipt.faultId}-${requestedBy}`,
|
||||
previousPid: previous.pid,
|
||||
@@ -516,8 +541,8 @@ export class ChaosSupervisor {
|
||||
const times = await runFreezeFault({ kind, rootPid: server.pid!, holdMs, record: timeFault, freezer: new ProcessFreezer(),
|
||||
...(kind === "TM2" ? { age: (onLockTimeout: () => Promise<void>) => this.age(receipt.faultId, onLockTimeout) } : {}),
|
||||
cancelled: () => this.shuttingDown,
|
||||
onFrozen: pids => this.timeline.add("fault_server_frozen", { faultId: receipt.faultId, pids, holdMs }),
|
||||
onThawed: frozenMs => this.timeline.add("fault_server_thawed", { faultId: receipt.faultId, frozenMs }),
|
||||
onFrozen: pids => { this.markFrozen(server.pid, Date.now(), true); this.timeline.add("fault_server_frozen", { faultId: receipt.faultId, pids, holdMs }); },
|
||||
onThawed: frozenMs => { this.markFrozen(server.pid, Date.now(), false); this.timeline.add("fault_server_thawed", { faultId: receipt.faultId, frozenMs }); },
|
||||
waitHealthy: () => this.waitForHealth(server, receipt.phases),
|
||||
exitStatus: () => this.exited(server) ? { code: server.exitCode, signal: server.signalCode } : null,
|
||||
replaceExited: async () => {
|
||||
@@ -532,6 +557,117 @@ export class ChaosSupervisor {
|
||||
receipt.healthyAt = times.healthyAt;
|
||||
}
|
||||
|
||||
/**
|
||||
* MS1–MS3 (multi-server.ts): a second server B on its own port, with the primary's database, instance
|
||||
* directory, storage and secrets. The primary keeps its port, so tasks and agents keep using it.
|
||||
*/
|
||||
private async multiServer(kind: MultiServerKind, primary: ChildProcess, holdMs: number, receipt: FaultReceipt): Promise<void> {
|
||||
const record: MultiServerRecord = { kind, secondary: null, steps: [] };
|
||||
receipt.multiServer = record;
|
||||
await runMultiServerFault(kind, holdMs, record, {
|
||||
now: () => Date.now(),
|
||||
sleep: ms => delay(ms),
|
||||
cancelled: () => this.shuttingDown,
|
||||
secondaryFailure: () => this.secondary?.exit ?? null,
|
||||
startSecondary: async current => {
|
||||
const child = await this.spawnSecondary(current);
|
||||
current.secondary!.readyAt = await this.waitForSecondaryReady(child, current.secondary!.port);
|
||||
this.timeline.add("multi_server_secondary_ready", { faultId: receipt.faultId, pid: child.pid, port: current.secondary!.port });
|
||||
},
|
||||
stopSecondary: async current => {
|
||||
const secondary = this.secondary;
|
||||
if (!secondary) return;
|
||||
// An exited B still has its owned tree cleaned up; a live one gets the graceful stop.
|
||||
try { await (secondary.exit ? this.stopper.crashForRestart(secondary.child) : this.stopper(secondary.child, "SIGTERM")); }
|
||||
finally {
|
||||
this.secondary = null;
|
||||
await new Promise<void>(resolve => secondary.log.end(resolve));
|
||||
if (current.secondary) current.secondary.stoppedAt = Date.now();
|
||||
this.timeline.add("multi_server_secondary_stopped", { faultId: receipt.faultId, pid: secondary.child.pid });
|
||||
}
|
||||
},
|
||||
freezePrimary: async (freezeMs, current) => {
|
||||
// TM2's tree freeze without aging: the primary and its runners stop; the time passes for real.
|
||||
const timeFault: TimeFaultRecord = { kind: "TM2" };
|
||||
current.primaryFreeze = timeFault;
|
||||
const times = await runFreezeFault({ kind: "TM2", rootPid: primary.pid!, holdMs: freezeMs, record: timeFault, freezer: new ProcessFreezer(),
|
||||
cancelled: () => this.shuttingDown,
|
||||
onFrozen: pids => { this.markFrozen(primary.pid, Date.now(), true); this.timeline.add("fault_server_frozen", { faultId: receipt.faultId, pids, holdMs: freezeMs }); },
|
||||
onThawed: frozenMs => { this.markFrozen(primary.pid, Date.now(), false); this.timeline.add("fault_server_thawed", { faultId: receipt.faultId, frozenMs }); },
|
||||
waitHealthy: () => this.waitForHealth(primary, receipt.phases),
|
||||
exitStatus: () => this.exited(primary) ? { code: primary.exitCode, signal: primary.signalCode } : null,
|
||||
replaceExited: async () => {
|
||||
this.timeline.add("time_fault_server_exited_after_thaw", { faultId: receipt.faultId, ...timeFault.serverExitedAfterThaw });
|
||||
try { await this.stopper.crashForRestart(primary); } finally { if (this.child === primary) this.child = null; }
|
||||
},
|
||||
startReplacement: async () => this.waitForHealth(await this.spawnServer(), receipt.phases),
|
||||
});
|
||||
receipt.stoppedAt = times.stoppedAt;
|
||||
receipt.restartedAt = times.restartedAt;
|
||||
receipt.healthyAt = times.healthyAt;
|
||||
},
|
||||
restartPrimary: async current => {
|
||||
const restart: NonNullable<MultiServerRecord["primaryRestart"]> = { stoppedAt: null, restartedAt: null, readyAt: null, replacementPid: null };
|
||||
current.primaryRestart = restart;
|
||||
await this.stopper(primary, "SIGTERM");
|
||||
if (this.child === primary) this.child = null;
|
||||
await this.waitForHealthToStop();
|
||||
receipt.stoppedAt = restart.stoppedAt = Date.now();
|
||||
this.timeline.add("fault_server_down", { faultId: receipt.faultId });
|
||||
receipt.restartedAt = restart.restartedAt = Date.now();
|
||||
const replacement = await this.spawnServer();
|
||||
restart.replacementPid = replacement.pid ?? null;
|
||||
receipt.healthyAt = await this.waitForHealth(replacement, receipt.phases);
|
||||
restart.readyAt = await this.waitForReadyPhase(receipt.phases);
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
private async spawnSecondary(record: MultiServerRecord): Promise<ChildProcess> {
|
||||
if (this.shuttingDown) throw new Error("Refusing to start a second server during harness shutdown");
|
||||
if (this.secondary) throw new Error("A second server is already running");
|
||||
const port = await reserveRunnerE2EServerPort();
|
||||
const log = createWriteStream(path.join(path.dirname(this.paths.serverLogPath), "server-b.log"), { flags: "a", mode: 0o600 });
|
||||
const root = this.serverRoot;
|
||||
const entry = path.join(root, "tests/runner-e2e/server-entry.ts");
|
||||
// Same database, instance directory, storage and secrets as the primary; its own port and API URL.
|
||||
const env = { ...this.environment, PORT: String(port), ...waveServerUrlEnvironment(port) };
|
||||
const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]),
|
||||
{ cwd: root, env, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached });
|
||||
const window = this.trackWindow(child, "secondary");
|
||||
record.secondary = { pid: child.pid ?? null, port, spawnedAt: Date.now(), readyAt: null, stoppedAt: null };
|
||||
const secondary = { child, port, log, exit: null as string | null };
|
||||
this.secondary = secondary;
|
||||
child.stdout?.on("data", chunk => log.write(chunk));
|
||||
child.stderr?.on("data", chunk => log.write(chunk));
|
||||
child.once("error", error => { this.spawnErrors.set(child, error); });
|
||||
child.once("exit", (code, signal) => {
|
||||
if (window) window.exitedAt = Date.now();
|
||||
log.write(`\n[restart-chaos] second server pid=${child.pid} exited code=${code} signal=${signal}\n`);
|
||||
if (!this.expectedStops.has(child) && !this.shuttingDown) {
|
||||
secondary.exit = `code=${code} signal=${signal}`;
|
||||
this.timeline.add("multi_server_secondary_exited", { pid: child.pid, code, signal });
|
||||
}
|
||||
});
|
||||
await this.stopper.watch(child);
|
||||
this.timeline.add("multi_server_secondary_spawned", { pid: child.pid, port });
|
||||
return child;
|
||||
}
|
||||
|
||||
/** B is ready when its own /api/ready says so; it never changes the primary's readiness. */
|
||||
private async waitForSecondaryReady(child: ChildProcess, port: number): Promise<number> {
|
||||
const baseUrl = `http://127.0.0.1:${port}`;
|
||||
await this.waitForHealth(child, undefined, baseUrl);
|
||||
const deadline = Date.now() + READY_TIMEOUT_MS;
|
||||
while (Date.now() < deadline) {
|
||||
if (this.shuttingDown) throw new Error("Harness shutdown interrupted the second server's start");
|
||||
if (this.exited(child)) throw new Error(`Second server exited during startup (code=${child.exitCode} signal=${child.signalCode})`);
|
||||
if (await this.readPhase(baseUrl) === "ready") return Date.now();
|
||||
await delay(250);
|
||||
}
|
||||
throw new Error(`Second server did not report ready within ${READY_TIMEOUT_MS}ms`);
|
||||
}
|
||||
|
||||
/**
|
||||
* K14: move the server from the older build to this checkout. Only the first K14 switches; a later
|
||||
* one restarts the current build (recorded with `switched: false`).
|
||||
|
||||
@@ -8,8 +8,9 @@ import type { TimeFaultRecord } from "./time-faults.js";
|
||||
* K14 (upgrade mid-run): a graceful stop, then the replacement starts from the current build (`--upgrade-from`).
|
||||
* TM1–TM3 (time faults, time-faults.ts): freeze the server, sleep with aged rows, or an aged outage.
|
||||
* RR1–RR3 (repeated restarts, repeated-restarts.ts): a K1, then more restarts while the replacement recovers.
|
||||
* MS1–MS3 (two servers, one database, multi-server.ts): a second server beside the first.
|
||||
*/
|
||||
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3"] as const;
|
||||
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3"] as const;
|
||||
export type FaultKind = (typeof FAULT_KINDS)[number];
|
||||
/** The shortest seeded K4 outage (scheduler.ts K4_HOLD_RANGE_MS), also the K4 hold after a provider fault. */
|
||||
export const K4_MIN_HOLD_MS = 120_000;
|
||||
@@ -256,7 +257,8 @@ export interface FaultReceipt {
|
||||
kind: FaultKind;
|
||||
/** Task that scheduled the fault; every other running task is also exposed. */
|
||||
requestedBy: string;
|
||||
signal: "SIGKILL" | "SIGTERM" | "SIGSTOP";
|
||||
/** "none": MS1 sends no signal to the primary server. */
|
||||
signal: "SIGKILL" | "SIGTERM" | "SIGSTOP" | "none";
|
||||
holdMs: number;
|
||||
requestedAt: number;
|
||||
startedAt: number;
|
||||
@@ -277,6 +279,8 @@ export interface FaultReceipt {
|
||||
timeFault?: TimeFaultRecord;
|
||||
/** RR1–RR3: every stop and replacement of the sequence, the first kill included. */
|
||||
repeatedRestart?: RepeatedRestartRecord;
|
||||
/** MS1–MS3: the second server and what happened to the first. */
|
||||
multiServer?: import("./multi-server.js").MultiServerRecord;
|
||||
serverLogOffsetAtStart: number;
|
||||
serverLogOffsetAtReady: number | null;
|
||||
error: string | null;
|
||||
|
||||
Reference in new issue
Block a user