mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
test(restart-chaos): add the release swap cell RS1 with user-view grading (#21126)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) proves that agent
tasks survive control-plane restarts
> - Staging takes a release about every 40 minutes as an overlap swap: a
new server starts beside the old one on the same database, both run for
8–9 minutes, then the old one gets SIGTERM
> - On 2026-11-12 staging failed or interrupted 577 of 1,000 runs across
these swaps, but the harness restarts one server at a time, grades each
task once and never repeats the swap
> - This pull request adds the release swap cell RS1: repeated overlap
swaps on one database, with the outgoing server starting new runs during
each overlap, and one end-of-wave grade that reads each task the way the
board shows it
> - The benefit is one number per wave, tasks completed without human
action over tasks started, that measures what staging users see
## Linked Issues or Issue Description
**Area**
Restart chaos harness (`tests/restart-chaos/`), test-only.
**Problem**
The harness did not reproduce the staging release shape. It had no
repeated swaps, no run started by the outgoing server during the
overlap, and no grade of every task at the end of a multi-swap wave.
**Proposed solution**
Add the `--release-swap` cell (RS1). It schedules K15 rolling redeploys
on a fixed clock, drives probe tasks through the outgoing server,
records per-swap evidence, and grades every task once after a settle
period.
**Why it matters**
The release cadence will not slow down. The harness must catch the
failures that the swap causes before staging does.
## What Changed
- `release-swap-schedule.ts`: the `--release-swap` options and the
`--release-swap-short` preset (3 swaps, 12 min apart, 510 s overlap)
(count 6, spacing 40 min, overlap 510 s, settle 15 min), their expansion
into a K15 overlap storm (adds `--k15-fresh-home --ephemeral-server` on
Daytona), the swap plan and the `ReleaseSwapRun` clock (swap k at k ×
spacing, never while a swap is unsettled; tasks launch until the last
swap settled).
- `release-swap-driver.ts`: the per-swap driver. It creates probe tasks
through the outgoing server's own port when the incoming server spawns,
when it enters startup recovery, and so that a run starts about 20 s
before the outgoing server's SIGTERM. It samples runs and their owning
controller PID every 2 s and records which server started each run. It
has a per-swap plug-in list; a plug-in can choose the checkout the
incoming server runs (for the later mixed-version variant).
- `release-swap-grade.ts`: the pure grade. One verdict per task (`pass`,
`needed_retry` with whether the click worked, `open_no_owner`,
`blocked_non_person`, `waiting_on_person`, `still_working`), the
per-swap numbers (failed runs, runs lost while the old server was alive,
time from SIGTERM to each surviving run's next event), duplicate runs
and duplicated side effects, the one-number summary, and one failure
label per failing task (`nothing pending`, `blocked for a non-person
reason`, `needed Retry` with the click result).
- `release-swap-finish.ts`: the end-of-wave collection. It waits for the
settle period, reads every task with a run from the API and the wave
database, clicks Retry where the board offers it (board UI when served,
else the same API call as the observer), waits again, and writes
`release-swap.json` and `release-swap-summary.md`.
- Small hooks in shared files: `supervisor.ts` calls `swapHook` at five
phases of the rolling redeploy and lets the hook choose the next server
root (the file stays at 1,000 lines); `config.ts`, `scheduler.ts`,
`types.ts` and `main.ts` wire the option, the plan, the clock, launches
during the overlap and the final grade.
- README section for the cell, its options and its grading.
- Unit tests for the scheduler, the option parsing, the grade
classifier, the per-swap numbers and the overlap evidence.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`
(58 files, 704 tests pass, after merging master).
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- Local smokes through the wave queue (`--env local --adapters opencode
--workloads A1,T1 --release-swap --release-swap-count 2
--release-swap-spacing-minutes 5 --release-swap-overlap-seconds 90
--release-swap-settle-minutes 2 --concurrency 2 --tasks 40`):
- `w-20261112T151735-s7101` (first head): 10 / 16 tasks completed
without human action; 26 min wall. It showed that native runs had no
owner in the sampler (fixed here).
- `w-20261112T154840-s7102` (owner fix): 8 / 13 tasks completed without
human action; 24 min wall. The late probe's run was started by the
outgoing server 20.0 s before its SIGTERM.
- Both smokes caught the same product failure: runs the outgoing server
hands off at shutdown (`restartSuspendedRunIds`) are classified
`awaitingEvidence` by the incoming server's startup recovery, even while
the outgoing server is still alive, and are never settled (`I2_settled`
reconciliation violations). They stay `running` with no process, and the
runs queued behind them are never dispatched. This PR does not fix it
(harness only).
- `w-20261112T163456-s7103` (probe agents, drain): 28 / 29 completed;
the late probe started on the outgoing server 20.0 s and 20.2 s before
SIGTERM in both swaps.
- `w-20261112T171115-s7104`, with 40 preparation probes per swap
overloaded the probe agent (late probe dispatched by the new server 30 s
after SIGTERM); the cap is now 6 per swap.
- CI notes on earlier heads: `ci / e2e shard (1/8)` failed once in
`task-loading-frame` (UI, outside this diff; re-run). `ci / General
tests (server (11/12))` failed in
`src/__tests__/storage-s3-multipart-throttle.test.ts` ("halves new
parts…", expected 3 to be 4), outside this diff; it is a known master
failure being fixed separately and is not touched here.
- Not run: the Daytona smoke. No published runner image matches this
checkout's content ID; an image build for master is queued.
## Risks
- Low risk: test-only harness code. No product code changes.
- Waves without `--release-swap` keep their plans and behavior. The only
shared-path change is the K15 overlap redeploy, which now calls an
optional hook that is unset in other waves.
- The probe tasks and the sampling add a few API calls and one database
query every 2 s during each swap.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits). Extended reasoning on.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b8207c4ea9
commit
cdefac664b
15 files changed
+1905
-18
No files matched your search
@@ -26,7 +26,7 @@ Options: `--triggers time,comment,retry,queued,tool,result` (fault timing, below
|
||||
default 3000), `--poll-ms` (2000–5000), `--keep` (keep the temporary instance),
|
||||
`--k4-hold-max-seconds` (raise the seeded K4 hold upper end from 300 s, up to the 600 s outage cap).
|
||||
`--faults K0` plans cells with no fault, for workloads whose shape is the fault (W2, below). `--fail-recovery-step`,
|
||||
`--long-turn-minutes`, `--k15-overlap-seconds`, `--k15-fresh-home`, `--ephemeral-server`, `--runner-bind-fault` and `--workload staging-mix` are described in their own sections.
|
||||
`--long-turn-minutes`, `--k15-overlap-seconds`, `--k15-fresh-home`, `--ephemeral-server`, `--runner-bind-fault`, `--workload staging-mix` and `--release-swap` are described in their own sections.
|
||||
`--fault-after-seconds <n>` fires every planned fault `n` seconds after its task launches (5–3600; with `--triggers time`), keeping
|
||||
the rest of the seeded plan, for example to restart a Daytona R1 task after its workspace preparation.
|
||||
`--matrix` crosses every selected workload with every fault kind (below), and `--server-from <dir>`
|
||||
@@ -542,6 +542,105 @@ the Runner's listen port for 15 s. The server waits until that listener has boun
|
||||
really took this path. Every run then takes it once per server generation, and the run must still finish without
|
||||
Retry. It needs `--env daytona`.
|
||||
|
||||
## Release swap (RS1, `--release-swap`)
|
||||
|
||||
```sh
|
||||
# Smoke: 2 swaps, 5 minutes apart, 90 s overlap, 2-minute settle.
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters opencode --workloads A1,T1 \
|
||||
--release-swap --release-swap-count 2 --release-swap-spacing-minutes 5 --release-swap-overlap-seconds 90 \
|
||||
--release-swap-settle-minutes 2 --concurrency 2 --tasks 40 --seed 7101
|
||||
# The default wave: 6 swaps, 40 minutes apart, 8.5-minute overlap, 15-minute settle, staging's volume-free shape.
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --storage s3 \
|
||||
--daytona-warm --daytona-cpu 2 --adapters opencode,codex,claude --release-swap --concurrency 12 --tasks 200 --seed 7100
|
||||
# The short form for hourly comparisons of fix PRs: 3 swaps, 12 minutes apart, the same 8.5-minute overlap, 15-minute settle.
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --storage s3 \
|
||||
--daytona-warm --daytona-cpu 2 --adapters opencode,codex,claude --release-swap-short --concurrency 12 --tasks 200 --seed 7110
|
||||
```
|
||||
|
||||
`--release-swap` runs the full default (6 swaps, 40 minutes apart). `--release-swap-short` runs 3 swaps 12 minutes
|
||||
apart with the full 510 s overlap, so two servers run for most of each cycle. Either takes the `--release-swap-*`
|
||||
options below, which override the preset. Workload sizing (turn length, concurrency) comes from the workload flags.
|
||||
|
||||
Staging takes a release about every 40 minutes as an overlap swap: the incoming server B starts with a new
|
||||
deployment identity and process space against the same database and storage while the outgoing server A keeps
|
||||
serving and driving runs; both run for 8–9 minutes; then A gets SIGTERM. RS1 repeats that swap on one database,
|
||||
so state saved during swap N is resumed after swaps N+1 and N+2.
|
||||
|
||||
- **Each swap** is a K15 rolling redeploy (`redeploy.ts`, the edge proxy of `--k15-overlap-seconds`). B boots
|
||||
beside A, the edge sends new connections to B once B reports `ready`, both stay up for the overlap, and A
|
||||
gets a graceful SIGTERM. B becomes A for the next swap. On `--env daytona` the cell adds `--k15-fresh-home`
|
||||
and `--ephemeral-server` (staging's volume-free shape, which needs `--storage s3`); locally both servers
|
||||
share the instance home.
|
||||
- **Schedule** (`release-swap-schedule.ts`): swap *k* starts *k* × spacing after the wave starts, never before
|
||||
the previous one settled. Tasks keep launching from the `--tasks` pool, during the overlap too, until the last
|
||||
swap settled; then they drain. The next swap never starts before the previous outgoing server has exited, and the
|
||||
spacing must leave the overlap plus 2 minutes for the incoming server's boot. The cell plans its own swaps, so it refuses `--faults`, `--storm-minutes`,
|
||||
`--k15-overlap-seconds`, `--k15-hold-seconds`, `--replay`, `--crash-point`, `--matrix` and `--upgrade-from`.
|
||||
- **The outgoing server starts runs** (`release-swap-driver.ts`, `release-swap-setup.ts`). The harness hires one
|
||||
probe agent per adapter, a copy of the seeded agent with its own concurrency (10 locally, 4 on Daytona), so
|
||||
probes never queue behind the workload. It creates small probe tasks (the A1 brief) for it through A's own port,
|
||||
not the edge: one when B is spawned, then one every 3 s locally (20 s on Daytona, at most 6 or 8) and one when B
|
||||
enters its startup recovery pass (`/api/ready` reports `recovering`) for as long as B boots, while the run sampler
|
||||
reads every 500 ms, and one timed so that its run
|
||||
starts about 20 s before A's SIGTERM (the wake goes out 20 s plus the swap's measured wake-to-start latency
|
||||
early). Every 2 s from B's spawn
|
||||
until 60 s after A stopped, the harness samples the wave's runs and their owning controller's PID. Each swap
|
||||
records which runs A and B started during the overlap, whether the late probe's run was started by A within the
|
||||
60 s before its SIGTERM, and whether a run A started was still preparing (`execution_stage` `preparing` or
|
||||
`dispatching`) while B's recovery pass ran. The owner is the run's controller boot (`heartbeat_runs` for legacy
|
||||
runs, `native_run_finalizations` for native runs). The recovery pass is the time B's `/api/ready` reports
|
||||
`recovering`; a swap where that phase was never seen has no recovery evidence. A run the product starts
|
||||
elsewhere is reported as a finding, not forced. A swap whose outgoing server is never stopped (B failed to
|
||||
start, the wave ended, or the harness was interrupted) is marked abandoned, and its sampling stops.
|
||||
- **Plug-ins.** `ReleaseSwapDriver.plugins` is the per-swap hook list. Each plug-in sees every phase
|
||||
(`before_spawn`, `spawned`, `switched`, `stopping`, `stopped`); a plug-in that returns a checkout directory at
|
||||
`before_spawn` makes B (and every later server) run that build, for the mixed-version variant.
|
||||
|
||||
| Option | Default | Range |
|
||||
| --- | --- | --- |
|
||||
| `--release-swap-count` | 6 | 1–24 |
|
||||
| `--release-swap-spacing-minutes` | 40 | 2–120; must exceed the overlap plus 2 minutes |
|
||||
| `--release-swap-overlap-seconds` | 510 | 30–600 |
|
||||
| `--release-swap-settle-minutes` | 15 | 1–60 |
|
||||
| `--release-swap-drain-minutes` | the longest task budget plus the settle | 1–240 |
|
||||
|
||||
**Disk.** On Daytona (`--k15-fresh-home`) each swap removes the outgoing generation's instance home once that
|
||||
server has stopped (the first home holds the database and stays). Every swap records the temporary root's size
|
||||
(`release_swap_disk`). The wave's temporary root is removed when the wave ends; what stays is `results/<wave>`.
|
||||
|
||||
**Grading** (`release-swap-grade.ts`, `release-swap-finish.ts`) runs once, after the last swap plus the settle
|
||||
period and a bounded drain: the grader waits until no started task has a queued or running run with activity in the
|
||||
last 5 minutes, or until the drain cap. A task still running at the cap is not counted in the number; the headline
|
||||
reports how many with the age of their last activity, and the wave fails. The grade uses the same 5-minute threshold:
|
||||
after the drain, a run with no event for 5 minutes, a queued wake, a scheduled retry or a working notice with no
|
||||
progressing run behind it is not pending work, so the task is `nothing pending`. The single read then covers every task of the wave's company that had a run (probes and agent-created subtasks included), the
|
||||
way the board shows it:
|
||||
|
||||
| Verdict | When |
|
||||
| --- | --- |
|
||||
| `pass` | Done, and nobody clicked Retry. Failed runs the system recovered from do not count against it. |
|
||||
| `needed_retry` | Retry was clicked, during the wave by the observer or now by the grader (the board's Retry button when the UI is served, else the same API call). The result says whether the click worked: accepted, then a later run succeeded or the task is done, checked again after another settle period. |
|
||||
| `open_no_owner` | To do or in progress with no live run, no queued wake and no named reason a person must act. |
|
||||
| `blocked_non_person` | Blocked, held by an execution blocker or a board recovery action, or interrupted, for a reason that is not a person's. A board request such as `choose_model`, `assign_agent`, `resume` or `confirm_run_stopped` counts here: the board asks a person, but it is not one of the reasons below. |
|
||||
| `waiting_on_person` | Rests on approval or a question (pending records in the database), a dependency, review, a usage-limit wait, budget or a shared credential. |
|
||||
| `still_working` | Still has a live run, a queued wake, a scheduled retry or a system recovery action after the settle period. |
|
||||
|
||||
Every failing task carries exactly one label, counted per wave on the summary's `Failing:` line: `nothing pending`
|
||||
(`open_no_owner`, or cancelled with no person asking), `blocked for a non-person reason` (`blocked_non_person`), or
|
||||
`needed Retry (click worked | click did not work | click result pending)`. A queued or running run with no run event
|
||||
and no update for the settle period is not a live owner. A task whose view cannot be read in time is left out of the
|
||||
grade, and the headline says how many: the number is then incomplete and the wave fails.
|
||||
|
||||
Only `pass` counts as completed. Each grade names its reason, and the summary counts the non-passing tasks by
|
||||
verdict and reason, so a wave shows which failure each task rests in. The first line of `release-swap-summary.md` (also the top of `SUMMARY.md`) is
|
||||
the wave's one number: `<pass> / <started> tasks completed without human action`. The table under it gives per
|
||||
swap the runs that failed (`failed`, `timed_out`, `interrupted`) between that swap and the next, the runs that
|
||||
failed with a lost-ownership code while the outgoing server was still alive and owned them, and the time from
|
||||
A's SIGTERM to the next event of each run that was live across it and did not fail. `release-swap.json` holds
|
||||
every grade, every failed run with its error code, message and last error events, each swap's run samples and
|
||||
probes, and the duplicates: two runs of one task executing at once, two Runner processes for one run, repeated
|
||||
identical agent comments and repeated subtask titles. The wave fails unless every task passed.
|
||||
|
||||
## Absolute symlink in the workspace (F1)
|
||||
|
||||
```sh
|
||||
|
||||
@@ -16,6 +16,7 @@ import { parseTimeFaultOptions } from "./time-faults-config.js";
|
||||
import { parseStorageTrigger } from "./storage-trigger.js";
|
||||
import { parseNativeTrigger, parseNativeTriggerOutage } from "./native-trigger.js";
|
||||
import { parseRepeatedRestartOptions } from "./repeated-restarts.js";
|
||||
import { expandReleaseSwapArgs } from "./release-swap-schedule.js";
|
||||
import { daytonaRetainsSandboxes } from "./ttft-baseline-stats.js";
|
||||
import { PROVIDER_FAULT_KINDS, PROVIDER_FAULT_WORKLOADS, TOOL_COMPLETED_WORKLOADS, UNINTERRUPTED_PROVIDER_MESSAGES, isProviderFault, parseProviderDelays, parseProviderMessages, type ProviderFaultKind } from "./provider-faults.js";
|
||||
import { SESSION_FAULT_KINDS, SESSION_FAULT_WORKLOADS, isSessionFault, type SessionFaultKind } from "./session-faults.js";
|
||||
@@ -93,7 +94,7 @@ function integer(raw: string | undefined, name: string, fallback: number, min: n
|
||||
|
||||
export function parseCommand(argv: readonly string[], now = Date.now()): ParsedCommand {
|
||||
// Tolerate a forwarded literal `--` (for example from a package runner).
|
||||
const args = argv[0] === "--" ? argv.slice(1) : [...argv];
|
||||
const { args, releaseSwap } = expandReleaseSwapArgs(argv[0] === "--" ? argv.slice(1) : [...argv]);
|
||||
const { values } = parseArgs({
|
||||
args,
|
||||
strict: true,
|
||||
@@ -420,6 +421,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
if (values["storage-trigger"] !== undefined) options.storageTrigger = parseStorageTrigger(values["storage-trigger"], options);
|
||||
if (values["native-trigger"] !== undefined) options.nativeTrigger = parseNativeTrigger(values["native-trigger"], options);
|
||||
if (values["native-trigger-outage"] !== undefined) options.nativeTrigger = { ...options.nativeTrigger!, outageMs: parseNativeTriggerOutage(values["native-trigger-outage"], options) };
|
||||
if (releaseSwap) options.releaseSwap = releaseSwap;
|
||||
return { mode: "wave", options: applyStagingMix(options, values), keep };
|
||||
}
|
||||
|
||||
|
||||
@@ -44,7 +44,7 @@ import { withT1CrossingCheck } from "./workloads.js";
|
||||
import { parseRecoveryStepReceipts, withRecoveryStepCheck } from "./recovery-step-faults.js";
|
||||
import { captureWaveRunnerLogs, executeInWaveSandboxes } from "./daytona-runner-logs.js";
|
||||
import { isClassRFault, killedPidsFromOutput, providerKillCommand, watchLabels, withClassRCheck, type ProviderKillRecord } from "./class-r.js";
|
||||
import { StormRun, TASK_FAULT_LOG, observedFaultSchedule, planWave, replayOnCurrentBuild, replayRecordFor, replayWave, type TaskFaultLog } from "./scheduler.js";
|
||||
import { TASK_FAULT_LOG, observedFaultSchedule, planWave, replayOnCurrentBuild, replayRecordFor, replayWave, type TaskFaultLog } from "./scheduler.js";
|
||||
import { chaosLifecycleOverrides, seedChaosLab } from "./seed.js";
|
||||
import { ChaosSupervisor } from "./supervisor.js";
|
||||
import { planStagingMixWave } from "./staging-mix.js";
|
||||
@@ -54,6 +54,7 @@ import type { RestartStep } from "./repeated-restarts.js";
|
||||
import { isBrokenStdoutError, markTimelineEchoBroken, Timeline } from "./timeline.js";
|
||||
import { guardedRunnerOutputs, worktreeFingerprint } from "./wave-guard.js";
|
||||
import { checkCrossServer, readCrossServerEvidence } from "./multi-server.js";
|
||||
import { ReleaseSwapWave, stormRunFor } from "./release-swap-setup.js";
|
||||
import { runnerBindFaultServerEnvironment, sessionFaultServerEnvironment } from "./session-faults.js";
|
||||
import { controllerLogNames, gradeWarmShutdown, isWarmWave, warmShutdownMarkdown, type WarmShutdownCleanup } from "./warm-shutdown.js";
|
||||
import type { FaultKind, FaultReceipt, OracleArtifacts, TriggerId, WavePlan } from "./types.js";
|
||||
@@ -462,6 +463,7 @@ async function main(): Promise<number> {
|
||||
};
|
||||
} });
|
||||
spendTarget = { api, companyId: lab.companyId };
|
||||
const releaseSwap = plan.options.releaseSwap ? await ReleaseSwapWave.start({ plan, lab, api, timeline, supervisor, databaseUrl: waveDatabaseUrl(), cancelled: () => interrupted }) : null;
|
||||
// --db-pressure: contend with the wave's running run rows from outside the server (contention.ts).
|
||||
if (plan.options.dbPressure) {
|
||||
pressure = new RunRowPressure(waveDatabaseUrl(), lab.companyId, plan.options.dbPressure).start();
|
||||
@@ -479,8 +481,7 @@ async function main(): Promise<number> {
|
||||
const active: TaskObserver[] = [];
|
||||
const lastPoll = new Map<TaskObserver, number>();
|
||||
let lastDoneAt = Date.now();
|
||||
// K13: storm faults fire on their own clock while tasks keep launching; then the storm drains.
|
||||
const storm = plan.storm ? new StormRun(plan.storm, Date.now()) : null;
|
||||
const storm = stormRunFor(plan, Date.now());
|
||||
if (storageProxy && (plan.options.storageWindows?.length || plan.options.storageFaults?.length)) {
|
||||
const windows = plan.options.storageWindows ?? planStorageFaults({ kinds: plan.options.storageFaults!, seed: plan.seed,
|
||||
spanMs: plan.storm?.durationMs ?? plan.tasks.length * 4 * 60_000, ...(plan.options.storageFaultMs ? { durationMs: plan.options.storageFaultMs } : {}) });
|
||||
@@ -651,12 +652,12 @@ async function main(): Promise<number> {
|
||||
}
|
||||
}
|
||||
// Recheck the storm clock per launch: ensureServing or a slow launch can cross the storm's end.
|
||||
while (!crashPoint && active.length < plan.options.concurrency && pending.length && supervisor.serverReady && !supervisor.faultInProgress && !storageTrigger?.busy && !nativeTrigger?.busy
|
||||
while (!crashPoint && active.length < plan.options.concurrency && pending.length && (supervisor.serverReady && !supervisor.faultInProgress || releaseSwap?.driver.launchOpen === true) && !storageTrigger?.busy && !nativeTrigger?.busy
|
||||
&& (!storm || storm.mayLaunch(Date.now())) && (!mix || mix.mayLaunch(Date.now(), pending[0]!.plan))
|
||||
// `--matrix`: no-fault baselines never share the server with a fault cell.
|
||||
&& (!plan.options.matrix || matrixMayLaunch(pending[0]!.plan, active.filter(observer => !observer.done).map(observer => observer.plan)))) {
|
||||
const next = pending.shift()!;
|
||||
launched.push(next);
|
||||
launched.push(next); releaseSwap?.noteLaunch(next);
|
||||
launching = next.launch().finally(() => { launching = null; });
|
||||
await launching;
|
||||
if (plan.options.storageWindows && storageChaos && launched.length === 1) storageChaos.start();
|
||||
@@ -768,6 +769,7 @@ async function main(): Promise<number> {
|
||||
for (const observer of launched) observer.markUnfinished();
|
||||
}
|
||||
|
||||
if (releaseSwap && !(await releaseSwap.finish({ board, observers, launched, waveDir }))) anyFailure = true;
|
||||
// R6: let finalizers settle, then sweep for anything still pending.
|
||||
const quietFor = Date.now() - lastDoneAt;
|
||||
if (quietFor < QUIET_BEFORE_SWEEP_MS) await delay(QUIET_BEFORE_SWEEP_MS - quietFor);
|
||||
@@ -985,7 +987,7 @@ async function main(): Promise<number> {
|
||||
if (lines.length) {
|
||||
await appendLedger(resultsRoot, lines);
|
||||
const summary = await writeSummary(resultsRoot, plan.waveId);
|
||||
await writeFile(path.join(waveDir, "SUMMARY.md"), [summary, warmShutdownSection, speedSection, storageSection].filter(Boolean).join("\n"), { mode: 0o600 });
|
||||
await writeFile(path.join(waveDir, "SUMMARY.md"), [await readFile(path.join(waveDir, "release-swap-summary.md"), "utf8").catch(() => null), summary, warmShutdownSection, speedSection, storageSection].filter(Boolean).join("\n"), { mode: 0o600 });
|
||||
console.info(`\n${summary}`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { ReleaseSwapDriver, hireProbeAgents, homeCleanupPlugin, type SwapPhaseInfo, type SwapRecord, type SwapRunReader } from "./release-swap-driver.js";
|
||||
|
||||
const info = (phase: SwapPhaseInfo["phase"], overrides: Partial<SwapPhaseInfo> = {}): SwapPhaseInfo => ({ faultId: "f01", phase, at: Date.now(),
|
||||
fromUrl: "http://127.0.0.1:9", toUrl: phase === "before_spawn" ? null : "http://127.0.0.1:9", fromPid: 10, toPid: phase === "before_spawn" ? null : 20,
|
||||
overlapMs: 60_000, ...overrides });
|
||||
|
||||
function driver(input: { rows?: Awaited<ReturnType<SwapRunReader["runsSince"]>>; abandonAfterMs?: number; prepIntervalMs?: number; prepProbeCap?: number } = {}) {
|
||||
const events: Array<{ kind: string; data: Record<string, unknown> }> = [];
|
||||
const posts: Array<{ url: string; route: string; body: unknown }> = [];
|
||||
let issue = 0;
|
||||
const swaps = new ReleaseSwapDriver({ options: { swaps: 1, spacingMs: 600_000, overlapMs: 60_000, settleMs: 60_000 }, companyId: "c1",
|
||||
adapters: ["opencode"], agents: { opencode: { id: "agent-1" } }, probeAgents: { opencode: { id: "probe-1" } }, sampleMs: 10, abandonAfterMs: input.abandonAfterMs,
|
||||
prepIntervalMs: input.prepIntervalMs, prepProbeCap: input.prepProbeCap,
|
||||
reader: { runsSince: async () => input.rows ?? [] },
|
||||
record: (kind, data) => { events.push({ kind, data }); },
|
||||
apiFor: url => ({ post: async <T>(route: string, body?: unknown) => { posts.push({ url, route, body }); issue += 1; return { id: `issue-${issue}`, identifier: `CHA-${issue}` } as T; } }) });
|
||||
return { swaps, events, posts };
|
||||
}
|
||||
|
||||
describe("release swap driver", () => {
|
||||
it("creates the first probe through the outgoing server's own port and lets a plug-in choose the incoming build", async () => {
|
||||
const { swaps, posts } = driver();
|
||||
swaps.plugins.push({ name: "mixed", onPhase: async (_swap, phase) => phase.phase === "before_spawn" ? "/checkouts/previous" : undefined });
|
||||
expect(await swaps.hook(info("before_spawn"))).toBe("/checkouts/previous");
|
||||
await swaps.hook(info("spawned"));
|
||||
await swaps.hook(info("stopping"));
|
||||
await swaps.hook(info("stopped", { at: Date.now() - 61_000 }));
|
||||
await swaps.drain();
|
||||
expect(posts[0]).toMatchObject({ url: "http://127.0.0.1:9", route: "/api/companies/c1/issues", body: { assigneeAgentId: "probe-1", status: "todo" } });
|
||||
expect(swaps.swaps[0]).toMatchObject({ serverRoot: "/checkouts/previous", abandoned: null, toPid: 20 });
|
||||
expect(swaps.swaps[0]!.probes[0]).toMatchObject({ kind: "prep", issueId: "issue-1", identifier: "CHA-1" });
|
||||
expect(swaps.swaps[0]!.evidence).not.toBeNull();
|
||||
});
|
||||
|
||||
it("keeps creating preparation probes on the outgoing server while the incoming server boots, up to the cap", async () => {
|
||||
const { swaps, posts } = driver({ prepIntervalMs: 20, prepProbeCap: 4 });
|
||||
await swaps.hook(info("before_spawn"));
|
||||
await swaps.hook(info("spawned"));
|
||||
await new Promise(resolve => setTimeout(resolve, 2_500));
|
||||
await swaps.hook(info("stopping"));
|
||||
await swaps.hook(info("stopped", { at: Date.now() - 61_000 }));
|
||||
await swaps.drain();
|
||||
expect(posts).toHaveLength(4);
|
||||
expect(swaps.swaps[0]!.probes.every(probe => probe.kind === "prep")).toBe(true);
|
||||
});
|
||||
|
||||
it("hires a probe agent per adapter like the seeded one, with its own concurrency, and removes each finished generation's home", async () => {
|
||||
const bodies: any[] = [];
|
||||
const hired = await hireProbeAgents({ post: async <T>(_route: string, body?: unknown) => { bodies.push(body); return { id: "p1" } as T; } }, "c1",
|
||||
{ opencode: { id: "a1", name: "Chaos opencode", body: { adapterType: "x", runtimeConfig: { heartbeat: { enabled: false, maxConcurrentRuns: 2 } } } } }, 10);
|
||||
expect(hired.opencode?.id).toBe("p1");
|
||||
expect(bodies[0]).toMatchObject({ name: "Chaos opencode swap probe", adapterType: "x", runtimeConfig: { heartbeat: { enabled: false, maxConcurrentRuns: 10 } } });
|
||||
const removed: string[] = [];
|
||||
const plugin = homeCleanupPlugin({ temporaryRoot: "/tmp/w", freshHome: true, remove: async directory => { removed.push(directory); },
|
||||
diskUsageKb: async () => 42, record: () => {} });
|
||||
const swap = (index: number) => ({ index }) as SwapRecord;
|
||||
await plugin.onPhase(swap(1), info("stopped"));
|
||||
await plugin.onPhase(swap(2), info("spawned"));
|
||||
await plugin.onPhase(swap(2), info("stopped"));
|
||||
await plugin.onPhase(swap(3), info("stopped"));
|
||||
expect(removed).toEqual(["/tmp/w/home-g1", "/tmp/w/home-g2"]);
|
||||
});
|
||||
|
||||
it("ends a swap whose incoming server never came up, so draining cannot hang", async () => {
|
||||
const { swaps, events } = driver({ abandonAfterMs: 50 });
|
||||
await swaps.hook(info("before_spawn"));
|
||||
await swaps.hook(info("spawned"));
|
||||
await new Promise(resolve => setTimeout(resolve, 300));
|
||||
expect(swaps.swaps[0]!.abandoned).toMatch(/not stopped within/);
|
||||
await swaps.drain();
|
||||
expect(swaps.swaps[0]!.evidence?.findings[0]).toMatch(/^swap abandoned/);
|
||||
expect(swaps.launchOpen).toBe(false);
|
||||
expect(events.some(event => event.kind === "release_swap_abandoned")).toBe(true);
|
||||
});
|
||||
|
||||
it("abandons the open swap when the wave finishes or the harness is interrupted while draining", async () => {
|
||||
const finished = driver();
|
||||
await finished.swaps.hook(info("before_spawn"));
|
||||
await finished.swaps.hook(info("spawned"));
|
||||
await finished.swaps.drain();
|
||||
expect(finished.swaps.swaps[0]!.abandoned).toMatch(/wave finished/);
|
||||
const next = driver();
|
||||
await next.swaps.hook(info("before_spawn"));
|
||||
await next.swaps.hook(info("spawned"));
|
||||
await next.swaps.hook(info("before_spawn", { faultId: "f02" }));
|
||||
expect(next.swaps.swaps[0]!.abandoned).toMatch(/new swap \(f02\)/);
|
||||
await next.swaps.drain(() => true);
|
||||
expect(next.swaps.swaps[1]!.abandoned).not.toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,408 @@
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { ChaosApi } from "./api.js";
|
||||
import { workloadDefinitions } from "./workloads.js";
|
||||
import { RELEASE_SWAP_LATE_LEAD_MS, type ReleaseSwapOptions } from "./release-swap-schedule.js";
|
||||
import type { AdapterId } from "./types.js";
|
||||
import type { Db } from "../../packages/db/src/index.js";
|
||||
|
||||
export type SwapPhase = "before_spawn" | "spawned" | "switched" | "stopping" | "stopped";
|
||||
|
||||
export interface SwapPhaseInfo {
|
||||
faultId: string;
|
||||
phase: SwapPhase;
|
||||
at: number;
|
||||
fromUrl: string;
|
||||
toUrl: string | null;
|
||||
fromPid: number | null;
|
||||
toPid: number | null;
|
||||
overlapMs: number;
|
||||
}
|
||||
|
||||
export interface SwapPlugin {
|
||||
name: string;
|
||||
onPhase(swap: SwapRecord, info: SwapPhaseInfo): Promise<string | void>;
|
||||
}
|
||||
|
||||
export interface ProbeRecord {
|
||||
kind: "prep" | "late";
|
||||
adapter: AdapterId;
|
||||
nonce: string;
|
||||
requestedAt: number;
|
||||
createdAt: number | null;
|
||||
issueId: string | null;
|
||||
identifier: string | null;
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
export interface RunSampleRow {
|
||||
at: number;
|
||||
status: string;
|
||||
ownerPid: number | null;
|
||||
executionStage: string | null;
|
||||
nativePhase: string | null;
|
||||
lastOutputSeq: number;
|
||||
}
|
||||
|
||||
export interface RunTrace {
|
||||
runId: string;
|
||||
issueId: string | null;
|
||||
createdAt: number | null;
|
||||
startedAt: number | null;
|
||||
finishedAt: number | null;
|
||||
errorCode: string | null;
|
||||
samples: RunSampleRow[];
|
||||
}
|
||||
|
||||
export interface SwapEvidence {
|
||||
lateRunId: string | null;
|
||||
lateStartedBy: "old" | "new" | "unknown" | null;
|
||||
lateStartOffsetMs: number | null;
|
||||
lateOk: boolean;
|
||||
prepRunIds: string[];
|
||||
prepOk: boolean;
|
||||
recoveryWindow: { from: number; to: number } | null;
|
||||
startedByOld: string[];
|
||||
startedByNew: string[];
|
||||
findings: string[];
|
||||
}
|
||||
|
||||
export interface SwapRecord {
|
||||
index: number;
|
||||
faultId: string;
|
||||
fromPid: number | null;
|
||||
toPid: number | null;
|
||||
fromUrl: string;
|
||||
toUrl: string | null;
|
||||
serverRoot: string | null;
|
||||
requestedAt: number;
|
||||
spawnedAt: number | null;
|
||||
switchedAt: number | null;
|
||||
overlapEndsAt: number | null;
|
||||
stopRequestedAt: number | null;
|
||||
stoppedAt: number | null;
|
||||
incomingPhases: Array<{ phase: string; at: number }>;
|
||||
probes: ProbeRecord[];
|
||||
runs: Record<string, RunTrace>;
|
||||
sampleErrors: number;
|
||||
plugins: Array<{ name: string; phase: SwapPhase; error: string | null }>;
|
||||
abandoned: string | null;
|
||||
evidence: SwapEvidence | null;
|
||||
}
|
||||
|
||||
export interface SwapRunReader {
|
||||
runsSince(companyId: string, since: number): Promise<Array<Omit<RunTrace, "samples"> & RunSampleRow>>;
|
||||
}
|
||||
|
||||
const PREPARING_STAGES = new Set(["preparing", "dispatching", "dispatch"]);
|
||||
const LATE_WINDOW_MS = 60_000;
|
||||
const SAMPLE_MS = 2_000;
|
||||
const SAMPLE_TAIL_MS = 60_000;
|
||||
const BOOT_SAMPLE_MS = 500;
|
||||
export const SWAP_ABANDON_AFTER_MS = 15 * 60_000;
|
||||
export const PREP_INTERVAL_MS = 2_000;
|
||||
export const PREP_PROBE_CAP = 6;
|
||||
export const PROBE_AGENT_CONCURRENCY = 10;
|
||||
|
||||
export function isPreparing(sample: RunSampleRow): boolean {
|
||||
if (sample.status !== "running") return false;
|
||||
if (sample.executionStage) return PREPARING_STAGES.has(sample.executionStage);
|
||||
return sample.lastOutputSeq === 0;
|
||||
}
|
||||
|
||||
function firstOwner(trace: RunTrace): number | null {
|
||||
return trace.samples.find(sample => sample.ownerPid !== null)?.ownerPid ?? null;
|
||||
}
|
||||
|
||||
function startedBy(trace: RunTrace, swap: Pick<SwapRecord, "fromPid" | "toPid">): "old" | "new" | "unknown" {
|
||||
const owner = firstOwner(trace);
|
||||
if (owner !== null && owner === swap.fromPid) return "old";
|
||||
if (owner !== null && owner === swap.toPid) return "new";
|
||||
return "unknown";
|
||||
}
|
||||
|
||||
export function recoveryWindow(swap: Pick<SwapRecord, "incomingPhases" | "spawnedAt" | "switchedAt">): { from: number; to: number } | null {
|
||||
const recovering = swap.incomingPhases.find(phase => phase.phase === "recovering")?.at ?? null;
|
||||
const ready = swap.incomingPhases.find(phase => phase.phase === "ready")?.at ?? swap.switchedAt;
|
||||
if (recovering === null || ready === null) return null;
|
||||
return { from: recovering, to: Math.max(recovering, ready) };
|
||||
}
|
||||
|
||||
function preparationSpan(trace: RunTrace): { from: number; to: number } | null {
|
||||
const preparing = trace.samples.filter(isPreparing);
|
||||
if (!preparing.length) return null;
|
||||
const from = trace.startedAt ?? preparing[0]!.at;
|
||||
return { from: Math.min(from, preparing[0]!.at), to: preparing[preparing.length - 1]!.at };
|
||||
}
|
||||
|
||||
export function swapEvidence(swap: SwapRecord): SwapEvidence {
|
||||
const traces = Object.values(swap.runs);
|
||||
const findings: string[] = swap.abandoned ? [`swap abandoned: ${swap.abandoned}`] : [];
|
||||
const windowStart = swap.spawnedAt ?? swap.requestedAt;
|
||||
const windowEnd = swap.stopRequestedAt ?? Number.POSITIVE_INFINITY;
|
||||
const startedInOverlap = traces.filter(trace => trace.startedAt !== null && trace.startedAt >= windowStart && trace.startedAt <= windowEnd);
|
||||
const startedByOld = startedInOverlap.filter(trace => startedBy(trace, swap) === "old").map(trace => trace.runId);
|
||||
const startedByNew = startedInOverlap.filter(trace => startedBy(trace, swap) === "new").map(trace => trace.runId);
|
||||
const late = swap.probes.find(probe => probe.kind === "late" && probe.issueId);
|
||||
const lateTrace = late ? traces.filter(trace => trace.issueId === late.issueId && trace.startedAt !== null)
|
||||
.sort((left, right) => left.startedAt! - right.startedAt!)[0] ?? null : null;
|
||||
const lateStartedBy = lateTrace ? startedBy(lateTrace, swap) : null;
|
||||
const lateStartOffsetMs = lateTrace && swap.stopRequestedAt !== null ? lateTrace.startedAt! - swap.stopRequestedAt : null;
|
||||
const lateOk = lateStartedBy === "old" && lateStartOffsetMs !== null && lateStartOffsetMs <= 0 && lateStartOffsetMs >= -LATE_WINDOW_MS;
|
||||
if (!late) findings.push("late probe was not created on the outgoing server");
|
||||
else if (!lateTrace) findings.push(`late probe ${late.identifier ?? late.issueId} had no run started before sampling ended`);
|
||||
else if (lateStartedBy !== "old") findings.push(`late probe run ${lateTrace.runId} was started by the ${lateStartedBy} server`);
|
||||
else if (!lateOk) findings.push(`late probe run ${lateTrace.runId} started ${Math.round((lateStartOffsetMs ?? 0) / 1000)} s from the outgoing server's SIGTERM`);
|
||||
const window = recoveryWindow(swap);
|
||||
const prepRunIds = window ? traces.filter(trace => {
|
||||
if (startedBy(trace, swap) !== "old") return false;
|
||||
const span = preparationSpan(trace);
|
||||
return span !== null && span.from <= window.to && span.to >= window.from;
|
||||
}).map(trace => trace.runId) : [];
|
||||
if (!window) findings.push("the incoming server's startup recovery pass was not observed (no `recovering` phase before `ready`)");
|
||||
else if (!prepRunIds.length) findings.push("no run of the outgoing server was seen in preparation during the incoming server's startup recovery");
|
||||
return { lateRunId: lateTrace?.runId ?? null, lateStartedBy, lateStartOffsetMs, lateOk, prepRunIds, prepOk: prepRunIds.length > 0,
|
||||
recoveryWindow: window, startedByOld, startedByNew, findings };
|
||||
}
|
||||
|
||||
export function mergeRunSample(runs: Record<string, RunTrace>, row: Omit<RunTrace, "samples"> & RunSampleRow): void {
|
||||
const trace = runs[row.runId] ??= { runId: row.runId, issueId: row.issueId, createdAt: row.createdAt, startedAt: row.startedAt,
|
||||
finishedAt: row.finishedAt, errorCode: row.errorCode, samples: [] };
|
||||
Object.assign(trace, { issueId: row.issueId, createdAt: row.createdAt, startedAt: row.startedAt, finishedAt: row.finishedAt, errorCode: row.errorCode });
|
||||
const sample: RunSampleRow = { at: row.at, status: row.status, ownerPid: row.ownerPid, executionStage: row.executionStage,
|
||||
nativePhase: row.nativePhase, lastOutputSeq: row.lastOutputSeq };
|
||||
const last = trace.samples[trace.samples.length - 1];
|
||||
if (last && last.status === sample.status && last.ownerPid === sample.ownerPid && last.executionStage === sample.executionStage
|
||||
&& last.nativePhase === sample.nativePhase && (last.lastOutputSeq > 0) === (sample.lastOutputSeq > 0)) {
|
||||
if (isPreparing(sample)) trace.samples.push(sample);
|
||||
return;
|
||||
}
|
||||
trace.samples.push(sample);
|
||||
}
|
||||
|
||||
export interface ReleaseSwapDriverInput {
|
||||
options: ReleaseSwapOptions;
|
||||
companyId: string;
|
||||
adapters: readonly AdapterId[];
|
||||
agents: Partial<Record<AdapterId, { id: string }>>;
|
||||
reader: SwapRunReader;
|
||||
record: (kind: string, data: Record<string, unknown>) => void;
|
||||
apiFor?: (baseUrl: string) => Pick<ChaosApi, "post">;
|
||||
sampleMs?: number;
|
||||
abandonAfterMs?: number;
|
||||
probeAgents?: Partial<Record<AdapterId, { id: string }>>;
|
||||
prepIntervalMs?: number;
|
||||
prepProbeCap?: number;
|
||||
}
|
||||
|
||||
export class ReleaseSwapDriver {
|
||||
readonly swaps: SwapRecord[] = [];
|
||||
readonly plugins: SwapPlugin[] = [];
|
||||
launchOpen = false;
|
||||
private current: SwapRecord | null = null;
|
||||
private background: Promise<unknown>[] = [];
|
||||
private probeCount = 0;
|
||||
private closed = false;
|
||||
|
||||
constructor(private readonly input: ReleaseSwapDriverInput) {}
|
||||
|
||||
readonly hook = async (info: SwapPhaseInfo): Promise<string | void> => {
|
||||
let swap = this.current;
|
||||
if (info.phase === "before_spawn" || !swap) {
|
||||
if (swap) this.abandon(swap, `a new swap (${info.faultId}) began before this one stopped the outgoing server`);
|
||||
swap = { index: this.swaps.length + 1, faultId: info.faultId, fromPid: info.fromPid, toPid: null, fromUrl: info.fromUrl, toUrl: null,
|
||||
serverRoot: null, requestedAt: info.at, spawnedAt: null, switchedAt: null, overlapEndsAt: null, stopRequestedAt: null, stoppedAt: null,
|
||||
incomingPhases: [], probes: [], runs: {}, sampleErrors: 0, plugins: [], abandoned: null, evidence: null };
|
||||
this.swaps.push(swap);
|
||||
this.current = swap;
|
||||
}
|
||||
swap.toUrl = info.toUrl ?? swap.toUrl;
|
||||
swap.toPid = info.toPid ?? swap.toPid;
|
||||
let root: string | void = undefined;
|
||||
for (const plugin of this.plugins) {
|
||||
try {
|
||||
const chosen = await plugin.onPhase(swap, info);
|
||||
if (typeof chosen === "string") root = chosen;
|
||||
swap.plugins.push({ name: plugin.name, phase: info.phase, error: null });
|
||||
} catch (error) {
|
||||
swap.plugins.push({ name: plugin.name, phase: info.phase, error: error instanceof Error ? error.message : String(error) });
|
||||
}
|
||||
}
|
||||
if (typeof root === "string") swap.serverRoot = root;
|
||||
this.input.record("release_swap_phase", { swap: swap.index, faultId: info.faultId, phase: info.phase, fromPid: info.fromPid, toPid: info.toPid,
|
||||
...(root ? { serverRoot: root } : {}) });
|
||||
if (info.phase === "spawned") this.onSpawned(swap, info);
|
||||
if (info.phase === "switched") this.onSwitched(swap, info);
|
||||
if (info.phase === "stopping") { swap.stopRequestedAt = info.at; this.launchOpen = false; }
|
||||
if (info.phase === "stopped") { swap.stoppedAt = info.at; this.current = null; }
|
||||
return root;
|
||||
};
|
||||
|
||||
private onSpawned(swap: SwapRecord, info: SwapPhaseInfo): void {
|
||||
swap.spawnedAt = info.at;
|
||||
this.background.push(this.probe(swap, "prep"));
|
||||
this.background.push(this.watchIncoming(swap));
|
||||
this.background.push(this.sample(swap));
|
||||
}
|
||||
|
||||
private onSwitched(swap: SwapRecord, info: SwapPhaseInfo): void {
|
||||
swap.switchedAt = info.at;
|
||||
swap.overlapEndsAt = info.at + info.overlapMs;
|
||||
this.launchOpen = true;
|
||||
this.background.push((async () => {
|
||||
const latency = this.startLatency(swap) ?? 5_000;
|
||||
for (let wait = swap.overlapEndsAt! - RELEASE_SWAP_LATE_LEAD_MS - latency - Date.now(); wait > 0 && !this.closed && swap.abandoned === null;
|
||||
wait = swap.overlapEndsAt! - RELEASE_SWAP_LATE_LEAD_MS - latency - Date.now()) await delay(Math.min(1_000, wait));
|
||||
if (swap.stopRequestedAt === null && swap.abandoned === null) await this.probe(swap, "late");
|
||||
})());
|
||||
}
|
||||
|
||||
private startLatency(swap: SwapRecord): number | null {
|
||||
const latencies = swap.probes.flatMap(probe => {
|
||||
const trace = Object.values(swap.runs).find(candidate => candidate.issueId === probe.issueId && candidate.startedAt !== null);
|
||||
return trace && probe.createdAt !== null ? [Math.max(0, trace.startedAt! - probe.createdAt)] : [];
|
||||
});
|
||||
return latencies.length ? latencies.sort((left, right) => left - right)[Math.floor(latencies.length / 2)]! : null;
|
||||
}
|
||||
|
||||
private ended(swap: SwapRecord): boolean {
|
||||
if (swap.abandoned === null && swap.stoppedAt === null && Date.now() > swap.requestedAt + (this.input.abandonAfterMs ?? this.input.options.overlapMs + SWAP_ABANDON_AFTER_MS)) {
|
||||
this.abandon(swap, "the outgoing server was not stopped within the swap's time bound (the incoming server failed or the harness stopped)");
|
||||
}
|
||||
return swap.abandoned !== null || (this.closed && (swap.stoppedAt === null || Date.now() > swap.stoppedAt + SAMPLE_TAIL_MS));
|
||||
}
|
||||
|
||||
abandon(swap: SwapRecord, reason: string): void {
|
||||
if (swap.abandoned !== null || swap.stoppedAt !== null) return;
|
||||
swap.abandoned = reason;
|
||||
if (this.current === swap) { this.current = null; this.launchOpen = false; }
|
||||
this.input.record("release_swap_abandoned", { swap: swap.index, faultId: swap.faultId, reason });
|
||||
}
|
||||
|
||||
private async watchIncoming(swap: SwapRecord): Promise<void> {
|
||||
let prepProbes = 1;
|
||||
let lastProbeAt = Date.now();
|
||||
const interval = this.input.prepIntervalMs ?? PREP_INTERVAL_MS;
|
||||
const cap = this.input.prepProbeCap ?? PREP_PROBE_CAP;
|
||||
while (swap.toUrl && swap.stopRequestedAt === null && !this.ended(swap)) {
|
||||
const phase = await fetch(`${swap.toUrl}/api/ready`, { signal: AbortSignal.timeout(1_000), cache: "no-store" })
|
||||
.then(async response => String(((await response.json()) as { status?: string }).status ?? response.status)).catch(() => "unreachable");
|
||||
const last = swap.incomingPhases[swap.incomingPhases.length - 1]?.phase;
|
||||
if (phase !== last) {
|
||||
swap.incomingPhases.push({ phase, at: Date.now() });
|
||||
if (phase === "recovering" && prepProbes < cap) { prepProbes += 1; lastProbeAt = Date.now(); this.background.push(this.probe(swap, "prep")); }
|
||||
}
|
||||
if (phase === "ready") return;
|
||||
if (prepProbes < cap && Date.now() - lastProbeAt >= interval) { prepProbes += 1; lastProbeAt = Date.now(); this.background.push(this.probe(swap, "prep")); }
|
||||
await delay(250);
|
||||
}
|
||||
}
|
||||
|
||||
private async sample(swap: SwapRecord): Promise<void> {
|
||||
const since = swap.requestedAt;
|
||||
for (;;) {
|
||||
try {
|
||||
const rows = await this.input.reader.runsSince(this.input.companyId, since);
|
||||
for (const row of rows) mergeRunSample(swap.runs, row);
|
||||
} catch {
|
||||
swap.sampleErrors += 1;
|
||||
}
|
||||
if (this.ended(swap) || (swap.stoppedAt !== null && Date.now() > swap.stoppedAt + SAMPLE_TAIL_MS)) break;
|
||||
const booting = swap.switchedAt === null && !swap.incomingPhases.some(phase => phase.phase === "ready");
|
||||
await delay(this.input.sampleMs ?? (booting ? BOOT_SAMPLE_MS : SAMPLE_MS));
|
||||
}
|
||||
swap.evidence = swapEvidence(swap);
|
||||
this.input.record("release_swap_evidence", { swap: swap.index, faultId: swap.faultId, lateOk: swap.evidence.lateOk, prepOk: swap.evidence.prepOk,
|
||||
lateRunId: swap.evidence.lateRunId, lateStartOffsetMs: swap.evidence.lateStartOffsetMs, prepRunIds: swap.evidence.prepRunIds,
|
||||
startedByOld: swap.evidence.startedByOld.length, startedByNew: swap.evidence.startedByNew.length, findings: swap.evidence.findings });
|
||||
}
|
||||
|
||||
private async probe(swap: SwapRecord, kind: ProbeRecord["kind"]): Promise<void> {
|
||||
const agents = this.input.probeAgents ?? this.input.agents;
|
||||
const adapters = this.input.adapters.filter(adapter => agents[adapter]);
|
||||
const adapter = adapters[this.probeCount % Math.max(1, adapters.length)]!;
|
||||
this.probeCount += 1;
|
||||
const nonce = randomBytes(4).toString("hex");
|
||||
const probe: ProbeRecord = { kind, adapter, nonce, requestedAt: Date.now(), createdAt: null, issueId: null, identifier: null, error: null };
|
||||
swap.probes.push(probe);
|
||||
const definition = workloadDefinitions.A1;
|
||||
const api = this.input.apiFor?.(swap.fromUrl) ?? new ChaosApi(swap.fromUrl);
|
||||
try {
|
||||
const issue = await api.post<{ id: string; identifier?: string | null }>(`/api/companies/${this.input.companyId}/issues`, {
|
||||
title: `Release swap ${swap.index} ${kind} probe ${nonce}`, description: definition.description(nonce),
|
||||
status: "todo", priority: "medium", assigneeAgentId: agents[adapter]!.id });
|
||||
probe.issueId = String(issue.id);
|
||||
probe.identifier = issue.identifier ?? null;
|
||||
probe.createdAt = Date.now();
|
||||
} catch (error) {
|
||||
probe.error = error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
this.input.record("release_swap_probe", { swap: swap.index, kind, adapter, issueId: probe.issueId, identifier: probe.identifier, error: probe.error });
|
||||
}
|
||||
|
||||
async drain(cancelled: () => boolean = () => false): Promise<void> {
|
||||
this.closed = true;
|
||||
if (this.current) this.abandonIfStalled(this.current);
|
||||
const settled = Promise.allSettled(this.background);
|
||||
while (!cancelled()) {
|
||||
const done = await Promise.race([settled.then(() => true), delay(1_000).then(() => false)]);
|
||||
if (done) return;
|
||||
}
|
||||
for (const swap of this.swaps) this.abandon(swap, "the harness was interrupted");
|
||||
}
|
||||
|
||||
private abandonIfStalled(swap: SwapRecord): void {
|
||||
if (swap.stopRequestedAt === null) this.abandon(swap, "the wave finished before the swap stopped the outgoing server");
|
||||
}
|
||||
}
|
||||
|
||||
export const RUN_OWNER_BOOT_ID = "coalesce(r.controller_boot_id::text, f.controller_boot_id, f.lease_holder->>'bootId')";
|
||||
|
||||
export function pgRunReader(db: Db): SwapRunReader {
|
||||
const ms = (value: unknown) => value === null || value === undefined ? null : Number(value);
|
||||
return {
|
||||
async runsSince(companyId, since) {
|
||||
const rows = await db.$client.unsafe(
|
||||
`select r.id::text as "runId", r.issue_id::text as "issueId", r.status, r.error_code as "errorCode",
|
||||
floor(extract(epoch from r.created_at) * 1000) as "createdAt", floor(extract(epoch from r.started_at) * 1000) as "startedAt",
|
||||
floor(extract(epoch from r.finished_at) * 1000) as "finishedAt", ci.pid as "ownerPid", r.execution_stage as "executionStage",
|
||||
r.native_phase as "nativePhase", r.last_output_seq as "lastOutputSeq", floor(extract(epoch from clock_timestamp()) * 1000) as "at"
|
||||
from heartbeat_runs r left join native_run_finalizations f on f.run_id = r.id
|
||||
left join controller_instances ci on ci.boot_id = ${RUN_OWNER_BOOT_ID}
|
||||
where r.company_id::text = $1 and (r.created_at >= to_timestamp($2::double precision / 1000) or r.status in ('queued', 'running'))`,
|
||||
[companyId, since]) as Array<Record<string, unknown>>;
|
||||
return rows.map(row => ({ runId: String(row.runId), issueId: row.issueId === null ? null : String(row.issueId), status: String(row.status),
|
||||
errorCode: row.errorCode === null ? null : String(row.errorCode), createdAt: ms(row.createdAt), startedAt: ms(row.startedAt),
|
||||
finishedAt: ms(row.finishedAt), ownerPid: ms(row.ownerPid), executionStage: row.executionStage === null ? null : String(row.executionStage),
|
||||
nativePhase: row.nativePhase === null ? null : String(row.nativePhase), lastOutputSeq: Number(row.lastOutputSeq ?? 0), at: Number(row.at) }));
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export async function hireProbeAgents(api: Pick<ChaosApi, "post">, companyId: string,
|
||||
agents: Partial<Record<AdapterId, { id: string; name: string; body?: Record<string, unknown> }>>, maxConcurrentRuns: number):
|
||||
Promise<Partial<Record<AdapterId, { id: string; createdAt: number }>>> {
|
||||
const hired: Partial<Record<AdapterId, { id: string; createdAt: number }>> = {};
|
||||
for (const [adapter, source] of Object.entries(agents) as Array<[AdapterId, { id: string; name: string; body?: Record<string, unknown> }]>) {
|
||||
if (!source.body) continue;
|
||||
const runtimeConfig = (source.body.runtimeConfig ?? {}) as Record<string, unknown>;
|
||||
const created = await api.post<{ id: string }>(`/api/companies/${companyId}/agents`, { ...source.body, name: `${source.name} swap probe`,
|
||||
runtimeConfig: { ...runtimeConfig, heartbeat: { ...((runtimeConfig.heartbeat ?? {}) as Record<string, unknown>), maxConcurrentRuns } } });
|
||||
hired[adapter] = { id: String(created.id), createdAt: Date.now() };
|
||||
}
|
||||
return hired;
|
||||
}
|
||||
|
||||
export function homeCleanupPlugin(input: { temporaryRoot: string; freshHome: boolean; remove: (directory: string) => Promise<void>;
|
||||
diskUsageKb: (directory: string) => Promise<number | null>; record: (kind: string, data: Record<string, unknown>) => void }): SwapPlugin {
|
||||
return {
|
||||
name: "home-cleanup",
|
||||
async onPhase(swap, info) {
|
||||
if (info.phase !== "stopped") return;
|
||||
const generation = swap.index - 1;
|
||||
const removed = input.freshHome && generation >= 1 ? `${input.temporaryRoot}/home-g${generation}` : null;
|
||||
if (removed) await input.remove(removed);
|
||||
input.record("release_swap_disk", { swap: swap.index, removed, temporaryRootKb: await input.diskUsageKb(input.temporaryRoot) });
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { finishReleaseSwap, type ReleaseSwapFinishInput } from "./release-swap-finish.js";
|
||||
|
||||
const dirs: string[] = [];
|
||||
afterEach(async () => { for (const dir of dirs.splice(0)) await rm(dir, { recursive: true, force: true }); });
|
||||
|
||||
function fakeDb(tables: { issues: Array<Record<string, unknown>>; runs: () => Array<Record<string, unknown>>; questions?: string[] }) {
|
||||
return { $client: { unsafe: async (sql: string) => {
|
||||
if (sql.includes("from issues i")) return tables.issues;
|
||||
if (sql.includes("issue_thread_interactions")) return (tables.questions ?? []).map(issueId => ({ issueId }));
|
||||
if (sql.includes("issue_approvals")) return [];
|
||||
if (sql.includes("agent_wakeup_requests")) return [];
|
||||
if (sql.includes("from heartbeat_runs r") && sql.includes("left(r.error")) return tables.runs();
|
||||
return [];
|
||||
} } } as unknown as ReleaseSwapFinishInput["db"];
|
||||
}
|
||||
|
||||
async function finish(overrides: Partial<ReleaseSwapFinishInput>) {
|
||||
const waveDir = await mkdtemp(path.join(os.tmpdir(), "release-swap-finish-"));
|
||||
dirs.push(waveDir);
|
||||
const events: string[] = [];
|
||||
const result = await finishReleaseSwap({ driver: { swaps: [], drain: async () => {} }, settleMs: 200, companyId: "c1",
|
||||
api: { get: async () => ({}), post: async () => ({}) } as unknown as ReleaseSwapFinishInput["api"], db: fakeDb({ issues: [], runs: () => [] }),
|
||||
board: null, timeline: [], issueIdForTask: new Map(), serverWindows: [], waveDir, cancelled: () => false,
|
||||
record: kind => { events.push(kind); }, pollMs: 10, ...overrides });
|
||||
return { result, events, json: JSON.parse(await readFile(path.join(waveDir, "release-swap.json"), "utf8")) };
|
||||
}
|
||||
|
||||
describe("release swap end-of-wave grade", () => {
|
||||
it("clicks Retry where the board offers it and grades the task as needing Retry, with the click working", async () => {
|
||||
let retried = false;
|
||||
const posts: Array<{ route: string; body: any }> = [];
|
||||
const api = {
|
||||
get: async (route: string) => route === "/api/issues/i2"
|
||||
? (retried ? { status: "done" } : { status: "todo", interruption: { state: "failed", canRetry: true, observedRunId: "r2", responsibility: "user" } })
|
||||
: { status: "done" },
|
||||
post: async (route: string, body: any) => { posts.push({ route, body }); retried = true; return { accepted: true }; },
|
||||
} as unknown as ReleaseSwapFinishInput["api"];
|
||||
const runs = () => [
|
||||
{ runId: "r1", issueId: "i1", status: "succeeded", createdAt: 1, startedAt: 2, finishedAt: 3, ownerPid: 10 },
|
||||
{ runId: "r2", issueId: "i2", status: "failed", errorCode: "process_lost", createdAt: 1, startedAt: 2, finishedAt: 3, ownerPid: 10 },
|
||||
...(retried ? [{ runId: "r3", issueId: "i2", status: "succeeded", createdAt: Date.now(), startedAt: Date.now(), finishedAt: Date.now(), ownerPid: 20 }] : []),
|
||||
];
|
||||
const { result, events, json } = await finish({ api, db: fakeDb({ issues: [{ id: "i1", identifier: "CHA-1", title: "a" }, { id: "i2", identifier: "CHA-2", title: "b" }], runs }) });
|
||||
expect(posts).toEqual([{ route: "/api/issues/i2/retry", body: { idempotencyKey: expect.any(String), observedRunId: "r2" } }]);
|
||||
expect(result.headline).toBe("1 / 2 tasks completed without human action");
|
||||
expect(result.pass).toBe(false);
|
||||
expect(json.result.retry).toEqual({ tasks: 1, worked: 1, didNotWork: 0, pending: 0 });
|
||||
expect(json.grades.find((grade: any) => grade.issueId === "i2")).toMatchObject({ verdict: "needed_retry", status: "done" });
|
||||
expect(json.failedRuns[0]).toMatchObject({ runId: "r2", errorCode: "process_lost" });
|
||||
expect(events).toContain("release_swap_retry_clicked");
|
||||
});
|
||||
|
||||
it("leaves a task whose view read failed out of the grade and marks the number incomplete", async () => {
|
||||
const api = { get: async (route: string) => { if (route === "/api/issues/u1") throw new Error("GET /api/issues/u1 unreachable: timeout"); return { status: "done" }; },
|
||||
post: async () => ({}) } as unknown as ReleaseSwapFinishInput["api"];
|
||||
const { result, events, json } = await finish({ api, db: fakeDb({ issues: [{ id: "d1", identifier: "CHA-1", title: "d" }, { id: "u1", identifier: "CHA-2", title: "u" }],
|
||||
runs: () => [{ runId: "r1", issueId: "d1", status: "succeeded" }, { runId: "r2", issueId: "u1", status: "succeeded" }] }) });
|
||||
expect(result).toMatchObject({ pass: false, headline: "1 / 1 tasks completed without human action (incomplete: 1 task(s) could not be read and are not counted)" });
|
||||
expect(json.grades.map((grade: any) => grade.issueId)).toEqual(["d1"]);
|
||||
expect(events).toContain("release_swap_views_unreadable");
|
||||
});
|
||||
|
||||
it("waits out runs still making progress after the settle, up to the drain cap, before the single read", async () => {
|
||||
let reads = 0;
|
||||
const db = { $client: { unsafe: async (sql: string) => {
|
||||
if (sql.includes("from issues i")) return [{ id: "d1", identifier: "CHA-1", title: "d" }];
|
||||
if (sql.includes("left(r.error")) { reads += 1; return [{ runId: "r1", issueId: "d1", status: reads < 3 ? "running" : "succeeded", lastActivityAt: Date.now() }]; }
|
||||
return [];
|
||||
} } } as unknown as ReleaseSwapFinishInput["db"];
|
||||
const quiet = await finish({ db, drainCapMs: 60_000, api: { get: async () => ({ status: "done" }), post: async () => ({}) } as unknown as ReleaseSwapFinishInput["api"] });
|
||||
expect(reads).toBeGreaterThanOrEqual(3);
|
||||
expect(quiet.result.headline).toBe("1 / 1 tasks completed without human action");
|
||||
const busy = { $client: { unsafe: async (sql: string) => {
|
||||
if (sql.includes("from issues i")) return [{ id: "d1", identifier: "CHA-1", title: "d" }];
|
||||
if (sql.includes("left(r.error")) return [{ runId: "r1", issueId: "d1", status: "running", createdAt: 0, lastActivityAt: Date.now() }];
|
||||
return [];
|
||||
} } } as unknown as ReleaseSwapFinishInput["db"];
|
||||
const started = Date.now();
|
||||
const capped = await finish({ db: busy, drainCapMs: 400, api: { get: async () => ({ status: "in_progress" }), post: async () => ({}) } as unknown as ReleaseSwapFinishInput["api"] });
|
||||
expect(Date.now() - started).toBeLessThan(5_000);
|
||||
expect(capped.events).toContain("release_swap_drained");
|
||||
expect(capped.result.headline).toMatch(/^0 \/ 0 tasks completed without human action; 1 still running at the drain cap/);
|
||||
expect(capped.result.pass).toBe(false);
|
||||
});
|
||||
|
||||
it("grades a pending question from the database as a person's wait, and passes a clean wave", async () => {
|
||||
const { json } = await finish({ api: { get: async () => ({ status: "blocked" }), post: async () => ({}) } as unknown as ReleaseSwapFinishInput["api"],
|
||||
db: fakeDb({ issues: [{ id: "q1", identifier: "CHA-3", title: "q" }], runs: () => [{ runId: "r1", issueId: "q1", status: "succeeded" }], questions: ["q1"] }) });
|
||||
expect(json.grades[0]).toMatchObject({ verdict: "waiting_on_person", personReason: "question" });
|
||||
const clean = await finish({ api: { get: async () => ({ status: "done" }), post: async () => ({}) } as unknown as ReleaseSwapFinishInput["api"],
|
||||
db: fakeDb({ issues: [{ id: "d1", identifier: "CHA-4", title: "d" }], runs: () => [{ runId: "r1", issueId: "d1", status: "succeeded" }] }) });
|
||||
expect(clean.result).toMatchObject({ headline: "1 / 1 tasks completed without human action", pass: true });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,271 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import type { ChaosApi } from "./api.js";
|
||||
import { checkCrossServer, readCrossServerEvidence, type ServerProcessWindow } from "./multi-server.js";
|
||||
import { RUN_OWNER_BOOT_ID, type ReleaseSwapDriver } from "./release-swap-driver.js";
|
||||
import {
|
||||
gradeReleaseSwapTask, overlappingRuns, releaseSwapMarkdown, summarizeGrades, swapMetrics,
|
||||
type DuplicateFacts, type GradeTaskInput, type RetryClick, type SwapRunFact, type TaskGrade,
|
||||
} from "./release-swap-grade.js";
|
||||
import type { TimelineEntry } from "./timeline.js";
|
||||
import type { Db } from "../../packages/db/src/index.js";
|
||||
|
||||
type Row = Record<string, unknown>;
|
||||
type Json = Record<string, any>;
|
||||
|
||||
export interface ReleaseSwapFinishInput {
|
||||
driver: Pick<ReleaseSwapDriver, "drain" | "swaps">;
|
||||
settleMs: number;
|
||||
companyId: string;
|
||||
api: Pick<ChaosApi, "get" | "post">;
|
||||
db: Pick<Db, "$client">;
|
||||
board: { clickRetry(identifier: string, issueId: string): Promise<{ accepted: boolean | null; httpStatus: number | null; error: string | null }> } | null;
|
||||
timeline: readonly TimelineEntry[];
|
||||
issueIdForTask: ReadonlyMap<string, string>;
|
||||
serverWindows: readonly ServerProcessWindow[];
|
||||
waveDir: string;
|
||||
cancelled: () => boolean;
|
||||
record: (kind: string, data: Record<string, unknown>) => void;
|
||||
pollMs?: number;
|
||||
drainCapMs?: number;
|
||||
}
|
||||
|
||||
type Client = Pick<Db, "$client">;
|
||||
const VIEW_READERS = 8;
|
||||
const VIEW_DEADLINE_MS = 120_000;
|
||||
|
||||
const num = (value: unknown) => value === null || value === undefined ? null : Number(value);
|
||||
const text = (value: unknown) => value === null || value === undefined ? null : String(value);
|
||||
|
||||
async function waitUntil(at: number, cancelled: () => boolean): Promise<void> {
|
||||
while (!cancelled() && Date.now() < at) await delay(Math.min(5_000, at - Date.now()));
|
||||
}
|
||||
|
||||
export const PROGRESS_WINDOW_MS = 5 * 60_000;
|
||||
export const progressWindowMs = (settleMs: number) => Math.min(settleMs, PROGRESS_WINDOW_MS);
|
||||
|
||||
async function drainRuns(input: ReleaseSwapFinishInput, lastStop: number): Promise<{ reason: "quiet" | "cap" | "cancelled"; at: string; progressing: number }> {
|
||||
const settledAt = lastStop + input.settleMs;
|
||||
const capAt = Math.max(settledAt, lastStop + (input.drainCapMs ?? input.settleMs));
|
||||
const window = progressWindowMs(input.settleMs);
|
||||
input.record("release_swap_settling", { settleUntil: new Date(settledAt).toISOString(), capAt: new Date(capAt).toISOString() });
|
||||
await waitUntil(settledAt, input.cancelled);
|
||||
for (;;) {
|
||||
const now = Date.now();
|
||||
const progressing = (await companyRuns(input.db, input.companyId)).filter(run => ["queued", "running", "scheduled_retry"].includes(String(run.status))
|
||||
&& (num(run.lastActivityAt) ?? 0) >= now - window).length;
|
||||
if (input.cancelled()) return { reason: "cancelled", at: new Date(now).toISOString(), progressing };
|
||||
if (!progressing) return { reason: "quiet", at: new Date(now).toISOString(), progressing };
|
||||
if (now >= capAt) return { reason: "cap", at: new Date(now).toISOString(), progressing };
|
||||
await delay(Math.min(input.pollMs ?? 30_000, Math.max(1, capAt - now)));
|
||||
}
|
||||
}
|
||||
|
||||
async function companyRuns(db: Client, companyId: string): Promise<Row[]> {
|
||||
return await db.$client.unsafe(
|
||||
`select r.id::text as "runId", r.issue_id::text as "issueId", r.status, r.error_code as "errorCode", left(r.error, 400) as error,
|
||||
floor(extract(epoch from r.created_at) * 1000) as "createdAt", floor(extract(epoch from r.started_at) * 1000) as "startedAt",
|
||||
floor(extract(epoch from r.finished_at) * 1000) as "finishedAt", ci.pid as "ownerPid",
|
||||
floor(extract(epoch from coalesce((select max(e.created_at) from heartbeat_run_events e where e.run_id = r.id), r.started_at, r.created_at)) * 1000) as "lastActivityAt"
|
||||
from heartbeat_runs r left join native_run_finalizations f on f.run_id = r.id
|
||||
left join controller_instances ci on ci.boot_id = ${RUN_OWNER_BOOT_ID}
|
||||
where r.company_id::text = $1`, [companyId]) as Row[];
|
||||
}
|
||||
|
||||
async function startedIssues(db: Client, companyId: string): Promise<Row[]> {
|
||||
return await db.$client.unsafe(
|
||||
`select i.id::text as id, i.identifier, i.title from issues i
|
||||
where i.company_id::text = $1 and i.hidden_at is null and exists (select 1 from heartbeat_runs r where r.issue_id = i.id)
|
||||
order by i.created_at`, [companyId]) as Row[];
|
||||
}
|
||||
|
||||
async function pendingPersonItems(db: Client, companyId: string): Promise<{ questions: Set<string>; approvals: Set<string> }> {
|
||||
const questions = await db.$client.unsafe(
|
||||
`select distinct issue_id::text as "issueId" from issue_thread_interactions where company_id::text = $1 and status = 'pending'`, [companyId]) as Row[];
|
||||
const approvals = await db.$client.unsafe(
|
||||
`select distinct ia.issue_id::text as "issueId" from issue_approvals ia join approvals a on a.id = ia.approval_id
|
||||
where ia.company_id::text = $1 and a.status in ('pending', 'revision_requested')`, [companyId]) as Row[];
|
||||
return { questions: new Set(questions.map(row => String(row.issueId))), approvals: new Set(approvals.map(row => String(row.issueId))) };
|
||||
}
|
||||
|
||||
async function queuedWakes(db: Client, companyId: string): Promise<Map<string, number>> {
|
||||
const rows = await db.$client.unsafe(
|
||||
`select payload->>'issueId' as "issueId", count(*)::int as count from agent_wakeup_requests
|
||||
where company_id::text = $1 and status in ('queued', 'claimed', 'deferred_issue_execution', 'deferred_comment_wake')
|
||||
and not (status = 'claimed' and run_id is not null)
|
||||
group by payload->>'issueId'`, [companyId]) as Row[];
|
||||
return new Map(rows.flatMap(row => row.issueId ? [[String(row.issueId), Number(row.count)] as const] : []));
|
||||
}
|
||||
|
||||
function view(issue: Json): Omit<GradeTaskInput, "issueId" | "identifier" | "title" | "runs" | "queuedWakes" | "retryClicks" | "pendingApproval" | "pendingQuestion"> {
|
||||
const interruption = issue.interruption;
|
||||
return {
|
||||
status: String(issue.status),
|
||||
scheduledRetry: Boolean(issue.scheduledRetry),
|
||||
scheduledRetryAt: issue.scheduledRetry?.scheduledRetryAt ? Date.parse(String(issue.scheduledRetry.scheduledRetryAt)) : null,
|
||||
interruption: interruption ? { state: String(interruption.state), canRetry: interruption.canRetry === true, retryWait: Boolean(interruption.retryWait),
|
||||
userAction: text(interruption.userAction), responsibility: text(interruption.responsibility), providerFailureCategory: text(interruption.providerFailure?.category) } : null,
|
||||
activeRecoveryOwner: text(issue.activeRecoveryAction?.ownerType),
|
||||
executionBlocker: Boolean(issue.executionBlocker),
|
||||
openBlockers: (issue.blockedBy ?? []).filter((blocker: Json) => blocker.status !== "done" && blocker.status !== "cancelled").length,
|
||||
};
|
||||
}
|
||||
|
||||
function observedClicks(timeline: readonly TimelineEntry[], issueIdForTask: ReadonlyMap<string, string>): Map<string, RetryClick[]> {
|
||||
const clicks = new Map<string, RetryClick[]>();
|
||||
for (const entry of timeline) {
|
||||
if (entry.kind !== "task_retry_clicked" && entry.kind !== "task_child_retry_clicked") continue;
|
||||
const issueId = entry.kind === "task_child_retry_clicked" ? text(entry.childIssueId) : issueIdForTask.get(String(entry.taskId)) ?? null;
|
||||
if (!issueId) continue;
|
||||
clicks.set(issueId, [...(clicks.get(issueId) ?? []), { at: Date.parse(entry.at), accepted: entry.accepted === true ? true : entry.accepted === false ? false : null, via: "observer" }]);
|
||||
}
|
||||
return clicks;
|
||||
}
|
||||
|
||||
async function clickRetry(input: ReleaseSwapFinishInput, issueId: string, ref: string, observedRunId: string | null): Promise<RetryClick> {
|
||||
const at = Date.now();
|
||||
if (input.board) {
|
||||
const click = await input.board.clickRetry(ref, issueId).catch(error => ({ accepted: false, httpStatus: null, error: String(error) }));
|
||||
input.record("release_swap_retry_clicked", { issueId, via: "ui", accepted: click.accepted, httpStatus: click.httpStatus, error: click.error });
|
||||
return { at, accepted: click.accepted, via: "grader" };
|
||||
}
|
||||
try {
|
||||
const result = await input.api.post<{ accepted?: boolean }>(`/api/issues/${issueId}/retry`, { idempotencyKey: randomUUID(), observedRunId });
|
||||
input.record("release_swap_retry_clicked", { issueId, via: "api", accepted: result?.accepted === true });
|
||||
return { at, accepted: result?.accepted === true, via: "grader" };
|
||||
} catch (error) {
|
||||
input.record("release_swap_retry_clicked", { issueId, via: "api", accepted: false, error: String(error) });
|
||||
return { at, accepted: false, via: "grader" };
|
||||
}
|
||||
}
|
||||
|
||||
async function failureChains(db: Client, runIds: readonly string[]): Promise<Map<string, string[]>> {
|
||||
if (!runIds.length) return new Map();
|
||||
const rows = await db.$client.unsafe(
|
||||
`select run_id::text as "runId", event_type as "eventType", level, left(message, 240) as message from (
|
||||
select e.*, row_number() over (partition by e.run_id order by e.created_at desc, e.id desc) as position
|
||||
from heartbeat_run_events e
|
||||
where e.run_id::text = any($1) and (e.level in ('error', 'warn') or e.event_type ilike '%fail%' or e.event_type ilike '%error%' or e.event_type ilike '%lost%')
|
||||
) ranked where position <= 6 order by "runId", created_at`, [runIds]) as Row[];
|
||||
const chains = new Map<string, string[]>();
|
||||
for (const row of rows) chains.set(String(row.runId), [...(chains.get(String(row.runId)) ?? []), `${row.eventType}${row.message ? `: ${row.message}` : ""}`]);
|
||||
return chains;
|
||||
}
|
||||
|
||||
async function firstEventsAfter(db: Client, stops: ReadonlyArray<{ index: number; at: number; runIds: string[] }>): Promise<Map<string, number>> {
|
||||
const found = new Map<string, number>();
|
||||
for (const stop of stops) {
|
||||
if (!stop.runIds.length) continue;
|
||||
const rows = await db.$client.unsafe(
|
||||
`select run_id::text as "runId", floor(extract(epoch from min(created_at)) * 1000) as at from heartbeat_run_events
|
||||
where run_id::text = any($1) and created_at > to_timestamp($2::double precision / 1000) group by run_id`, [stop.runIds, stop.at]) as Row[];
|
||||
for (const row of rows) found.set(`${stop.index}:${row.runId}`, Number(row.at));
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
async function duplicateFacts(input: ReleaseSwapFinishInput, facts: readonly SwapRunFact[], waveEnd: number): Promise<DuplicateFacts> {
|
||||
const cross = await readCrossServerEvidence(input.db as Db, input.companyId)
|
||||
.then(evidence => checkCrossServer({ ...evidence, windows: input.serverWindows })).catch(() => null);
|
||||
const comments = await input.db.$client.unsafe(
|
||||
`select issue_id::text as "issueId", count(*)::int as count from issue_comments
|
||||
where company_id::text = $1 and author_agent_id is not null group by issue_id, author_agent_id, body having count(*) > 1`, [input.companyId]) as Row[];
|
||||
const children = await input.db.$client.unsafe(
|
||||
`select parent_id::text as "parentId", title, count(*)::int as count from issues
|
||||
where company_id::text = $1 and parent_id is not null and hidden_at is null group by parent_id, title having count(*) > 1`, [input.companyId]) as Row[];
|
||||
return {
|
||||
overlappingRuns: overlappingRuns(facts, waveEnd),
|
||||
doubleExecutors: cross?.violations.filter(violation => violation.rule === "MS_double_executor").map(violation => violation.runId) ?? [],
|
||||
duplicateComments: comments.map(row => ({ issueId: String(row.issueId), count: Number(row.count) })),
|
||||
duplicateChildren: children.map(row => ({ parentId: String(row.parentId), title: String(row.title), count: Number(row.count) })),
|
||||
};
|
||||
}
|
||||
|
||||
export async function finishReleaseSwap(input: ReleaseSwapFinishInput): Promise<{ headline: string; pass: boolean; markdown: string }> {
|
||||
const { driver, db, companyId } = input;
|
||||
await driver.drain(input.cancelled);
|
||||
const lastStop = driver.swaps.length ? Math.max(...driver.swaps.map(swap => swap.stoppedAt ?? swap.requestedAt)) : Date.now();
|
||||
const drained = await drainRuns(input, lastStop);
|
||||
input.record("release_swap_drained", drained);
|
||||
const clicks = observedClicks(input.timeline, input.issueIdForTask);
|
||||
const views = new Map<string, Json>();
|
||||
const readViews = async (ids: readonly string[]) => {
|
||||
const deadline = Date.now() + VIEW_DEADLINE_MS;
|
||||
const queue = [...ids];
|
||||
await Promise.all(Array.from({ length: Math.min(VIEW_READERS, queue.length) }, async () => {
|
||||
for (let id = queue.shift(); id !== undefined; id = queue.shift()) {
|
||||
const left = deadline - Date.now();
|
||||
views.set(id, left <= 0 ? { status: "unreadable", error: "view deadline passed" }
|
||||
: await input.api.get<Json>(`/api/issues/${id}`, Math.min(10_000, left)).catch(error => ({ status: "unreadable", error: String(error) })));
|
||||
}
|
||||
}));
|
||||
};
|
||||
let issues = await startedIssues(db, companyId);
|
||||
await readViews(issues.map(issue => String(issue.id)));
|
||||
const retried: string[] = [];
|
||||
for (const issue of issues) {
|
||||
const id = String(issue.id);
|
||||
const current = views.get(id)!;
|
||||
if (current.status === "done" || current.interruption?.canRetry !== true || current.interruption?.retryWait) continue;
|
||||
const click = await clickRetry(input, id, text(issue.identifier) ?? id, text(current.interruption?.observedRunId));
|
||||
clicks.set(id, [...(clicks.get(id) ?? []), click]);
|
||||
retried.push(id);
|
||||
}
|
||||
if (retried.length) {
|
||||
const until = Date.now() + input.settleMs;
|
||||
while (!input.cancelled() && Date.now() < until) {
|
||||
await delay(input.pollMs ?? 15_000);
|
||||
await readViews(retried);
|
||||
if (retried.every(id => views.get(id)?.status === "done")) break;
|
||||
}
|
||||
issues = await startedIssues(db, companyId);
|
||||
await readViews(issues.map(issue => String(issue.id)));
|
||||
}
|
||||
const person = await pendingPersonItems(db, companyId);
|
||||
const runs = await companyRuns(db, companyId);
|
||||
const wakes = await queuedWakes(db, companyId);
|
||||
const runsByIssue = new Map<string, Row[]>();
|
||||
for (const run of runs) if (run.issueId) runsByIssue.set(String(run.issueId), [...(runsByIssue.get(String(run.issueId)) ?? []), run]);
|
||||
const now = Date.now();
|
||||
const unreadable = issues.filter(issue => (views.get(String(issue.id)) ?? { status: "unreadable" }).status === "unreadable");
|
||||
if (unreadable.length) input.record("release_swap_views_unreadable", { count: unreadable.length, issueIds: unreadable.map(issue => String(issue.id)) });
|
||||
const grades: TaskGrade[] = issues.filter(issue => !unreadable.includes(issue)).map(issue => {
|
||||
const id = String(issue.id);
|
||||
const current = views.get(id)!;
|
||||
return gradeReleaseSwapTask({ now, staleAfterMs: progressWindowMs(input.settleMs), issueId: id, identifier: text(issue.identifier), title: String(issue.title), ...view(current),
|
||||
runs: (runsByIssue.get(id) ?? []).map(run => ({ runId: String(run.runId), status: String(run.status), errorCode: text(run.errorCode),
|
||||
createdAt: num(run.createdAt), startedAt: num(run.startedAt), finishedAt: num(run.finishedAt), lastActivityAt: num(run.lastActivityAt) })),
|
||||
queuedWakes: wakes.get(id) ?? 0, retryClicks: clicks.get(id) ?? [], pendingApproval: person.approvals.has(id), pendingQuestion: person.questions.has(id) });
|
||||
});
|
||||
const traceOwner = new Map<string, number>();
|
||||
for (const swap of driver.swaps) for (const trace of Object.values(swap.runs)) {
|
||||
const owned = trace.samples.filter(sample => sample.ownerPid !== null && (trace.finishedAt === null || sample.at <= trace.finishedAt));
|
||||
if (owned.length) traceOwner.set(trace.runId, owned[owned.length - 1]!.ownerPid!);
|
||||
}
|
||||
const facts: SwapRunFact[] = runs.map(run => ({ runId: String(run.runId), issueId: text(run.issueId), status: String(run.status), errorCode: text(run.errorCode),
|
||||
startedAt: num(run.startedAt), finishedAt: num(run.finishedAt), ownerPidBeforeEnd: traceOwner.get(String(run.runId)) ?? num(run.ownerPid) }));
|
||||
const waveEnd = Date.now();
|
||||
const windows = driver.swaps.map(swap => ({ index: swap.index, requestedAt: swap.requestedAt, spawnedAt: swap.spawnedAt, stopRequestedAt: swap.stopRequestedAt,
|
||||
stoppedAt: swap.stoppedAt, fromPid: swap.fromPid }));
|
||||
const stops = windows.flatMap(swap => swap.stopRequestedAt === null ? [] : [{ index: swap.index, at: swap.stopRequestedAt,
|
||||
runIds: facts.filter(fact => fact.startedAt !== null && fact.startedAt < swap.stopRequestedAt! && (fact.finishedAt === null || fact.finishedAt > swap.stopRequestedAt!))
|
||||
.map(fact => fact.runId) }]);
|
||||
const metrics = swapMetrics(windows, facts, await firstEventsAfter(db, stops), waveEnd);
|
||||
const duplicates = await duplicateFacts(input, facts, waveEnd);
|
||||
const result = summarizeGrades(grades, unreadable.length, drained.reason);
|
||||
const failed = runs.filter(run => ["failed", "timed_out", "interrupted"].includes(String(run.status)));
|
||||
const chains = await failureChains(db, failed.map(run => String(run.runId)));
|
||||
const failedRuns = failed.map(run => ({ runId: String(run.runId), issueId: text(run.issueId), status: String(run.status), errorCode: text(run.errorCode),
|
||||
error: text(run.error), finishedAt: num(run.finishedAt), ownerPid: traceOwner.get(String(run.runId)) ?? num(run.ownerPid),
|
||||
swap: [...windows].reverse().find(swap => (num(run.finishedAt) ?? 0) >= swap.requestedAt)?.index ?? null, chain: chains.get(String(run.runId)) ?? [] }));
|
||||
const evidence = driver.swaps.map(swap => swap.evidence ? { index: swap.index, ...swap.evidence } : null);
|
||||
const markdown = releaseSwapMarkdown(result, metrics, evidence, duplicates, grades);
|
||||
await writeFile(path.join(input.waveDir, "release-swap.json"), `${JSON.stringify({ schema: "paperclip.restart-chaos.release-swap/v1", result, metrics,
|
||||
swaps: driver.swaps, duplicates, failedRuns, grades }, null, 2)}\n`, { mode: 0o600 });
|
||||
await writeFile(path.join(input.waveDir, "release-swap-summary.md"), markdown, { mode: 0o600 });
|
||||
input.record("release_swap_graded", { headline: result.headline, byVerdict: result.byVerdict, retry: result.retry,
|
||||
failedPerSwap: metrics.map(metric => metric.failedRuns.length), lostWhileOldAlive: metrics.map(metric => metric.lostWhileOldAlive.length) });
|
||||
return { headline: result.headline, pass: result.uncollected === 0 && result.stillRunning === 0 && result.started > 0 && result.completedWithoutHuman === result.started, markdown };
|
||||
}
|
||||
@@ -0,0 +1,225 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { gradeReleaseSwapTask, overlappingRuns, personReason, releaseSwapMarkdown, retryLabel, summarizeGrades, swapMetrics, type GradeTaskInput, type SwapRunFact } from "./release-swap-grade.js";
|
||||
import { mergeRunSample, swapEvidence, type SwapRecord } from "./release-swap-driver.js";
|
||||
|
||||
const task = (overrides: Partial<GradeTaskInput> = {}): GradeTaskInput => ({
|
||||
issueId: "i1", identifier: "CHA-1", title: "t", status: "done", runs: [{ runId: "r1", status: "succeeded", errorCode: null, createdAt: 1, startedAt: 2, finishedAt: 3 }],
|
||||
queuedWakes: 0, scheduledRetry: false, interruption: null, activeRecoveryOwner: null, executionBlocker: false, openBlockers: 0,
|
||||
pendingApproval: false, pendingQuestion: false, retryClicks: [], ...overrides,
|
||||
});
|
||||
const failed = { runId: "r0", status: "failed", errorCode: "process_lost", createdAt: 0, startedAt: 0, finishedAt: 1 };
|
||||
const interruption = (overrides: Partial<NonNullable<GradeTaskInput["interruption"]>> = {}) =>
|
||||
({ state: "failed", canRetry: false, retryWait: false, userAction: null, providerFailureCategory: null, ...overrides });
|
||||
|
||||
describe("release swap task grade", () => {
|
||||
it("passes a task done with no person, even after a failed run the system recovered", () => {
|
||||
expect(gradeReleaseSwapTask(task()).verdict).toBe("pass");
|
||||
const recovered = gradeReleaseSwapTask(task({ runs: [failed, ...task().runs] }));
|
||||
expect(recovered).toMatchObject({ verdict: "pass", failedRuns: ["r0"] });
|
||||
});
|
||||
|
||||
it("fails a task that needed Retry and says whether the click worked", () => {
|
||||
const worked = gradeReleaseSwapTask(task({ runs: [failed, { runId: "r2", status: "succeeded", errorCode: null, createdAt: 50, startedAt: 51, finishedAt: 60 }],
|
||||
retryClicks: [{ at: 40, accepted: true, via: "observer" }] }));
|
||||
expect(worked).toMatchObject({ verdict: "needed_retry", retry: { clicks: 1, accepted: 1, worked: true } });
|
||||
const refused = gradeReleaseSwapTask(task({ status: "todo", runs: [failed], retryClicks: [{ at: 40, accepted: false, via: "grader" }] }));
|
||||
expect(refused.retry).toEqual({ clicks: 1, accepted: 0, worked: false });
|
||||
const running = gradeReleaseSwapTask(task({ status: "in_progress", runs: [failed, { runId: "r3", status: "running", errorCode: null, createdAt: 50, startedAt: 51, finishedAt: null }],
|
||||
retryClicks: [{ at: 40, accepted: true, via: "grader" }] }));
|
||||
expect(running.retry?.worked).toBeNull();
|
||||
const stuck = gradeReleaseSwapTask(task({ status: "todo", runs: [failed, { runId: "r4", status: "failed", errorCode: "x", createdAt: 50, startedAt: 51, finishedAt: 52 }],
|
||||
retryClicks: [{ at: 40, accepted: true, via: "grader" }] }));
|
||||
expect(stuck.retry?.worked).toBe(false);
|
||||
});
|
||||
|
||||
it("fails an open task with no live run, no queued wake and no named reason", () => {
|
||||
expect(gradeReleaseSwapTask(task({ status: "in_progress", runs: [failed] })).verdict).toBe("open_no_owner");
|
||||
expect(gradeReleaseSwapTask(task({ status: "todo", runs: [failed] })).verdict).toBe("open_no_owner");
|
||||
});
|
||||
|
||||
it("keeps a task with a live run, a queued wake or a system retry as still working, not passed", () => {
|
||||
expect(gradeReleaseSwapTask(task({ status: "in_progress", runs: [{ ...failed, status: "queued" }] })).verdict).toBe("still_working");
|
||||
expect(gradeReleaseSwapTask(task({ status: "todo", runs: [failed], queuedWakes: 1 })).verdict).toBe("still_working");
|
||||
expect(gradeReleaseSwapTask(task({ status: "todo", runs: [failed], interruption: interruption({ state: "retry_scheduled" }) })).verdict).toBe("still_working");
|
||||
expect(gradeReleaseSwapTask(task({ status: "todo", runs: [failed], activeRecoveryOwner: "agent" })).verdict).toBe("still_working");
|
||||
});
|
||||
|
||||
it("fails a task blocked for a reason that is not a person's", () => {
|
||||
const confirm = gradeReleaseSwapTask(task({ status: "blocked", runs: [failed], interruption: interruption({ state: "waiting_for_user", userAction: "confirm_run_stopped" }) }));
|
||||
expect(confirm).toMatchObject({ verdict: "blocked_non_person", personReason: null });
|
||||
expect(confirm.detail).toContain("confirm_run_stopped");
|
||||
expect(gradeReleaseSwapTask(task({ status: "blocked", runs: [failed] })).verdict).toBe("blocked_non_person");
|
||||
expect(gradeReleaseSwapTask(task({ status: "in_progress", runs: [failed], executionBlocker: true })).verdict).toBe("blocked_non_person");
|
||||
expect(gradeReleaseSwapTask(task({ status: "todo", runs: [failed], activeRecoveryOwner: "board" })).verdict).toBe("blocked_non_person");
|
||||
});
|
||||
|
||||
it("names a person's reason and grades the wait apart from the failures", () => {
|
||||
const cases: Array<[Partial<GradeTaskInput>, string]> = [
|
||||
[{ interruption: interruption({ state: "waiting_for_user", userAction: "approval" }) }, "approval"],
|
||||
[{ pendingApproval: true }, "approval"],
|
||||
[{ pendingQuestion: true }, "question"],
|
||||
[{ openBlockers: 1 }, "dependency"],
|
||||
[{ status: "in_review" }, "review"],
|
||||
[{ interruption: interruption({ providerFailureCategory: "rate_limit" }) }, "usage_limit"],
|
||||
[{ interruption: interruption({ state: "waiting_for_user", userAction: "budget" }) }, "budget"],
|
||||
[{ interruption: interruption({ state: "waiting_for_user", userAction: "reconnect" }) }, "shared_credential"],
|
||||
];
|
||||
for (const [overrides, reason] of cases) {
|
||||
const input = task({ status: "blocked", runs: [failed], ...overrides });
|
||||
expect(personReason(input)).toBe(reason);
|
||||
expect(gradeReleaseSwapTask(input).verdict).toBe("waiting_on_person");
|
||||
}
|
||||
});
|
||||
|
||||
it("names a user-responsibility wait that is not a grade person reason and counts it by reason", () => {
|
||||
const model = gradeReleaseSwapTask(task({ status: "blocked", runs: [failed],
|
||||
interruption: interruption({ state: "waiting_for_user", userAction: "choose_model", responsibility: "user" }) }));
|
||||
expect(model).toMatchObject({ verdict: "blocked_non_person", personReason: null, reason: "user action choose_model" });
|
||||
expect(model.detail).toContain("not one of the grade's person reasons");
|
||||
const question = gradeReleaseSwapTask(task({ status: "blocked", runs: [failed], pendingQuestion: true }));
|
||||
expect(summarizeGrades([model, question, gradeReleaseSwapTask(task())]).byReason)
|
||||
.toEqual({ "blocked_non_person: user action choose_model": 1, "waiting_on_person: question": 1 });
|
||||
});
|
||||
|
||||
it("labels every failing task with exactly one of the three reported labels", () => {
|
||||
const grades = [
|
||||
gradeReleaseSwapTask(task({ status: "todo", runs: [failed] })),
|
||||
gradeReleaseSwapTask(task({ status: "blocked", runs: [failed] })),
|
||||
gradeReleaseSwapTask(task({ runs: [failed, { runId: "r2", status: "succeeded", errorCode: null, createdAt: 50, startedAt: 51, finishedAt: 60 }], retryClicks: [{ at: 40, accepted: true, via: "grader" }] })),
|
||||
gradeReleaseSwapTask(task({ status: "todo", runs: [failed], retryClicks: [{ at: 40, accepted: false, via: "grader" }] })),
|
||||
gradeReleaseSwapTask(task({ status: "cancelled", runs: [failed] })),
|
||||
];
|
||||
expect(grades.map(retryLabel)).toEqual(["nothing pending", "blocked for a non-person reason", "needed Retry (click worked)",
|
||||
"needed Retry (click did not work)", "nothing pending"]);
|
||||
expect(retryLabel(gradeReleaseSwapTask(task()))).toBeNull();
|
||||
expect(summarizeGrades(grades).byLabel).toEqual({ "nothing pending": 2, "blocked for a non-person reason": 1,
|
||||
"needed Retry (click worked)": 1, "needed Retry (click did not work)": 1 });
|
||||
});
|
||||
|
||||
it("does not count a run with no activity for the settle period as a live owner", () => {
|
||||
const stuck = task({ status: "in_progress", now: 2_000_000, staleAfterMs: 900_000, interruption: interruption({ state: "working" }),
|
||||
runs: [{ runId: "r9", status: "running", errorCode: null, createdAt: 0, startedAt: 1, finishedAt: null, lastActivityAt: 1_000_000 }] });
|
||||
const grade = gradeReleaseSwapTask(stuck);
|
||||
expect(grade).toMatchObject({ verdict: "open_no_owner", label: "nothing pending" });
|
||||
expect(grade.detail).toContain("run r9 running with no activity for 17 min");
|
||||
expect(gradeReleaseSwapTask({ ...stuck, runs: [{ ...stuck.runs[0]!, lastActivityAt: 1_500_000 }] }).verdict).toBe("still_working");
|
||||
const retry = gradeReleaseSwapTask({ ...stuck, scheduledRetry: true, runs: [failed, { ...stuck.runs[0]!, status: "queued" }] });
|
||||
expect(retry).toMatchObject({ verdict: "open_no_owner", label: "nothing pending" });
|
||||
expect(gradeReleaseSwapTask({ ...stuck, scheduledRetry: true, runs: [failed] }).verdict).toBe("open_no_owner");
|
||||
});
|
||||
|
||||
it("reports an accepted Retry as pending while its scheduled retry is not yet due, and as not working once it is past due", () => {
|
||||
const now = 10_000_000;
|
||||
const base = { status: "todo", now, staleAfterMs: 300_000, runs: [failed], scheduledRetry: true, retryClicks: [{ at: 5, accepted: true, via: "grader" as const }] };
|
||||
expect(gradeReleaseSwapTask(task({ ...base, scheduledRetryAt: now + 60_000 })).retry?.worked).toBeNull();
|
||||
expect(gradeReleaseSwapTask(task({ ...base, scheduledRetryAt: now - 60_000 })).retry?.worked).toBe(false);
|
||||
expect(gradeReleaseSwapTask(task({ ...base, scheduledRetry: false, queuedWakes: 1 })).retry?.worked).toBeNull();
|
||||
});
|
||||
|
||||
it("uses one staleness threshold after the drain: a 6-minute-silent run or a queued wake alone is nothing pending", () => {
|
||||
const now = 10_000_000;
|
||||
const silent = gradeReleaseSwapTask(task({ status: "in_progress", now, staleAfterMs: 300_000,
|
||||
runs: [{ runId: "r6", status: "running", errorCode: null, createdAt: 0, startedAt: 1, finishedAt: null, lastActivityAt: now - 360_000 }] }));
|
||||
expect(silent).toMatchObject({ verdict: "open_no_owner", label: "nothing pending", lastActivityAgeMs: null });
|
||||
expect(silent.detail).toContain("run r6 running with no activity for 6 min");
|
||||
const wake = gradeReleaseSwapTask(task({ status: "todo", now, staleAfterMs: 300_000, runs: [failed], queuedWakes: 1 }));
|
||||
expect(wake).toMatchObject({ verdict: "open_no_owner", label: "nothing pending" });
|
||||
expect(wake.detail).toContain("1 queued wake(s) never dispatched");
|
||||
expect(gradeReleaseSwapTask(task({ status: "todo", runs: [failed], queuedWakes: 1 })).verdict).toBe("still_working");
|
||||
});
|
||||
|
||||
it("keeps a task still running past the drain cap out of the number and reports it with its last activity", () => {
|
||||
const running = gradeReleaseSwapTask(task({ status: "in_progress", now: 1_000_000, staleAfterMs: 900_000,
|
||||
runs: [{ runId: "r5", status: "running", errorCode: null, createdAt: 0, startedAt: 1, finishedAt: null, lastActivityAt: 940_000 }] }));
|
||||
expect(running).toMatchObject({ verdict: "still_working", lastActivityAgeMs: 60_000, label: null });
|
||||
expect(running.detail).toContain("last activity 60 s ago");
|
||||
expect(summarizeGrades([gradeReleaseSwapTask(task()), running], 0, "cap")).toMatchObject({ started: 1, stillRunning: 1, drainReason: "cap",
|
||||
headline: "1 / 1 tasks completed without human action; 1 still running at the drain cap, not counted (last activity 60 s ago)" });
|
||||
});
|
||||
|
||||
it("says the number is incomplete when task views could not be read", () => {
|
||||
expect(summarizeGrades([gradeReleaseSwapTask(task())], 2)).toMatchObject({ uncollected: 2,
|
||||
headline: "1 / 1 tasks completed without human action (incomplete: 2 task(s) could not be read and are not counted)" });
|
||||
});
|
||||
|
||||
it("publishes one number: tasks completed without human action over tasks started", () => {
|
||||
const grades = [task(), task({ issueId: "i2" }), task({ issueId: "i3", status: "todo", runs: [failed] }),
|
||||
task({ issueId: "i4", runs: [failed], retryClicks: [{ at: 0, accepted: true, via: "grader" }] })].map(gradeReleaseSwapTask);
|
||||
const result = summarizeGrades(grades);
|
||||
expect(result.headline).toBe("2 / 4 tasks completed without human action");
|
||||
expect(result.byVerdict).toMatchObject({ pass: 2, open_no_owner: 1, needed_retry: 1 });
|
||||
const markdown = releaseSwapMarkdown(result, [], [], { overlappingRuns: [], doubleExecutors: [], duplicateComments: [], duplicateChildren: [] }, grades);
|
||||
expect(markdown.split("\n")[0]).toBe("2 / 4 tasks completed without human action");
|
||||
});
|
||||
});
|
||||
|
||||
describe("release swap per-swap numbers", () => {
|
||||
const swaps = [
|
||||
{ index: 1, requestedAt: 1_000, spawnedAt: 1_100, stopRequestedAt: 2_000, stoppedAt: 2_100, fromPid: 10 },
|
||||
{ index: 2, requestedAt: 5_000, spawnedAt: 5_100, stopRequestedAt: 6_000, stoppedAt: 6_100, fromPid: 11 },
|
||||
];
|
||||
const fact = (overrides: Partial<SwapRunFact>): SwapRunFact => ({ runId: "r", issueId: "i", status: "succeeded", errorCode: null, startedAt: 0, finishedAt: null, ownerPidBeforeEnd: null, ...overrides });
|
||||
|
||||
it("counts failed runs per swap, runs lost while the old server lived, and the gap to each survivor's next event", () => {
|
||||
const runs = [
|
||||
fact({ runId: "lost", status: "failed", errorCode: "process_lost", startedAt: 500, finishedAt: 1_500, ownerPidBeforeEnd: 10 }),
|
||||
fact({ runId: "shutdown", status: "failed", errorCode: "server_shutdown_interrupted", startedAt: 500, finishedAt: 2_050, ownerPidBeforeEnd: 10 }),
|
||||
fact({ runId: "late", status: "failed", errorCode: "controller_lease_lost", startedAt: 5_500, finishedAt: 7_000, ownerPidBeforeEnd: 11 }),
|
||||
fact({ runId: "survivor", status: "succeeded", startedAt: 1_900, finishedAt: 6_500 }),
|
||||
fact({ runId: "quiet", status: "running", startedAt: 1_950, finishedAt: null }),
|
||||
];
|
||||
const events = new Map([["1:survivor", 2_400], ["2:survivor", 6_300]]);
|
||||
const [first, second] = swapMetrics(swaps, runs, events, 9_000);
|
||||
expect(first).toMatchObject({ failedRuns: ["lost", "shutdown"], lostWhileOldAlive: ["lost"], nextEventP50Ms: 400, nextEventMaxMs: 400, survivorsWithoutEvent: 1 });
|
||||
expect(first!.survivors.map(survivor => survivor.runId)).toEqual(["survivor", "quiet"]);
|
||||
expect(second).toMatchObject({ failedRuns: ["late"], lostWhileOldAlive: [], nextEventP50Ms: 300 });
|
||||
});
|
||||
|
||||
it("finds a run nested inside a long earlier run, not only neighbours", () => {
|
||||
expect(overlappingRuns([fact({ runId: "A", startedAt: 0, finishedAt: 100 }), fact({ runId: "B", startedAt: 10, finishedAt: 20 }),
|
||||
fact({ runId: "C", startedAt: 30, finishedAt: 50 })], 1_000)).toEqual([{ issueId: "i", runIds: ["A", "B"] }, { issueId: "i", runIds: ["A", "C"] }]);
|
||||
});
|
||||
|
||||
it("finds two runs of one task executing at once", () => {
|
||||
expect(overlappingRuns([fact({ runId: "a", startedAt: 0, finishedAt: 100 }), fact({ runId: "b", startedAt: 50, finishedAt: 200 }),
|
||||
fact({ runId: "c", startedAt: 200, finishedAt: 300 }), fact({ runId: "d", issueId: "other", startedAt: 0, finishedAt: null })], 1_000))
|
||||
.toEqual([{ issueId: "i", runIds: ["a", "b"] }]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("release swap overlap evidence", () => {
|
||||
const swap = (): SwapRecord => ({ index: 1, faultId: "f01", fromPid: 10, toPid: 20, fromUrl: "a", toUrl: "b", serverRoot: null, requestedAt: 0,
|
||||
spawnedAt: 100, switchedAt: 3_000, overlapEndsAt: 93_000, stopRequestedAt: 93_000, stoppedAt: 95_000,
|
||||
incomingPhases: [{ phase: "starting", at: 500 }, { phase: "recovering", at: 2_000 }, { phase: "ready", at: 2_600 }],
|
||||
probes: [{ kind: "prep", adapter: "codex", nonce: "n", requestedAt: 100, createdAt: 200, issueId: "prep", identifier: null, error: null },
|
||||
{ kind: "late", adapter: "codex", nonce: "m", requestedAt: 60_000, createdAt: 60_100, issueId: "late", identifier: "CHA-9", error: null }],
|
||||
runs: {}, sampleErrors: 0, plugins: [], abandoned: null, evidence: null });
|
||||
const row = (runId: string, issueId: string, at: number, status: string, ownerPid: number | null, executionStage: string | null, startedAt: number) =>
|
||||
({ runId, issueId, createdAt: startedAt, startedAt, finishedAt: null, errorCode: null, at, status, ownerPid, executionStage, nativePhase: null, lastOutputSeq: 0 });
|
||||
|
||||
it("proves the old server started a run about 20 s before its SIGTERM and had one preparing during the new server's recovery", () => {
|
||||
const record = swap();
|
||||
for (const sample of [row("p1", "prep", 1_000, "running", 10, "preparing", 900), row("p1", "prep", 2_400, "running", 10, "preparing", 900),
|
||||
row("p1", "prep", 4_000, "running", 10, "executing", 900), row("l1", "late", 74_000, "running", 10, "preparing", 73_500)]) mergeRunSample(record.runs, sample);
|
||||
const evidence = swapEvidence(record);
|
||||
expect(evidence).toMatchObject({ lateRunId: "l1", lateStartedBy: "old", lateStartOffsetMs: -19_500, lateOk: true, prepRunIds: ["p1"], prepOk: true,
|
||||
startedByOld: ["p1", "l1"], startedByNew: [], findings: [] });
|
||||
});
|
||||
|
||||
it("does not count the incoming server's boot as its recovery pass when no recovering phase was seen", () => {
|
||||
const record = { ...swap(), incomingPhases: [{ phase: "starting", at: 500 }, { phase: "ready", at: 2_600 }] };
|
||||
mergeRunSample(record.runs, row("p1", "prep", 1_000, "running", 10, "preparing", 900));
|
||||
const evidence = swapEvidence(record);
|
||||
expect(evidence).toMatchObject({ recoveryWindow: null, prepOk: false });
|
||||
expect(evidence.findings).toContain("the incoming server's startup recovery pass was not observed (no `recovering` phase before `ready`)");
|
||||
});
|
||||
|
||||
it("reports a finding instead of a pass when the new server took the runs", () => {
|
||||
const record = swap();
|
||||
for (const sample of [row("p1", "prep", 1_000, "running", 20, "preparing", 900), row("l1", "late", 74_000, "running", 20, "preparing", 73_500)]) mergeRunSample(record.runs, sample);
|
||||
const evidence = swapEvidence(record);
|
||||
expect(evidence).toMatchObject({ lateOk: false, prepOk: false, startedByNew: ["p1", "l1"] });
|
||||
expect(evidence.findings).toEqual(["late probe run l1 was started by the new server",
|
||||
"no run of the outgoing server was seen in preparation during the incoming server's startup recovery"]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,330 @@
|
||||
export type TaskVerdict = "pass" | "needed_retry" | "open_no_owner" | "blocked_non_person" | "waiting_on_person" | "still_working" | "cancelled";
|
||||
export type PersonReason = "approval" | "question" | "dependency" | "review" | "usage_limit" | "budget" | "shared_credential";
|
||||
|
||||
export interface GradeRun {
|
||||
runId: string;
|
||||
status: string;
|
||||
errorCode: string | null;
|
||||
createdAt: number | null;
|
||||
startedAt: number | null;
|
||||
finishedAt: number | null;
|
||||
lastActivityAt?: number | null;
|
||||
}
|
||||
|
||||
export interface GradeInterruption {
|
||||
state: string;
|
||||
canRetry: boolean;
|
||||
retryWait: boolean;
|
||||
userAction: string | null;
|
||||
responsibility?: string | null;
|
||||
providerFailureCategory: string | null;
|
||||
}
|
||||
|
||||
export interface RetryClick {
|
||||
at: number;
|
||||
accepted: boolean | null;
|
||||
via: "observer" | "grader";
|
||||
}
|
||||
|
||||
export interface GradeTaskInput {
|
||||
issueId: string;
|
||||
identifier: string | null;
|
||||
title: string;
|
||||
status: string;
|
||||
runs: GradeRun[];
|
||||
queuedWakes: number;
|
||||
scheduledRetry: boolean;
|
||||
interruption: GradeInterruption | null;
|
||||
activeRecoveryOwner: string | null;
|
||||
executionBlocker: boolean;
|
||||
openBlockers: number;
|
||||
pendingApproval: boolean;
|
||||
pendingQuestion: boolean;
|
||||
retryClicks: RetryClick[];
|
||||
now?: number;
|
||||
staleAfterMs?: number;
|
||||
scheduledRetryAt?: number | null;
|
||||
}
|
||||
|
||||
export type FailureLabel = "nothing pending" | "blocked for a non-person reason" | "needed Retry";
|
||||
|
||||
export interface TaskGrade {
|
||||
issueId: string;
|
||||
identifier: string | null;
|
||||
title: string;
|
||||
verdict: TaskVerdict;
|
||||
detail: string;
|
||||
status: string;
|
||||
retry: { clicks: number; accepted: number; worked: boolean | null } | null;
|
||||
personReason: PersonReason | null;
|
||||
reason: string | null;
|
||||
label: FailureLabel | null;
|
||||
lastActivityAgeMs: number | null;
|
||||
failedRuns: string[];
|
||||
}
|
||||
|
||||
const LIVE_RUN = new Set(["queued", "running", "scheduled_retry"]);
|
||||
const FAILED_RUN = new Set(["failed", "timed_out", "interrupted"]);
|
||||
const SUCCEEDED_RUN = new Set(["succeeded"]);
|
||||
|
||||
export function personReason(task: GradeTaskInput): PersonReason | null {
|
||||
const action = task.interruption?.userAction ?? null;
|
||||
if (action === "approval" || task.pendingApproval) return "approval";
|
||||
if (action === "budget") return "budget";
|
||||
if (action === "reconnect") return "shared_credential";
|
||||
const category = task.interruption?.providerFailureCategory ?? null;
|
||||
if (category === "rate_limit" || category === "quota_exhausted") return "usage_limit";
|
||||
if (category === "authentication" || category === "permission") return "shared_credential";
|
||||
if (task.pendingQuestion) return "question";
|
||||
if (task.status === "in_review") return "review";
|
||||
if (task.openBlockers > 0) return "dependency";
|
||||
return null;
|
||||
}
|
||||
|
||||
export function staleRuns(task: GradeTaskInput): GradeRun[] {
|
||||
if (task.now === undefined || task.staleAfterMs === undefined) return [];
|
||||
return task.runs.filter(run => LIVE_RUN.has(run.status) && (run.lastActivityAt ?? run.createdAt ?? 0) < task.now! - task.staleAfterMs!);
|
||||
}
|
||||
|
||||
function liveOwner(task: GradeTaskInput): string | null {
|
||||
const stale = new Set(staleRuns(task).map(run => run.runId));
|
||||
const live = task.runs.find(run => LIVE_RUN.has(run.status) && !stale.has(run.runId));
|
||||
if (live) return `run ${live.runId} ${live.status}`;
|
||||
if (task.now !== undefined) return null;
|
||||
if (task.queuedWakes > 0) return `${task.queuedWakes} queued wake(s)`;
|
||||
if (task.scheduledRetry && !stale.size) return "scheduled retry";
|
||||
const interruption = task.interruption;
|
||||
if (!stale.size && interruption && (interruption.state === "working" || interruption.state === "retry_scheduled" || interruption.retryWait)) {
|
||||
return `interruption ${interruption.retryWait ? "retry waiting on the system" : interruption.state}`;
|
||||
}
|
||||
if (task.activeRecoveryOwner && task.activeRecoveryOwner !== "board") return `${task.activeRecoveryOwner} recovery action`;
|
||||
return null;
|
||||
}
|
||||
|
||||
function retryPending(task: GradeTaskInput): boolean {
|
||||
const stale = new Set(staleRuns(task).map(run => run.runId));
|
||||
if (task.runs.some(run => LIVE_RUN.has(run.status) && !stale.has(run.runId))) return true;
|
||||
if (task.queuedWakes > 0) return true;
|
||||
if (task.scheduledRetry && (task.scheduledRetryAt == null || task.now === undefined || task.scheduledRetryAt > task.now)) return true;
|
||||
const interruption = task.interruption;
|
||||
return Boolean(interruption && (interruption.state === "working" || interruption.state === "retry_scheduled" || interruption.retryWait));
|
||||
}
|
||||
|
||||
function retryOutcome(task: GradeTaskInput): TaskGrade["retry"] {
|
||||
if (!task.retryClicks.length) return null;
|
||||
const accepted = task.retryClicks.filter(click => click.accepted === true);
|
||||
const first = Math.min(...accepted.map(click => click.at));
|
||||
const worked = accepted.length === 0 ? false
|
||||
: task.status === "done" || task.runs.some(run => SUCCEEDED_RUN.has(run.status) && (run.createdAt ?? 0) >= first)
|
||||
? true : retryPending(task) ? null : false;
|
||||
return { clicks: task.retryClicks.length, accepted: accepted.length, worked };
|
||||
}
|
||||
|
||||
function namedReason(task: GradeTaskInput, person: PersonReason | null): string | null {
|
||||
if (person) return person;
|
||||
const interruption = task.interruption;
|
||||
if (interruption?.userAction) return `user action ${interruption.userAction}`;
|
||||
if (interruption?.responsibility === "user") return `user responsibility, ${interruption.state}`;
|
||||
if (interruption) return `interruption ${interruption.state}`;
|
||||
if (task.executionBlocker) return "execution blocker";
|
||||
if (task.activeRecoveryOwner) return `${task.activeRecoveryOwner} recovery action`;
|
||||
return null;
|
||||
}
|
||||
|
||||
export function gradeReleaseSwapTask(task: GradeTaskInput): TaskGrade {
|
||||
const failedRuns = task.runs.filter(run => FAILED_RUN.has(run.status)).map(run => run.runId);
|
||||
const person = personReason(task);
|
||||
const base = { issueId: task.issueId, identifier: task.identifier, title: task.title, status: task.status, failedRuns,
|
||||
retry: retryOutcome(task), personReason: person, reason: namedReason(task, person) };
|
||||
const label = (value: TaskVerdict): FailureLabel | null => value === "needed_retry" ? "needed Retry"
|
||||
: value === "open_no_owner" || value === "cancelled" ? "nothing pending" : value === "blocked_non_person" ? "blocked for a non-person reason" : null;
|
||||
const staleIds = new Set(staleRuns(task).map(run => run.runId));
|
||||
const liveActivity = task.runs.filter(run => LIVE_RUN.has(run.status) && !staleIds.has(run.runId)).map(run => run.lastActivityAt ?? run.createdAt ?? null).filter((at): at is number => at !== null);
|
||||
const lastActivityAgeMs = task.now !== undefined && liveActivity.length ? task.now - Math.max(...liveActivity) : null;
|
||||
const verdict = (value: TaskVerdict, detail: string): TaskGrade => ({ ...base, verdict: value, detail, label: label(value), lastActivityAgeMs });
|
||||
const stale = staleRuns(task);
|
||||
const idle = task.now === undefined ? [] : [
|
||||
...stale.map(run => `run ${run.runId} ${run.status} with no activity for ${Math.round((task.now! - (run.lastActivityAt ?? run.createdAt ?? 0)) / 60_000)} min`),
|
||||
...(task.queuedWakes ? [`${task.queuedWakes} queued wake(s) never dispatched`] : []), ...(task.scheduledRetry ? ["a scheduled retry that never ran"] : []),
|
||||
...(task.interruption && (task.interruption.state === "working" || task.interruption.state === "retry_scheduled" || task.interruption.retryWait)
|
||||
? [`the board shows ${task.interruption.retryWait ? "a Retry waiting on the system" : task.interruption.state}`] : []),
|
||||
...(task.activeRecoveryOwner && task.activeRecoveryOwner !== "board" ? [`a ${task.activeRecoveryOwner} recovery action that made no progress`] : [])];
|
||||
const staleNote = idle.length ? `; ${idle.join(", ")}` : "";
|
||||
if (task.retryClicks.length) {
|
||||
const retry = base.retry!;
|
||||
return verdict("needed_retry", `${retry.clicks} Retry click(s), ${retry.accepted} accepted; ${retry.worked === null ? "still running" : retry.worked ? "worked" : "did not work"}; now ${task.status}`);
|
||||
}
|
||||
if (task.status === "done") return verdict("pass", failedRuns.length ? `done after ${failedRuns.length} failed run(s) recovered without a person` : "done");
|
||||
if (task.status === "cancelled") return verdict("cancelled", "cancelled with no person asking");
|
||||
const owner = liveOwner(task);
|
||||
if (owner) return verdict("still_working", `${task.status}: ${owner}${lastActivityAgeMs !== null ? `, last activity ${Math.round(lastActivityAgeMs / 1000)} s ago` : ""}`);
|
||||
if (base.personReason) return verdict("waiting_on_person", `${task.status}: waits on ${base.personReason}`);
|
||||
const interruption = task.interruption;
|
||||
if (task.status === "blocked" || task.executionBlocker || task.activeRecoveryOwner === "board"
|
||||
|| (interruption && (interruption.state === "failed" || interruption.state === "waiting_for_user"))) {
|
||||
const why = base.reason ?? "blocked with no named reason";
|
||||
const note = interruption?.responsibility === "user" ? " (the board asks a person, but this is not one of the grade's person reasons)" : "";
|
||||
return verdict("blocked_non_person", `${task.status}: ${why}${note}${staleNote}`);
|
||||
}
|
||||
return verdict("open_no_owner", `${task.status} with no live run, no queued wake and no named reason${staleNote}`);
|
||||
}
|
||||
|
||||
export interface SwapWindow {
|
||||
index: number;
|
||||
requestedAt: number;
|
||||
spawnedAt: number | null;
|
||||
stopRequestedAt: number | null;
|
||||
stoppedAt: number | null;
|
||||
fromPid: number | null;
|
||||
}
|
||||
|
||||
export interface SwapRunFact {
|
||||
runId: string;
|
||||
issueId: string | null;
|
||||
status: string;
|
||||
errorCode: string | null;
|
||||
startedAt: number | null;
|
||||
finishedAt: number | null;
|
||||
ownerPidBeforeEnd: number | null;
|
||||
}
|
||||
|
||||
export const LOST_RUN_CODES: ReadonlySet<string> = new Set(["process_lost", "native_execution_ownership_unverified", "controller_lease_lost",
|
||||
"runner_remote_process_identity_unavailable", "native_continuation_requires_reconciliation", "native_session_interrupted"]);
|
||||
|
||||
export interface SwapMetrics {
|
||||
index: number;
|
||||
failedRuns: string[];
|
||||
lostWhileOldAlive: string[];
|
||||
survivors: Array<{ runId: string; nextEventAfterMs: number | null }>;
|
||||
nextEventP50Ms: number | null;
|
||||
nextEventMaxMs: number | null;
|
||||
survivorsWithoutEvent: number;
|
||||
}
|
||||
|
||||
function percentile(values: number[], fraction: number): number | null {
|
||||
if (!values.length) return null;
|
||||
const sorted = [...values].sort((left, right) => left - right);
|
||||
return sorted[Math.min(sorted.length - 1, Math.floor(fraction * sorted.length))]!;
|
||||
}
|
||||
|
||||
export function swapMetrics(swaps: readonly SwapWindow[], runs: readonly SwapRunFact[], firstEventAfterStop: ReadonlyMap<string, number>, waveEnd: number): SwapMetrics[] {
|
||||
return swaps.map((swap, position) => {
|
||||
const until = swaps[position + 1]?.requestedAt ?? waveEnd;
|
||||
const failedRuns = runs.filter(run => FAILED_RUN.has(run.status) && run.finishedAt !== null && run.finishedAt >= swap.requestedAt && run.finishedAt < until)
|
||||
.map(run => run.runId);
|
||||
const aliveUntil = swap.stopRequestedAt ?? swap.stoppedAt;
|
||||
const lostWhileOldAlive = runs.filter(run => run.errorCode !== null && LOST_RUN_CODES.has(run.errorCode) && run.finishedAt !== null
|
||||
&& swap.fromPid !== null && run.ownerPidBeforeEnd === swap.fromPid && aliveUntil !== null
|
||||
&& run.finishedAt >= (swap.spawnedAt ?? swap.requestedAt) && run.finishedAt < aliveUntil).map(run => run.runId);
|
||||
const stop = swap.stopRequestedAt;
|
||||
const survivors = stop === null ? [] : runs.filter(run => run.startedAt !== null && run.startedAt < stop
|
||||
&& (run.finishedAt === null || run.finishedAt > stop) && !FAILED_RUN.has(run.status) && run.status !== "cancelled")
|
||||
.map(run => {
|
||||
const at = firstEventAfterStop.get(`${swap.index}:${run.runId}`);
|
||||
return { runId: run.runId, nextEventAfterMs: at === undefined ? null : at - stop };
|
||||
});
|
||||
const delays = survivors.flatMap(survivor => survivor.nextEventAfterMs === null ? [] : [survivor.nextEventAfterMs]);
|
||||
return { index: swap.index, failedRuns, lostWhileOldAlive, survivors, nextEventP50Ms: percentile(delays, 0.5),
|
||||
nextEventMaxMs: delays.length ? Math.max(...delays) : null, survivorsWithoutEvent: survivors.length - delays.length };
|
||||
});
|
||||
}
|
||||
|
||||
export interface DuplicateFacts {
|
||||
overlappingRuns: Array<{ issueId: string; runIds: [string, string] }>;
|
||||
doubleExecutors: string[];
|
||||
duplicateComments: Array<{ issueId: string; count: number }>;
|
||||
duplicateChildren: Array<{ parentId: string; title: string; count: number }>;
|
||||
}
|
||||
|
||||
export function overlappingRuns(runs: ReadonlyArray<SwapRunFact>, waveEnd: number): DuplicateFacts["overlappingRuns"] {
|
||||
const byIssue = new Map<string, SwapRunFact[]>();
|
||||
for (const run of runs) {
|
||||
if (!run.issueId || run.startedAt === null) continue;
|
||||
byIssue.set(run.issueId, [...(byIssue.get(run.issueId) ?? []), run]);
|
||||
}
|
||||
const found: DuplicateFacts["overlappingRuns"] = [];
|
||||
for (const [issueId, list] of byIssue) {
|
||||
const sorted = [...list].sort((left, right) => left.startedAt! - right.startedAt!);
|
||||
const open: SwapRunFact[] = [];
|
||||
for (const next of sorted) {
|
||||
const still = open.filter(earlier => next.startedAt! < (earlier.finishedAt ?? waveEnd));
|
||||
for (const earlier of still) found.push({ issueId, runIds: [earlier.runId, next.runId] });
|
||||
open.splice(0, open.length, ...still, next);
|
||||
}
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
export interface ReleaseSwapResult {
|
||||
headline: string;
|
||||
completedWithoutHuman: number;
|
||||
started: number;
|
||||
byVerdict: Record<TaskVerdict, number>;
|
||||
byReason: Record<string, number>;
|
||||
byLabel: Record<string, number>;
|
||||
uncollected: number;
|
||||
stillRunning: number;
|
||||
drainReason: "quiet" | "cap" | "cancelled" | null;
|
||||
retry: { tasks: number; worked: number; didNotWork: number; pending: number };
|
||||
}
|
||||
|
||||
export function retryLabel(grade: TaskGrade): string | null {
|
||||
if (!grade.label) return null;
|
||||
if (grade.label !== "needed Retry") return grade.label;
|
||||
const worked = grade.retry?.worked;
|
||||
return `needed Retry (${worked === true ? "click worked" : worked === false ? "click did not work" : "click result pending"})`;
|
||||
}
|
||||
|
||||
export function summarizeGrades(grades: readonly TaskGrade[], uncollected = 0, drainReason: "quiet" | "cap" | "cancelled" | null = null): ReleaseSwapResult {
|
||||
const byVerdict = { pass: 0, needed_retry: 0, open_no_owner: 0, blocked_non_person: 0, waiting_on_person: 0, still_working: 0, cancelled: 0 } as Record<TaskVerdict, number>;
|
||||
for (const grade of grades) byVerdict[grade.verdict] += 1;
|
||||
const byReason: Record<string, number> = {};
|
||||
for (const grade of grades) if (grade.verdict !== "pass") {
|
||||
const key = `${grade.verdict}: ${grade.reason ?? "none"}`;
|
||||
byReason[key] = (byReason[key] ?? 0) + 1;
|
||||
}
|
||||
const byLabel: Record<string, number> = {};
|
||||
for (const grade of grades) {
|
||||
const key = retryLabel(grade);
|
||||
if (key) byLabel[key] = (byLabel[key] ?? 0) + 1;
|
||||
}
|
||||
const retried = grades.filter(grade => grade.retry);
|
||||
const completedWithoutHuman = byVerdict.pass;
|
||||
const started = grades.length - byVerdict.still_working;
|
||||
const running = grades.filter(grade => grade.verdict === "still_working");
|
||||
const ages = running.flatMap(grade => grade.lastActivityAgeMs === null ? [] : [`${Math.round(grade.lastActivityAgeMs / 1000)} s`]).join(", ");
|
||||
const when = drainReason === "cap" ? "at the drain cap" : drainReason === "cancelled" ? "when the harness was interrupted" : "at the end-of-wave read";
|
||||
const past = running.length ? `; ${running.length} still running ${when}, not counted${ages ? ` (last activity ${ages} ago)` : ""}` : "";
|
||||
const incomplete = uncollected ? ` (incomplete: ${uncollected} task(s) could not be read and are not counted)` : "";
|
||||
return { headline: `${completedWithoutHuman} / ${started} tasks completed without human action${past}${incomplete}`, completedWithoutHuman, started,
|
||||
stillRunning: running.length, drainReason,
|
||||
byVerdict, byReason, byLabel, uncollected, retry: { tasks: retried.length, worked: retried.filter(grade => grade.retry!.worked === true).length,
|
||||
didNotWork: retried.filter(grade => grade.retry!.worked === false).length, pending: retried.filter(grade => grade.retry!.worked === null).length } };
|
||||
}
|
||||
|
||||
export function releaseSwapMarkdown(result: ReleaseSwapResult, metrics: readonly SwapMetrics[], evidence: ReadonlyArray<{ index: number; lateOk: boolean; prepOk: boolean; findings: string[] } | null>,
|
||||
duplicates: DuplicateFacts, grades: readonly TaskGrade[]): string {
|
||||
const seconds = (ms: number | null) => ms === null ? "–" : `${Math.round(ms / 1000)} s`;
|
||||
const lines = [
|
||||
result.headline,
|
||||
"",
|
||||
`Verdicts: ${Object.entries(result.byVerdict).filter(([, count]) => count).map(([verdict, count]) => `${verdict} ${count}`).join(", ") || "none"}.`,
|
||||
`Failing: ${Object.entries(result.byLabel).map(([label, count]) => `${label} ${count}`).join("; ") || "none"}.`,
|
||||
`Reasons: ${Object.entries(result.byReason).map(([reason, count]) => `${reason} ${count}`).join("; ") || "none"}.`,
|
||||
`Retry: ${result.retry.tasks} task(s) needed it; worked ${result.retry.worked}, did not work ${result.retry.didNotWork}, still running ${result.retry.pending}.`,
|
||||
`Duplicates: ${duplicates.overlappingRuns.length} overlapping run pair(s), ${duplicates.doubleExecutors.length} double executor(s), `
|
||||
+ `${duplicates.duplicateComments.length} duplicated agent comment(s), ${duplicates.duplicateChildren.length} duplicated subtask title(s).`,
|
||||
"",
|
||||
"| Swap | Failed runs | Lost while old alive | Survivors | Next event p50 | Next event max | No event | Late start on old | Prep during recovery |",
|
||||
"| --- | --- | --- | --- | --- | --- | --- | --- | --- |",
|
||||
...metrics.map((metric, position) => {
|
||||
const swap = evidence[position];
|
||||
return `| ${metric.index} | ${metric.failedRuns.length} | ${metric.lostWhileOldAlive.length} | ${metric.survivors.length} | ${seconds(metric.nextEventP50Ms)} | `
|
||||
+ `${seconds(metric.nextEventMaxMs)} | ${metric.survivorsWithoutEvent} | ${swap ? (swap.lateOk ? "yes" : "no") : "–"} | ${swap ? (swap.prepOk ? "yes" : "no") : "–"} |`;
|
||||
}),
|
||||
"",
|
||||
...evidence.flatMap(swap => swap?.findings.map(finding => `- Swap ${swap.index}: ${finding}`) ?? []),
|
||||
...grades.filter(grade => grade.verdict !== "pass").map(grade => `- ${grade.identifier ?? grade.issueId} ${retryLabel(grade) ?? grade.verdict}: ${grade.detail}`),
|
||||
];
|
||||
return `${lines.join("\n")}\n`;
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { parseCommand } from "./config.js";
|
||||
import { planWave } from "./scheduler.js";
|
||||
import { OVERLAP_LAUNCH_READY_MARGIN_MS, ReleaseSwapRun, RELEASE_SWAP_DEFAULTS, RELEASE_SWAP_SHORT, expandReleaseSwapArgs, overlapLaunchExtensionMs, planReleaseSwaps, swapDueAt } from "./release-swap-schedule.js";
|
||||
|
||||
describe("release swap options", () => {
|
||||
it("expands the defaults into a storm of overlapping K15 swaps", () => {
|
||||
const { args, releaseSwap } = expandReleaseSwapArgs(["--env", "local", "--release-swap"]);
|
||||
expect(releaseSwap).toEqual(RELEASE_SWAP_DEFAULTS);
|
||||
expect(releaseSwap).toMatchObject({ swaps: 6, spacingMs: 40 * 60_000, overlapMs: 510_000, settleMs: 15 * 60_000 });
|
||||
expect(args).toEqual(["--env", "local", "--storm-minutes", "240", "--faults", "K15", "--k15-overlap-seconds", "510"]);
|
||||
});
|
||||
|
||||
it("takes the smoke shape and the staging volume-free shape on Daytona", () => {
|
||||
const { args, releaseSwap } = expandReleaseSwapArgs(["--env=daytona", "--release-swap", "--release-swap-count", "2",
|
||||
"--release-swap-spacing-minutes=5", "--release-swap-overlap-seconds", "90", "--release-swap-settle-minutes", "2"]);
|
||||
expect(releaseSwap).toEqual({ swaps: 2, spacingMs: 300_000, overlapMs: 90_000, settleMs: 120_000 });
|
||||
expect(args).toEqual(["--env=daytona", "--storm-minutes", "10", "--faults", "K15", "--k15-overlap-seconds", "90", "--k15-fresh-home", "--ephemeral-server"]);
|
||||
});
|
||||
|
||||
it("takes the short hourly preset: 3 swaps 12 minutes apart with the full two-server overlap, and lets flags override it", () => {
|
||||
const { args, releaseSwap } = expandReleaseSwapArgs(["--env", "local", "--release-swap-short"]);
|
||||
expect(releaseSwap).toEqual({ swaps: 3, spacingMs: 720_000, overlapMs: 510_000, settleMs: 900_000 });
|
||||
expect(args).toEqual(["--env", "local", "--storm-minutes", "36", "--faults", "K15", "--k15-overlap-seconds", "510"]);
|
||||
expect(expandReleaseSwapArgs(["--release-swap-short", "--release-swap-settle-minutes", "5"]).releaseSwap).toMatchObject({ swaps: 3, settleMs: 300_000 });
|
||||
expect(() => expandReleaseSwapArgs(["--release-swap-short", "--release-swap-overlap-seconds", "601"])).toThrow(/between 30 and 600/);
|
||||
expect(() => expandReleaseSwapArgs(["--release-swap-short", "--release-swap-spacing-minutes", "10"])).toThrow(/spacing/);
|
||||
const run = new ReleaseSwapRun(planReleaseSwaps(releaseSwap!), 0);
|
||||
expect(run.commit(run.due(720_000)!, 720_000)).toBe(true);
|
||||
expect(run.due(1_440_000)).toBeNull();
|
||||
run.settled(1_500_000);
|
||||
expect(run.due(1_500_000)).toBe(run.plan.faults[1]);
|
||||
});
|
||||
|
||||
it("refuses flags the cell plans itself, stray shape flags, and an overlap the spacing cannot hold", () => {
|
||||
expect(() => expandReleaseSwapArgs(["--release-swap", "--faults", "K1"])).toThrow(/--faults/);
|
||||
expect(() => expandReleaseSwapArgs(["--release-swap-count", "2"])).toThrow(/add --release-swap/);
|
||||
expect(() => expandReleaseSwapArgs(["--release-swap", "--release-swap-overlap-seconds", "601"])).toThrow(/between 30 and 600/);
|
||||
expect(() => expandReleaseSwapArgs(["--release-swap", "--release-swap-spacing-minutes", "9", "--release-swap-overlap-seconds", "510"])).toThrow(/spacing/);
|
||||
expect(expandReleaseSwapArgs(["--env", "local"])).toEqual({ args: ["--env", "local"] });
|
||||
});
|
||||
|
||||
it("parses into a storm wave whose tasks the swaps cross", () => {
|
||||
const command = parseCommand(["--env", "local", "--adapters", "codex", "--workloads", "A1", "--seed", "7", "--release-swap",
|
||||
"--release-swap-count", "2", "--release-swap-spacing-minutes", "5", "--release-swap-overlap-seconds", "90", "--tasks", "4"], 0);
|
||||
if (command.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(command.options).toMatchObject({ faults: ["K15"], k15Overlap: true, k15HoldMs: [90_000, 90_000], stormMinutes: 10,
|
||||
releaseSwap: { swaps: 2, spacingMs: 300_000, overlapMs: 90_000, settleMs: 15 * 60_000 } });
|
||||
const plan = planWave("w-20261112T000000-s7", command.options);
|
||||
expect(plan.storm).toEqual(planReleaseSwaps(command.options.releaseSwap!));
|
||||
expect(plan.tasks.every(task => task.fault === "K13")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("release swap scheduler", () => {
|
||||
const plan = planReleaseSwaps({ swaps: 3, spacingMs: 600_000, overlapMs: 90_000, settleMs: 60_000 });
|
||||
|
||||
it("fires each swap on the spacing clock from the wave start, never while one is unsettled", () => {
|
||||
const run = new ReleaseSwapRun(plan, 0);
|
||||
expect(run.due(599_999)).toBeNull();
|
||||
const first = run.due(600_000)!;
|
||||
expect(first).toMatchObject({ kind: "K15", holdMs: 90_000 });
|
||||
expect(run.commit(first, 600_000)).toBe(true);
|
||||
expect(run.due(1_300_000)).toBeNull();
|
||||
run.settled(800_000);
|
||||
expect(run.due(1_199_999)).toBeNull();
|
||||
expect(run.due(1_200_000)).toBe(plan.faults[1]);
|
||||
expect(swapDueAt(0, 600_000, 2)).toBe(1_800_000);
|
||||
});
|
||||
|
||||
it("keeps launching until the last swap settled, then drains", () => {
|
||||
const run = new ReleaseSwapRun(plan, 0);
|
||||
for (let index = 0; index < 3; index += 1) {
|
||||
const at = swapDueAt(0, 600_000, index);
|
||||
expect(run.mayLaunch(at)).toBe(true);
|
||||
expect(run.endIfDue(at)).toBe(false);
|
||||
expect(run.commit(run.due(at)!, at)).toBe(true);
|
||||
if (index < 2) run.settled(at + 200_000);
|
||||
}
|
||||
expect(run.endIfDue(5_000_000)).toBe(false);
|
||||
expect(run.exhaustedEarly(5_000_000)).toBe(true);
|
||||
run.settled(2_000_000);
|
||||
expect(run.due(9_000_000)).toBeNull();
|
||||
expect(run.endIfDue(2_000_001)).toBe(true);
|
||||
expect(run.mayLaunch(2_000_001)).toBe(false);
|
||||
expect(run.exhaustedEarly(2_000_001)).toBe(false);
|
||||
expect(run.endIfDue(2_000_002)).toBe(false);
|
||||
});
|
||||
|
||||
it("reports no task-pool shortfall once every swap settled, even before the drain is noticed", () => {
|
||||
const run = new ReleaseSwapRun(plan, 0);
|
||||
for (let index = 0; index < 3; index += 1) {
|
||||
const at = swapDueAt(0, 600_000, index);
|
||||
run.commit(run.due(at)!, at);
|
||||
run.settled(at + 100_000);
|
||||
}
|
||||
expect(run.ended).toBe(false);
|
||||
expect(run.exhaustedEarly(2_000_000)).toBe(false);
|
||||
});
|
||||
|
||||
it("extends a task launched during the overlap by the time it overlapped the swap, for the full and short presets", () => {
|
||||
for (const preset of [RELEASE_SWAP_DEFAULTS, RELEASE_SWAP_SHORT]) {
|
||||
const switchedAt = 1_000_000;
|
||||
const stoppedAt = switchedAt + preset.overlapMs + 2_000;
|
||||
expect(overlapLaunchExtensionMs(switchedAt, stoppedAt)).toBe(preset.overlapMs + 2_000 + OVERLAP_LAUNCH_READY_MARGIN_MS);
|
||||
expect(overlapLaunchExtensionMs(stoppedAt - 20_000, stoppedAt)).toBe(20_000 + OVERLAP_LAUNCH_READY_MARGIN_MS);
|
||||
expect(preset.overlapMs).toBeGreaterThanOrEqual(510_000);
|
||||
}
|
||||
});
|
||||
|
||||
it("does not commit a swap that is not due", () => {
|
||||
const run = new ReleaseSwapRun(plan, 0);
|
||||
expect(run.commit(plan.faults[0]!, 10)).toBe(false);
|
||||
expect(run.state.fired).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,145 @@
|
||||
import type { StormFault, StormPlan } from "./types.js";
|
||||
|
||||
export interface ReleaseSwapOptions {
|
||||
swaps: number;
|
||||
spacingMs: number;
|
||||
overlapMs: number;
|
||||
settleMs: number;
|
||||
drainMs?: number;
|
||||
}
|
||||
|
||||
export const RELEASE_SWAP_DEFAULTS: ReleaseSwapOptions = { swaps: 6, spacingMs: 40 * 60_000, overlapMs: 510_000, settleMs: 15 * 60_000 };
|
||||
export const RELEASE_SWAP_SHORT: ReleaseSwapOptions = { swaps: 3, spacingMs: 12 * 60_000, overlapMs: 510_000, settleMs: 15 * 60_000 };
|
||||
export const RELEASE_SWAP_BOOT_ALLOWANCE_MS = 120_000;
|
||||
export const RELEASE_SWAP_LATE_LEAD_MS = 20_000;
|
||||
|
||||
const FLAGS = {
|
||||
"release-swap-count": { key: "swaps", unit: 1, min: 1, max: 24 },
|
||||
"release-swap-spacing-minutes": { key: "spacingMs", unit: 60_000, min: 2, max: 120 },
|
||||
"release-swap-overlap-seconds": { key: "overlapMs", unit: 1_000, min: 30, max: 600 },
|
||||
"release-swap-settle-minutes": { key: "settleMs", unit: 60_000, min: 1, max: 60 },
|
||||
"release-swap-drain-minutes": { key: "drainMs", unit: 60_000, min: 1, max: 240 },
|
||||
} as const;
|
||||
|
||||
const REPLACED = ["storm-minutes", "faults", "k15-overlap-seconds", "k15-hold-seconds", "replay", "crash-point", "matrix", "upgrade-from"];
|
||||
|
||||
function flagName(arg: string): string | null {
|
||||
return arg.startsWith("--") ? arg.slice(2).split("=")[0]! : null;
|
||||
}
|
||||
|
||||
export function expandReleaseSwapArgs(input: readonly string[]): { args: string[]; releaseSwap?: ReleaseSwapOptions } {
|
||||
const rest: string[] = [];
|
||||
const values: Partial<Record<keyof typeof FLAGS, string>> = {};
|
||||
let enabled = false;
|
||||
let short = false;
|
||||
for (let index = 0; index < input.length; index += 1) {
|
||||
const arg = input[index]!;
|
||||
const name = flagName(arg);
|
||||
if (name === "release-swap" || name === "release-swap-short") {
|
||||
if (arg.includes("=")) throw new Error(`--${name} takes no value`);
|
||||
enabled = true;
|
||||
short ||= name === "release-swap-short";
|
||||
continue;
|
||||
}
|
||||
if (name && name in FLAGS) {
|
||||
const inline = arg.includes("=") ? arg.slice(arg.indexOf("=") + 1) : undefined;
|
||||
const value = inline ?? input[index + 1];
|
||||
if (inline === undefined) index += 1;
|
||||
if (value === undefined || value.startsWith("--")) throw new Error(`--${name} needs a value`);
|
||||
values[name as keyof typeof FLAGS] = value;
|
||||
continue;
|
||||
}
|
||||
rest.push(arg);
|
||||
}
|
||||
if (!enabled) {
|
||||
const stray = Object.keys(values);
|
||||
if (stray.length) throw new Error(`--${stray[0]} shapes the release swap; add --release-swap`);
|
||||
return { args: rest };
|
||||
}
|
||||
const given = new Set(rest.map(flagName).filter((name): name is string => name !== null));
|
||||
const clash = REPLACED.find(name => given.has(name));
|
||||
if (clash) throw new Error(`--release-swap plans its own K15 swaps; do not combine it with --${clash}`);
|
||||
const options: ReleaseSwapOptions = { ...(short ? RELEASE_SWAP_SHORT : RELEASE_SWAP_DEFAULTS) };
|
||||
for (const [flag, spec] of Object.entries(FLAGS) as Array<[keyof typeof FLAGS, (typeof FLAGS)[keyof typeof FLAGS]]>) {
|
||||
const raw = values[flag];
|
||||
if (raw === undefined) continue;
|
||||
if (!/^[0-9]+$/.test(raw)) throw new Error(`--${flag} must be a whole number`);
|
||||
const value = Number(raw);
|
||||
if (value < spec.min || value > spec.max) throw new Error(`--${flag} must be between ${spec.min} and ${spec.max}`);
|
||||
options[spec.key] = value * spec.unit;
|
||||
}
|
||||
if (options.spacingMs < options.overlapMs + RELEASE_SWAP_BOOT_ALLOWANCE_MS) {
|
||||
throw new Error(`--release-swap-spacing-minutes must leave the overlap plus ${RELEASE_SWAP_BOOT_ALLOWANCE_MS / 1000} s for the incoming server's boot`);
|
||||
}
|
||||
const envIndex = rest.findIndex(arg => arg === "--env" || arg.startsWith("--env="));
|
||||
const env = envIndex < 0 ? "local" : rest[envIndex]!.includes("=") ? rest[envIndex]!.split("=")[1] : rest[envIndex + 1];
|
||||
const added = ["--storm-minutes", String(Math.min(240, Math.ceil((options.swaps * options.spacingMs) / 60_000))), "--faults", "K15",
|
||||
"--k15-overlap-seconds", String(Math.round(options.overlapMs / 1000))];
|
||||
if (env === "daytona") {
|
||||
if (!given.has("k15-fresh-home")) added.push("--k15-fresh-home");
|
||||
if (!given.has("ephemeral-server")) added.push("--ephemeral-server");
|
||||
}
|
||||
return { args: [...rest, ...added], releaseSwap: options };
|
||||
}
|
||||
|
||||
export function planReleaseSwaps(options: ReleaseSwapOptions): StormPlan {
|
||||
const faults: StormFault[] = Array.from({ length: options.swaps }, () => ({ kind: "K15", holdMs: options.overlapMs, gapMs: options.spacingMs }));
|
||||
return { durationMs: options.swaps * options.spacingMs, faults };
|
||||
}
|
||||
|
||||
export function swapDueAt(startedAt: number, spacingMs: number, index: number): number {
|
||||
return startedAt + (index + 1) * spacingMs;
|
||||
}
|
||||
|
||||
export class ReleaseSwapRun {
|
||||
readonly state: { startedAt: number; fired: number; lastSettledAt: number | null };
|
||||
ended = false;
|
||||
private unsettled = false;
|
||||
|
||||
constructor(readonly plan: StormPlan, startedAt: number) {
|
||||
this.state = { startedAt, fired: 0, lastSettledAt: null };
|
||||
}
|
||||
|
||||
private get complete(): boolean {
|
||||
return this.state.fired >= this.plan.faults.length && !this.unsettled;
|
||||
}
|
||||
|
||||
endIfDue(_now: number): boolean {
|
||||
if (this.ended || !this.complete) return false;
|
||||
this.ended = true;
|
||||
return true;
|
||||
}
|
||||
|
||||
mayLaunch(_now: number): boolean {
|
||||
return !this.ended;
|
||||
}
|
||||
|
||||
due(now: number): StormFault | null {
|
||||
if (this.ended || this.unsettled) return null;
|
||||
const next = this.plan.faults[this.state.fired];
|
||||
if (!next) return null;
|
||||
return now >= swapDueAt(this.state.startedAt, next.gapMs, this.state.fired) ? next : null;
|
||||
}
|
||||
|
||||
commit(fault: StormFault, now: number): boolean {
|
||||
if (this.due(now) !== fault) return false;
|
||||
this.state.fired += 1;
|
||||
this.unsettled = true;
|
||||
return true;
|
||||
}
|
||||
|
||||
settled(now: number) {
|
||||
this.state.lastSettledAt = now;
|
||||
this.unsettled = false;
|
||||
}
|
||||
|
||||
exhaustedEarly(_now: number): boolean {
|
||||
return !this.complete;
|
||||
}
|
||||
}
|
||||
|
||||
export const OVERLAP_LAUNCH_READY_MARGIN_MS = 30_000;
|
||||
|
||||
export function overlapLaunchExtensionMs(launchedAt: number, stoppedAt: number): number {
|
||||
return Math.max(0, stoppedAt - launchedAt) + OVERLAP_LAUNCH_READY_MARGIN_MS;
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { rm } from "node:fs/promises";
|
||||
import { promisify } from "node:util";
|
||||
import type { ChaosApi } from "./api.js";
|
||||
import { PREP_PROBE_CAP, PROBE_AGENT_CONCURRENCY, ReleaseSwapDriver, hireProbeAgents, homeCleanupPlugin, pgRunReader } from "./release-swap-driver.js";
|
||||
import { ReleaseSwapRun, overlapLaunchExtensionMs, type ReleaseSwapOptions } from "./release-swap-schedule.js";
|
||||
import { finishReleaseSwap, type ReleaseSwapFinishInput } from "./release-swap-finish.js";
|
||||
import { StormRun } from "./scheduler.js";
|
||||
import { setOverlapProbe } from "./staging-mix.js";
|
||||
import type { ChaosSupervisor } from "./supervisor.js";
|
||||
import type { TaskObserver } from "./observer.js";
|
||||
import { waitForAgentsReady, type ChaosLab } from "./seed.js";
|
||||
import type { Timeline } from "./timeline.js";
|
||||
import type { AdapterId, EnvironmentKind, WavePlan } from "./types.js";
|
||||
import { createDb, type Db } from "../../packages/db/src/index.js";
|
||||
|
||||
const run = promisify(execFile);
|
||||
|
||||
export const DAYTONA_PROBE_SHAPE = { concurrency: 4, prepIntervalMs: 20_000, prepProbeCap: 8 };
|
||||
export const LOCAL_PROBE_SHAPE = { concurrency: PROBE_AGENT_CONCURRENCY, prepIntervalMs: 3_000, prepProbeCap: PREP_PROBE_CAP };
|
||||
|
||||
export async function diskUsageKb(directory: string): Promise<number | null> {
|
||||
try {
|
||||
const { stdout } = await run("du", ["-sk", directory], { timeout: 60_000 });
|
||||
return Number(stdout.trim().split(/\s+/)[0]);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function startReleaseSwap(input: { options: ReleaseSwapOptions; env: EnvironmentKind; adapters: readonly AdapterId[]; lab: ChaosLab;
|
||||
api: ChaosApi; db: Db; timeline: Timeline; temporaryRoot: string; freshHome: boolean; cancelled: () => boolean }): Promise<ReleaseSwapDriver> {
|
||||
const shape = input.env === "daytona" ? DAYTONA_PROBE_SHAPE : LOCAL_PROBE_SHAPE;
|
||||
const record = (kind: string, data: Record<string, unknown>) => { input.timeline.add(kind, data); };
|
||||
const agents = Object.fromEntries(input.adapters.flatMap(adapter => input.lab.agents[adapter] ? [[adapter, input.lab.agents[adapter]]] : []));
|
||||
const probeAgents = await hireProbeAgents(input.api, input.lab.companyId, agents, shape.concurrency);
|
||||
await waitForAgentsReady(input.api, probeAgents, input.timeline, input.env === "daytona" ? 10 * 60_000 : undefined, 1_000, input.cancelled);
|
||||
record("release_swap_probe_agents", { agents: Object.fromEntries(Object.entries(probeAgents).map(([adapter, agent]) => [adapter, agent!.id])),
|
||||
maxConcurrentRuns: shape.concurrency, prepIntervalMs: shape.prepIntervalMs, prepProbeCap: shape.prepProbeCap });
|
||||
const driver = new ReleaseSwapDriver({ options: input.options, companyId: input.lab.companyId, adapters: input.adapters, agents: input.lab.agents,
|
||||
probeAgents, prepIntervalMs: shape.prepIntervalMs, prepProbeCap: shape.prepProbeCap, reader: pgRunReader(input.db), record });
|
||||
driver.plugins.push(homeCleanupPlugin({ temporaryRoot: input.temporaryRoot, freshHome: input.freshHome,
|
||||
remove: directory => rm(directory, { recursive: true, force: true }), diskUsageKb, record }));
|
||||
return driver;
|
||||
}
|
||||
|
||||
export function stormRunFor(plan: WavePlan, now: number): StormRun | ReleaseSwapRun | null {
|
||||
if (!plan.storm) return null;
|
||||
return plan.options.releaseSwap ? new ReleaseSwapRun(plan.storm, now) : new StormRun(plan.storm, now);
|
||||
}
|
||||
|
||||
export class ReleaseSwapWave {
|
||||
private constructor(readonly driver: ReleaseSwapDriver, private readonly db: Db, private readonly plan: WavePlan, private readonly lab: ChaosLab,
|
||||
private readonly api: ChaosApi, private readonly timeline: Timeline, private readonly supervisor: Pick<ChaosSupervisor, "serverWindows">,
|
||||
private readonly cancelled: () => boolean) {}
|
||||
|
||||
static async start(input: { plan: WavePlan; lab: ChaosLab; api: ChaosApi; timeline: Timeline; databaseUrl: string;
|
||||
supervisor: Pick<ChaosSupervisor, "serverWindows" | "swapHook" | "paths">; cancelled: () => boolean }): Promise<ReleaseSwapWave> {
|
||||
const options = input.plan.options;
|
||||
const db = createDb(input.databaseUrl);
|
||||
const driver = await startReleaseSwap({ options: options.releaseSwap!, env: options.env, adapters: options.adapters, lab: input.lab, api: input.api, db,
|
||||
timeline: input.timeline, temporaryRoot: input.supervisor.paths.temporaryRoot, freshHome: options.k15FreshHome === true, cancelled: input.cancelled });
|
||||
input.supervisor.swapHook = driver.hook;
|
||||
setOverlapProbe(() => driver.launchOpen);
|
||||
const wave = new ReleaseSwapWave(driver, db, input.plan, input.lab, input.api, input.timeline, input.supervisor, input.cancelled);
|
||||
driver.plugins.push({ name: "overlap-launch-deadlines", onPhase: async (_swap, info) => { if (info.phase === "stopped") wave.extendOverlapLaunches(info.at); } });
|
||||
return wave;
|
||||
}
|
||||
|
||||
private overlapLaunches: Array<{ observer: Pick<TaskObserver, "extendDeadline" | "plan">; at: number }> = [];
|
||||
|
||||
noteLaunch(observer: Pick<TaskObserver, "extendDeadline" | "plan">, at = Date.now()): void {
|
||||
if (this.driver.launchOpen) this.overlapLaunches.push({ observer, at });
|
||||
}
|
||||
|
||||
extendOverlapLaunches(stoppedAt: number): void {
|
||||
for (const { observer, at } of this.overlapLaunches.splice(0)) {
|
||||
const ms = overlapLaunchExtensionMs(at, stoppedAt);
|
||||
observer.extendDeadline(ms);
|
||||
this.timeline.add("release_swap_deadline_extended", { taskId: observer.plan.taskId, ms });
|
||||
}
|
||||
}
|
||||
|
||||
async finish(input: { board: ReleaseSwapFinishInput["board"]; observers: readonly TaskObserver[]; launched: readonly TaskObserver[]; waveDir: string }): Promise<boolean> {
|
||||
const options = this.plan.options.releaseSwap!;
|
||||
try {
|
||||
const graded = await finishReleaseSwap({ driver: this.driver, settleMs: options.settleMs, companyId: this.lab.companyId, api: this.api, db: this.db, board: input.board,
|
||||
drainCapMs: options.drainMs ?? Math.max(0, ...input.observers.map(observer => observer.definition.budgetMs)) + options.settleMs,
|
||||
timeline: this.timeline.entries, issueIdForTask: new Map(input.launched.flatMap(observer => observer.record.issueId ? [[observer.plan.taskId, observer.record.issueId] as const] : [])),
|
||||
serverWindows: this.supervisor.serverWindows, waveDir: input.waveDir, cancelled: this.cancelled, record: (kind, data) => { this.timeline.add(kind, data); } });
|
||||
return graded.pass;
|
||||
} finally {
|
||||
setOverlapProbe(null);
|
||||
await this.db.$client.end({ timeout: 5 }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js";
|
||||
import { isMultiServerFault, multiServerHoldMs } from "./multi-server.js";
|
||||
import { isRepeatedRestartFault, planRestartSteps, type RestartStep } from "./repeated-restarts.js";
|
||||
import { k15HoldMs } from "./redeploy.js";
|
||||
import { planReleaseSwaps } from "./release-swap-schedule.js";
|
||||
import { CR5_FIRE_WINDOW_MS, classRHoldMs, isClassRFault } from "./class-r.js";
|
||||
import { type AdapterId, type FaultKind, type ScheduledFault, type StormFault, type StormPlan, type TaskFaultKind, type TaskPlan, type TriggerId, type WavePlan, type WaveOptions, type WorkloadId } from "./types.js";
|
||||
import type { ReplayRef } from "./config.js";
|
||||
@@ -132,7 +133,8 @@ export function planWave(waveId: string, options: WaveOptions, createdAt = new D
|
||||
createdAt: createdAt.toISOString(), options, tasks: withProviderFaultRules(planTasks(options), options)
|
||||
// `--long-turn-minutes` is part of each task's plan, so a replay keeps it.
|
||||
.map(task => options.longTurnMinutes ? { ...task, longTurnMinutes: options.longTurnMinutes } : task),
|
||||
...(options.stormMinutes ? { storm: planStorm(options.seed, options.stormMinutes, options.faults, options.k4HoldMaxMs, options.k15HoldMs) } : {}) };
|
||||
...(options.stormMinutes ? { storm: options.releaseSwap ? planReleaseSwaps(options.releaseSwap)
|
||||
: planStorm(options.seed, options.stormMinutes, options.faults, options.k4HoldMaxMs, options.k15HoldMs) } : {}) };
|
||||
}
|
||||
|
||||
/** K13: the gap between one storm fault settling and the next one firing. */
|
||||
|
||||
@@ -42,6 +42,7 @@ import { isMultiServerFault, runMultiServerFault, type MultiServerKind, type Mul
|
||||
import { generationHome, nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs, redeployReceipt, type RedeployIdentity } from "./redeploy.js";
|
||||
import { startEdgeProxy, type EdgeProxy } from "./edge-proxy.js";
|
||||
import { ephemeralServerEnvironment } from "./ephemeral-server.js";
|
||||
import type { SwapPhase, SwapPhaseInfo } from "./release-swap-driver.js";
|
||||
import { CLASS_R_PRE_FAULT_DEADLINE_MS, classRRedeploys, isClassRFault, runPluginStall, withReplacementPluginStall, type ClassRHooks } from "./class-r.js";
|
||||
|
||||
/** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */
|
||||
@@ -172,9 +173,9 @@ export class ChaosSupervisor {
|
||||
* before the next deploy starts, while the server is down. It must not throw.
|
||||
*/
|
||||
redeployEvidence: ((faultId: string, phase: "stopped" | "before_start") => Promise<void>) | null = null;
|
||||
/** K15 deploy shape (redeploy.ts `parseK15DeployShape`); main.ts sets it before `start`. */
|
||||
deployShape: { overlap: boolean; freshHome: boolean } = { overlap: false, freshHome: false };
|
||||
ephemeralServer = false;
|
||||
swapHook: ((info: SwapPhaseInfo) => Promise<string | void>) | null = null;
|
||||
/** `--k15-fresh-home`: the instance home and config of the serving generation (generation 0 uses `paths`). */
|
||||
private serverHome: { home: string; configPath: string };
|
||||
/** `--k15-overlap-seconds`: the edge on the wave's port, and the serving generation's own port behind it. */
|
||||
@@ -518,7 +519,6 @@ export class ChaosSupervisor {
|
||||
}
|
||||
if (kind === "TM1" || kind === "TM2") await this.freeze(kind, previous, holdMs, receipt);
|
||||
else if (isMultiServerFault(kind)) await this.multiServer(kind, previous, holdMs, receipt);
|
||||
// `--k15-overlap-seconds`: the next deploy boots beside the old server; its hold is the overlap.
|
||||
else if (kind === "K15" && this.deployShape.overlap) await this.rollingRedeploy(previous, holdMs, receipt);
|
||||
else if (kind === "CR3" || kind === "CR5") {
|
||||
const stall = await runPluginStall({ serverPid: previous.pid, holdMs, cancelled: () => this.shuttingDown,
|
||||
@@ -798,20 +798,18 @@ export class ChaosSupervisor {
|
||||
return { home, configPath };
|
||||
}
|
||||
|
||||
/**
|
||||
* `--k15-overlap-seconds`: the staging deploy shape. The next generation boots beside the old server on its own
|
||||
* port; once it is ready, the edge sends new connections to it; the old server keeps running for the overlap and
|
||||
* then gets SIGTERM. There is no outage.
|
||||
*/
|
||||
private async rollingRedeploy(previous: ChildProcess, overlapMs: number, receipt: FaultReceipt): Promise<void> {
|
||||
if (!this.edge) throw new Error("--k15-overlap-seconds needs the edge proxy started with the wave");
|
||||
const fromPort = this.backendPort;
|
||||
const swap = async (phase: SwapPhase, next?: ChildProcess) => this.swapHook?.({ faultId: receipt.faultId, phase, at: Date.now(), overlapMs, fromUrl: `http://127.0.0.1:${fromPort}`,
|
||||
toUrl: phase === "before_spawn" ? null : `http://127.0.0.1:${this.backendPort}`, fromPid: previous.pid ?? null, toPid: next?.pid ?? null });
|
||||
const nextRoot = await swap("before_spawn"); if (typeof nextRoot === "string") this.serverRoot = nextRoot;
|
||||
receipt.redeploy = await this.advanceDeploy(receipt.faultId);
|
||||
receipt.restartedAt = Date.now();
|
||||
this.timeline.add("fault_restarting", { faultId: receipt.faultId, overlapMs });
|
||||
const next = await this.spawnServer();
|
||||
// Bound to the next generation: the old server keeps serving during the overlap and must not count hits.
|
||||
this.armPendingRecoveryStep(receipt, next.pid);
|
||||
await swap("spawned", next);
|
||||
const backendUrl = `http://127.0.0.1:${this.backendPort}`;
|
||||
receipt.healthyAt = await this.waitForHealth(next, receipt.phases, backendUrl);
|
||||
const deadline = Date.now() + READY_TIMEOUT_MS;
|
||||
@@ -823,10 +821,12 @@ export class ChaosSupervisor {
|
||||
}
|
||||
this.edge.target = this.backendPort;
|
||||
this.timeline.add("k15_traffic_switched", { faultId: receipt.faultId, fromPort, toPort: this.backendPort, overlapMs });
|
||||
await swap("switched", next);
|
||||
for (const until = Date.now() + overlapMs; Date.now() < until && !this.shuttingDown;) await delay(Math.min(1_000, until - Date.now()));
|
||||
await this.stopper(previous, "SIGTERM");
|
||||
await swap("stopping", next); await this.stopper(previous, "SIGTERM");
|
||||
receipt.stoppedAt = Date.now();
|
||||
this.timeline.add("fault_server_down", { faultId: receipt.faultId, previousPid: previous.pid, overlapMs });
|
||||
await swap("stopped", next);
|
||||
await this.redeployEvidence?.(receipt.faultId, "stopped");
|
||||
}
|
||||
|
||||
|
||||
@@ -89,6 +89,7 @@ export interface WaveOptions {
|
||||
/** `--k15-fresh-home`: each deploy generation gets a new, empty instance home and no volume (redeploy.ts). */
|
||||
k15FreshHome?: boolean;
|
||||
ephemeralServer?: boolean;
|
||||
releaseSwap?: import("./release-swap-schedule.js").ReleaseSwapOptions;
|
||||
runnerBindFault?: boolean;
|
||||
/** `--faults K0`: cells with no fault, for workloads whose shape is the fault (W2, W2Q, W2C). */
|
||||
baseline?: boolean;
|
||||
|
||||
Reference in new issue
Block a user