From 17b9e1025230a7ab3356cb4479a99c79542a75cc Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 7 Nov 2026 19:55:02 -0600 Subject: [PATCH] docs(plans): Phase 2 reconciler data; let chaos K4 holds reach 600 s (#19927) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Control-plane restarts leave in-progress records that each subsystem releases in its own way. The durable-ownership plan replaces those paths with one shared path. > - Phase 1 added a report-only reconciler. Its light local waves showed zero violations, and that proves little. > - Phase 2 moves kinds to the shared path. It needs data that shows which kinds really leak, and the speed numbers to protect. > - This pull request records that data from harsher waves. It also fixes the harness so that K4 outages past the reconnect grace qualify. > - The benefit is that Phase 2 moves kinds in order of measured risk, against a known speed baseline. ## Linked Issues or Issue Description Refs #19831 (durable-ownership plan), #19861 (report-only reconciler), #19863 (speed budgets), #19797 (K4 hold cap). **What is the problem?** Phase 2 needs per-kind invariant data and speed data from harsh restart waves. Only light local waves existed. In addition, a K4 hold over about 405 s always failed its fault receipt, so the 600 s cap from #19797 could not be used. **Where is it documented?** `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` (Phase 2 rows) and `tests/restart-chaos/README.md`. **What should change?** Record the data in a dated plan doc. Let chaos K4 holds run up to 600 s and still qualify. ## What Changed - Added `doc/plans/2026-11-07-phase2-reconciler-data.md`. It covers 5 unmasked local waves (two storms, a crash loop, two K4 runs). It gives per-kind violation counts with example rows, owner verdicts, TTFT, boot and restart recovery against the budgets, Phase 3 gate notes, and what is not covered (Daytona). - Added a short pointer to that doc in the durable-ownership plan. - `tests/restart-chaos`: new flag `--k4-hold-max-seconds` (120–600). It raises the upper end of the seeded K4 hold. The default range (120–300 s) does not change. - `tests/runner-e2e/controller-outage.ts`: the receipt's ready bound is now a parameter (`CONTROLLER_READY_BY_MS`, default 420 s). Runner suites keep 420 s. The chaos supervisor passes hold + 420 s, so a hold past the Runner's reconnect grace qualifies. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 12 files, 135 tests pass, including the new flag test. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/controller-outage.test.ts`: 36 pass, including the new long-hold receipt test. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and `tests/runner-e2e`: pass. - Live: the K4 wave at seed 9203 failed 3 of 6 tasks with the old bound (fault error "did not produce a qualifying replacement receipt"). It passed 6 of 6 with this fix, with holds of 294–579 s. ## Risks - Low risk. The changes are test harness code and docs only. The runner suites keep the 420 s ready bound by default. - The data is local only. The doc says which kinds the waves do not reach. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). The model ran the chaos waves and wrote the analysis and code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge πŸ€– Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...07-durable-ownership-and-reconciliation.md | 4 + .../2026-11-07-phase2-reconciler-data.md | 175 ++++++++++++++++++ tests/restart-chaos/README.md | 3 +- tests/restart-chaos/config.ts | 6 +- tests/restart-chaos/scheduler.test.ts | 14 ++ tests/restart-chaos/scheduler.ts | 17 +- tests/restart-chaos/supervisor.ts | 6 +- tests/restart-chaos/types.ts | 2 + tests/runner-e2e/controller-outage.test.ts | 7 + tests/runner-e2e/controller-outage.ts | 13 +- 10 files changed, 236 insertions(+), 11 deletions(-) create mode 100644 doc/plans/2026-11-07-phase2-reconciler-data.md diff --git a/doc/plans/2026-11-07-durable-ownership-and-reconciliation.md b/doc/plans/2026-11-07-durable-ownership-and-reconciliation.md index 9aaf87c9c03..8032c06acfc 100644 --- a/doc/plans/2026-11-07-durable-ownership-and-reconciliation.md +++ b/doc/plans/2026-11-07-durable-ownership-and-reconciliation.md @@ -124,6 +124,10 @@ What the data does and does not show: - Track 1 now stamps the PID space on `native_run_finalizations` (#19868) and `heartbeat_runs` (#19871), and the registry reads those stamps. Rows written before them stay `unprovable`. Phase 2 needs the same stamp on every kind it moves before that kind can settle by proof. - Restart-crossing dispatches exceeded the TTFT and boot budgets because those metrics spanned the outage. The budget check now judges such dispatches only on restart-recovery latency. +### Phase 2 data: harsh local waves (2026-11-07) + +Storms, crash loops and outages up to 588 s, all unmasked, are in [`2026-11-07-phase2-reconciler-data.md`](2026-11-07-phase2-reconciler-data.md). Existing paths settle most run-lifecycle records within 45 s of `ready`. This is not a bound: one environment lease took 119 s, and some native claims stayed proven gone during a load spike. One leak was found: a claimed wake whose run ended without settling it. The unowned pending-cleanup queue and unstamped accounting rows are the main I1 findings. Daytona was not run in this pass (see the data doc). + ## Relationship to the zero-downtime upgrade work Separate work moves the control plane toward zero-downtime upgrades on an ephemeral filesystem. PostgreSQL holds execution authority and durable protocol state. Object storage holds workspace archives, provider checkpoints, logs, traces and skill assets. Replacement servers reuse stable encryption and signing secrets. Most of the foundations are on master; #19830 is still open. Entry points: diff --git a/doc/plans/2026-11-07-phase2-reconciler-data.md b/doc/plans/2026-11-07-phase2-reconciler-data.md new file mode 100644 index 00000000000..d7e0c023e42 --- /dev/null +++ b/doc/plans/2026-11-07-phase2-reconciler-data.md @@ -0,0 +1,175 @@ +# Phase 2 reconciler data: harsh local chaos waves + +Status: Phase 2, step 1 deliverable, 2026-11-07. +Plan: [`2026-11-07-durable-ownership-and-reconciliation.md`](2026-11-07-durable-ownership-and-reconciliation.md). +Inventory: [`2026-11-07-ownership-inventory.md`](2026-11-07-ownership-inventory.md). Budgets: `server/src/services/native-runtime/native-speed-budgets.ts` (#19863). + +## Summary + +Phase 1's light waves showed zero invariant violations. Five harsher local waves, all unmasked, change the picture in three ways: + +1. **The existing per-subsystem paths settle most run-lifecycle records within 45 s of `ready`, but this is not a bound.** Most owners proven gone were seen while the control plane was down or recovering, and they were settled before the first sampler pass more than 45 s after `ready`. There are three exceptions: + - one active environment lease of a terminal run settled 119 s after `ready`; + - in the s9204 storm, six native claims were still proven gone during the load-61 window; + - the leak in finding 2. + + The 20 s sampling interval limits all of these numbers. +2. **One kind leaks: a claimed wake whose run ended without settling it.** Native restart recovery blocked a run (`native_restart_recovery_blocked`, `execution_recovery_budget_exhausted`) and ended it, but left the run's assignment wake `claimed`. It stayed that way for more than 24 minutes, until the wave ended. Nothing releases it. Rule I2 caught it. This is the first Phase 2 settle target the data found. Its PR is listed under "Step 2" below. +3. **Unowned and unstamped records are the main I1 finding:** + - `environment_pending_cleanup` (61 records across the valid waves). These are cleanup jobs that are queued but not claimed, so they have no owner by design. + - `usage_receipt_spool` files (4 records). + - `run_accounting_debt` rows. They are always `unprovable`, because they carry no owner stamp. + +Long holds (K4 up to 579 s, so outages up to 588 s, past the Runner's 7-minute reconnect grace) all recovered: 6 of 6 tasks passed. Crash loops passed too (6 of 6). + +TTFT, boot and restart recovery stay within the p50 budgets for cold starts and for recovery. The exceptions are claude cold TTFT, at the budget (16.3–17.5 s against 16.5 s), and warm follow-ups on claude (about 3Γ—) and codex (slightly over). See "Speed". + +Daytona did not run in this pass. A follow-up adds one Daytona wave. See "Not covered". + +## Waves + +All waves ran local, unmasked, with `--keep`, on Apple M5 Max (18 cores) alongside other sessions. Every fault receipt records the host load (`loadAvg1m`). + +| Wave | Base | Shape | Graded | Crossed a fault | Passed | Faults | +|---|---|---|---|---|---|---| +| `w-20261107T221244-s9201` | `ee556a266d` + harness flag | K13 storm 45 min, 5 in flight, triggers `time,result` | 58 | 30 | 51 | 16 (K1 Γ—2, K2 Γ—10, K6 Γ—4), load 3.5–10.2 | +| `w-20261107T221303-s9202` | same | K6 crash loop, 6 tasks, 3 adapters | 6 | 5 | 6 | 5, load 4–10 | +| `w-20261107T221304-s9203` (first run) | same | K4, holds up to 600 s | 6 | 5 | 3 | 5. Invalid: harness defect, see below | +| `w-20261107T225750-s9203` (rerun) | `ee556a266d` + harness fix | K4, holds 294–579 s | 6 | 6 | 6 | 6, outages 306–588 s | +| `w-20261107T231436-s9204` | `94718ce168` + harness | K13 storm 40 min, 6 in flight, K4 holds up to 600 s mixed in | 42 | 31 | 36 | 13 (K1 Γ—5, K2 Γ—5, K4 Γ—2 at 421 and 215 s, K6 Γ—1) | + +Failure classes: + +- **Harness (T):** the first K4 run failed 3 tasks on R1. Its outage receipt required `ready` within 420 s of the stop, so every hold over about 405 s failed the fault. #19797 raised the hold cap to 600 s but never made it usable. The fix is in this PR: runner suites keep the 420 s bound, and chaos passes hold + 420 s. +- **Storm shape (not P):** 8 B5 tasks failed R1 across both storms. Each crossed 10–16 faults. A 4-minute `bench.sh` killed every 2–4 minutes cannot finish within its budget, and every one of these issues still ended `done`. This is the long-tool boundary (B6) under a storm, not an ownership record. +- **Agent quality (Q):** 3 T-oracle failures (`brief_contains_followup_marker`). Two crossed no fault. +- **Load (E):** in the s9204 storm, fault f12 fired at load 61 and f13 at load 24. f13's server did not become healthy within 180 s. My own concurrent test run caused the load. The last A1 R1 failure falls in this window, and so do the sampler's I2 hits on six native claims. +- **Product (P), new:** the stuck claimed wake (finding 2). + +## Per-kind data + +There are two sources: +- the product reconciler's `recovery.invariant_violation` rows in each kept database; +- a read-only sampler that ran the Phase 1 pass (`record: false`) against each wave's database every 20 s. + +The sampler counts in the second table skip passes that fall inside a fault window (from fault start to `ready` + 45 s), because nothing can settle while the server is down. + +Violations recorded by the reconciler, counted as distinct records: + +| Rule | Kind | s9201 | s9202 | s9203 rerun | s9204 | Example | Meaning | +|---|---|---|---|---|---|---|---| +| I1 | `environment_pending_cleanup` | 47 | 4 | 4 | 6 | `00cb373d…` (run `13d667d1…`) | Unclaimed cleanup obligations name no owner by design. A modelling gap, not a leak: every one reached `expired` within about 1 min. | +| I1 | `usage_receipt_spool` | 3 | 0 | 0 | 1 | `2357b5be…` | Spool files carry no owner stamp. | +| I2 | `wake_request_claimed` | 0 | 0 | 1 | 0 | wake `6b3c4cce…`, run `7c45860b…` | Run failed at 23:19:41 (`native_restart_recovery_blocked`). Wake still `claimed` at the end of the wave (23:44+). A real leak. | +| I4 | `wake_request` | 0 | 0 | 1 | 0 | `f7082368…` | False positive. The wake failed "Process lost… retrying once", and retry run `1820a14b…` delivered it and succeeded. The report's refusal regex matches "retry". | + +There were no I3 violations: no run held admission past the 10-minute accounting closure in any wave. There were no I7 violations. + +Owner verdicts from the sampler outside fault windows (record-passes): + +| Kind | Peak in progress | Gone | Alive | Unprovable (reason) | +|---|---|---|---|---| +| `environment_run_lease_active` | 56 | 1 (`owner_run_terminal`, settled 119 s after ready) | 3,636 | 0 | +| `issue_execution_lock` / `issue_checkout_lock` | 8 | 0 | 846 each | 0 | +| `wake_request_claimed` | 8 | 2 (the leak above) | 846 | 0 | +| `native_executor_claim` | 6 | 1 (load window) | 285 | 0 | +| `native_restart_recovery_claim` | 7 | 5 (load window) | 548 | 0 | +| `native_accepted_result_lease` | 1 | 0 | 3 | 0 | +| `run_accounting_debt` | 3 | 0 | 0 | 44 (`no_owner_stamp`) | +| `legacy_controller_lease` | 1 | 0 | 0 | 4 (`missing_identity`) | +| `environment_pending_cleanup` | 29 | 0 | 0 | 78 (`no_owner_stamp`) | + +Gone owners seen while the server was down or recovering, in record-passes over all waves: + +| Kind | Record-passes | +|---|---| +| `native_restart_recovery_claim` | 72 | +| `native_executor_claim` | 56 | +| `wake_request_claimed` | 61 (K4 rerun) | +| `native_accepted_result_lease` | 3 | + +Except for the claimed wake, one environment lease (119 s after `ready`) and the load window above, existing paths settled them before the first pass more than 45 s after `ready`. Pending-cleanup leases were retried and released 46–58 s after their run finished. That is far below the inventory's 5-minute age rule, which never fired. + +What this does and does not show: +- These waves reach the run-lifecycle kinds, the issue locks and the wake queue. They do not reach the remote, workspace-operation, sandbox or accounting kinds that hurt the overnight campaign most: teardown claims, create intents, writer lanes and the age-only list in the inventory. Those need Daytona waves or the crash-point suite. +- "Within 45 s" describes most records, with the exceptions above. It is a sampling observation (20 s passes), not a measured latency or a guarantee. + +## Speed against the #19863 budgets + +Runs that no fault crossed, merged over both storms (n in brackets), in seconds: + +| Adapter | Start | TTFT p50 (budget p50) | TTFT p95 (budget p95) | Boot p50 (budget p50) | Boot p95 (budget p95) | +|---|---|---|---|---|---| +| claude | cold | 16.3–17.5 [18] (16.5) | 23.5–30.8 (30.0) | 4.2 (9.0) | 4.8–6.8 (20.5) | +| claude | warm | 27.6–29.9 [13] (9.5) | 54.4–76.6 (28.5) | 1.8–2.0 (3.5) | 4.2–5.3 (16.5) | +| codex | cold | 4.8–4.9 [20] (6.0) | 6.2–7.6 (17.0) | 1.6–1.8 (2.5) | 3.1–3.6 (7.0) | +| codex | warm | 10.2–11.0 [12] (10.0) | 18.4–72.8 (28.5) | 2.5–2.6 (5.0) | 4.2–4.7 (16.5) | +| opencode | cold | 10.8–11.0 [23] (17.5) | 16.3–22.1 (49.0) | 6.7 (12.0) | 9.2–9.9 (37.0) | +| opencode | warm | 19.3 [2] (26.5) | 94.5 (88.0) | 6.1 (22.0) | 6.4 (65.5) | + +Ranges give the two storms' values. The crash-loop and outage waves had only 1–2 uncrossed runs per cell, all inside these ranges. + +Restart recovery, from `ready` to the recovered run's next provider event. A sample counts only when that event comes before the next fault starts. Otherwise the run has no next event for this fault, because a later event measures a later fault. + +| Adapter | p50 s1 / s2 (budget 15.0 / 6.5 / 25.5) | p95 s1 / s2 (budget 22.5 / 8.0 / 38.5) | Samples | +|---|---|---|---| +| claude | 5.0 / 4.2 | 9.4 / 5.4 | 21 / 17 | +| codex | 4.5 / 4.6 | 9.1 / 9.1 | 36 / 24 | +| opencode | 10.8 / 10.3 | 14.3 / 13.4 | 28 / 18 | + +Crash loop (s9202): claude 3.4 / 4.7 s (n 5), opencode 9.5 / 10.4 s (n 6); no codex run crossed a fault. + +K4 rerun (s9203, outages 306–588 s): of 10 runs in flight at a fault, only 2 emitted a provider event before the next fault: claude 5.8 s and opencode 10.8 s, both within budget. The other 8 (all 4 codex runs included) had no event before the next fault. After an outage past the reconnect grace, the run continues through a retry or a new run, not by reattaching, so this metric does not measure it. A first version of this table counted events after later faults, which inflated the s9204 p95 to 42–49 s. + +Findings: +- **Warm follow-up TTFT is over budget on claude (about 3Γ—) and slightly on codex.** In C1 the follow-up comment is posted while the first turn is still running. TTFT is anchored at message acceptance, so it includes waiting for the live turn to finish. The baseline measured idle follow-ups. This needs a TTFT anchor that separates "queued behind the live turn" from "turn start to first token" before it is called a regression. +- Codex restart-recovery p95 (9.1 s in both storms) is just over its 8.0 s budget, with 4 run-log breaches (max 9.4 s) in the first storm. +- Recovery after an outage past the reconnect grace needs its own metric: from `ready` to the first provider event of the run that continues the task (the retry), not of the interrupted run. +- No change in this phase adds work before `turn.start`. The settle paths run in restart recovery and in the background reconciler. + +## Step 2: kinds moved to the shared path + +Order of the moves, highest risk first: +- **Accepted-result executor lease** (`native_accepted_result_lease`). It is an accepted-result obligation, and the storm saw its owners proven gone. It had its own liveness proof (`native-dead-controller-result-lease.ts`, #19713), which the move deletes. PR: feat/ownership-settle-accepted-result-lease. +- **Claimed wake** (`wake_request_claimed`). This is the leak the data found. Settle copies the finished run's outcome onto the claim once the run, and for native runs its coordinator, is final. PR: feat/ownership-settle-claimed-wake. + +Both release only on proof: +- for the lease, `proveOwnerGone` returns gone; +- for the wake, the owner run's terminal row. + +An unprovable owner is never released. The lease kind keeps the one-release legacy fallback for rows saved before #19868, and logs each use (#19883). An ephemeral server asks the board to confirm the stop instead. + +## Phase 3 gates the data bears on + +Phase 3 is not started. It needs a separate go-ahead. + +- **`waitingRunCount` and `recoveringRunCount`** (budget-exposure.ts, budgets.ts). Across 112 graded, unmasked tasks in the valid waves, no run held admission past the accounting closure: there were no I3 violations, and accounting debt peaked at 3 rows and always cleared. Locally the data gives no evidence that these gates prevent anything. Removing them is the planned direction (decision 1), and these waves raise no objection. They do not cover remote accounting or late receipts. +- **Scope budget hold and the 15 s `budget_evaluation_work` gate.** These waves never exercised them; no run hit a hard stop. No evidence either way. +- **`hasPendingRunCleanup` successor block.** It fails closed when the run row is missing. No I7 violations were seen, so it never stranded a task in these waves. No evidence either way. + +## Not covered + +- **Daytona.** No signed runner image existed for the tested commit. + - An apparent content-ID mismatch with CI run 42422571919 was a misreading: that run was dispatched from master but built target commit `a37a2e4`, and its Daytona image step was skipped. + - Local content IDs reproduce CI's for the same commit; for example, `35d41f11` gives `11475796…` on both. + - A follow-up builds an `image_only` image and adds one Daytona wave. +- The remote, workspace-operation, sandbox and accounting kinds (see "Per-kind data"). +- Crash points that sit between two durable writes. Random waves hit 20 ms–1 s windows only by luck. The crash-point suite (Phase 5) is the tool for those. + +## Reproduce + +```sh +node scripts/agent-preflight.mjs +set -a; source ~/.secrets; set +a # in the harness command's own shell only +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude,opencode \ + --concurrency 6 --storm-minutes 40 --triggers time,result --k4-hold-max-seconds 600 --seed 9204 --keep +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude,opencode \ + --faults K4 --k4-hold-max-seconds 600 --tasks 6 --concurrency 3 --seed 9203 --keep +``` + +To count violations, copy the kept instance's database and query it: + +```sql +select details->>'rule', details->>'kind', count(distinct entity_id) +from activity_log where action = 'recovery.invariant_violation' group by 1, 2; +``` diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 6238aedc3a4..04e8f0065bf 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -23,7 +23,8 @@ node scripts/typecheck-test-projects.mjs tests/restart-chaos # typecheck Options: `--triggers time,comment,retry,queued,tool,result` (fault timing, below), `--workloads A1,B1,B5,C1`, `--budget-cents` (company hard cap, -default 3000), `--poll-ms` (2000–5000), `--keep` (keep the temporary instance). +default 3000), `--poll-ms` (2000–5000), `--keep` (keep the temporary instance), +`--k4-hold-max-seconds` (raise the seeded K4 hold upper end from 300 s, up to the 600 s outage cap). `--rc06-masked` keeps the company budget amount but disables its hard stop on the isolated Chaos Lab company, so one stuck accounting row (RC-06) cannot diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts index 09a8859f9d1..11a724d478d 100644 --- a/tests/restart-chaos/config.ts +++ b/tests/restart-chaos/config.ts @@ -87,13 +87,14 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC "rc06-masked": { type: "boolean" }, "storm-minutes": { type: "string" }, lifecycle: { type: "string" }, + "k4-hold-max-seconds": { type: "string" }, }, }); const keep = values.keep === true; const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000); const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000); if (values.replay !== undefined) { - for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "storm-minutes", "lifecycle"] as const) { + for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "storm-minutes", "lifecycle", "k4-hold-max-seconds"] as const) { if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`); } return { mode: "replay", ref: parseReplayRef(values.replay), keep, @@ -131,6 +132,9 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC ...(env === "daytona" ? { daytonaImage, daytonaCpu: integer(values["daytona-cpu"], "daytona-cpu", DAYTONA_DEFAULT_CPU, 1, 4) } : {}), ...(values["rc06-masked"] ? { masks: ["rc06"] } : {}), ...(lifecycle ? { lifecycle } : {}), + // Seeded K4 hold upper end; the controller-outage cap is 600 s (#19797). + ...(values["k4-hold-max-seconds"] !== undefined + ? { k4HoldMaxMs: integer(values["k4-hold-max-seconds"], "k4-hold-max-seconds", 300, 120, 600) * 1000 } : {}), }; if (env !== "daytona" && values["daytona-cpu"] !== undefined) throw new Error("--daytona-cpu applies only to --env daytona"); if (options.daytonaCpu !== undefined) { diff --git a/tests/restart-chaos/scheduler.test.ts b/tests/restart-chaos/scheduler.test.ts index ccf644be452..473825a52ef 100644 --- a/tests/restart-chaos/scheduler.test.ts +++ b/tests/restart-chaos/scheduler.test.ts @@ -32,6 +32,20 @@ describe("scheduler determinism", () => { } }); + it("raises the K4 hold upper end with --k4-hold-max-seconds, up to the 600 s outage cap", () => { + const command = parseCommand(["--faults", "K4", "--k4-hold-max-seconds", "600", "--seed", "5"]); + if (command.mode !== "wave") throw new Error("expected a wave"); + expect(command.options.k4HoldMaxMs).toBe(600_000); + const holds = planTasks({ ...command.options, tasks: 40 }).map(task => task.holdMs); + expect(Math.min(...holds)).toBeGreaterThanOrEqual(K4_HOLD_RANGE_MS[0]); + expect(Math.max(...holds)).toBeLessThanOrEqual(600_000); + expect(Math.max(...holds)).toBeGreaterThan(K4_HOLD_RANGE_MS[1]); + const storm = planStorm(5, 120, ["K4"], 600_000); + expect(Math.max(...storm.faults.map(fault => fault.holdMs))).toBeGreaterThan(K4_HOLD_RANGE_MS[1]); + expect(() => parseCommand(["--k4-hold-max-seconds", "601"])).toThrow(/between 120 and 600/); + expect(() => parseCommand(["--replay", "w-20261106T101112-s42/t03", "--k4-hold-max-seconds", "400"])).toThrow(/do not combine/); + }); + it("covers every workload Γ— adapter Γ— fault cell once per cycle", () => { const cells = options.workloads.length * options.adapters.length * options.faults.length; const tasks = planTasks({ ...options, tasks: cells }); diff --git a/tests/restart-chaos/scheduler.ts b/tests/restart-chaos/scheduler.ts index c331e740dc1..9d3ed10700c 100644 --- a/tests/restart-chaos/scheduler.ts +++ b/tests/restart-chaos/scheduler.ts @@ -29,6 +29,13 @@ export function createRng(seed: number) { export const K4_HOLD_RANGE_MS: readonly [number, number] = [120_000, 300_000]; +/** The seeded K4 hold range; `--k4-hold-max-seconds` raises its upper end + * (up to the 600 s controller-outage cap) to reach the 7-minute reconnect + * grace and longer outages. The default range is unchanged. */ +export function k4HoldRange(maxMs?: number): readonly [number, number] { + return maxMs === undefined ? K4_HOLD_RANGE_MS : [K4_HOLD_RANGE_MS[0], Math.max(K4_HOLD_RANGE_MS[0], maxMs)]; +} + export const TRIGGER_DELAY_RANGE_MS: Record, readonly [number, number]> = { comment: [0, 5_000], retry: [0, 2_000], @@ -64,7 +71,7 @@ function stormCellFaults(options: Pick, * Pairwise-friendly sampling: cycle a seeded shuffle of the full * workload Γ— adapter Γ— fault product so small waves still spread across cells. */ -export function planTasks(options: Pick): TaskPlan[] { +export function planTasks(options: Pick): TaskPlan[] { const rng = createRng(options.seed); const triggers = options.triggers?.length ? options.triggers : ["time" as const]; for (const trigger of triggers) { @@ -95,7 +102,7 @@ export function planTasks(options: Pick>> 0); const durationMs = minutes * 60_000; @@ -136,7 +143,7 @@ export function planStorm(seed: number, minutes: number, faults: readonly FaultK for (let planned = 0; planned < durationMs;) { const gapMs = rng.int(...STORM_GAP_RANGE_MS); const kind = faults[rng.int(0, faults.length - 1)]!; - sequence.push({ kind, gapMs, holdMs: kind === "K4" ? rng.int(...K4_HOLD_RANGE_MS) : 0 }); + sequence.push({ kind, gapMs, holdMs: kind === "K4" ? rng.int(...k4HoldRange(k4HoldMaxMs)) : 0 }); planned += gapMs; } return { durationMs, faults: sequence }; diff --git a/tests/restart-chaos/supervisor.ts b/tests/restart-chaos/supervisor.ts index c27ab268a17..d6dfb874484 100644 --- a/tests/restart-chaos/supervisor.ts +++ b/tests/restart-chaos/supervisor.ts @@ -22,7 +22,7 @@ import os from "node:os"; import path from "node:path"; import { setTimeout as delay } from "node:timers/promises"; import { prepareControllerDatabase } from "../runner-e2e/controller-database.js"; -import { replaceController, type ControllerOutageReceipt } from "../runner-e2e/controller-outage.js"; +import { CONTROLLER_READY_BY_MS, replaceController, type ControllerOutageReceipt } from "../runner-e2e/controller-outage.js"; import { assertIsolatedServerEnvironment, buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment, resolvePaperclipRunnerBinaryForHarness, @@ -259,6 +259,10 @@ export class ChaosSupervisor { requestId: `chaos-${receipt.faultId}-${requestedBy}`, previousPid: previous.pid, outage: { signal: receipt.signal, holdMs }, + // A K4 hold may outlast the Runner's 7-minute reconnect grace on + // purpose (--k4-hold-max-seconds); the replacement still gets the + // normal startup allowance after the hold. + readyByMs: holdMs + CONTROLLER_READY_BY_MS, stop: async signal => { if (signal === "SIGKILL") await this.stopper.crashForRestart(previous); else await this.stopper(previous, "SIGTERM"); diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts index 02074096ae8..e1106d82eea 100644 --- a/tests/restart-chaos/types.ts +++ b/tests/restart-chaos/types.ts @@ -34,6 +34,8 @@ export interface WaveOptions { daytonaImage?: string; /** `--env daytona`: vCPU per sandbox (see seed.ts chaosDaytonaShape). */ daytonaCpu?: number; + /** `--k4-hold-max-seconds`: upper end of the seeded K4 hold (default 300 s, cap 600 s). */ + k4HoldMaxMs?: number; /** * Known product signatures this wave deliberately masks. "rc06" keeps the * company budget amount but disables its hard stop, so a stuck diff --git a/tests/runner-e2e/controller-outage.test.ts b/tests/runner-e2e/controller-outage.test.ts index f95778b617c..c2e1049ae99 100644 --- a/tests/runner-e2e/controller-outage.test.ts +++ b/tests/runner-e2e/controller-outage.test.ts @@ -105,6 +105,13 @@ describe("controller outage qualification", () => { signal: "SIGKILL", holdMs: 300000, previousPid: 1, replacementPid: 2, startedAt: new Date().toISOString(), stoppedAfterMs: 0, cacheDiscarded: true, replacementStartedAfterMs: 300000, readyAfterMs: 420000 })).toBe(false); }); + it("accepts a hold past the reconnect grace only with an explicit later ready bound", () => { + const receipt = { schema: "paperclip.e2e.controller-outage/v1", requestId: "long", signal: "SIGKILL", holdMs: 579_090, + previousPid: 1, replacementPid: 2, startedAt: new Date().toISOString(), stoppedAfterMs: 200, cacheDiscarded: true, + replacementStartedAfterMs: 579_400, readyAfterMs: 593_500 }; + expect(validControllerOutageReceipt(receipt)).toBe(false); + expect(validControllerOutageReceipt(receipt, undefined, undefined, undefined, 579_090 + 420_000)).toBe(true); + }); }); it.each(["absent", "present"])("retires a %s old controller cache", async state => { diff --git a/tests/runner-e2e/controller-outage.ts b/tests/runner-e2e/controller-outage.ts index 2f07b834089..d8d0ade93bc 100644 --- a/tests/runner-e2e/controller-outage.ts +++ b/tests/runner-e2e/controller-outage.ts @@ -33,6 +33,10 @@ export interface ControllerOutageReceipt extends ControllerOutage { * inside that grace, so this does not loosen the qualification contract. */ export const MAX_CONTROLLER_OUTAGE_MS = 600_000; +/** Runner suites need the replacement ready inside the Runner's 7-minute + * reconnect grace, counted from the start of the outage. Chaos holds that + * deliberately outlast the grace pass a later bound to `replaceController`. */ +export const CONTROLLER_READY_BY_MS = 420_000; export function parseControllerOutage(value: unknown): ControllerOutage { if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("Invalid controller outage"); @@ -67,7 +71,8 @@ export function controllerOutageForExecution(execution: MatrixExecution, env: No /** A fault receipt is required in addition to the runner's continuity receipts. */ export function validControllerOutageReceipt(value: unknown, expected?: ControllerOutage, requestId?: string, expectedArtifacts?: { previous: Pick; - replacement: Pick }): value is ControllerOutageReceipt { + replacement: Pick }, + readyByMs = CONTROLLER_READY_BY_MS): value is ControllerOutageReceipt { if (!value || typeof value !== "object" || Array.isArray(value)) return false; const r = value as ControllerOutageReceipt; try { parseControllerOutage(r); } catch { return false; } @@ -81,7 +86,7 @@ export function validControllerOutageReceipt(value: unknown, expected?: Controll && durations.every(ms => typeof ms === "number" && Number.isFinite(ms) && ms >= 0) && r.replacementStartedAfterMs! - r.stoppedAfterMs! >= r.holdMs && r.readyAfterMs! >= r.replacementStartedAfterMs! - && r.readyAfterMs! < 420_000 + && r.readyAfterMs! < readyByMs && ((!r.previousArtifact && !r.replacementArtifact) || Boolean(r.previousArtifact && r.replacementArtifact && [r.previousArtifact, r.replacementArtifact].every(a => /^[a-f0-9]{40}$/.test(a.revision) && /^[a-f0-9]{64}$/.test(a.treeSha256) && typeof a.version === "string" @@ -113,6 +118,8 @@ export async function replaceController(input: { replacementArtifact?: () => ServingArtifact | undefined; afterStopped?: (receipt: ControllerOutageReceipt) => Promise; afterReady?: (receipt: ControllerOutageReceipt, record: () => Promise) => Promise; + /** Upper bound on `readyAfterMs`; defaults to the Runner reconnect grace. */ + readyByMs?: number; }): Promise { const outage = parseControllerOutage(input.outage); const started = performance.now(); @@ -149,7 +156,7 @@ export async function replaceController(input: { if (!receipt.replacementArtifact) throw new Error("Replacement has no verified serving artifact"); } await record(); - if (!validControllerOutageReceipt(receipt, outage, input.requestId)) throw new Error("Controller outage did not produce a qualifying replacement receipt"); + if (!validControllerOutageReceipt(receipt, outage, input.requestId, undefined, input.readyByMs)) throw new Error("Controller outage did not produce a qualifying replacement receipt"); await input.afterReady?.(receipt, record); return receipt; }