diff --git a/doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md b/doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md index 2550b6ff024..7c0bb57e730 100644 --- a/doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md +++ b/doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md @@ -307,3 +307,41 @@ What this shows: - The next scenario to build is **K15 with `--upgrade-from` on Daytona**: start with the older checkout and its image, redeploy to the current checkout, and keep the older runner in the retained sandboxes. Today the Daytona image check refuses that mixed state on purpose. + +### Follow-up: long mid-tool outages and server stalls (PR #20711) + +Built after review by the class R and class L fix owners: + +- `--k15-hold-seconds` holds a K15 outage, because staging deploys were down 4–7 minutes. +- The T1 workload keeps a shell tool running across the fault, and `fault_crossed_tool` proves it. +- Sandbox-side Runner evidence is captured in `results//daytona-runner/`. +- TM1 stalls now run on Daytona, with `--tm1-hold-seconds`. + +Class L turned out not to be a deploy. The server process stalled for about 100 s at 08:19Z on Nov 10. + +| Wave | Revision | Scenario | Result | +| --- | --- | --- | --- | +| `w-20261111T015441-s9310` | 8f6ca25976 | K15 hold 267–408 s, T1, OpenCode | 4 of 4 pass. The Runner was alive through every outage. | +| `w-20261111T022637-s9311` | 8f6ca25976 | K15 hold 300 s, T1, OpenCode and Claude | 4 of 4 pass. The stop captures show the T1 loop running. | +| `w-20261111T025307-s9320` | a1ec05a1a4 | K15 hold 460–482 s, past the Runner's 420 s reconnect grace | 4 of 4 pass. The same `runnerd` PID was alive 491–549 s after the stop, every sandbox was `started`, and there was no `transport_reconnect_grace_exceeded`. | +| `w-20261111T034105-s9330` | a1ec05a1a4 | TM1 70–120 s at `--fault-after-seconds 10`, A1, all three adapters | **RED, 1 of 3 fail.** An OpenCode run was stalled at `native.process_start_requested`. Afterwards it waited forever on `native.recovery.transition` with `reason: controller_still_alive` and `restartKind: hard`. The controller it waited for was its own live, unrestarted process. No Retry was offered. | +| `w-20261111T040439-s9331` | a1ec05a1a4 | TM1 70–120 s mid-tool, T1, all three adapters | **RED, 3 of 3 fail.** After the stall, `native.session.execute` ended, but the run never got another event: no terminal and no recovery. It stayed `running` for more than 20 minutes with no Retry. Each task crossed three back-to-back stalls. | +| `w-20261111T043759-s9340` | a1ec05a1a4 | One TM1 stall of 90 s mid-tool, T1, Codex | 1 of 1 pass. `fault_crossed_tool` confirms the stall landed in the loop. The mid-tool wedge needs repeated stalls; one did not cause it. | + +Conclusions: + +- Class R's Runner loss (`runner_remote_process_identity_unavailable`) does **not** reproduce on current + master with the e2e image, even past the reconnect grace. +- Staging differs from these waves in four ways: + - its sandboxes come from the managed fleet snapshot (`fleet-sbx-…`, last updated Nov 5), not the e2e image; + - it ran older runner builds; + - Railway drains the server before it stops; + - the cloud manages the sandbox lifecycle. +- The next step needs staging-side `runnerd.stderr.log` from a retained sandbox of a failed run. +- A control-plane stall **without** a restart is a new open product class. A native run whose lease + lapses while its own server is frozen is never taken back by that same server: + - **stall during preparation:** the run waits forever on `controller_still_alive`; + - **stall mid-turn:** the run is orphaned while still `running`. This took three back-to-back stalls of about 90 s; a single 90 s stall recovered. +- #20694 fixed the legacy `preparing` case only. +- Code pointer: `server/src/services/native-runtime/native-restart-recovery.ts` (the same-boot owner is not + recognized as the server itself). diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index f0e1731365d..2de445fb04e 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -337,6 +337,58 @@ Limits: - Combine K15 with `--adapters opencode` and `--daytona-reuse-lease` or `--daytona-warm` to match staging's workload mix. +### Long mid-tool redeploy (`--k15-hold-seconds`, T1, Runner logs) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \ + --adapters opencode --workloads T1 --faults K15 --k15-hold-seconds 240-480 --concurrency 2 --tasks 4 --seed 9310 +``` + +A staging deploy usually lands mid-turn and keeps the server down for 4–7 +minutes. In staging class R, recovery then found the remote Runner gone +(`runner_remote_process_identity_unavailable`). + +- **`--k15-hold-seconds ` or `-`** (0–600 s) seeds each K15's + outage. The server stops with SIGTERM and stays down for the hold. Then the + next deploy generation starts. Without the option, a K15 restarts at once. + The range is drawn only for K15 cells, so other seeds keep their plans. +- **T1** (opt-in workload) has the agent run one shell command in the + foreground for about 100 s (50 ticks of 2 s). The command prints the loop's + own clock window. The agent then saves `brief`, whose first line is + `TICKS-=50 start= end=`. Its fault window (35–80 s + after launch) aims to land while the tool runs in the sandbox, and stays short + of provider shell-tool timeouts. + + The `fault_crossed_tool` check (skipped for `K0` baselines and provider faults; + a storm task counts the storm's faults) fails the task unless one of its own faults + started inside the reported window, with 5 s of clock slack. A fault that hit + preparation, with the loop run afterwards, tested nothing mid-tool, as a + provider-fault task whose fault never applied. +- **Runner evidence:** on a Daytona K15 wave, the harness captures every + started sandbox of the wave three times: + - right after the server stopped (`-stopped`); + - after the hold, just before the next deploy starts (`-before_start`); + - at wave end (`wave-end`). + + Each capture is written to `results//daytona-runner/