From 62b7733161c578165fa5bb0195ec8becaab8a003 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 00:29:06 -0600 Subject: [PATCH] test(restart-chaos): long mid-tool K15 outages, Runner evidence, and TM1 stalls on Daytona (#20711) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart-chaos harness (`tests/restart-chaos/`) restarts or stalls the control plane while real agent tasks run. > - The staging census (#20679, `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`) found two open classes that the harness could not create. In class R, a deploy lands mid-turn, the server stays down for 4–7 minutes, and recovery then finds the remote Runner gone (`runner_remote_process_identity_unavailable`). In class L, the server stalls for about 100 s without a restart, and legacy runs that are still `preparing` end up waiting for a manual Retry. > - K15 restarted at once. No Daytona workload kept a tool running across the fault. TM1 was local-only with a fixed range. Nothing recorded what the remote Runner did while the controller was away. > - This pull request adds a K15 outage hold, a mid-tool workload, sandbox-side Runner evidence, and TM1 stalls on Daytona with a configurable hold. > - The benefit: these staging failure shapes can now be reproduced, and each wave records the Runner's own state. ## Linked Issues or Issue Description Refs #20679 **What happened?** On staging, two failure shapes had no matching chaos scenario: - Class R: a deploy landed mid-turn and the server stayed down for 4–7 minutes. Recovery then found the remote Runner gone. Nothing captured why the Runner exited. - Class L: a 100 s server stall without a restart lapsed every controller lease. Legacy `preparing` runs then waited for a manual Retry. **What did you expect?** The harness should create a long mid-turn deploy outage and a stall without a restart on Daytona. It should also save the sandbox-side Runner evidence. **Steps to reproduce** 1. Use the census report and the staging run records listed in it. 2. Compare them with the harness options before this change: K15 has no hold, TM1 is local-only, and there is no Daytona tool workload. ## What Changed - `--k15-hold-seconds ` (0–600). A K15 stops the server with SIGTERM, keeps it down for the seeded hold, then starts the next deploy generation. The range is drawn only for K15 cells, so existing seeds keep their plans. Storms use it too. - New opt-in workload `T1`. The agent runs one shell loop in the foreground for about 100 s, then saves `TICKS-=50`. Its fault window, 35–80 s after launch, lands mid-tool. - New `daytona-runner-logs.ts`, a read-only probe of each wave sandbox through the Daytona toolbox. It runs only on sandboxes with the wave's labels, and only on `started` ones. It records every process, memory, disk, `dmesg` OOM lines, and the tails of `runnerd.stderr.log`, `runner-process.identity` and `runner-state.json`. It redacts known secrets. It runs right after a K15 stop, just before the next deploy starts, and at wave end. Output goes to `results//daytona-runner/` and to `runner_logs_captured` in the timeline. The probe never stops, starts or deletes a sandbox, and a failed capture is recorded instead of thrown. - TM1 now runs on `--env daytona`, and `--tm1-hold-seconds ` (10–600) replaces its default 95–150 s range with the same single draw. TM2 and TM3 stay local. - Tests: `k15-long-hold.test.ts` and `tm1-stall.test.ts`. The README is updated. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 45 files and 531 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Daytona waves, all with T1 and K15: | Wave | Revision and image | Hold | Result | | --- | --- | --- | --- | | `w-20261111T015441-s9310` | master 8f6ca25976, image `bc7ca92d…` | 267–408 s, OpenCode | 4 of 4 pass | | `w-20261111T022637-s9311` | master 8f6ca25976, image `bc7ca92d…` | 300 s, OpenCode and Claude | 4 of 4 pass. Every stop capture shows the T1 loop running. | | `w-20261111T025307-s9320` | master a1ec05a1a4, image `09321542…` | 460–482 s, past the Runner's 7-minute reconnect grace | 4 of 4 pass | - In every pre-restart capture, the same `runnerd` PID was alive for the whole outage (etimes 491–549 s in the last wave). Every sandbox was `started`. There was no `transport_reconnect_grace_exceeded` and no OOM. - So the class R Runner loss does not reproduce on current master with the e2e image. The census document lists the remaining differences from staging. - TM1 Daytona waves on master a1ec05a1a4 found a **new product gap** (RED; the harness only reports it): - `w-20261111T034105-s9330` (TM1 at `--fault-after-seconds 10`): 2 of 3 pass. The failing OpenCode run was stalled at `native.process_start_requested`. Afterwards it waited forever with `controller_still_alive` / `restartKind: hard`, where the controller it waited for was its own live server. - `w-20261111T040439-s9331` (TM1 mid-tool, T1): 0 of 3 pass. After the stall the run had no further events and stayed `running` for more than 20 minutes, with no Retry. - Review fixes in 3540bfe700: matrix cells take both holds, T1's `fault_crossed_tool` check, and one 45 s deadline per Runner capture. The suite is now 45 files and 531 tests. ## Risks - Low risk. This change is test harness and docs only, with no product code. - The Runner probe executes one fixed read-only command in this wave's own sandboxes through the Daytona toolbox. It never targets fleet sandboxes, because it filters on the wave's company and environment labels. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude --- ...1-11-staging-failures-vs-chaos-coverage.md | 38 +++ tests/restart-chaos/README.md | 75 +++++- tests/restart-chaos/config.ts | 10 +- tests/restart-chaos/daytona-runner-logs.ts | 137 +++++++++++ tests/restart-chaos/journey-matrix.ts | 9 +- tests/restart-chaos/journeys.ts | 2 +- tests/restart-chaos/k15-long-hold.test.ts | 222 ++++++++++++++++++ tests/restart-chaos/main.ts | 20 +- tests/restart-chaos/redeploy.ts | 26 ++ tests/restart-chaos/scheduler.ts | 16 +- tests/restart-chaos/supervisor.ts | 13 +- tests/restart-chaos/time-faults-config.ts | 43 +++- tests/restart-chaos/tm1-stall.test.ts | 40 ++++ tests/restart-chaos/types.ts | 7 +- tests/restart-chaos/workloads.ts | 67 ++++++ 15 files changed, 698 insertions(+), 27 deletions(-) create mode 100644 tests/restart-chaos/daytona-runner-logs.ts create mode 100644 tests/restart-chaos/k15-long-hold.test.ts create mode 100644 tests/restart-chaos/tm1-stall.test.ts diff --git a/doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md b/doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md index 2550b6ff024..7c0bb57e730 100644 --- a/doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md +++ b/doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md @@ -307,3 +307,41 @@ What this shows: - The next scenario to build is **K15 with `--upgrade-from` on Daytona**: start with the older checkout and its image, redeploy to the current checkout, and keep the older runner in the retained sandboxes. Today the Daytona image check refuses that mixed state on purpose. + +### Follow-up: long mid-tool outages and server stalls (PR #20711) + +Built after review by the class R and class L fix owners: + +- `--k15-hold-seconds` holds a K15 outage, because staging deploys were down 4–7 minutes. +- The T1 workload keeps a shell tool running across the fault, and `fault_crossed_tool` proves it. +- Sandbox-side Runner evidence is captured in `results//daytona-runner/`. +- TM1 stalls now run on Daytona, with `--tm1-hold-seconds`. + +Class L turned out not to be a deploy. The server process stalled for about 100 s at 08:19Z on Nov 10. + +| Wave | Revision | Scenario | Result | +| --- | --- | --- | --- | +| `w-20261111T015441-s9310` | 8f6ca25976 | K15 hold 267–408 s, T1, OpenCode | 4 of 4 pass. The Runner was alive through every outage. | +| `w-20261111T022637-s9311` | 8f6ca25976 | K15 hold 300 s, T1, OpenCode and Claude | 4 of 4 pass. The stop captures show the T1 loop running. | +| `w-20261111T025307-s9320` | a1ec05a1a4 | K15 hold 460–482 s, past the Runner's 420 s reconnect grace | 4 of 4 pass. The same `runnerd` PID was alive 491–549 s after the stop, every sandbox was `started`, and there was no `transport_reconnect_grace_exceeded`. | +| `w-20261111T034105-s9330` | a1ec05a1a4 | TM1 70–120 s at `--fault-after-seconds 10`, A1, all three adapters | **RED, 1 of 3 fail.** An OpenCode run was stalled at `native.process_start_requested`. Afterwards it waited forever on `native.recovery.transition` with `reason: controller_still_alive` and `restartKind: hard`. The controller it waited for was its own live, unrestarted process. No Retry was offered. | +| `w-20261111T040439-s9331` | a1ec05a1a4 | TM1 70–120 s mid-tool, T1, all three adapters | **RED, 3 of 3 fail.** After the stall, `native.session.execute` ended, but the run never got another event: no terminal and no recovery. It stayed `running` for more than 20 minutes with no Retry. Each task crossed three back-to-back stalls. | +| `w-20261111T043759-s9340` | a1ec05a1a4 | One TM1 stall of 90 s mid-tool, T1, Codex | 1 of 1 pass. `fault_crossed_tool` confirms the stall landed in the loop. The mid-tool wedge needs repeated stalls; one did not cause it. | + +Conclusions: + +- Class R's Runner loss (`runner_remote_process_identity_unavailable`) does **not** reproduce on current + master with the e2e image, even past the reconnect grace. +- Staging differs from these waves in four ways: + - its sandboxes come from the managed fleet snapshot (`fleet-sbx-…`, last updated Nov 5), not the e2e image; + - it ran older runner builds; + - Railway drains the server before it stops; + - the cloud manages the sandbox lifecycle. +- The next step needs staging-side `runnerd.stderr.log` from a retained sandbox of a failed run. +- A control-plane stall **without** a restart is a new open product class. A native run whose lease + lapses while its own server is frozen is never taken back by that same server: + - **stall during preparation:** the run waits forever on `controller_still_alive`; + - **stall mid-turn:** the run is orphaned while still `running`. This took three back-to-back stalls of about 90 s; a single 90 s stall recovered. +- #20694 fixed the legacy `preparing` case only. +- Code pointer: `server/src/services/native-runtime/native-restart-recovery.ts` (the same-boot owner is not + recognized as the server itself). diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index f0e1731365d..2de445fb04e 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -337,6 +337,58 @@ Limits: - Combine K15 with `--adapters opencode` and `--daytona-reuse-lease` or `--daytona-warm` to match staging's workload mix. +### Long mid-tool redeploy (`--k15-hold-seconds`, T1, Runner logs) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \ + --adapters opencode --workloads T1 --faults K15 --k15-hold-seconds 240-480 --concurrency 2 --tasks 4 --seed 9310 +``` + +A staging deploy usually lands mid-turn and keeps the server down for 4–7 +minutes. In staging class R, recovery then found the remote Runner gone +(`runner_remote_process_identity_unavailable`). + +- **`--k15-hold-seconds ` or `-`** (0–600 s) seeds each K15's + outage. The server stops with SIGTERM and stays down for the hold. Then the + next deploy generation starts. Without the option, a K15 restarts at once. + The range is drawn only for K15 cells, so other seeds keep their plans. +- **T1** (opt-in workload) has the agent run one shell command in the + foreground for about 100 s (50 ticks of 2 s). The command prints the loop's + own clock window. The agent then saves `brief`, whose first line is + `TICKS-=50 start= end=`. Its fault window (35–80 s + after launch) aims to land while the tool runs in the sandbox, and stays short + of provider shell-tool timeouts. + + The `fault_crossed_tool` check (skipped for `K0` baselines and provider faults; + a storm task counts the storm's faults) fails the task unless one of its own faults + started inside the reported window, with 5 s of clock slack. A fault that hit + preparation, with the loop run afterwards, tested nothing mid-tool, as a + provider-fault task whose fault never applied. +- **Runner evidence:** on a Daytona K15 wave, the harness captures every + started sandbox of the wave three times: + - right after the server stopped (`-stopped`); + - after the hold, just before the next deploy starts (`-before_start`); + - at wave end (`wave-end`). + + Each capture is written to `results//daytona-runner/