From 0bf3236bc84c9cc01594ec6b0720b620ce384040 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 15:50:59 -0600 Subject: [PATCH] test(restart-chaos): staging class R cells CR1-CR3 and sandbox capture on every Daytona wave (#20909) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) reproduces staging failures before users meet them > - Staging fails with `runner_remote_process_identity_unavailable` when the remote Runner or its provider is gone after a deploy, a stall or a re-wake, and nobody has seen the sandbox side > - The harness had no cell for a provider-only death or a plugin RPC stall, and its sandbox capture ran only on K15 waves > - This pull request adds the staging class R cells CR1–CR3, and captures the sandbox side on every Daytona wave > - The benefit is that a wave that reproduces the refusal also keeps the evidence of why the Runner or provider died ## Linked Issues or Issue Description **What is the problem?** The harness cannot answer why a remote runnerd or its provider dies. Its capture ran only around K15 restarts. It recorded no process start times, sockets or provider log files. It had no fault that kills only the provider or stalls a plugin RPC. **What should change?** Add cells for each staging shape, and capture the sandbox before, during and after each one, on every Daytona wave. **Context** Refs #20907 (keeps runnerd's stderr in the sandbox, and reports it on failed recovery). Refs #20893 (W2 re-wake cells). ## What Changed - `class-r.ts` (new) adds three faults, selected with `--faults`: - **CR1:** a K15 redeploy with a held outage. `--cr1-hold-seconds`, 0–600, default 360. - **CR2:** SIGKILL of only the provider process under the run's `paperclip-runnerd`, inside its sandbox. Runnerd stays alive. By default the controller stays up; `--cr2-hold-seconds` adds a K15 outage. - **CR3:** SIGSTOP of the server's Daytona plugin worker. `--cr3-stall-seconds`, 41–600, default 45. Each CR task must pass the `class_r_fault_applied` check. - `daytona-runner-logs.ts` adds `CLASS_R_PROBE`. It reads process start times and `wchan`, sockets, the runtime file listing, and the tails of the Runner's and the provider's state and log files. It also adds `executeInWaveSandboxes`, the fixed-script runner that CR2 uses. - `main.ts` and `supervisor.ts`: - every Daytona wave captures with the class R probe when a task rests and at wave end; - CR cells also capture before the fault, after the stop, before the restart, and every 15 s for 180 s after ready; - `--sandbox-watch-seconds` captures on a clock for the whole wave. - The README has a new "Staging class R cells" section. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 49 files, 600 tests pass, including the new `class-r.test.ts`. It covers option parsing, planning, the kill command's quoting and its refusal of anything but a run id, plugin worker selection, the watch schedule and the applied check. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Daytona waves run from this branch plus #20907. The results are in the follow-up comment. ## Risks - This change is limited to the harness. CR2 is its only in-sandbox mutation: a SIGKILL of the provider, run by a fixed script only in sandboxes labelled with the wave's company and environment. The script never signals runnerd. - More captures add Daytona toolbox calls on every Daytona wave. Each capture has a 45 s deadline, and a failed capture is recorded and never fails the wave. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, Daytona waves). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude --- tests/restart-chaos/README.md | 70 ++++++ tests/restart-chaos/class-r.test.ts | 192 ++++++++++++++ tests/restart-chaos/class-r.ts | 279 +++++++++++++++++++++ tests/restart-chaos/config.ts | 12 +- tests/restart-chaos/daytona-runner-logs.ts | 93 ++++++- tests/restart-chaos/journey-matrix.ts | 6 +- tests/restart-chaos/main.ts | 87 ++++++- tests/restart-chaos/redeploy.ts | 8 +- tests/restart-chaos/scheduler.ts | 6 +- tests/restart-chaos/supervisor.ts | 45 +++- tests/restart-chaos/time-faults.ts | 2 +- tests/restart-chaos/types.ts | 13 +- 12 files changed, 793 insertions(+), 20 deletions(-) create mode 100644 tests/restart-chaos/class-r.test.ts create mode 100644 tests/restart-chaos/class-r.ts diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 6e0da141e08..8402bf5ccb6 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -425,6 +425,76 @@ for a while after the new one was up. Plain K15 restarts on the same instance ho Pass is the usual grade: no Retry (R3), no stuck record (R6), and the oracle. Staging's terminal errors here were `ENOENT` on a runtime asset, `Agent directory operation is already active` after a 9.5-minute `controller_lease_held` wait. +### Staging class R cells (CR1–CR3, `class-r.ts`) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \ + --adapters opencode,claude --workloads T1 --faults CR1 --concurrency 1 --tasks 2 --seed 9501 +``` + +Staging failed with `runner_remote_process_identity_unavailable` in 9 runs. In 8 of them: + +1. A deploy kept the controller away for 5–7 minutes. +2. The Runner in the Daytona sandbox survived, and the new controller adopted it. +3. About 40 s later, `session.snapshot` failed because the provider process was dead or unresponsive. + +The ninth run had no deploy. A plugin `environmentExecute` RPC stalled for more than 40 s. These cells +reproduce each part. The class is named R on staging. The cells are named CR1–CR3 because the R1 workload +and the R1–R6 grading rules already exist. + +| Fault | What it does | Option | +| --- | --- | --- | +| CR1 | The K15 redeploy: SIGTERM, then a held outage, then the next deploy generation. | `--cr1-hold-seconds ` or `-`, 0–600. Default 360. | +| CR2 | SIGKILL of the provider process only, inside the task's sandbox. The Runner stays alive. | `--cr2-hold-seconds `, 0–600. Default 0: the controller stays up. With a hold, a K15 redeploy holds the controller down, and the kill happens right after the stop. | +| CR3 | SIGSTOP of the server's Daytona plugin worker and its children. Every plugin RPC, including `environmentExecute`, stalls. There is no deploy. | `--cr3-stall-seconds `, 41–600. Default 45. | + +- CR2 kills only the processes under the run's `paperclip-runnerd` whose arguments match the adapter's + provider command. For Claude (ACPX) that is `--output-format stream-json`. For OpenCode it is + `serve --hostname 127.0.0.1 --port`. The Runner is found by the run id in its listen path. +- CR2 is the only class R action that changes anything in a sandbox. It runs one fixed script, only in + sandboxes that carry the wave's company and environment labels. +- The run id comes from the task's last sample before the fault. Each kill's output is saved to + `daytona-runner/provider-kill--.json`. +- Each CR task must pass `class_r_fault_applied`. The fault must not have failed, CR2 must have killed a + provider process, and CR3's stall must have ended. A cell whose fault never applied tested nothing. +- Use T1, so that the fault lands while a tool runs, as it did on staging. Use `--concurrency 1` for CR3, + because the stall hits every task's plugin RPCs. + +**Sandbox capture.** Every Daytona wave, not only a CR wave, runs the class R probe (`CLASS_R_PROBE`) in +every started wave sandbox. Staging's Runner loss also reproduces with no fault at all (W2 re-wake). Its +output goes to `results//daytona-runner/