diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 6e0da141e08..8402bf5ccb6 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -425,6 +425,76 @@ for a while after the new one was up. Plain K15 restarts on the same instance ho Pass is the usual grade: no Retry (R3), no stuck record (R6), and the oracle. Staging's terminal errors here were `ENOENT` on a runtime asset, `Agent directory operation is already active` after a 9.5-minute `controller_lease_held` wait. +### Staging class R cells (CR1–CR3, `class-r.ts`) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \ + --adapters opencode,claude --workloads T1 --faults CR1 --concurrency 1 --tasks 2 --seed 9501 +``` + +Staging failed with `runner_remote_process_identity_unavailable` in 9 runs. In 8 of them: + +1. A deploy kept the controller away for 5–7 minutes. +2. The Runner in the Daytona sandbox survived, and the new controller adopted it. +3. About 40 s later, `session.snapshot` failed because the provider process was dead or unresponsive. + +The ninth run had no deploy. A plugin `environmentExecute` RPC stalled for more than 40 s. These cells +reproduce each part. The class is named R on staging. The cells are named CR1–CR3 because the R1 workload +and the R1–R6 grading rules already exist. + +| Fault | What it does | Option | +| --- | --- | --- | +| CR1 | The K15 redeploy: SIGTERM, then a held outage, then the next deploy generation. | `--cr1-hold-seconds ` or `-`, 0–600. Default 360. | +| CR2 | SIGKILL of the provider process only, inside the task's sandbox. The Runner stays alive. | `--cr2-hold-seconds `, 0–600. Default 0: the controller stays up. With a hold, a K15 redeploy holds the controller down, and the kill happens right after the stop. | +| CR3 | SIGSTOP of the server's Daytona plugin worker and its children. Every plugin RPC, including `environmentExecute`, stalls. There is no deploy. | `--cr3-stall-seconds `, 41–600. Default 45. | + +- CR2 kills only the processes under the run's `paperclip-runnerd` whose arguments match the adapter's + provider command. For Claude (ACPX) that is `--output-format stream-json`. For OpenCode it is + `serve --hostname 127.0.0.1 --port`. The Runner is found by the run id in its listen path. +- CR2 is the only class R action that changes anything in a sandbox. It runs one fixed script, only in + sandboxes that carry the wave's company and environment labels. +- The run id comes from the task's last sample before the fault. Each kill's output is saved to + `daytona-runner/provider-kill--.json`. +- Each CR task must pass `class_r_fault_applied`. The fault must not have failed, CR2 must have killed a + provider process, and CR3's stall must have ended. A cell whose fault never applied tested nothing. +- Use T1, so that the fault lands while a tool runs, as it did on staging. Use `--concurrency 1` for CR3, + because the stall hits every task's plugin RPCs. + +**Sandbox capture.** Every Daytona wave, not only a CR wave, runs the class R probe (`CLASS_R_PROBE`) in +every started wave sandbox. Staging's Runner loss also reproduces with no fault at all (W2 re-wake). Its +output goes to `results//daytona-runner/