## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The restart-chaos harness (`tests/restart-chaos/`) restarts or
stalls the control plane while real agent tasks run.
> - The staging census (#20679,
`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`) found two
open classes that the harness could not create. In class R, a deploy
lands mid-turn, the server stays down for 4–7 minutes, and recovery then
finds the remote Runner gone
(`runner_remote_process_identity_unavailable`). In class L, the server
stalls for about 100 s without a restart, and legacy runs that are still
`preparing` end up waiting for a manual Retry.
> - K15 restarted at once. No Daytona workload kept a tool running
across the fault. TM1 was local-only with a fixed range. Nothing
recorded what the remote Runner did while the controller was away.
> - This pull request adds a K15 outage hold, a mid-tool workload,
sandbox-side Runner evidence, and TM1 stalls on Daytona with a
configurable hold.
> - The benefit: these staging failure shapes can now be reproduced, and
each wave records the Runner's own state.
## Linked Issues or Issue Description
Refs #20679
**What happened?**
On staging, two failure shapes had no matching chaos scenario:
- Class R: a deploy landed mid-turn and the server stayed down for 4–7
minutes. Recovery then found the remote Runner gone. Nothing captured
why the Runner exited.
- Class L: a 100 s server stall without a restart lapsed every
controller lease. Legacy `preparing` runs then waited for a manual
Retry.
**What did you expect?**
The harness should create a long mid-turn deploy outage and a stall
without a restart on Daytona. It should also save the sandbox-side
Runner evidence.
**Steps to reproduce**
1. Use the census report and the staging run records listed in it.
2. Compare them with the harness options before this change: K15 has no
hold, TM1 is local-only, and there is no Daytona tool workload.
## What Changed
- `--k15-hold-seconds <n|min-max>` (0–600). A K15 stops the server with
SIGTERM, keeps it down for the seeded hold, then starts the next deploy
generation. The range is drawn only for K15 cells, so existing seeds
keep their plans. Storms use it too.
- New opt-in workload `T1`. The agent runs one shell loop in the
foreground for about 100 s, then saves `TICKS-<nonce>=50`. Its fault
window, 35–80 s after launch, lands mid-tool.
- New `daytona-runner-logs.ts`, a read-only probe of each wave sandbox
through the Daytona toolbox. It runs only on sandboxes with the wave's
labels, and only on `started` ones. It records every process, memory,
disk, `dmesg` OOM lines, and the tails of `runnerd.stderr.log`,
`runner-process.identity` and `runner-state.json`. It redacts known
secrets. It runs right after a K15 stop, just before the next deploy
starts, and at wave end. Output goes to `results/<wave>/daytona-runner/`
and to `runner_logs_captured` in the timeline. The probe never stops,
starts or deletes a sandbox, and a failed capture is recorded instead of
thrown.
- TM1 now runs on `--env daytona`, and `--tm1-hold-seconds <n|min-max>`
(10–600) replaces its default 95–150 s range with the same single draw.
TM2 and TM3 stay local.
- Tests: `k15-long-hold.test.ts` and `tm1-stall.test.ts`. The README is
updated.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
45 files and 531 tests pass.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Daytona waves, all with T1 and K15:
| Wave | Revision and image | Hold | Result |
| --- | --- | --- | --- |
| `w-20261111T015441-s9310` | master 8f6ca25976, image `bc7ca92d…` |
267–408 s, OpenCode | 4 of 4 pass |
| `w-20261111T022637-s9311` | master 8f6ca25976, image `bc7ca92d…` | 300
s, OpenCode and Claude | 4 of 4 pass. Every stop capture shows the T1
loop running. |
| `w-20261111T025307-s9320` | master a1ec05a1a4, image `09321542…` |
460–482 s, past the Runner's 7-minute reconnect grace | 4 of 4 pass |
- In every pre-restart capture, the same `runnerd` PID was alive for the
whole outage (etimes 491–549 s in the last wave). Every sandbox was
`started`. There was no `transport_reconnect_grace_exceeded` and no OOM.
- So the class R Runner loss does not reproduce on current master with
the e2e image. The census document lists the remaining differences from
staging.
- TM1 Daytona waves on master a1ec05a1a4 found a **new product gap**
(RED; the harness only reports it):
- `w-20261111T034105-s9330` (TM1 at `--fault-after-seconds 10`): 2 of 3
pass. The failing OpenCode run was stalled at
`native.process_start_requested`. Afterwards it waited forever with
`controller_still_alive` / `restartKind: hard`, where the controller it
waited for was its own live server.
- `w-20261111T040439-s9331` (TM1 mid-tool, T1): 0 of 3 pass. After the
stall the run had no further events and stayed `running` for more than
20 minutes, with no Retry.
- Review fixes in 3540bfe700: matrix cells take both holds, T1's
`fault_crossed_tool` check, and one 45 s deadline per Runner capture.
The suite is now 45 files and 531 tests.
## Risks
- Low risk. This change is test harness and docs only, with no product
code.
- The Runner probe executes one fixed read-only command in this wave's
own sandboxes through the Daytona toolbox. It never targets fleet
sandboxes, because it filters on the wave's company and environment
labels.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, GitHub CLI).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
