mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
DottaandClaude
0bf3236bc8
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) reproduces staging failures before users meet them > - Staging fails with `runner_remote_process_identity_unavailable` when the remote Runner or its provider is gone after a deploy, a stall or a re-wake, and nobody has seen the sandbox side > - The harness had no cell for a provider-only death or a plugin RPC stall, and its sandbox capture ran only on K15 waves > - This pull request adds the staging class R cells CR1–CR3, and captures the sandbox side on every Daytona wave > - The benefit is that a wave that reproduces the refusal also keeps the evidence of why the Runner or provider died ## Linked Issues or Issue Description **What is the problem?** The harness cannot answer why a remote runnerd or its provider dies. Its capture ran only around K15 restarts. It recorded no process start times, sockets or provider log files. It had no fault that kills only the provider or stalls a plugin RPC. **What should change?** Add cells for each staging shape, and capture the sandbox before, during and after each one, on every Daytona wave. **Context** Refs #20907 (keeps runnerd's stderr in the sandbox, and reports it on failed recovery). Refs #20893 (W2 re-wake cells). ## What Changed - `class-r.ts` (new) adds three faults, selected with `--faults`: - **CR1:** a K15 redeploy with a held outage. `--cr1-hold-seconds`, 0–600, default 360. - **CR2:** SIGKILL of only the provider process under the run's `paperclip-runnerd`, inside its sandbox. Runnerd stays alive. By default the controller stays up; `--cr2-hold-seconds` adds a K15 outage. - **CR3:** SIGSTOP of the server's Daytona plugin worker. `--cr3-stall-seconds`, 41–600, default 45. Each CR task must pass the `class_r_fault_applied` check. - `daytona-runner-logs.ts` adds `CLASS_R_PROBE`. It reads process start times and `wchan`, sockets, the runtime file listing, and the tails of the Runner's and the provider's state and log files. It also adds `executeInWaveSandboxes`, the fixed-script runner that CR2 uses. - `main.ts` and `supervisor.ts`: - every Daytona wave captures with the class R probe when a task rests and at wave end; - CR cells also capture before the fault, after the stop, before the restart, and every 15 s for 180 s after ready; - `--sandbox-watch-seconds` captures on a clock for the whole wave. - The README has a new "Staging class R cells" section. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 49 files, 600 tests pass, including the new `class-r.test.ts`. It covers option parsing, planning, the kill command's quoting and its refusal of anything but a run id, plugin worker selection, the watch schedule and the applied check. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Daytona waves run from this branch plus #20907. The results are in the follow-up comment. ## Risks - This change is limited to the harness. CR2 is its only in-sandbox mutation: a SIGKILL of the provider, run by a fixed script only in sandboxes labelled with the wave's company and environment. The script never signals runnerd. - More captures add Daytona toolbox calls on every Daytona wave. Each capture has a 45 s deadline, and a failed capture is recorded and never fails the wave. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, Daytona waves). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>