mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) restarts the control plane during real agent tasks and grades each task > - On staging (2026-11-12), 577 of 1,000 runs failed or were interrupted across overlap deploys every 40 minutes > - The harness missed this: its waves run a few short tasks, while staging runs about 30 agents at once with long tool turns, subtasks, comments and Retries during each deploy > - This pull request adds a staging-like workload (`--workload staging-mix`) that runs under any existing restart cell today, and under the coming release-swap cell > - The benefit is one honest number per wave (tasks completed without human action / tasks started) under a load shaped like staging ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart chaos harness workloads. Every current workload is a few short tasks, so a wave never has 30 runs, a queue, long mid-tool turns, subtasks, or comments and Retries during a deploy overlap. **Subsystem affected** Test harness only: `tests/restart-chaos/`. No product code changes. **Current behavior** A wave plans a fixed list of short tasks with one fault each. The harness holds every board action while a fault is in progress, so no comment or Retry ever lands inside a deploy overlap. **Proposed behavior** `--workload staging-mix` plans a mix of OpenCode, Codex and Claude tasks (default 60/25/15). Tasks keep arriving for the whole wave. Each task runs 15–45 minutes of foreground tool rounds, parents create subtasks, and follow-up comments and Retries wait for an overlap window. Each task has a deliverable that the harness checks from the result. **Reason and benefit** Release-swap waves need a workload shaped like staging. A pass then means the work was done, not only that a status says `done`. ## What Changed - `staging-mix.ts`: the options (`--workload staging-mix`, `--mix-*`), the seeded plan (interleaved adapter ratio, arrivals over the whole wave, one fault slot per `--mix-fault-spacing-minutes`), the SM task (rounds of a hashing shell command, subtasks, `DONE-<nonce>` comment) and its oracle. - The oracle recomputes each round's chained POSIX `cksum` hash. It fails a missing, short or wrong round, a repeated round, comment or subtask (duplicated side effects), a follow-up that did not reach the ledger, and a parent that did not read its subtasks' results. - The follow-up and Retry step waits for an overlap window, for at most `--mix-hold-minutes`. A swap driver registers the window with `setOverlapProbe`. Until then, the window is a fault in progress while the server still answers (K15 overlap). Retries go through the observer's existing click path, so R3 still counts them. - `staging-mix-run.ts`: the arrival gate, the end of arrivals, and the resource guard. The guard refuses `--mix-runs` above the local cap, samples the harness process tree's memory and CPU and the host's available memory every 30 s, and holds launches while memory is low. - The three agents' `maxConcurrentRuns` split the runs by the ratio. The harness keeps runs plus `--mix-queue` tasks in flight, so a queue forms in Paperclip. - Small hooks in `config.ts`, `types.ts`, `main.ts`, `observer.ts`, `seed.ts`, `workloads.ts` and `journeys.ts`. `supervisor.ts` is unchanged. - README section "Staging mix", and `staging-mix.test.ts` (18 unit tests). ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 54 files, 657 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - The unit tests compare the TypeScript `cksum` with the system `cksum`, and run the round command in a real shell. - Live smoke and sizing results: see the comment on this PR. ## Risks - Low risk. The change is in the test harness only. Existing waves do not set `--workload`, so their plans and grades are unchanged. - A 30-run Daytona wave needs Daytona vCPU headroom above the default 40 vCPU budget. The org is shared with the live fleet, so `RESTART_CHAOS_DAYTONA_VCPU_BUDGET` must be raised only to the real headroom. - Fault slots fire relative to each carrier task's real launch, so a slot can drift when launches wait for memory or the in-flight cap. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>