Files
PaperClipAI/tests
DottaandClaude Opus 5.5 0bac3510db test(restart-chaos): add the staging-mix workload for release-swap waves (#21125)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) restarts the
control plane during real agent tasks and grades each task
> - On staging (2026-11-12), 577 of 1,000 runs failed or were
interrupted across overlap deploys every 40 minutes
> - The harness missed this: its waves run a few short tasks, while
staging runs about 30 agents at once with long tool turns, subtasks,
comments and Retries during each deploy
> - This pull request adds a staging-like workload (`--workload
staging-mix`) that runs under any existing restart cell today, and under
the coming release-swap cell
> - The benefit is one honest number per wave (tasks completed without
human action / tasks started) under a load shaped like staging

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The restart chaos harness workloads. Every current workload is a few
short tasks, so a wave never has 30 runs, a queue, long mid-tool turns,
subtasks, or comments and Retries during a deploy overlap.

**Subsystem affected**
Test harness only: `tests/restart-chaos/`. No product code changes.

**Current behavior**
A wave plans a fixed list of short tasks with one fault each. The
harness holds every board action while a fault is in progress, so no
comment or Retry ever lands inside a deploy overlap.

**Proposed behavior**
`--workload staging-mix` plans a mix of OpenCode, Codex and Claude tasks
(default 60/25/15). Tasks keep arriving for the whole wave. Each task
runs 15–45 minutes of foreground tool rounds, parents create subtasks,
and follow-up comments and Retries wait for an overlap window. Each task
has a deliverable that the harness checks from the result.

**Reason and benefit**
Release-swap waves need a workload shaped like staging. A pass then
means the work was done, not only that a status says `done`.

## What Changed

- `staging-mix.ts`: the options (`--workload staging-mix`, `--mix-*`),
the seeded plan (interleaved adapter ratio, arrivals over the whole
wave, one fault slot per `--mix-fault-spacing-minutes`), the SM task
(rounds of a hashing shell command, subtasks, `DONE-<nonce>` comment)
and its oracle.
- The oracle recomputes each round's chained POSIX `cksum` hash. It
fails a missing, short or wrong round, a repeated round, comment or
subtask (duplicated side effects), a follow-up that did not reach the
ledger, and a parent that did not read its subtasks' results.
- The follow-up and Retry step waits for an overlap window, for at most
`--mix-hold-minutes`. A swap driver registers the window with
`setOverlapProbe`. Until then, the window is a fault in progress while
the server still answers (K15 overlap). Retries go through the
observer's existing click path, so R3 still counts them.
- `staging-mix-run.ts`: the arrival gate, the end of arrivals, and the
resource guard. The guard refuses `--mix-runs` above the local cap,
samples the harness process tree's memory and CPU and the host's
available memory every 30 s, and holds launches while memory is low.
- The three agents' `maxConcurrentRuns` split the runs by the ratio. The
harness keeps runs plus `--mix-queue` tasks in flight, so a queue forms
in Paperclip.
- Small hooks in `config.ts`, `types.ts`, `main.ts`, `observer.ts`,
`seed.ts`, `workloads.ts` and `journeys.ts`. `supervisor.ts` is
unchanged.
- README section "Staging mix", and `staging-mix.test.ts` (18 unit
tests).

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
54 files, 657 tests pass.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- The unit tests compare the TypeScript `cksum` with the system `cksum`,
and run the round command in a real shell.
- Live smoke and sizing results: see the comment on this PR.

## Risks

- Low risk. The change is in the test harness only. Existing waves do
not set `--workload`, so their plans and grades are unchanged.
- A 30-run Daytona wave needs Daytona vCPU headroom above the default 40
vCPU budget. The org is shared with the live fleet, so
`RESTART_CHAOS_DAYTONA_VCPU_BUDGET` must be raised only to the real
headroom.
- Fault slots fire relative to each carrier task's real launch, so a
slot can drift when launches wait for memory or the in-flight cap.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits). Extended reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:18:33 -06:00
..