Files
PaperClipAI/tests
DottaandClaude Opus 5.5 d1d71a7755 test(restart-chaos): staging twin cells, follow-up/queue/subtask grading, and full Daytona cleanup (#21325)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos/`) proves that a
release swap (two servers that overlap, then the old one stops) does not
break running agent work
> - The staging twin runs that swap with two local servers and real
Daytona sandboxes, and it is the gate before any hosted proof
> - A loaded twin wave (48 tasks, 20 runs plus 20 queued, 2 swaps)
reported 31 per-task failures and 2 extra turns, but the product did the
correct thing in almost all of them
> - Three harness defects made the noise: the graders called a required
follow-up run an extra turn, called a healthy queued run an orphan, and
called a parent that waited for its subtasks a stall. Cleanup also left
archived Daytona sandboxes in the shared org on every wave
> - This pull request lands the twin harness work and fixes the three
grader defects and the cleanup gap, each with a red/green harness test
> - The benefit is a stop-rule grade that shows only real failures
(here: 1 Retry banner) and makes the follow-up wait visible, and a wave
that leaves no sandbox behind

## Linked Issues or Issue Description

Refs #21175 and #21227 (the server's comment and child-result delivery
rules; this harness takes their outcome from the server's wake record
and does not re-check them).

**What happened?**

Wave `w-20261113T052703-s7301` (loaded twin L1, on Daytona) graded 31
tasks FAIL and listed 2 extra turns. The release-swap grade was 82 / 82
tasks completed without human action.

- t007 and t010 (OpenCode): a person posted a follow-up comment during
the first run (05:44:56 and 05:49:21). OpenCode takes no input mid-turn.
The first run finished `done` at 05:53 and said "No board comments
arrived". The deferred wake was promoted, and the second run answered
the comment after 16 to 17 minutes in the 20-deep queue. The graders
counted the second run as an extra turn and as a `completion_once`
duplicate.
- t021 to t048: each waited about 483 s queued for a slot in the
deliberate 20-deep queue. U1 allows 5 minutes queued plus 3 minutes
without an owner, so all 28 failed `no_owner_no_action`.
- t001: a parent ended its turn `yielded` to wait for its two subtasks.
The observer saw no run for 3 minutes, called it a stall, and rested it
`in_progress`. The parent completed later through its
`issue_children_completed` wake. R1, R3, R5 and U1 failed.
- Cleanup: every Daytona wave left 1 to 8 archived `paperclip-create-*`
setup sandboxes and reported `remainingSandboxes: []`. Daytona does not
delete archived sandboxes.

**Expected behavior**

- A run that starts with a person's comment that no earlier run received
is required. Only a run that starts with no undelivered input is an
extra turn.
- A run queued while every slot of its agent is taken is healthy for the
declared queue time. A run queued while its agent has a free slot still
fails.
- A parent that waits for open subtasks is in a named wait, not a stall.
- Cleanup deletes the sandboxes that this wave created and fails when
one remains.

**Steps to reproduce**

1. Run the L1 loaded twin (command in the README section "Staging twin
cells").
2. Read `staging-mix-summary.md` and `SUMMARY.md`.

## What Changed

- Cherry-picked the 8 twin harness commits from
`fastloop/staging-twin-loaded`: staging-mix under the release-swap
driver on Daytona, `--server-from` on Daytona, probe agents verified
while the first tasks run, image matched to the `--server-from`
checkout, and the knobs `RESTART_CHAOS_K15_KILL_OLD_AFTER_SECONDS` /
`_KILL_OLD_SWAPS`, `RESTART_CHAOS_K15_FREEZE_NEW_SECONDS` /
`_FREEZE_SWAPS`, `RESTART_CHAOS_MIX_BANNER_REST_SECONDS` and
`RESTART_CHAOS_MIX_FIRST_SLOT_SECONDS`. A misplaced doc block in
`supervisor.ts` is moved back to its function. No code comments are
added.
- Cleanup (`daytona-sweep.ts`): after the environment delete, the
harness deletes every sandbox that carries both of this wave's UUID
labels (company and environment), archived ones included.
`assertHarnessMayMutate` still refuses any sandbox without both labels
and any `fleet-sbx-*` snapshot. The `paperclip-create-*` name refusal is
removed, because every plugin sandbox (the fleet's included) has that
name, so the labels are the real ownership proof. A sandbox left after
deletion fails the wave (`undeletedSandboxes`, warm-shutdown verdict,
`ttft-baseline`).
- Follow-ups (`follow-up-delivery.ts`, `wake-evidence.ts`,
`staging-mix-retry.ts`, `staging-mix.ts`): delivery comes from the
server's own record. A comment that the server judged delivered never
becomes a run. A run that the server started from a wake carrying a
person's comment posted after the first run started (a retry included)
is required, not an extra turn. `completion_once` keeps the first
completion as the baseline and does not count a later completion by such
a run (truth-table test). The grader does not re-check the server's
delivery rule; the server's tests cover it. The summary adds `extra
turns: N` and `follow-up answered in a later run: n, queued wait
p50/max, posted to answered p50/max`.
- Declared queue (`universal.ts`, `agent-slots.ts`,
`wave-grade-inputs.ts`): with `--mix-queue` above 0, U1 allows
ceil(queue / runs) × (longest turn + 5 min) queued (27 minutes for L1).
Only queued time while the run's agent has a free slot counts against
the 5-minute bound. Slot use comes from a new
`wake-evidence/agent-slots.json` (run start/finish and each agent's
`maxConcurrentRuns`).
- Subtask wait (`child-wait.ts`, `observer.ts`, `grader.ts`): an idle
parent after a `yielded` or `blocked` turn, until its
`issue_children_completed` wake, is a named wait (`waitingOnChildren`)
for R5 and U1. The observer no longer gives up on a parent while a
subtask is open. An older record that rested during that wait takes the
completion from the wake evidence.
- Review follow-ups: the first staging-mix slot moves earlier without
adding a swap; each sandbox delete pass has a deadline; a frozen
incoming server is always resumed (`server-freeze.ts`); the first
completion is always the baseline for `completion_once` (truth-table
test); a follow-up run that failed before its turn keeps the answer owed
for its retry; `agent-slots.json` saves the server's effective run cap
(`maxConcurrentRunsSql`); the wave-end cleanup flow moved to
`daytona-cleanup.ts` (`main.ts` stays under 1,000 lines).
- README: one-command entries for the twin cell (G, G2, G3) and L1, the
knobs, the stop-rule grading, the new summary lines, the U1 queue rule,
the subtask wait and the cleanup rule.

## Verification

- Red/green tests (red = new tests against the previous source):
- Cleanup, `daytona-sweep.test.ts`: red 4 failed (for example "expected
[] to deeply equal [ 'archived', 'archiving', 'started' ]": nothing was
deleted). Green: pass.
- Follow-ups, declared queue, subtask wait (`staging-mix-retry.test.ts`,
`universal.test.ts`, `child-wait.test.ts`, `wake-evidence.test.ts`): red
6 failed (for example the t010 run listed as an extra turn, U1 fail for
the t030 queue shape, t001 "rested_not_done"). Green: pass. The tests
use the real timings of t001, t010 and t030.
- Full harness suite: `npx vitest run --config
tests/restart-chaos/vitest.config.ts`: 71 files, 811 tests pass. `npx
tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- `pnpm verify` exits 2 (the harness README is a global CI input).
Required CI supplies the broad coverage.
- One-off offline analysis (a scratch script, not part of this PR): the
saved wave `w-20261113T052703-s7301` regraded with these graders, from a
copy of its results:

| | Before | After |
| --- | --- | --- |
| Per-task FAIL | 31 (t001 R1, t007 and t010 `completion_once`, t021 to
t048 U1) | 0 |
| Extra turns | 2 (t007, t010) | 0 |
| Retry banner shown | 1 (t020, 3 s, completed without a click) | 1
(t020) |
| Follow-up answered in a later run | not shown | 4: t001, t007, t010,
t012. Queued wait p50 983 s, max 1739 s. Posted to answered p50 1420 s,
max 2255 s |

t001 and t012 started their follow-up run before completion, so they
were never extra turns. They are listed because the wait is real. This
wave predates `agent-slots.json`, so the regrade used only the declared
queue bound for U1.
- No Daytona wave ran for this PR. There is no live proof here that the
cleanup deletes archived sandboxes in the real org or that the new
evidence file is written on a live wave. The delete path uses the same
`DELETE /sandbox/<id>` call as the Daytona SDK's `deleteSandbox`.

## Risks

- The harness now deletes Daytona sandboxes in an org that the Cloud
fleet shares. The guard is the wave's two freshly minted UUID labels,
filtered on the server and checked again on every row, plus the
`fleet-sbx-*` refusal. A sandbox without both labels is never deleted.
- When the server wrongly re-delivers a comment that a finished run
already received, the harness counts that run as a required follow-up,
not as an extra turn. The server's own delivery tests catch that class,
not this grader. `completion_once` still fails when that run also
reports completion and is not the first.
- The declared queue bound is generous (27 minutes for L1). A stuck
queue with full slots for longer than that still fails. A queue with a
free slot fails after 5 minutes when slot evidence exists. Without slot
evidence only the declared bound applies.
- Test harness only. No product code changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:03:08 -06:00
..