mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) proves that a release swap (two servers that overlap, then the old one stops) does not break running agent work > - The staging twin runs that swap with two local servers and real Daytona sandboxes, and it is the gate before any hosted proof > - A loaded twin wave (48 tasks, 20 runs plus 20 queued, 2 swaps) reported 31 per-task failures and 2 extra turns, but the product did the correct thing in almost all of them > - Three harness defects made the noise: the graders called a required follow-up run an extra turn, called a healthy queued run an orphan, and called a parent that waited for its subtasks a stall. Cleanup also left archived Daytona sandboxes in the shared org on every wave > - This pull request lands the twin harness work and fixes the three grader defects and the cleanup gap, each with a red/green harness test > - The benefit is a stop-rule grade that shows only real failures (here: 1 Retry banner) and makes the follow-up wait visible, and a wave that leaves no sandbox behind ## Linked Issues or Issue Description Refs #21175 and #21227 (the server's comment and child-result delivery rules; this harness takes their outcome from the server's wake record and does not re-check them). **What happened?** Wave `w-20261113T052703-s7301` (loaded twin L1, on Daytona) graded 31 tasks FAIL and listed 2 extra turns. The release-swap grade was 82 / 82 tasks completed without human action. - t007 and t010 (OpenCode): a person posted a follow-up comment during the first run (05:44:56 and 05:49:21). OpenCode takes no input mid-turn. The first run finished `done` at 05:53 and said "No board comments arrived". The deferred wake was promoted, and the second run answered the comment after 16 to 17 minutes in the 20-deep queue. The graders counted the second run as an extra turn and as a `completion_once` duplicate. - t021 to t048: each waited about 483 s queued for a slot in the deliberate 20-deep queue. U1 allows 5 minutes queued plus 3 minutes without an owner, so all 28 failed `no_owner_no_action`. - t001: a parent ended its turn `yielded` to wait for its two subtasks. The observer saw no run for 3 minutes, called it a stall, and rested it `in_progress`. The parent completed later through its `issue_children_completed` wake. R1, R3, R5 and U1 failed. - Cleanup: every Daytona wave left 1 to 8 archived `paperclip-create-*` setup sandboxes and reported `remainingSandboxes: []`. Daytona does not delete archived sandboxes. **Expected behavior** - A run that starts with a person's comment that no earlier run received is required. Only a run that starts with no undelivered input is an extra turn. - A run queued while every slot of its agent is taken is healthy for the declared queue time. A run queued while its agent has a free slot still fails. - A parent that waits for open subtasks is in a named wait, not a stall. - Cleanup deletes the sandboxes that this wave created and fails when one remains. **Steps to reproduce** 1. Run the L1 loaded twin (command in the README section "Staging twin cells"). 2. Read `staging-mix-summary.md` and `SUMMARY.md`. ## What Changed - Cherry-picked the 8 twin harness commits from `fastloop/staging-twin-loaded`: staging-mix under the release-swap driver on Daytona, `--server-from` on Daytona, probe agents verified while the first tasks run, image matched to the `--server-from` checkout, and the knobs `RESTART_CHAOS_K15_KILL_OLD_AFTER_SECONDS` / `_KILL_OLD_SWAPS`, `RESTART_CHAOS_K15_FREEZE_NEW_SECONDS` / `_FREEZE_SWAPS`, `RESTART_CHAOS_MIX_BANNER_REST_SECONDS` and `RESTART_CHAOS_MIX_FIRST_SLOT_SECONDS`. A misplaced doc block in `supervisor.ts` is moved back to its function. No code comments are added. - Cleanup (`daytona-sweep.ts`): after the environment delete, the harness deletes every sandbox that carries both of this wave's UUID labels (company and environment), archived ones included. `assertHarnessMayMutate` still refuses any sandbox without both labels and any `fleet-sbx-*` snapshot. The `paperclip-create-*` name refusal is removed, because every plugin sandbox (the fleet's included) has that name, so the labels are the real ownership proof. A sandbox left after deletion fails the wave (`undeletedSandboxes`, warm-shutdown verdict, `ttft-baseline`). - Follow-ups (`follow-up-delivery.ts`, `wake-evidence.ts`, `staging-mix-retry.ts`, `staging-mix.ts`): delivery comes from the server's own record. A comment that the server judged delivered never becomes a run. A run that the server started from a wake carrying a person's comment posted after the first run started (a retry included) is required, not an extra turn. `completion_once` keeps the first completion as the baseline and does not count a later completion by such a run (truth-table test). The grader does not re-check the server's delivery rule; the server's tests cover it. The summary adds `extra turns: N` and `follow-up answered in a later run: n, queued wait p50/max, posted to answered p50/max`. - Declared queue (`universal.ts`, `agent-slots.ts`, `wave-grade-inputs.ts`): with `--mix-queue` above 0, U1 allows ceil(queue / runs) × (longest turn + 5 min) queued (27 minutes for L1). Only queued time while the run's agent has a free slot counts against the 5-minute bound. Slot use comes from a new `wake-evidence/agent-slots.json` (run start/finish and each agent's `maxConcurrentRuns`). - Subtask wait (`child-wait.ts`, `observer.ts`, `grader.ts`): an idle parent after a `yielded` or `blocked` turn, until its `issue_children_completed` wake, is a named wait (`waitingOnChildren`) for R5 and U1. The observer no longer gives up on a parent while a subtask is open. An older record that rested during that wait takes the completion from the wake evidence. - Review follow-ups: the first staging-mix slot moves earlier without adding a swap; each sandbox delete pass has a deadline; a frozen incoming server is always resumed (`server-freeze.ts`); the first completion is always the baseline for `completion_once` (truth-table test); a follow-up run that failed before its turn keeps the answer owed for its retry; `agent-slots.json` saves the server's effective run cap (`maxConcurrentRunsSql`); the wave-end cleanup flow moved to `daytona-cleanup.ts` (`main.ts` stays under 1,000 lines). - README: one-command entries for the twin cell (G, G2, G3) and L1, the knobs, the stop-rule grading, the new summary lines, the U1 queue rule, the subtask wait and the cleanup rule. ## Verification - Red/green tests (red = new tests against the previous source): - Cleanup, `daytona-sweep.test.ts`: red 4 failed (for example "expected [] to deeply equal [ 'archived', 'archiving', 'started' ]": nothing was deleted). Green: pass. - Follow-ups, declared queue, subtask wait (`staging-mix-retry.test.ts`, `universal.test.ts`, `child-wait.test.ts`, `wake-evidence.test.ts`): red 6 failed (for example the t010 run listed as an extra turn, U1 fail for the t030 queue shape, t001 "rested_not_done"). Green: pass. The tests use the real timings of t001, t010 and t030. - Full harness suite: `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 71 files, 811 tests pass. `npx tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean. - `pnpm verify` exits 2 (the harness README is a global CI input). Required CI supplies the broad coverage. - One-off offline analysis (a scratch script, not part of this PR): the saved wave `w-20261113T052703-s7301` regraded with these graders, from a copy of its results: | | Before | After | | --- | --- | --- | | Per-task FAIL | 31 (t001 R1, t007 and t010 `completion_once`, t021 to t048 U1) | 0 | | Extra turns | 2 (t007, t010) | 0 | | Retry banner shown | 1 (t020, 3 s, completed without a click) | 1 (t020) | | Follow-up answered in a later run | not shown | 4: t001, t007, t010, t012. Queued wait p50 983 s, max 1739 s. Posted to answered p50 1420 s, max 2255 s | t001 and t012 started their follow-up run before completion, so they were never extra turns. They are listed because the wait is real. This wave predates `agent-slots.json`, so the regrade used only the declared queue bound for U1. - No Daytona wave ran for this PR. There is no live proof here that the cleanup deletes archived sandboxes in the real org or that the new evidence file is written on a live wave. The delete path uses the same `DELETE /sandbox/<id>` call as the Daytona SDK's `deleteSandbox`. ## Risks - The harness now deletes Daytona sandboxes in an org that the Cloud fleet shares. The guard is the wave's two freshly minted UUID labels, filtered on the server and checked again on every row, plus the `fleet-sbx-*` refusal. A sandbox without both labels is never deleted. - When the server wrongly re-delivers a comment that a finished run already received, the harness counts that run as a required follow-up, not as an extra turn. The server's own delivery tests catch that class, not this grader. `completion_once` still fails when that run also reports completion and is not the first. - The declared queue bound is generous (27 minutes for L1). A stuck queue with full slots for longer than that still fails. A queue with a free slot fails after 5 minutes when slot evidence exists. Without slot evidence only the declared bound applies. - Test harness only. No product code changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>