PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-12 06:10:44 +01:00

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	bf67e2fbd9	test(chaos): honor 429 retry delays on harness reads (#20194 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness collects each wave's speed from the board API after the tasks rest, including the M1 subtasks' runs > - The server rate-limits concurrent issue-list requests per client and answers 429 with `retryAfterSeconds` > - After a 95-task storm, the harness requested every M1 parent's subtask list at once. The server returned 429, speed collection failed, and the wave exited 1 with no speed grade > - This pull request makes harness GET requests honor a 429's retry delay with a bounded number of attempts, and collects subtask speed one parent at a time > - The benefit is that a busy wave keeps its speed grade, and the harness behaves like a polite API client ## Linked Issues or Issue Description Refs #20071, #20147 What happened? Wave `w-20261109T055902-s9402` (K13 storm, A1/C1/M1, 95 tasks) finished its tasks. Speed collection then failed with: `GET /api/companies/…/issues?view=index&parentIds=…&limit=20 returned 429: {"error":"Too many concurrent issue-list requests for this actor/client","retryAfterSeconds":1}`. `main.ts` ran `childSpeedTasks()` for every observer in parallel. Expected behavior - The harness waits and retries a rate-limited read. - It does not flood the issue-list endpoint while collecting speed. - A busy wave still gets graded. ## What Changed - `api.ts`: `ChaosApi` retries a GET that returns 429. - It waits for the delay from `Retry-After` or the body's `retryAfterSeconds` (default 1 s, capped at 10 s), scaled by the attempt number. - It makes at most 5 attempts by default. - Writes (POST/PUT/PATCH/DELETE) are never retried, so a user action is never repeated. - `rateLimitDelayMs` is exported for tests. - `main.ts`: subtask speed is collected one observer at a time instead of with `Promise.all`. - New `api.test.ts` against a real local HTTP server: - a GET that gets 429 twice, then 200, succeeds; - a POST that gets 429 is sent once; - retries stop at the bound; - delay parsing (header, body, cap, default). ## Verification - `npx vitest run` in `tests/restart-chaos`: 216 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - A persistently rate-limited read now takes up to about 15 s before it fails, instead of failing at once. - Collecting subtask speed one parent at a time adds a few seconds to the end of large M1 waves. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 01:06:41 -06:00
DottaandClaude Opus 5.5	0ddc9f1ef1	test(chaos): count an M1 subtask stall only when its recovery needs the board (#20178 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The M1 chaos oracle checks that a parent gets exactly one `issue_blocker_stalled` wake per stalled subtask (#20147) > - The product wakes a parent only for a stall that needs the board. The harness recorded a stall for any blocked subtask, whatever its latest recovery action > - In the Daytona M1 wave, a subtask blocked twice on agent- and system-owned recovery that resolved by itself. The harness counted two stalls and failed the parent for getting no wake, a false failure > - This pull request records a stall only when the subtask's latest recovery action needs the board, matching the product's definition > - The benefit is that `parent_woken_once_per_stall` fails only for a wake the product owed ## Linked Issues or Issue Description Refs #20147, #20098 What happened? In wave `w-20261109T041248-s9501`, t10's subtask 195b5fd7 was blocked twice. Action 56d036e0 was agent-owned (`native_session_interrupted`) and was cancelled. Action 53eb91b0 was system-owned (`native_continuation_requires_reconciliation`) and resolved with outcome cancelled. Neither needed the board, so the product correctly sent no stall wake. The harness still failed `parent_woken_once_per_stall` with 2 stalls and 0 wakes. Expected behavior A stall counts when the subtask's latest recovery action is: - open (active or escalated) and board-owned, or escalated; or - board-owned and resolved with outcome blocked, with the subtask still blocked. That is the definition the product's blocker-stall wake uses. ## What Changed - `observeChildStalls` reads the parent's `stalledDependencies` from `GET /api/issues/:id`. That is the product's own stalled-dependency view (`listStalledDependencies`, built with the same `readStalls` the blocker-stall wake uses). - Each `relation: "child"` entry is recorded as a stall (`stalledIssueId`, `recoveryAction.id`). The harness no longer infers stalls from `/recovery-actions`. That endpoint lists only open actions and selected holds, and can omit a board action resolved with outcome blocked. - Clicking the board Retry on a blocked subtask is unchanged: it still happens whenever the product offers Retry, and it still counts as a needed Retry. - Tests: - a stall the product lists is recorded; - for t10's two non-board recovery states, the product lists nothing, so no stall is recorded, and the subtask's Retry is still clicked. ## Verification - `npx vitest run` in `tests/restart-chaos`: 212 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - The harness checks `stalledDependencies` every 10 s. A stall that appears and clears inside one interval is not recorded, so its wake goes unchecked. It is never reported as a false failure. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 23:36:40 -06:00
DottaandClaude Opus 5.5	61fa13c6d8	test(chaos): add S1, a board Stop across a restart then a continue comment (#20161 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness drives real user actions across control-plane restarts: comments, Retry, approvals and budget resumes > - The board's Stop button on a running task was not exercised. The cancel route stamps the stop as operator-initiated so that recovery stands down instead of re-waking the agent > - A restart right after a Stop is where recovery could wrongly restart work the operator stopped, or lose the operator's later "continue" direction > - This pull request adds S1: stop the running run, fire a fault right after, then comment to continue > - The benefit is a graded check that a Stop holds across a restart and that a later continue comment finishes the task ## Linked Issues or Issue Description Refs #20071 What is missing? The campaign's scenario list includes mid-run stops. C1 covers a mid-run comment (a steer), but no workload clicks Stop. Without one, a regression where recovery restarts operator-stopped work, or drops the continue comment, would go unseen. Expected behavior - After the board stops a run and a restart follows, the stopped run stays cancelled. - Nothing runs until the operator comments. - The continue comment then finishes the task. ## What Changed - S1 workload ("Stop while running, then continue"): a brief task. At its follow-up time, while a run is running, the observer posts `POST /api/heartbeat-runs/:id/cancel` (the board's Stop) and records `stop`. - New `stop` trigger, for S1 only: the planned fault fires 0–2 s after the stop. - The continue comment is posted only after the stop has settled: at least 20 s after the stop, with no run running or queued and no fault in progress. It asks the agent to continue and add a `CONTINUE-<nonce>` line. - The oracle requires: - status done; - the brief with the chaos and continue markers; - `stop_took_effect`: the stopped run ended `cancelled`; - `nothing_ran_after_stop`: no run was created between the stop and the continue comment. - S1 is not a default workload. There is a README section. ## Verification - `npx vitest run` in `tests/restart-chaos`: 205 passed. New tests cover: - the S1 grade (pass, a run restarted after the stop, the stop not taking effect, no stop); - the trigger-to-workload pairing; - an observer test that clicks Stop once and posts the continue comment only after the 20 s settle. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - A live S1 wave runs after the current chaos wave. Its result will be posted here before merge. ## Risks - If the product moves the stopped run to a status other than `cancelled`, `stop_took_effect` fails. That check is intentional. - The continue comment waits for the stop to settle, so a run stuck `queued` delays it. R5 and the wave deadline still bound the wait. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 22:52:48 -06:00
DottaandClaude Opus 5.5	f316562045	test(chaos): wait for new agents to be ready before the first assignment (#20158 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades each run's time to first activity against the speed budgets > - The performance session traced an 8–12 s "first claim" delay on every wave's first cold runs. A brand-new agent spends about 8 s in `lifecycleState: verifying` while its harness runs a real hello probe, and queued runs start only when it becomes ready > - The chaos waves create agents and assign the first tasks in the same second, so those runs carry the one-time verification. Their cold first-activity and boot looked over budget even though dispatch after ready takes 40–60 ms > - This pull request makes the waves wait for each agent to be ready before the first assignment, records how long verification took, and fails the wave if an agent never verifies > - The benefit is that cold-start grades measure task startup, while agent verification stays visible as its own number ## Linked Issues or Issue Description Refs #20119 What happened? In waves `w-20261109T021322-s9801` (G1) and the K4/K6 wave, every new agent's first cold runs waited 8–12 s in `heartbeat.queue`. The performance session reproduced it locally: the agent is in `lifecycleState: verifying` while a hello probe runs, and its queued runs start on `onReady`. `ttft-baseline.ts` already waits for `ready` before launching tasks, so it never showed the delay. Expected behavior The chaos waves start their first task after the agent is ready, as the baseline does, and they report the verification time. ## What Changed - `seed.ts`: new `waitForAgentsReady` polls each new agent until `lifecycleState` is `ready`, with a 4-minute cap. - It records `seed_agent_ready` with `readyAfterMs` per agent on the timeline. - It throws, failing the wave, if an agent reports a `lifecycleError` or never becomes ready. - `seedChaosLab` runs it when `waitForReadyAgents` is set. `main.ts` sets it for every wave. `ttft-baseline.ts` keeps its own readiness logic, which skips unready adapters. - The seed test's fake API answers agent reads as ready. ## Verification - `npx vitest run` in `tests/restart-chaos`: 205 passed. New tests cover the wait, the verification-error failure and the timeout failure. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - Waves start about 8 s later, once per wave. - A product regression that keeps agents in `verifying` now fails the wave at seeding with a clear error, instead of showing as slow first tasks. - The one-time verification delay a user sees on a new agent's first task is no longer in the cold grade. It stays in the timeline (`seed_agent_ready`) for anyone tracking it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 21:32:11 -06:00
DottaandClaude Opus 5.5	0502cc9fac	test(chaos): retry a blocked M1 subtask and check the parent's stall wake (#20147 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades the M1 scenario: a parent task delegates two subtasks across restarts > - When a subtask's run failed recovery, the harness only clicked Retry on the parent. The parent's Retry is correctly held behind the blocked subtask (#20109), so the task waited until its deadline > - A real board user would also click the subtask's Retry. #20098 also added a wake for a parent whose subtask stalls, and the grade did not check it > - This pull request clicks the board Retry on a blocked subtask (graded as a needed Retry) and checks that the parent got exactly one stall wake per stall > - The benefit is that M1 grades the full user path and proves the stall wake from the product's own activity record ## Linked Issues or Issue Description Refs #20098, #20109, #20094 What happened? In wave `w-20261109T003126-s9701`, the M1 parent t05 rested blocked behind a blocked codex subtask. The harness clicked Retry on the parent. The product held that Retry ("waiting on a blocked task") as designed. Nothing retried the subtask, and the task waited 38 minutes until the wave deadline. The wave also could not show whether the parent got the new `issue_blocker_stalled` wake. Expected behavior The harness acts like the board: it clicks Retry on a blocked subtask that offers one, and it counts that as a needed Retry. The M1 oracle checks the parent's activity for one `issue.blocker_stalled_wake` per stalled subtask. ## What Changed - `observer.ts`: - For M1, the observer checks the parent's subtasks every 10 s while the parent is open, independent of the parent's rest window. - Each blocked subtask is recorded as a stall: the subtask plus its latest recovery action, from `GET /api/issues/:id/recovery-actions`. - When a blocked subtask's interruption offers Retry, the harness posts the board Retry. It clicks once per subtask; a click that was not accepted can be tried again on a later check. - The parent's `issue.blocker_stalled_wake` activity entries are collected as `stallWakes` through `GET /api/issues/:id/activity`. - `grader.ts`: R3 fails with `retry_needed_on_subtask_<n> (<ids>)` when a subtask needed a board Retry, unless the parent's own Retry already failed R3. - `workloads.ts`: the new M1 check `parent_woken_once_per_stall` is added when a subtask stalled. Each (subtask, recovery action) stall must have exactly one matching wake. A stall with an unknown action matches any wake for that subtask. ## Verification - `npx vitest run` in `tests/restart-chaos`: 203 passed. New tests cover: - an observer test: a blocked subtask gets one Retry click, and the parent is still watched; - the R3 subtask-retry grade; - the stall-wake check: woken once, not woken, woken twice, two stalls of one subtask with only one woken, and no stall. - G1, which also has subtasks, is unaffected: the existing governance tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - The harness now acts on subtasks, which it did not before. It clicks once per subtask and only when the product offers Retry. - The stall-wake check reads activity through the board API. If the API omits these entries for a board user, the check fails visibly instead of passing. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 20:12:03 -06:00
DottaandClaude Opus 5.5	89265f7312	test(chaos): add R1, a code change in a fresh clone of a real repository (#20111 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness proves that real agent tasks survive control-plane restarts > - Its coding workloads run in a tiny chaoscli fixture. Real work happens in large repositories, where workspace setup, copy and restore cost more and fail differently > - The campaign asks for real coding tasks in a fresh Paperclip clone, run under restarts > - This pull request adds the smallest support for that: a `--workspace-repo` option and an R1 workload > - The benefit is restart coverage for code changes in a real repository, graded by running the agent's own test and a hidden case ## Linked Issues or Issue Description Refs #20071, #20089 What is missing? The harness can only seed the generated chaoscli fixture as the project workspace. There is no way to run restart chaos against a code change in a real repository. Expected behavior An operator can name a git repository. The harness clones it fresh as the workspace, and a workload asks for a real code change in it with an oracle that runs the result. ## What Changed - `--workspace-repo <git repository>`: `seedChaosLab` clones it (shallow, from committed HEAD, `--no-hardlinks`) as the project workspace instead of the chaoscli fixture. Local only. - R1 workload: the agent adds `packages/shared/src/chaos/slug-<nonce>.ts` with `chaosSlug` and a `node:test` file, and runs it. The nonce names keep concurrent tasks from colliding in one repository. - The R1 oracle runs the agent's test with `node --test` and a hidden case against the module through Node's type stripping. - Collect commands can take an `argv` function of the nonce. - Config rules: R1 and `--workspace-repo` go together. B1 and B5 (which need the chaoscli fixture) do not combine with it. R1 is not a default workload. - README section "Code change in a real repository". ## Verification - `npx vitest run` in `tests/restart-chaos`: 195 passed, including new tests for the config pairing, R1 grading with the nonce argv, and a shallow clone of a fixture repository. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - I checked the oracle commands by hand on Node 26: `node --test` on a `.ts` test and `node --input-type=module -e 'import … from "./…ts"'` both work. - A live R1 wave against a Paperclip clone runs after the current chaos wave frees the harness slot. Its result will be posted here before merge. ## Risks - A shallow clone of a large repository adds seconds to wave seeding. That is one clone per wave. - The oracle uses Node's built-in type stripping. A runtime without it fails `node_test` and `hidden_case`, which shows as an oracle failure, not a false pass. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 16:55:02 -06:00
DottaandClaude Opus 5.5	42a5d71cb7	test(chaos): separate workspace provisioning from agent boot; grade outage-created runs as restart (#20119 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades speed against the native speed budgets, and a breach fails the wave > - In the real-repository R1 wave, the claude cold boot p95 was 31 s against 20.5 s, and cold first activity p95 was 37 s against 30 s > - The performance session traced both. The boot breach was the repository's own provisioning script on the first worktree creation (21–24 s), not Paperclip startup. The first-activity breach was a run created during an outage, behind the next fault > - This pull request reports provisioning on its own row, keeps it out of agent boot, and grades runs created during an outage as restart > - The benefit is that a speed breach points at Paperclip startup, not at project setup scripts or injected downtime ## Linked Issues or Issue Description Refs #20106, #20111 What happened? Wave `w-20261108T215822-s9805` (R1 in a Paperclip clone) reported a claude cold boot p95 of 31.0 s. The slow run spent 22.8 s in `heartbeat.prepare.workspace_provision`, the repository's `scripts/provision-worktree.sh`. Reuse takes 0.3 s. Its cold first-activity p95 came from run 630088a4, created 4 s after a fault killed its predecessor. That run queued until the next fault cleared, and #20106 still graded it cold because no fault started inside its own window. Expected behavior Project provisioning is reported, not charged to agent boot. A run created while a fault's outage is in progress is graded as restart. ## What Changed - `faultBeforeFirstActivity` takes outage windows and returns true when any window, from fault start to ready, overlaps the run's span from creation to its first activity. It used to take fault start instants only. - A run with `retryOfRunId` (the successor of an interrupted run) is graded as restart. - `collectWaveSpeed` reads each run's workspace operations. Provisioning counts only when a `workspace_provision` operation has `metadata.created: true`, meaning first-time creation of the workspace; reuse (about 0.5 s) is left in boot. - `bootWithoutProvisionMs` subtracts that first-time `heartbeat.prepare.workspace_provision` span from agent boot for the wave grade. Per-run boot breaches use it too. - New report-only row `workspace_provision` per adapter and start, shown only when the run created its workspace. It has no budget. - `runMetrics` and the TTFT baseline tool are unchanged. `BaselineRun` gains an optional `workspaceCreated`. ## Verification - `npx vitest run` in `tests/restart-chaos`: 195 passed. New cases cover a run created during an outage, a retry successor, provisioning counted only for `created: true` through `collectWaveSpeed`, the provisioning row and boot without provisioning. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - A slow provisioning step no longer fails the boot budget. It stays visible on its own row, with no budget, because it runs the project's own script. - A run created during any outage, and any retry successor (including a user Retry), is graded as restart even when it started quickly. Restart recovery still measures it. - One extra API read per run (workspace operations) during speed collection. A failed read counts as not created, so boot is never under-charged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 16:54:58 -06:00
DottaandClaude Opus 5.5	83a229ae8c	test(chaos): grade a fault-hit run as a restart, not a cold start (#20106 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades speed against the native speed budgets, and a regression fails the wave > - It grades a run's time to first activity as a cold or warm start, and restart recovery separately > - Fault receipts carry no run ids, so a run that a fault hit before its first activity was graded as a cold start > - In a K4/K6 wave, the two slowest codex cold samples were runs that a K6 SIGKILL hit before their first activity, so the cold p95 breach was not a cold start > - This pull request grades such a run as a restart > - The benefit is that cold and warm rows measure only starts with no fault, and restart cost stays in restart recovery ## Linked Issues or Issue Description Refs #20071, #20094 What happened? Wave `w-20261108T200620-s9701` reported a codex cold time-to-first-activity p95 of 29.9 s against a 17 s budget. The two slowest samples were runs that a K6 restart hit before their first activity. Expected behavior A run whose first-activity window contains a fault start is graded as a restart, not as a cold or warm start. Actual behavior The run was graded as a cold start, and the restart time counted as cold-start latency. ## What Changed - `faultBeforeFirstActivity(run, faultStarts)`: true when a fault started between the run's trace start and the end of its raw first-activity span (or first-token span when no activity span exists). - `collectWaveSpeed` sets such a run's phase to `restart` before it removes outage time. Cold, warm and single-run grading already skip restart runs. The run still counts in restart recovery through the fault's run timing. - A test covers the helper and the exclusion from the cold row. ## Verification - `npx vitest run wave-speed` in `tests/restart-chaos`: 10 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - A run with no first-activity or first-token span counts as hit when any fault starts after its trace start. Such a run has no cold sample to grade anyway. - A fault that starts just before a run's trace start does not count, and outage removal handles it as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 15:30:45 -06:00
DottaandClaude Opus 5.5	88835e624a	test(chaos): fail M1 fast on a blocked subtask and grade interrupted waves (#20094 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness proves that real agent tasks survive control-plane restarts > - In a K4/K6 wave with the M1 multi-agent scenario, one subtask failed. Its parent waited on it, and the harness kept watching for over 50 minutes > - When the operator stopped the wave, the harness wrote no grade files, so the 17 finished tasks had no grade > - Separately, master's harness test for time to first activity failed after #20077 changed how time is measured from run creation > - This pull request makes M1 fail fast on a blocked subtask, grades the finished tasks of an interrupted wave, and aligns the speed grade with #20077 > - The benefit is shorter waves with a grade for every task, and a green harness suite on master ## Linked Issues or Issue Description Refs #20071, #20077, #20088 What happened? Wave `w-20261108T181313-s9701` (local, codex and claude, A1/C1/M1, K4 and K6) ran 71 minutes against a planned 30. Task t05 (M1) delegated CHA-8 and CHA-9. The CHA-8 run failed after a K4 restart (the defect that #20088 fixes), and CHA-8 went to `blocked`. The observer kept waiting for CHA-8 to finish. After SIGINT, the harness threw before grading, so no task had a grade file. Also, `wave-speed.test.ts` fails on master: #20077 changed `runMetrics` to measure from the start of `heartbeat.queue`. Steps to reproduce 1. Run an M1 task whose subtask run fails and leaves the subtask `blocked`. 2. See the observer wait until the parent deadline. 3. Send SIGINT to the harness and see that no grade files are written. Expected behavior A blocked subtask ends the wait, and the parent is graded as a failure with a `task_child_blocked` timeline entry. An interrupted wave grades every finished task and fails the wave. ## What Changed - `observer.ts`: `childrenPending` treats a `blocked` subtask as settled, so the parent rests and is graded. The timeline records `task_child_blocked` with the subtask ids. - `observer.ts`: `markUnfinished` closes an open task at interrupt as unfinished (`timedOut`, last seen status) and records `task_unfinished_at_interrupt`. - `main.ts`: an interrupt now sets the harness error and grades the wave, instead of throwing before the grade. The wave still fails. - `wave-speed.ts`: `firstActivityMs` uses `runMetrics().firstActivityAfterRunCreatedMs` from #20077, so there is one definition of time from run creation. - `wave-speed.test.ts`: the fixture starts `heartbeat.queue` when the comment wait ends, as in real runs. ## Verification - `npx vitest run` in `tests/restart-chaos`: 172 passed. Before this change, the wave-speed test failed on master. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - A subtask that is `blocked` only for a short time and then resumes would end the parent's watch after the 60-second grace. The grade then shows a failure that a longer wait could avoid. M1 subtasks have no blockers, so `blocked` means a stuck subtask. - An interrupted wave grades tasks while its server is still up. If the server is already down, artifact collection fails, and those tasks are graded as failures. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 14:20:57 -06:00
DottaandClaude Opus 5.5	c3c226d165	test(restart-chaos): governance (approval, budget hard stop) and upgrade mid-run scenarios (#20089 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness kills and restarts the control plane during real agent tasks and grades every task > - It covers run lifecycle, comments, delegation and long tools, but not two areas that must also survive a restart: governance (approvals and budget hard stops) and an upgrade in the middle of a run > - An approval that disappears, runs twice or runs after a rejection, or a budget pause that dispatches work, breaks governance. An upgrade that cannot resume saved state breaks the upgrade-safe-saved-state rule > - This pull request adds the harness support for both scenarios, using only the paths a real board user and agent use > - The benefit is that the next chaos waves can test these boundaries and report failures with evidence ## Linked Issues or Issue Description Refs #20084, #20067 What happened? The restart chaos harness had no scenario for an approval-gated action pending across a restart, for a budget hard stop crossed mid-run with a restart, or for a server upgrade in the middle of a run. These boundaries were untested. Expected behavior - G1: a pending approval survives K1, K2 or K4. Approving it after the restart runs the gated action exactly once; rejecting it never runs it. - H1: a budget hard stop crossed mid-run pauses the company, nothing dispatches while paused (also across a restart), and resuming continues the work. - K14: a wave started on an older build and upgraded mid-turn resumes old-saved state on the new build with no Retry, no lost comments and no stuck records. Steps to reproduce 1. Run the harness commands in `tests/restart-chaos/README.md` ("Governance across restarts" and "Upgrade mid-run"). 2. Each task is graded R1–R6 plus its oracle. Paperclip version or commit master at `4f7eee10c2`. ## What Changed - G1 workload: approval across a restart. - The agent requests board approval through its `call_api` tool (`POST /api/companies/{companyId}/approvals`, `request_board_approval`, linked to its task), blocks, and ends its turn. - A new `approval` trigger fires the fault once the observer sees the approval pending. - After the restart, with the server ready, the observer acts as the board through `POST /api/approvals/:id/approve` or `/reject` (rejection also cancels the task). The decision is fixed by the nonce. - The oracle checks one request, still pending when decided after the restart, the decision, and the gated subtask exactly once on approval and never after rejection. - R1 and R3 grade a rejected task against `cancelled` (`expectedFinalStatus`). - H1 workload: budget hard stop across a restart. - Before launch the observer sets the company budget to observed spend plus one cent through `PATCH /api/companies/:id/budgets`. - A new `paused` trigger fires the fault once the company shows `paused` / `budget`. - After the restart the pause must hold 30 s with the server ready. The observer records runs that started while paused, then resolves the incident with `raise_budget_and_resume`. - The oracle checks the brief, the pause, no dispatch while paused, the resume after the fault, and an active company at rest. - H1 runs alone, at concurrency 1, and not with `--rc06-masked` (config errors). - K14 fault: upgrade mid-run, with `--upgrade-from <older checkout>`. - The server starts from the older checkout. - The first K14 stops it with SIGTERM and starts this checkout on the same database, instance directory, port and secrets. The runner binary, pinned Codex and source dependencies are prepared per checkout. - The receipt names both commits. - K14 is local-only and needs `--upgrade-from` (and the reverse). It is excluded from storm defaults. Preflight refuses an older checkout that is not prepared. - G1 and H1 are opt-in: they are not in the default workloads. - README sections for both scenarios. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: every new and changed test passes; 188 of 189 pass in total. The one failure, `wave-speed.test.ts` "judges time to first agent activity…", also fails on master: #20077 gave the local TTFT a budget after #20071's test expected none. This PR does not touch `wave-speed` or the speed budgets. The new `governance.test.ts` covers: - the board decides only after the fault the pending approval armed, with the server ready, for approve and reject; - a blocked task waiting on its approval is not counted as resting; - G1 and H1 oracle pass and fail cases (duplicate request, action twice or after rejection, decision or resume that crossed no restart, dispatch while paused); - H1 budget set, pause hold and incident resolve, against a fake board API; - trigger gating, opt-in workloads, H1 exclusivity, and K14 option pairing. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passed. - Not run yet: a live wave of G1, H1 or K14. Phase 3 holds the harness slot, and the smoke runs are requested there. Their results will be added here. ## Risks - Test harness only; no product code changes. - G1 relies on the agent following the instructions (request, block, act only when approved). A task where the agent does not follow them fails the oracle visibly. - K14 seeds the company with this checkout's harness code while the older server serves. This works for recent builds; a much older build could reject a seed call. - Live behavior is not proven until the smoke runs. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution, a read-only research subagent). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 13:40:39 -06:00
DottaandClaude Opus 5.5	fcdceaf5fb	test(restart-chaos): grade a stopped retain-on-failure restore source as retained (#20086 ) ## Thinking Path - The restart-chaos harness's Daytona sweep grades R6, sandbox leaks, after every wave. It counts a sandbox as kept on purpose when its lease says so: a retained lease, or a released reusable lease whose sandbox is stopped for reuse. - Paperclip also keeps a failed run's sandbox stopped as a workspace-restore source (`retain_on_failure`, owned by restore recovery). The sweep graded those as orphans, so the warm-reuse chaos gate for #20048 counted intentional, recoverable retention as leaks. - The coordinator's decision: grade it honestly. A stopped, owned, recoverable restore source is retained; a running one, or one that survives its environment's deletion (no lease left), is still a leak. ## Linked Issues or Issue Description Refs #20048 What happened? In Daytona warm wave w-20261108T170423-s9501, four "orphans" were `retain_on_failure` leases: released, sandbox stopped, kept as failed runs' workspace copies. The sweep counted them as R6 leaks, failing the gate for intended retention. Steps to reproduce 1. Have a released `retain_on_failure` lease whose Daytona sandbox is stopped. 2. Run `reconcileSandboxes`: the sandbox is listed in `orphanSandboxes`. Expected behavior It is listed in `retainedLeases`. A running restore source, or a sandbox with no lease, stays in `orphanSandboxes`. ## What Changed - `tests/restart-chaos/daytona-sweep.ts`: `keptForReuse` also accepts a released `retain_on_failure` lease whose sandbox is `stopped`; the doc comment explains both kinds of kept sandbox. - `tests/restart-chaos/daytona-sweep.test.ts`: a stopped restore source is retained; a running one, or a sandbox with no lease, is still an orphan. ## Verification - `npx vitest run --root tests/restart-chaos daytona-sweep`: 14/14 pass. ## Risks - Hiding a leak. Only stopped sandboxes behind a released restore lease qualify; they are owned by restore recovery and, with #20085, destroyed with their environment. Running sandboxes and lease-less sandboxes still fail R6. This is harness-only, with no product change. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 12:32:45 -06:00
DottaandClaude Opus 5.5	b32ef9f111	perf(budgets): judge time to first visible agent activity; keep time to first text as a reported metric (#20077 ) ## Thinking Path - The native speed budgets (plan invariant 8) judge how fast an agent responds. They judged `task.time_to_first_token`: time to the first deliverable text. - In the clean local restart storm (w-20261108T153612-s9401), Claude breached its TTFT budget (cold p50/p95 17.4/34.7 s against 16.5/30). The per-run spans showed this was not a control-plane wait: in all 49 cold runs (and 76% of warm), Claude started a tool call before writing any text. Its first visible activity came at 7.4 s p50 from run creation; first text came at 19.6 s. Codex never calls a tool first. - So first text charges model tool work as latency for tool-first adapters. What the user experiences as "the agent responded" is the first visible activity: a text delta, a tool-call start, or a thinking indicator. - The coordinator decided: budgets judge time to first visible agent activity from run creation, with the budget numbers unchanged; first text stays a reported metric with no budget for tool-first adapters. ## Linked Issues or Issue Description What happened? Local storm s9401: the Claude TTFT budget breached on every aggregate, while the control-plane steps (queue 0.05 s, prepare 0.3 s, runner startup ~4 s, first provider event 1.3–1.8 s) were well within budget. The rest of the first-text wait was Claude streaming tool input (about 5.5 s per call) and running tools before it wrote text. Steps to reproduce 1. Run a Claude task whose first action is a tool call (most real tasks). 2. Read the run log: `provider.time_to_first_agent_event` ends at the tool call; `task.time_to_first_token` ends many seconds later at the first text. 3. The speed budget judges the later one and reports a breach. Expected behavior The budget judges the first visible agent activity from run creation (`task.time_to_first_agent_activity`). Time to first text is still recorded and reported. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: when the first meaningful agent event commits (the existing `firstMeaningfulAgentEventKind`: text delta, tool-call start, or reasoning), record `task.time_to_first_agent_activity`. It uses the same detached anchor resolution as TTFT (accepted comment, answered question, durable wake request, or run creation), so it adds nothing to the turn path. - `server/src/services/native-runtime/native-speed-budgets.ts`: the metric is now named `time_to_first_activity` (was `ttft`), with identical budget numbers. The type comment documents the definition and why first text is unbudgeted. - `server/src/services/native-runtime/native-run-trace.ts`: the budget check judges `task.time_to_first_agent_activity` from run creation, keeping the existing input-wait handling. `task.time_to_first_token` is no longer budgeted. - `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`: report first activity from run creation (the budgeted value) next to TTFT. - Docs: `doc/observability.md` (new span) and `doc/plans/2026-11-07-ttft-baseline.md` (metric table). - Tests: - A tool-first turn records first activity at the tool call (5 s) while TTFT records first text (16 s). - A tool-first trace with slow first text but fast first activity is within budget. - The existing budget tests now judge the activity span. Observability path (AGENTS.md section 5.7): the new run-log span uses only the existing `eventKind` and `anchor` attributes. Its name survives run-log redaction intact; checked with `redactEventPayload`. The operator OTLP gate is unchanged. It is not Telemetry. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts`: 44/44 pass. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "first visible activity\|first assistant text once\|first-token timing"`: 7/7 pass. - `npx vitest run --root tests/restart-chaos`: 159/159 pass; `tsc` on `tests/restart-chaos` is clean. ## Risks - A metric rename. Consumers of `run.performance.budget_exceeded` see `metric: "time_to_first_activity"` instead of `"ttft"`. In-repo there are none besides the tests; Phase 3's chaos speed grade switches to the new span. - A looser budget for slow first text. Intentional: first text is reported, not judged. A regression in control-plane latency still shows up in first activity. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 12:32:42 -06:00
DottaandClaude Opus 5.5	a65626a2a2	test(chaos): fail waves on a needed Retry and on speed regressions (outage removed); add M1 multi-agent workload (#20071 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (tests/restart-chaos) kills and restarts the control plane during real agent tasks, and grades every task on reliability rules R1–R6 plus a task oracle > - Dotta's bar for today's campaign: restarts must continue on their own (Retry is only a fallback), the system must stay fast (speed budgets in `native-speed-budgets.ts`), and realistic scenarios must be covered > - The harness graded a task that needed a user Retry as a pass, graded speed only in crash-point mode, and counted a K4 controller hold as slow TTFT. It also had no multi-agent scenario > - This pull request closes those four gaps > - The benefit is that every chaos wave now fails on a needed Retry and on a real speed regression, but not on the fault's own outage ## Linked Issues or Issue Description Refs #19831 (durable ownership plan) and the restart chaos plan (`doc/plans/2026-11-06-control-plane-restart-chaos.md`). Used by today's campaign waves: - w-20261108T134237-s9401 re-graded to 104/107 under the Retry rule; - w-20261108T153612-s9401: 100/101, with a Speed section showing 6 breaches after outage removal. ## What Changed - R3 Retry rule (`grader.ts`): a task that needed a Retry fails R3 as `retry_needed (<failed runs' error codes>)`, and a second click fails as `retry_needed_<n>_clicks`. The harness still clicks Retry so the task can finish and be graded. - Per-wave speed grade (new `wave-speed.ts`, wired in `main.ts`): - It reads every task run's run log through the API and reports TTFT (from run created) and agent boot p50/p95 per adapter and start kind, restart recovery (ready → the run's next provider event), and single-run `run.performance.budget_exceeded` events, against the local budgets. - A breach fails the wave. - Output goes to `speed.json` and a Speed section in `SUMMARY.md`. Remote environments report numbers with no budget. - Outage removal: each performance span is shortened by its overlap with fault outage windows (fault start to replacement server ready) before it is judged. The summary reports how many spans and how much time were removed. - M1 workload (multi-agent delegation): - The parent task creates two subtasks, part 1 delegated to the other Chaos Lab agent. - Each subtask saves a marked document, and the parent writes a summary of both before it completes. - The observer collects the subtasks and their documents, and the oracle checks the parent links, assignees, statuses and markers. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 14 files, 163 tests pass. New tests cover the M1 grade, the R3 Retry rule (one click and two clicks), the wave speed grade (budget judgement, restart recovery, remote no-budget, fault→run attribution) and outage removal. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Real waves: - The R3 rule re-graded w-20261108T134237-s9401 to 104/107. - w-20261108T153612-s9401 produced the Speed section: 923 spans adjusted, 2,452 s of outage removed, 6 breaches. - M1 has not yet run in a live wave. It's next in the campaign rotation. ## Risks - Waves that passed before can now fail: on a needed Retry, or on a speed breach. That is the intended bar. - Speed is judged at wave concurrency on a shared host, which is noisier than the idle baseline the budgets came from. Breaches are reported per row, so a reviewer can see the margin. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 11:27:39 -06:00
DottaandClaude Opus 5.5	f679cf81dc	fix(runner): refuse a finish or block decided before earlier results reached the agent (#20069 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Restarts must be invisible to a task: an agent that was working keeps working after the control plane comes back > - Restart chaos t03 (Daytona warm wave, K4 holds) ended a task as blocked even though its document was saved > - The cause is in how a terminal decision (paperclip_block or paperclip_finish) is handled after a long controller outage, not in how the Runner journals tool calls > - This pull request refuses a terminal call that the agent made before it had seen results that already returned, so the agent continues with those results > - The benefit is that a controller outage no longer turns into a false "blocked" task ## Linked Issues or Issue Description Restart chaos t03, wave `w-20261108T142906-s9501` (Daytona, codex, A1, three back-to-back K4 holds of 136 s, 252 s and 152 s). Refs #19831 (durable ownership plan). What happened The kept instance's run log shows the order of events: 1. The turn is accepted at 14:29:57. The runner then delivers nothing to the controller until 14:39:27. The holds keep the controller down, and the Runner keeps Codex's `item/tool/call` requests until it is reattached. 2. During that outage Codex keeps running. Its `write_document` call stays pending. The model is sampled again about once a minute and says "The save request is also waiting on Paperclip". It then calls `paperclip_block`, with the reason `tool_unavailable`. 3. After the reattach the Runner delivers the backlog in order. `write_document` runs and returns (the document is created: sourceSeq 152). Then the stale block arrives (sourceSeq 247) before Codex has acknowledged the write result: there is no `tool.execution.completed` for it before the block. The task ends blocked. Codex behavior, checked directly against the pinned Codex 0.160 app-server. A local probe sent one dynamic tool call and left it unanswered: - Codex does not time out. It re-samples the model at about 60 s and 127 s ("The save call is still waiting to return"). - A reply sent 130 s later is accepted (`durationMs` 130003), and the model finishes ("DONE"). So the reply is not lost. The agent's terminal decision was made without it. Expected behavior - A terminal call decided while earlier Paperclip results were still unseen is refused with a retryable error. - The agent continues with those results. In t03 it would see that the document was saved, and finish. ## What Changed - `packages/paperclip-runner/src/live/terminal-call-freshness.ts`: `unseenReturnedToolCalls` reads the turn's committed run events up to the terminal call. It returns each dynamic call that has a returned `tool_result` but no provider `tool.execution.completed`. `staleTerminalCallResult` builds the retryable `terminal_call_before_results` error, which names those calls. - `runnerd-codex-transport.ts` `#handleSemanticToolInput`: when such calls exist, `paperclip_finish` and `paperclip_block` get that error instead of being dispatched. The recovery path after a restart uses the same handler. A long-running call with no result yet does not count. - Crash point `runner.semantic_tool.after_input_before_dispatch`, just before the controller dispatches a dynamic tool call. A crash-point hold (`--hold-ms`, `--crash-point-hold-ms`; a K4-style receipt) keeps the controller down after the self-kill. Together they reproduce t03 on Daytona. ## Verification - Red/green: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts -t "refuses a terminal call decided"`. - The case `seen: false` failed on master: the handler was called and the stale block was dispatched. It passes with the fix. - The case `seen: true` (the agent saw the result first) dispatches normally, both before and after. - `npx vitest run src/live/runnerd-codex-transport.test.ts`: 289 passed. `src/live/terminal-call-freshness.test.ts`: 5 passed. - `npx tsc --noEmit -p packages/paperclip-runner` passes, and `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Shared registry: 15 passed. Call-site test: 3 passed. Chaos runner tests: 12 passed. - Codex probe (local, about 2.5 min per run): reproduced the "still waiting" re-sampling and the accepted late reply described above. - Not yet run: the Daytona crash-point repro with a hold. It needs the shared harness slot, and I'll add the result here. The fix is controller TypeScript, which is not an input of the Daytona image, so the current image still matches. ## Risks - A refused terminal call costs the agent one more turn step. It applies only when a result returned that the agent had not seen, which in practice happens only after a controller outage. - The check reads committed run events. If the store evicted the earlier call's events, it finds nothing and keeps today's behavior. - Upgrade-safe run state: no persisted shape changes. The check reads events the Runner already commits. A run saved by the previous release resumes with the same history and the same answer. - ACPX and other providers emit no dynamic `tool.execution.*` events, so the check is a no-op for them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 10:43:21 -06:00
DottaandClaude Opus 5.5	5de2842620	fix(restart-chaos): bound each skopeo image copy attempt (#20056 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness downloads the signed Daytona runner image with skopeo before each Daytona wave > - #20021 made a failed copy retry from an empty directory, but the retry only runs when skopeo exits > - A hung blob read never exits, and a Daytona pull stalled for 80 minutes > - This pull request bounds each copy attempt, so a stall is killed and retried > - The benefit is that one network stall costs at most one bounded attempt, not the whole run ## Linked Issues or Issue Description Follow-up to #20021. No duplicate PR was found. What happened? A Daytona chaos run's image pull stalled inside `skopeo copy` for 80 minutes, even after #20021. `copyImageToDir` awaited `execFile` with no timeout, so a read that hung forever blocked the run forever. Expected behavior Each copy attempt has a time bound. A stalled attempt is killed and the next attempt starts from an empty directory. Steps to reproduce 1. Run a Daytona wave or crash point with an uncached image on a network where a registry blob read hangs. 2. Before this PR, `skopeo copy` never returns and the run waits indefinitely. Paperclip version or commit `master` before this PR. ## What Changed - `copyImageToDir` passes `timeout` (default 20 minutes, `RESTART_CHAOS_SKOPEO_ATTEMPT_TIMEOUT_MS`) and `killSignal: "SIGKILL"` to each `skopeo copy`. A timed-out attempt counts as a failed attempt and retries from an empty directory. - README updated. ## Verification - New test: a fake skopeo that writes a partial blob and then hangs is killed after the 500 ms test bound, and the second attempt succeeds, within 10 s. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/daytona-image.test.ts`: 18/18. `tsc -p tests/restart-chaos/tsconfig.json` is clean. ## Risks - Test harness only. A very slow but progressing pull longer than 20 minutes would now be retried; the bound is configurable. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 09:07:36 -06:00
DottaandClaude Opus 5.5	96badeaa42	fix(lifecycle): never repeat a passed harness verification; settle a dead verifier's lease at once (#20030 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A new or reconfigured agent runs a harness verification (a hello probe in a test sandbox) before it is ready, and its runs wait until it passes > - In the final Daytona restart chaos wave, a Codex agent's runs waited about 417 s for their first sandbox, while boot took only about 32 s > - Each control-plane restart killed the verifier. The next process waited out the 120 s lifecycle lease and then booted a fresh sandbox. Nothing recorded a configuration that had already passed. > - This pull request records a passed verification by what makes it valid and reuses it, and settles a dead verifier's lease on the shared ownership path at once > - The benefit is that a restart never repeats a passed verification, and never leaves runs waiting out a dead process's lease ## Linked Issues or Issue Description What happened? In the final Daytona chaos wave, the Codex agent was created at 09:55:46 UTC and its three runs were queued. Verification ran four times, in four sandboxes, because faults K1, K4 and K2 each killed the verifying server. After K2 the server was ready at 10:00:20, but verification restarted only at 10:02:13: it waited out the 120 s lifecycle lease. The first run sandboxes were created at 10:02:43, 417 s after launch. Steps to reproduce 1. Create a Codex agent and assign it a task at once, so the task waits for harness verification. 2. Kill the control plane (SIGKILL) while the agent is verifying. 3. The restarted server waits out the dead verifier's 120 s lifecycle lease, then verifies again in a new test target. The task's first token arrives about 130 s after the fault. Expected behavior - A passed verification of the same effective configuration in the same runner image is never repeated. - A verification is never restarted from scratch while its holder process is alive. When the holder is proven gone, its lease is settled at once. - Runs gate only on a configuration that has never passed. Root cause `verify()` (`server/src/services/agent-harness-verification.ts`) leased a fresh sandbox on every attempt and recorded nothing on a pass. The lifecycle lease (`modules/agent-lifecycle/adapters/postgres.ts`) named no holder process, so a restart could only wait for it to expire. ## What Changed - Restart chaos TTFT driver: a `verify` phase, reported in its own rows, and a `--verify-fault K1\|K2\|K6` option. The fault fires 1 s after `verifying` is observed, so it interrupts a held lease. The seed keeps each agent's creation body, so the driver can hire an identical agent. - Persisted pass: new columns `agents.harness_verified_digest` and `harness_verified_at` (migration 0448). The digest is an HMAC, keyed with the instance's decision-signing secret, over: - the fully resolved effective configuration: every value the test used, including `cwd`, resolved credentials, and agent-over-environment precedence; - the selected AI account and its credential generation (connection and grant ids and `updatedAt`, never the credential); - the environment and the runner image. The key means the stored value cannot confirm a guessed credential offline. For a sandbox, the runner image is an immutable image digest or a qualified Daytona snapshot ID (`sandboxBinding`, now exported). For local runs it is the server version. A mutable image or snapshot name, or a local environment redirected to the managed sandbox, never reuses a pass. - Server-only: `normalizeAgentBaseRow` and `redactAgentRowForResponse` strip the digest and its timestamp from agent API responses. - `verify()` still runs every account, policy and secret check, then returns complete on a matching digest without booting a test target. It records the digest on a pass before the lifecycle step completes, so a restart in between reuses it. - Lease holder: the lifecycle `claim()` stamps the claiming controller process on the lease (`lifecycleOperation.leaseHolder`, supplied through the lifecycle effects port) and clears the stamp with the lease. The compare-and-swap release lives in the lifecycle module (`releaseAgentLifecycleLeaseOfHolderInTransaction`). - Shared ownership path: a new `agent_lifecycle_lease` kind with `settleAgentLifecycleLease`. - When the reconciler proves the holder gone, settle clears exactly that lease (compare-and-swap on the holder), notifies lifecycle work, and logs `agent.lifecycle_lease_reconciled`. - A live holder is never taken over. The lifecycle state, operation and progress are untouched. ## Verification - `restart-verification.test.ts` (embedded Postgres, real lifecycle store and worker, real reconciler pass): - A verification held by a dead process is claimed and completed at once after the startup pass, with the agent ready in under 10 s instead of waiting out the 120 s lease. It fails without the new kind. - A live holder is never taken over. - The claim stamps the holder, and `defer` clears it. - `agent-harness-verification-preflight.test.ts`: - A second `verify()` of the same configuration allocates no target. A revoked account still fails before reuse. A changed model, `cwd` or literal credential tests again. It fails without the reuse. - A failed verification records nothing. - Ownership report, all lifecycle module tests, remote, provider-API and unit harness verification tests, and crash-point call sites: 13 files, 220 tests pass. - Targeted local run: the TTFT driver's new `verify` phase (`--phases verify --verify-fault K1`, Codex, 2 samples) hires a new agent, assigns it a task at once, kills the control plane (K1) while it verifies, and times fault to first agent token: - Before (master server code): 129.8 s and 129.8 s. The restarted server waited out the dead verifier's 120 s lease. Run-log TTFT from run creation was only 4.1 s. - After (this branch): 15.8 s and 17.2 s. The startup reconciler pass settled `agent_lifecycle_lease` and verification re-ran at once. ## Risks - A stale pass: the digest covers every resolved value, the selected account generation, the environment and an immutable runner image, so any change tests again. Rotating the instance signing secret invalidates every digest, so each agent verifies once more. - Migration: two nullable columns, no backfill. Existing agents verify once more, then reuse. - Settle: it runs only on a `gone` proof, compare-and-swap on the exact holder. It never changes the lifecycle state. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 08:10:40 -06:00
DottaandClaude Opus 5.5	bfa9002a3c	test(restart-chaos): reach the stop-and-retain crash point with --daytona-reuse-lease (#20049 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos crash-point suite kills the server at named points, then checks that recovery restores every invariant > - On Daytona, `environment.stop_retain.after_claim_before_stop` was never reached, because no workload produced a stop-and-retain release > - A per-turn run on a reusable sandbox lease ends every turn with `stop_and_reuse`, which releases through exactly that stop-only claim > - This pull request adds `--daytona-reuse-lease`, so the chaos Daytona environment keeps reusable sandboxes and the point is reachable > - The benefit is live coverage of the stop-and-retain crash window ## Linked Issues or Issue Description This is a follow-up from the Phase 2 durable-ownership data (`doc/plans/2026-11-07-phase2-reconciler-data.md`, "Not covered"). Related: #19937 (crash-point suite) and #20042 (stop-only claim fence renewal). No duplicate PR was found. What happened? The Daytona crash-point run `cp-20261108T104102` reported `environment.stop_retain.after_claim_before_stop` as `not_reached`. The point needs a stop-and-retain intent, and the chaos Daytona environment is created with ephemeral leases. Expected behavior A flag lets a Daytona wave or crash-point run use reusable leases. A per-turn run then stops and keeps its sandbox, and the point is reached and graded. Steps to reproduce 1. Run `node scripts/crash-points.mjs --env daytona --daytona-image <image> --points environment.stop_retain.after_claim_before_stop --adapters codex --workloads A1`. 2. Before this PR the point is `not_reached`. 3. With `--daytona-reuse-lease --lifecycle per_turn` it is reached. Paperclip version or commit `master` at `8475826d2d` (the signed image `c24059f1…` revision). ## What Changed - `--daytona-reuse-lease` (wave and crash-point suite, Daytona only) creates the chaos environment with `reuseLease: true`. The crash-point suite passes the flag through. - The R6 sandbox sweep counts a stopped sandbox behind a released `reuse_by_environment` lease as kept for reuse, not a leak. A later run on the same workspace resumes it, because reuse accepts released and retained leases. A running sandbox, or a stopped one behind an ephemeral lease, is still a leak. - The crash-point registry prerequisite and the README name the flag. ## Verification - Live, Daytona, codex A1, from the image revision `8475826d2d`: - `cp-20261108T132149`: the point was reached and settled with no invariant violations. R6 flagged the stopped reusable sandbox as a leak, which led to the sweep fix. - `cp-20261108T133351`, with the sweep fix: pass. 0 orphaned sandboxes, 0 held leases, the environment deleted, and nothing left running. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts` passes `daytona-sweep`, `scheduler` and `crash-points` (52/52). `tsc -p tests/restart-chaos/tsconfig.json` is clean. ## Risks - Test harness only; the registry change is prerequisite text. Without the flag, waves behave as before. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 08:07:19 -06:00
DottaandClaude Opus 5.5	c966334ec3	fix(recovery): settle a run-less lease once its stamped holder is proven gone (#20026 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane recovers in-progress records after a server restart through a shared ownership path > - Run-less sandbox leases (login, probe, harness verification) were released only by a five-minute idle sweep > - That sweep releases by age: it leaves a dead holder's sandbox for minutes and takes a live login session's sandbox > - This pull request settles a run-less lease only when its stamped holder is proven gone > - The benefit is no stuck sandbox after a restart, and no live session loses its sandbox ## Linked Issues or Issue Description Related: #19965 (named the owner of login/probe leases), #19964 (the run-lease settle this mirrors). No duplicate PR was found. What happened? The final Daytona chaos wave `w-20261108T083036-s9401` recorded I2 `environment_login_probe_lease` (`pid_dead`) on 2 records. These were the codex agent's harness-verification leases. A restart killed their holder. The leases stayed `active` until the five-minute idle sweep released them. The same sweep releases any run-less lease after five idle minutes, also when its holder is alive. Expected behavior A run-less lease is released as soon as its stamped holder is proven gone. A lease whose holder is alive is never released, however long it is idle. Steps to reproduce 1. Start a Daytona wave: `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image <image>` (K1/K4 faults). 2. Let a fault kill the server while an agent's harness verification holds its sandbox lease. 3. See the reconciler report I2 `environment_login_probe_lease` until the idle sweep releases it. Paperclip version or commit `master` at `8475826d2d`. ## What Changed - New `settleEnvironmentLoginProbeLease`, registered on `environment_login_probe_lease`. It runs only for a stamped holder that is proven gone, and compares and swaps on that exact stamp. - It moves the lease to `pending_cleanup` with the recovered-orphan reason, so the cleanup sweep destroys the sandbox in its next tick. - It leaves the lease alone when a legacy shutdown fence applies, or when another live lease holds the same provider resource. - The idle sweep for run-less leases now covers only legacy rows saved before holders were stamped (no `leaseController` key). Stamped rows are never released by age. - `holderUnchanged` is now exported from the run-lease settle so both settles share it. - The controller stamp (`leaseController`, and every other `ControllerProcess` stamp) now records the platform deployment when the platform names one (Railway: project, environment, service, deployment, volume). `controllerStamp` passes it to `proveOwnerGone`. A container redeploy changes the PID space, and the old stamp would read `different_process_space` (unprovable) forever. Now the existing `deployment_replaced` proof shows the old holder gone. This also covers `environment_run_lease_active` and the other kinds built on `controllerStamp`. - `currentDeploymentIdentity` in `owner-proof.ts` now delegates to `currentControllerDeployment`, so there is one parser. - Data doc: added the final Daytona wave, the six Daytona-only crash points (4 pass, 2 not reached, no violations), and the cause of the roughly 450 s codex TTFT. Runs waited behind the agent's harness verification, which restarted after each fault; boot itself took about 32 s. - Restart-chaos README: corrected which Daytona crash points a plain task reaches. ## Verification - Red, before the change: `environment-login-probe-lease-ownership.test.ts` failed 3 of 6. No settle happened, and the idle sweep released 3 live holders' leases. - Green: `pnpm exec vitest run` passes for `environment-login-probe-lease-ownership`, `environment-run-lease-ownership` and `heartbeat-orphaned-active-lease-sweep` (45/45), and for `environment-lease-owner-stamps`, `heartbeat-observed-workspace-preservation`, `ownership-reconciliation-report`, `heartbeat/recovery` and `workspace-recovery-warm-handoff`. - Redeploy: a stamp from a replaced Railway deployment settles with `deployment_replaced`. A stamp from the same deployment in another PID space stays `active`. This was red when `controllerStamp` dropped the deployment. - Live-owner counter-cases: the current process, another live PID, another PID space, and a lease re-stamped after the proof all stay `active`. A read-only pass never settles. - Not re-run live: the Daytona wave and crash points ran before this fix. The next Daytona wave should show no `environment_login_probe_lease` I2. ## Risks - A stamped run-less lease whose live holder never releases it, because of a bug in that process, is no longer released after five idle minutes. It is released once that process exits. This is the intended trade: never take a live holder's sandbox. - On an ephemeral server without a platform deployment identity (not Railway), a stamp from a replaced container is still unprovable. Such a run-less lease stays `active` until the provider expires the sandbox, and the reconciler reports it as unprovable. Before this PR, the five-minute sweep released it by age. This is the same open gap as unstamped legacy leases on an ephemeral server: there is no operator action yet. It is listed as a Phase 2 follow-up. - Legacy unstamped run-less leases keep the age fallback. These are rows written before #19965, and no current code writes them. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 06:43:45 -06:00
DottaandClaude Opus 5.5	022c73e475	fix(restart-chaos): retry a failed image copy from an empty directory; share the image cache (#20021 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness tests recovery on Daytona sandboxes > - The harness extracts the runner from the Daytona image with skopeo before a wave starts > - A skopeo blob read that timed out failed the whole run before any test started > - This pull request makes the image extraction retry cleanly and share one cache > - The benefit is that one network hiccup no longer costs a 20+ minute run ## Linked Issues or Issue Description Related: #19544 (Daytona chaos waves without Docker). No duplicate PR was found. What happened? The Daytona wave `w-20261108T082021` and the crash-point run `cp-20261108T101304` stopped before any test ran. skopeo reported `read: operation timed out` while it copied a blob. The retry then reported `not a containers image directory, don't want to overwrite`, because the destination already had partial files. The crash-point suite also used its own image cache, so it downloaded the image again on each run. Expected behavior A failed copy retries into an empty directory. Every wave and crash-point run reuses an image that another run already extracted. Steps to reproduce 1. Run `node scripts/crash-points.mjs --env daytona --daytona-image <image>` on a slow or unstable network. 2. Let skopeo time out on one blob read. 3. See the harness stop with exit 2 before any point runs. Paperclip version or commit `master` at `8475826d2d`. ## What Changed - With a shared cache, two runs can extract the same image at once. The final cache directory is created only by an atomic rename, and no run deletes a cache entry. A run that loses the rename adopts the valid entry. An invalid leftover stays in place, and the run publishes its extraction to a unique sibling directory. - `copyImageToDir` runs up to 3 `skopeo copy` attempts. Each attempt starts from an empty destination directory. - `defaultDaytonaImageCacheRoot()` is now the one cache for both the wave harness and the CLI: `tests/restart-chaos/results/daytona-images` by default, or `RESTART_CHAOS_IMAGE_CACHE` if set. Crash-point runs reuse an image that a wave already extracted. - README updated. - Three more tests: two runs against one cache through a barrier, a corrupt leftover, and two runs that both meet a corrupt leftover. - Unit tests use a fake skopeo that behaves like the real one: it refuses a non-empty destination and leaves partial blobs behind when it fails. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/daytona-image.test.ts`: 17/17 passed. - `tsc -p tests/restart-chaos/tsconfig.json`: no errors in `tests/restart-chaos`. - Live run: the six Daytona crash points were rerun with this patch, reusing the cached image. ## Risks - Low. Test harness only; no product code changes. - A shared cache means all runs depend on one receipt per image digest. Receipts are already written atomically by renaming the staging dir, and each one is keyed by the immutable digest. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 06:00:48 -06:00
DottaandClaude Opus 5.5	8475826d2d	fix(wake-queue): deliver user comments whose wake a restart lost or a stale continuation dropped (#19992 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user's follow-up comment on a task must reach the agent exactly once, through restarts (invariant 4) > - The final unmasked local chaos storm (96 tasks, 31 restarts) passed 94 tasks. The 2 failures were both lost user comments > - In t69, a restart killed the server between saving the comment and writing its wake. In t72, a comment's own run was cancelled as stale, because the earlier run completed the task first without reading the comment > - This pull request adds a reconciler rule that gives a lost comment its wake, and saves the comment before a stale cancellation > - The benefit is that a user's message is never silently dropped when a restart or a completion race hits at the wrong moment ## Linked Issues or Issue Description Refs #19956 (work-intent outbox), #19944, #19831 (plan, invariant 4). What happened? In chaos wave `w-20261108T055444-s9401` (local, unmasked, codex and claude, 30-min K13 storm, master `46bf913fb9`), 94 of 96 tasks passed. Two failed: - t69, R1 `rested_not_done`. The user's comment was saved at 06:14:59.358 and reopened the task to `todo`. A K1 SIGKILL fired 28 ms later, before the wake was written. No wake row exists. Recovery skips a `todo` task whose last run succeeded, so the task sat with no run until the wave ended. - t72, R4 `concurrent_runs`. A comment posted during a run queued its own run. That run was claimed at 06:16:04.874. The first run's completion marked the task `done` 2 ms later. The comment's run was then cancelled with `continuation_task_ownership_changed` before dispatch, and the comment was lost. Expected behavior Each accepted user comment is delivered once, unless the task is cancelled or reassigned, or a person closes it. Steps to reproduce `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay w-20261108T055444-s9401/t69` (or `.../t72`). ## What Changed - `recovery/ownership-invariant-report.ts`: a new reconciler-owned I4 rule, `user_comment_without_wake`. It finds a user comment that meets all of these conditions: - it is newer than every run of the task; - the task is assigned, is not a conversation, has no live run or lock, and has at least one earlier run; - no wake request carries the comment. A comment younger than 30 s is not reported yet, because it may still get its normal wake. Its end-of-grace time is returned as `nextDueAt`. - `native-runtime/native-finalization-reconciler.ts`: the backstop step is now exported as `applyOwnershipReportBackstops`. For each such comment it saves the missing outbox receipt (`executionWait: wake_lost`) and promotes it. - `work-intent-outbox.ts`: new `saveUndeliveredUserComments`, which the reconciler rule and the stale-run path both use. It saves only live user comments that no completed run received. It writes nothing for a task that is reassigned, is a conversation, or has a status the caller does not allow. - `heartbeat/queue.ts`: the saved-receipt pass that already sends pause-saved messages also sends `wake_lost` receipts. It uses the same per-comment consumption check and the same chat-input exclusion. - `heartbeat/stale-comment-continuation.ts` and `heartbeat.ts`: before a stale continuation is cancelled, its undelivered user comments are saved as a deferred receipt. The release drain then reopens the `done` task and delivers them once. This applies only when an agent run that started before the comment completed the task after it. A task that a person marks done is left closed. - `tests/restart-chaos/grader.ts`: R4 ignores runs cancelled before dispatch (`continuation_task_ownership_changed`) when it looks for overlapping execution. Those runs never executed. Comment consumption still counts them. ## Verification Each focused file ran alone in one process (`--maxWorkers=1`): - `heartbeat-held-comment-wake.test.ts` (22 pass). This includes two new cases: - t69 shape: the reconciler rule fires, the backstop delivers the comment once, and a second pass does nothing; - t72 shape: the stale run is cancelled, the task reopens, and the comment is delivered once. Both fail with the rule or the save disabled, and pass with them. - `ownership-reconciliation-report.test.ts` (23 pass). It includes the rule's boundaries: a carried comment, a too-young comment (deadline only) and an already-read comment. - `tests/restart-chaos/grader.test.ts` (17 pass). The new R4 case fails with the old grader. A cancelled run that is not a pre-dispatch cancellation still fails R4. - Unchanged and passing: comment wake batching (27), refused assignment wake (3), wake-queue postgres adapter (103), stale queue invalidation (68), deferred receipt work (45). The first version also reopened a task that a person had marked done, which broke one deferred-receipt test. That case is now excluded. - The server package typecheck and the restart-chaos harness typecheck pass. - Same-session chaos context on master: K6 crash loop `w-20261108T063326-s9402`: 6 of 6 pass. The K4 long-hold wave is still running. After merge, I'll replay only the targeted fault shapes (t69 and t72). ## Risks - The reconciler rule runs on each report pass. It is bounded by the 1-hour lookback and the per-kind row cap, and it excludes conversation tasks, so chat input keeps its own authorization. - A comment younger than 30 s waits up to the end of its grace period before the backstop acts. - The stale-run save reopens a `done` task only for a comment that the completing run could not have read. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 02:14:45 -06:00
DottaandClaude Opus 5.5	68d5fb5a6e	feat(chaos): check I6 and I8 per crash point; ready the Daytona points (#19984 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The on-demand crash-point suite (#19946) kills the control plane at named durable write boundaries and checks the plan's invariants > - Its first full run measured I1–I4 and I7, but I5, I6 and I8 were reported as not measured > - #19960 (generation stamping) and #19863 (speed budgets) now record what I6 and I8 need in the run log > - The six Daytona-only points have not run yet, and three of them cannot be reached by a plain task > - This pull request checks I6 and I8 per point, names the I5 gap, and readies the Daytona points for the coming Daytona wave > - The benefit is that every invariant is either asserted or a named gap, and a Daytona run explains any point it cannot reach ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5). Follows #19946 (the suite). Uses #19960 (event generations) and #19863 (speed budgets). Problem - The suite reported I5, I6 and I8 as "not measured". - A Daytona run would report three points as `not_reached` with no reason. - A Daytona run without its key or image failed only after a 3-minute boot. Expected behavior - I6 and I8 are checked at each point. - I5 is a named gap with its reason. - Daytona points say what they need, and a Daytona run checks its inputs before it boots. ## What Changed - `tests/restart-chaos/invariants.ts`: `runLogViolations` checks two invariants from the run log, scoped to the wave's company: - I6: a run that failed with an error that names a stale or earlier-generation event; - I8: each `run.performance.budget_exceeded` event, with the budget evidence (metric, measured, p95, adapter, start kind). Both are keyed by run or event id, so the per-point baseline diff charges each one to the point that produced it. I5 is reported as `notMeasured` with a reason (`I5_GAP`). - `packages/shared/src/crash-point-registry.ts`: an optional `prerequisite` on the three Daytona points a plain task cannot reach. These are the teardown claim (needs a failed release), the reusable destroy (needs a `reuse_by_environment` lease), and stop-and-retain. - `tests/restart-chaos/crash-points.ts`: - `--list` prints each point's `needs:`, and a `not_reached` result names it too; - `--env daytona` refuses to start without `DAYTONA_API_KEY` and an image. - Docs: the README and the plan describe the I5/I6/I8 coverage and the Daytona points. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`: 12 passed. - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and `npx tsc --noEmit -p packages/shared` both pass. - I ran `runLogViolations` against a throwaway Postgres 18 with seeded rows, then stopped and deleted it. It returned: - I8 for the budget breach, with its evidence; - I6 for the run of this company that failed on `stale_generation_event`. It excluded the other company's stale run, the `process_lost` run and a non-budget event. - `node scripts/crash-points.mjs --list --env daytona` lists 19 points, with `needs:` on the three. `--env daytona` with no key or image fails at once with the missing names. - No harness run: the harness slot is held for other work. The next suite run exercises the new checks. ## Risks - Test harness only; no product code. - I6 is a proxy: it matches the run's error text and code. A failure that names a generation for another reason would be reported, with its evidence, for a person to judge. - I8 can flag breaches on a loaded host. Each breach carries its measured and budget values, so a load-caused breach is visible as one. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 00:06:01 -06:00
DottaandClaude Opus 5.5	051834bdc8	feat(chaos): add the on-demand crash-point suite and its manual CI workflow (#19946 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - Random restart-chaos waves found these 20 ms–1 s windows only by luck > - #19937 added `crashPoint(name)`, and #19938 put it at 20 durable write boundaries > - Nothing yet runs a task, kills the server at a chosen point, restarts it, and checks the plan's invariants > - This pull request adds that on-demand suite, as a local command and as a manual CI workflow (not a required check, per Decision 2) > - The benefit is a repeatable work list: which boundary breaks which invariant, for Phases 2–4 ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5 and Decision 2). Refs #19861 (report-only reconciler used for I1–I7). Stacked on #19938 (call sites), which is stacked on #19937 (helper). Problem The named crash points cannot be exercised end to end. No command arms one point, runs real agent tasks across it, restarts, and grades the result. Expected behavior `node scripts/crash-points.mjs --points <glob> --kinds <id> --adapters <list>` runs one wave per selected point and reports each point as pass, fail (with failing R rules and violated invariants), not reached, or harness error. A `workflow_dispatch` workflow runs the same thing on demand. ## What Changed - Runtime arming. `PAPERCLIP_CRASH_POINT_CONTROL=<file>`, in the shared helper and the runner mirror. Each hit reads `<token> <name>[:<nth>]`, and a new token resets the count. It is test mode only: `NODE_ENV=production` ignores it, and the disarmed path is still one comparison. - `tests/restart-chaos/main.ts --crash-point <a>:<nth>,<b>:<nth>,...` (config, types, ledger): - one booted server and no Vite UI; - per point: take an invariant baseline, launch the point's batch, arm the point, then on rest disarm it and check invariants as violations that are new since the baseline; - every task is graded R1–R6 and T; - `crash-point.json` (v2) holds one entry per point. - `tests/restart-chaos/supervisor.ts`: - every spawn gets the control file; - a SIGKILL whose receipt names the child's pid is restarted like K1: the control file is emptied first, the owned tree is killed, and a replacement starts. A failed tree cleanup fails the fault; - the isolated instance pins `PAPERCLIP_IN_WORKTREE=false` (a `.paperclip/.env` above the checkout otherwise suppressed all scheduling); - `RESTART_CHAOS_HEALTH_TIMEOUT_MS`. - `tests/restart-chaos/invariants.ts`: - the Phase 1 report-only pass with `record: false`: two real-clock passes for I1 and I2, and one pass 15 minutes ahead for I3, I4 and I7; - spool kinds are read from the wave's instance; - an unfinished pass is a harness error; - each violation carries its wake and run state as evidence. - `tests/restart-chaos/crash-points.ts` and `scripts/crash-points.mjs`: - select points by glob, kind and environment, and run them in one wave; - a per-point outcome: pass, fail, not_reached or harness_error; - `SUMMARY.md` with a violation evidence column. - `.github/workflows/crash-points.yml`, manual only, under the paid-workflow contract: - a credential-free `authorize` job; - a `suite` job in `runner-e2e-paid` whose first step reauthorizes; - a frozen install with `--ignore-scripts` and no shared cache; - only the selected adapters' keys; - the workflow is added to both security-test allowlists. - Docs: "Crash-point suite" in `tests/restart-chaos/README.md`, and "Built (Phase 5)" plus the first work list in the plan. ## Verification - Focused tests: - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`; - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 tests, including runtime arming in a real child process; - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 72 tests; - `tests/runner-e2e/workflow-security.test.ts`: 19 tests. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Loop line: question "how long is one crash-point loop?" \| loop: `node scripts/crash-points.mjs --points <one>` (codex A1) \| elapsed 2.7 min end to end: boot 9 s, crash 9 s after arming, replacement ready in 6 s, rest about 2 min \| result: one point alone takes about 3 min. Before this change it took about 15 min (one cold boot per point, Vite UI, host at load 77). - Full local run, 14 points, codex A1, 11.8 min in all: - every point was reached (the server killed itself at each one, and the replacement came back); - 11 pass; - `workspace.finalize.after_start_before_copyback` and `workspace.finalize.after_copyback_before_done` fail I2 and R6: a `workspace_finalize` operation stays running after the run succeeded; - `wake.claim.after_run_running_before_wakeup_claimed` reports I4. Its evidence shows the wake failed with "…retrying once", and the retry delivered the work, so this is the known refusal-regex false positive. - The 6 Daytona-only points were not run. ## Risks - Test harness and an on-demand workflow only. No product code changes. - `.github/**` needs a CODEOWNER review. The workflow uses provider keys, so it refuses non-default branches and actors that are not on the allowlist, like `runner-full-stack-e2e.yml`. - `PAPERCLIP_IN_WORKTREE=false` is pinned in every chaos wave, not only crash-point waves. A chaos instance is never a worktree dev instance, so scheduling must not be suppressed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:41:15 -06:00
DottaandClaude Opus 5.5	4aed3056db	feat(ui): steer a queued follow-up into the live turn after a 5 s edit window (#19970 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person comments on a task while the agent works, the comment waits in a queue until the current run ends, and only the next run reads it > - Measured locally, a comment posted during a Codex tool call waited 54–68 s for a reply (median 64 s) > - Codex and Pi can take a message into the live turn ("Steer"), but today the person must click Steer on each queued message > - Restart recovery now delivers a steered message that a restart interrupted (#19962), so steering is safe to do automatically > - This pull request steers a queued follow-up into the live turn on its own after a 5-second window in which the person can edit or cancel it > - The benefit is a faster reply to mid-turn comments on Codex and Pi, with no new decision for the person ## Linked Issues or Issue Description Feature Send a queued follow-up into the running turn automatically when the runner supports steering. Problem A follow-up comment posted during a running turn waits for the whole turn to end and for a new run to start. Steering is available for Codex and Pi, but only as a manual button on each queued message. Proposed behavior After a 5-second edit window, the queued message is steered into the live turn. During the window the row shows "Sending in Xs" with Edit and Cancel. If steering is unavailable, the message stays queued as before. Claude, OpenCode and Cursor keep the queue. Steer and Interrupt stay available as buttons. Refs #19962 ## What Changed - `queued-message-auto-steer.ts`: one constant, `QUEUED_MESSAGE_AUTO_STEER_DELAY_MS = 5_000`, a deadline helper and an eligibility check. A message steers automatically only when all of these are true: - steering is available for the queue, and it is a `paperclip_runner_v1` queue with a queue id and a target run; - the person can edit the message; - it is not an interaction response or a response that requires a fresh session; - it is not optimistic, cancelled, already attempted, or open in the editor. - `TaskChatQueuedMessages`: countdown row "Sending in Xs" with Edit and Cancel in place of Steer and the "…" menu. The deadline is `max(message changed, first eligible) + delay`, so an old queued message does not send the moment its turn becomes steerable, and an edit restarts the window. When the deadline passes, the row uses the existing Steer action, so the server path, idempotency and fallback stay the same. Cancel returns the row to the normal Steer button. The countdown is announced once to screen readers, not on every tick. - `TaskChatThread` turns it on with the constant and passes the message being edited, so an open edit never sends. The prop stays off by default for other callers. - Storybook: `AutoSteerCountdown` (with a play test for Cancel) and `AutoSteerCountdownMobile` in "Composer / Steering and queued messages". - Restart chaos TTFT driver: a `midturn` phase measures comment-to-reply for a comment posted during a 40-second tool call, with `--midturn-mode queue\|steer` and `--midturn-fault K1\|K2\|K6`. It records the reply count, so exactly-once delivery is checked. ## Verification - `pnpm --filter @paperclipai/ui vitest run src/components/task-chat/TaskChatQueuedMessages.test.tsx src/components/TaskChatThread.test.tsx`: 434 passed. Seven new auto-steer tests use fake timers: countdown and send, Cancel, Edit restarts the window, an open edit never sends, ineligible queues stay manual, no instant send for an old message, one attempt per revision. They fail with the trigger disabled. - `node scripts/check-token-gates.mjs`: clean. - Storybook in the browser, desktop and mobile: the first row counts down and steers ("Message steered into the active turn."). The cancelled row shows the normal Steer button. On mobile the text truncates and the row fits. - Mid-turn latency, local, Codex, comment during a 40-second tool call: queue 54–68 s (median 64 s), steer 47–61 s (median 50 s). The steer was acknowledged in 0.8–0.9 s. The remaining time is the tool call and the model, not the hand-off. - Restart chaos, Codex, steer mode, faults K1, K2 and K6, 2 samples each, on #19962 plus this branch: 6 of 6 messages acknowledged, still unread at the fault, recovered, and answered exactly once (before #19962: 0 of 6). ## Risks - A message now leaves the queue after 5 seconds unless the person edits or cancels it. The window and the countdown make this visible, and steering is only automatic where the provider supports it. - Pi: a steer the model already read before a restart can arrive again after recovery, because Pi reports no read evidence (see #19962). Automatic steering makes this a little more likely. - If steering fails, the message stays queued and is delivered by the next run, as today. - UI only and a test harness. No schema, API or server change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, browser checks, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 22:59:44 -06:00
DottaandClaude Opus 5.5	f35eb1e5ef	fix(observability): judge the TTFT budget from run creation; report the wait behind a live turn (#19941 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Invariant 8 of the durable-ownership plan says turns must start fast. #19863 added per-run speed budgets that write `run.performance.budget_exceeded` to the run log. > - The Phase 2 chaos data (#19927) showed follow-up TTFT about 3x over the claude budget. > - The cause is measurement scope. A follow-up comment posted during a live turn waits for that turn to end, and TTFT is anchored at the comment. The budgets came from idle follow-ups. > - This pull request judges the TTFT budget from run creation and reports the wait separately. > - The benefit is that budget breaches point at real start-up slowness, not at time spent behind a turn the user is already watching. ## Linked Issues or Issue Description Refs #19863 (speed budgets), #19927 (Phase 2 data), #19852 (startup sub-spans), #19888 (warm default). What happened? In two local storm waves, C1 follow-up runs breached the claude warm TTFT budget (p50 27.6–29.9 s against 9.5 s). Their spans show `comment.to_run_created` at 6–62 s (waiting for the live turn), then session resume at 1.4–3.8 s and submit-to-first-event at 2–5 s. Codex without the wait was about 4.8 s, the same as the warm-follow-up baseline. What did you expect? The budget should judge start-up latency, which is what its baseline measured. The wait behind a live turn should be visible but not judged. Steps to reproduce 1. Run a local chaos storm with C1 tasks. 2. Read `run.performance.budget_exceeded` rows with metric `ttft` for `issue_commented` runs, and compare them with each run's `comment.to_run_created` span. ## What Changed - `native-run-trace.ts`: the TTFT budget check measures from run creation (start of `heartbeat.queue`, or end of the ingress span). The TTFT span itself does not change. - `native-speed-budgets.ts`: breaches can carry `inputWaitMs` (TTFT only). - `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`: the wait behind a live turn is its own measured step (`inputWait`), next to the full TTFT and the TTFT from run creation. The TTFT table gains a "Wait behind live turn" column. The full comment-to-first-token time stays the user-visible metric. - `doc/run-log-events.md` and the plan's speed-budget section describe the rule. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts`: 43 pass, 2 new. `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/ttft-baseline-stats.test.ts`: 12 pass, 1 new. They cover a follow-up under budget after a 60 s wait (no breach), a follow-up over budget (breach with `inputWaitMs`), and queueing after run creation still judged. - Server typecheck passes. ## Risks - Low. This is a run-log-only change (AGENTS.md §5.7): no Telemetry and no span attributes change. - A slow wait before run creation is no longer a TTFT budget breach. The TTFT span still records it, and the breach payload names it. Admission holds after run creation are still judged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:27:41 -06:00
DottaandClaude Opus 5.5	17b9e10252	docs(plans): Phase 2 reconciler data; let chaos K4 holds reach 600 s (#19927 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Control-plane restarts leave in-progress records that each subsystem releases in its own way. The durable-ownership plan replaces those paths with one shared path. > - Phase 1 added a report-only reconciler. Its light local waves showed zero violations, and that proves little. > - Phase 2 moves kinds to the shared path. It needs data that shows which kinds really leak, and the speed numbers to protect. > - This pull request records that data from harsher waves. It also fixes the harness so that K4 outages past the reconnect grace qualify. > - The benefit is that Phase 2 moves kinds in order of measured risk, against a known speed baseline. ## Linked Issues or Issue Description Refs #19831 (durable-ownership plan), #19861 (report-only reconciler), #19863 (speed budgets), #19797 (K4 hold cap). What is the problem? Phase 2 needs per-kind invariant data and speed data from harsh restart waves. Only light local waves existed. In addition, a K4 hold over about 405 s always failed its fault receipt, so the 600 s cap from #19797 could not be used. Where is it documented? `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` (Phase 2 rows) and `tests/restart-chaos/README.md`. What should change? Record the data in a dated plan doc. Let chaos K4 holds run up to 600 s and still qualify. ## What Changed - Added `doc/plans/2026-11-07-phase2-reconciler-data.md`. It covers 5 unmasked local waves (two storms, a crash loop, two K4 runs). It gives per-kind violation counts with example rows, owner verdicts, TTFT, boot and restart recovery against the budgets, Phase 3 gate notes, and what is not covered (Daytona). - Added a short pointer to that doc in the durable-ownership plan. - `tests/restart-chaos`: new flag `--k4-hold-max-seconds` (120–600). It raises the upper end of the seeded K4 hold. The default range (120–300 s) does not change. - `tests/runner-e2e/controller-outage.ts`: the receipt's ready bound is now a parameter (`CONTROLLER_READY_BY_MS`, default 420 s). Runner suites keep 420 s. The chaos supervisor passes hold + 420 s, so a hold past the Runner's reconnect grace qualifies. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 12 files, 135 tests pass, including the new flag test. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/controller-outage.test.ts`: 36 pass, including the new long-hold receipt test. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and `tests/runner-e2e`: pass. - Live: the K4 wave at seed 9203 failed 3 of 6 tasks with the old bound (fault error "did not produce a qualifying replacement receipt"). It passed 6 of 6 with this fix, with holds of 294–579 s. ## Risks - Low risk. The changes are test harness code and docs only. The runner suites keep the 420 s ready bound by default. - The data is local only. The doc says which kinds the waves do not reach. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). The model ran the chaos waves and wrote the analysis and code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 19:55:02 -06:00
DottaandClaude Opus 5.5	fc6e1b0346	perf(native): keep local Codex and OpenCode runners warm by default (#19888 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When someone messages an agent, the turn must start fast. Time to first token (TTFT) is a first-class metric. > - The Phase 0 baseline (#19846) found that every local turn launches a new runner and loads the provider session again, because `per_turn` is the default lifecycle. That made a follow-up message slower than a new task. > - Warm runners keep the runner and provider session alive between runs. #19874 fixed the defect that broke warm OpenCode follow-ups, and #19879 bounds an idle warm runner whose controller dies. > - This pull request makes warm the default for local Codex and OpenCode agents, and migrates existing agents that only carried the old creation default. > - The benefit: follow-up TTFT p50 drops from 12.0 s to 4.7 s (Codex) and from 21.7 s to 4.2 s (OpenCode), with no new restart-chaos failures. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline), #19852 (startup sub-spans), #19874 (warm completion contract), #19879 (warm reconnect grace). What existing behavior does this improve? The default runner lifecycle for native agents, and how a follow-up message starts. Current behavior Agent creation writes `lifecycleMode: "per_turn"` into every native agent. Every follow-up launches a new runner and restores the provider session: 4.2 s (Codex) and 16.3 s (OpenCode) at p50 before the turn starts. Proposed behavior New native agents store no lifecycle. On a local target, an unset lifecycle resolves to `warm` with the default 5-minute idle window for Codex and OpenCode. Other harnesses, remote targets and managed providers without warm support stay `per_turn`. An explicit choice always wins. Migration 0443 clears the stored creation default from existing agents, except where an edit set "Turn by turn" explicitly. Reason and benefit Measured locally (`doc/plans/2026-11-07-warm-follow-up-ttft.md`). Warm follow-ups are 2.6–5× faster for Codex and OpenCode, and restart recovery latency is equal or lower. ## What Changed - `native-lifecycle.ts`: an unset `agentMode` resolves to warm (5-minute idle window) for local `codex_app_server` and `opencode_server` agents. `runtime-selection.ts` passes `localTarget`. Explicit `per_turn` or `warm` wins. An admitted session keeps its persisted policy. ACPX Claude stays `per_turn`: it relaunches its provider process on every follow-up by design, so warm did not shorten its follow-up (10.7 s warm vs 10.5 s per_turn). - Creation and configuration writes no longer store `lifecycleMode: "per_turn"` for Codex, Claude or OpenCode. This covers `normalizePaperclipRunnerAdapterConfig` (runs on every create and update), `paperclipRunnerTransitionConfig` (switching to the native runner), `agent-runner-selection.ts`, `codex-local` `build-config.ts`, and the simplified new-agent setup (`paperclipRunnerCreationLifecycle` in `@paperclipai/shared`). Grok, Pi and Cursor keep their explicit value. - Agent form: the Runner lifecycle field shows the unset state as "Automatic". "Turn by turn" and "Warm session" remain. - Migration `0443_native_warm_follow_up_default.sql`: clears `lifecycleMode = 'per_turn'` from native Codex, OpenCode and Claude agents unless `agent_config_revisions` shows an edit that set it to `per_turn`. Each migrated agent gets an `agent.updated` activity entry with the old value (`actor_id = migration:native-warm-follow-up-default`). Creation records no revision, so a "Turn by turn" pick made at creation cannot be told apart from the old default. The product owner approved migrating those agents. Claude agents follow the default, which is `per_turn` today. - Restart chaos harness and TTFT driver: `--lifecycle per_turn\|warm\|default` (saved in the wave plan). `default` removes the catalog's pinned lifecycle, so a wave exercises the real default path. - Docs: `doc/DEVELOPING.md` "Runner lifecycle default", and the results in `doc/plans/2026-11-07-warm-follow-up-ttft.md`, which also documents baseline finding 4 (Claude's resumed turn-start gap is ACPX's per-turn process generation, not the provider). ## Verification - `server`: `native-lifecycle.test.ts` (new default cases: local Codex and OpenCode warm, Claude, remote, unknown target, unmeasured harness, managed Claude, admitted per-turn session, explicit choices, non-native adapter), `agent-runner-selection.test.ts`, `runtime-selection.test.ts` (335 pass). - `packages/db`: `native-warm-follow-up-default-migration.test.ts` on embedded Postgres. Default agents are cleared, the explicit edit is kept, other harnesses and legacy adapters are untouched, audit rows are written once, and a rerun is a no-op. Without the explicit-edit filter the test fails. `check:migrations` passes. - `codex-local` `build-config.test.ts` (65), `ui` `config-fields.test.tsx` (40), restart-chaos unit tests (134), `tsc --noEmit` for `server`. - `adapter-utils` `paperclip-runner-lifecycle-default.test.ts`: configuration writes and adapter conversion store no lifecycle for Codex, OpenCode and Claude, keep `per_turn` for the others, and preserve explicit choices. It fails without the fix. - End-to-end default check on this branch (`ttft-baseline.ts --lifecycle default`, codex and opencode, 2 samples): the agents read back with no stored lifecycle, and every follow-up reused the runner and provider session (no `runner.process.launch`). Follow-up TTFT was 3.6–4.8 s. This check found the per-write `per_turn` injection fixed here. - Restart chaos (codex and opencode, A1/B1/B5, K1/K2/K6, seed 4242), 8/8 PASS each with no open signatures: forced warm, forced per_turn, and `--lifecycle default` on this branch (agents created without a lifecycle). - `pnpm verify` selects broader CI (harness config is a global input), so required CI supplies the broad coverage. ## Risks - Behavior change: local Codex and OpenCode agents keep a runner and provider process for up to 5 minutes after a run. Instance sleep waits for that idle window. The process is reaped by the idle timer, by graceful shutdown, or by the #19879 reconnect grace if its controller dies. - Migration: agents whose "Turn by turn" was chosen at creation become Automatic. The activity entry records the old value; choosing "Turn by turn" restores it. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): a task session keeps the lifecycle policy it was admitted with, so sessions saved before this change stay `per_turn`. New sessions get the default. No run, checkpoint or launch state changes shape. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 14:37:36 -06:00
DottaandClaude Opus 5.5	6cfb69b4b0	docs(plans): Phase 0 ownership inventory and TTFT baseline (#19846 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart can land between two durable writes. It leaves an in-progress record owned by a dead process. About 69 restart-chaos PRs fixed that shape one subsystem at a time. > - The durable ownership plan (#19831) replaces those fixes with one owner stamp, one prover, write-ahead intents and one reconciler. Phase 0 must first list every in-progress record kind and measure how fast turns start today. > - Without the inventory, Phase 1 cannot choose one stamp or one proof. Without the baseline, no later phase can show that it kept turns fast (invariant 8). > - This pull request adds the Phase 0 inventory, the TTFT, boot and restart-recovery baseline, and the run-log-only driver that produced the baseline. > - The benefit is one reviewed map of 83 record kinds to plan sections A–G, and measured p50/p95 budgets per adapter. ## Linked Issues or Issue Description Refs #19831. Refs #19838 (the TTFT anchor that the baseline uses). What is the problem? Phase 0 of the durable ownership plan has no inventory of in-progress record kinds and no speed baseline. What is the expected outcome? Two reviewed documents in `doc/plans/`, plus a repeatable way to measure the baseline again in later phases. Why does it matter? Phase 1 cannot start without them, and every later phase's exit criteria compare against the baseline. ## What Changed - `doc/plans/2026-11-07-ownership-inventory.md`: - 83 durable in-progress record kinds (94 reader rows) across native runtime, environments, workspaces, accounting and the wake queue; - each kind with its table and column, writer, states, owner, release rule, proof, obligation, invariants and plan sections A–G; - the overlaps, the merge decisions and the zero-downtime upgrade record kinds. - `doc/plans/2026-11-07-ttft-baseline.md`: - TTFT, agent boot, pre-turn steps and restart recovery per adapter (codex, claude, opencode), local, cold and warm, p50/p95; - restart recovery from the overnight chaos evidence, the slowest steps ranked, and the gaps. - `tests/restart-chaos/ttft-baseline.ts`: an on-demand driver on the chaos supervisor. It reads the run log only, never Telemetry, and always stops its embedded Postgres. - `tests/restart-chaos/ttft-baseline-stats.ts` and its unit tests: pure percentile, boot, pre-turn ranking and recovery statistics. No runtime behavior changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/ttft-baseline-stats.test.ts` - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` - The baseline numbers come from the driver run named in the report. To reproduce them, follow the report's "Method" section. - The owner spot-checked the inventory's file:line claims at `f2d8a51027`. One example: `processLossRetryCount` has no writer, and `heartbeat/recovery.ts:2675` is its only reader. ## Risks - Low risk. This adds documentation and an opt-in test driver. Nothing runs in CI except the pure stats unit tests. - The driver spends a small amount of provider credit when someone runs it by hand. - The baseline sample is small (6 per cell). The local host was shared with other worktrees, so the p95 tails include load. The report states both. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use, code execution and five parallel read-only subagents for the inventory. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 10:47:31 -06:00
DottaandClaude Opus 5.5	6fb6220d96	fix(restart-chaos): rebuild the Daytona plugin when its sources change (#19798 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness runs Daytona waves with a local control plane that installs the bundled Daytona sandbox-provider plugin from `packages/plugins/sandbox-providers/daytona` > - That plugin is excluded from the pnpm workspace, so workspace builds and `scripts/agent-preflight.mjs` never rebuild it > - The harness built it only when `dist/` was missing. A long-lived chaos worktree that moved across many master commits kept serving the plugin it built on day one > - As a result, plugin-side fixes such as #19696 (reclaim sandboxes left by a restart during acquire) and #19754 (keep a runner sandbox alive through the reconnect grace) were not under test in those waves, while server and runner fixes were > - This pull request keys the plugin build on a content hash of its inputs and rebuilds when the hash changes > - The benefit is that every Daytona wave tests the plugin of the commit it claims to test ## Linked Issues or Issue Description Bug description `ensureDaytonaPlugin()` in `tests/restart-chaos/main.ts` returned early whenever `dist/manifest.js`, `dist/worker.js` and the Daytona SDK existed. Steps to reproduce Build the plugin on commit A, check out a later commit B that changes `packages/plugins/sandbox-providers/daytona/src`, and run a `--env daytona` wave. The wave installs the commit-A plugin. In one chaos worktree, `dist/plugin.js` was 20 hours older than `src/plugin.ts` and lacked `runnerReconnectGraceMs` from #19754. Expected behavior The harness rebuilds the plugin whenever its sources, lockfile or the plugin SDK source changed since the last build. Refs #19544 ## What Changed - `tests/restart-chaos/plugin-build.ts` (new): `pluginBuildInputHash` hashes the plugin's `package.json`, `pnpm-lock.yaml`, `tsconfig.json` and `src/`, plus extra inputs: the plugin SDK `src`, `package.json` and `tsconfig.json`, and the root `tsconfig.json` and `tsconfig.base.json` that both packages extend. It excludes `node_modules` and `dist`. `pluginBuildState` compares that hash with a stamp in `dist/.restart-chaos-build-input.sha256` and reports a reason. - `tests/restart-chaos/main.ts`: `ensureDaytonaPlugin` installs, links the SDK and builds when the state is stale, then writes the stamp. It records `daytona_plugin_build_state` and `daytona_plugin_built` in the timeline. - `tests/restart-chaos/plugin-build.test.ts` (new): an unstamped build is stale; the stamped build is up to date; a changed source, lockfile or SDK source is stale; changes to `node_modules` and `dist` are ignored; missing outputs or dependencies are stale. Other out-of-workspace packages, checked as requested: - `pnpm-workspace.yaml` excludes only `packages/plugins/sandbox-providers/`. - The restart chaos local mode (`--env local`) does not install a sandbox provider. - CI's runner-e2e "Prepare bundled Daytona plugin" step runs on a fresh checkout per job, so it cannot see a stale `dist/`. - The other sandbox providers are not used by these harnesses. The risk is therefore limited to long-lived local worktrees running Daytona waves, which this change covers. ## Verification - `vitest run --config tests/restart-chaos/vitest.config.ts`: 121/121 pass, including the new staleness tests. - `tsc -p tests/restart-chaos/tsconfig.json`: no errors in the changed files. - Manual: in the chaos worktree, `dist/plugin.js` (built 2026-11-06 11:16) lacked `runnerReconnectGraceMs`. After a rebuild at master `35d41f11bb` it contains it. ## Risks - This change touches the test harness only. - The first wave after this change rebuilds the plugin once in each worktree, because existing builds have no stamp. That takes a few seconds. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, GitHub CLI). Default context window, extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 03:44:25 -06:00
DottaandClaude Opus 5.5	644ee66f5d	test(restart-chaos): R5 live tools and processes; fail a wave whose worktree changes mid-wave (#19697 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) grades every task on R1–R6; R5 catches silent stalls, including a run that stays `running` with no live evidence > - A B5 task runs `bench.sh` for about four minutes with no output, so a healthy Retry run was graded `R5:silent_dead_run` while its runner process was alive and its tool was still running > - Separately, a runner `dist` rebuilt in the harness worktree during a wave made later restarts run different code and produced a false signature (RC-32's investigation) > - This pull request makes R5 treat a run as live while a tool it started has not ended or while its recorded local runner process still runs, and fails a wave whose worktree changes mid-wave > - The benefit is fewer false R5 failures and no silently mixed-code waves ## Linked Issues or Issue Description Refs #19629 Refs #19641 What is the problem? R5 flags a quiet but live run (a long tool call, a live runner process) as `silent_dead_run`. A build, install or branch switch in the harness worktree during a wave goes unnoticed, though every restart re-runs the control plane from that worktree. What is the expected behavior? A run with a tool in flight, or with its recorded local runner process still alive (same PID and start time), is live for R5. A wave whose checkout or runner build output changes during the wave fails as a harness error. Why does it matter? False R5 failures send fixers after non-bugs, and mixed-code waves produce signatures that are not product behavior. ## What Changed - `observer.ts`: the run-event reader counts tool starts minus tool ends (PRP item ids are not unique per tool call), and remembers the run's latest local process identity until recovery reports it stopped. Samples carry `toolInFlight` and `processAlive`. `localProcessAlive` checks `ps` for the same PID with the same start time, so a recycled PID never counts. Daytona runs are not checked (their process is remote). - `grader.ts`: `deadRunningRun` skips a run with a tool in flight or a live process. - `types.ts`: `RunSample.toolInFlight` and `RunSample.processAlive`. - `wave-guard.ts` (new) and `main.ts`: fingerprint `HEAD`, the tracked diff contents (`git diff --binary HEAD`), untracked file contents, and size/mtime of `packages/paperclip-runner/dist` and the runnerd binary. The baseline is taken after the harness's own setup (preflight rebuilds provider entrypoints). Re-check after each fault and at the end; fail the wave (`worktree_changed_mid_wave`) if it changed. - `main.ts`, `scheduler.ts`: `faults.json` and a new `task-faults.json` (each task's launch time and crossing faults) are written after every fault. `--replay` falls back to that log when a task has no grade file (an interrupted or wedged wave), so a storm task replays the faults that crossed it at the same offsets. - `README.md`: both grading notes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 111 tests pass. New tests cover: - a B5 run quiet for 4 minutes passes with a tool in flight or a live process, and fails otherwise - tool and process liveness come from run events and clear on tool end and process stop - `localProcessAlive` accepts this process's real PID and start time and rejects a different start time - the guard catches rebuilt dist, a re-edited dirty file, an edited untracked file and a rebuilt runner binary, in a real temporary git repo - a storm task with no grade file replays the faults that crossed it at their offsets from its launch - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. ## Risks - Low risk, test harness only. - The R5 exemption narrows R5. A run that is truly stuck while its process lives (a hung provider) now passes R5 and falls to R1's time budget instead. - One `ps` call per running run per poll on local waves. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 22:07:42 -06:00
DottaandClaude Opus 5.5	da850990f8	fix(restart-chaos): grade R4 on overlapping execution; wait out pending cleanup for R6 (#19710 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades every task on rules R1 to R6. R4 means "no duplicate execution" > - R4 counted every queued, running or scheduled-retry run in one poll as an active run, and failed the task when there were two > - A follow-up comment while a run is in progress normally queues a wake behind that run. The product cancels the wake when the running run finishes, so the wake never executes > - A Daytona wave failed an opencode task for exactly this case, so R4 reported a duplicate-execution failure that did not happen > - This pull request grades R4 on actual overlapping execution > - The benefit is that R4 failures again mean that two runs really executed together, or a comment was consumed twice ## Linked Issues or Issue Description Bug description `gradeR4` failed a task whenever one poll showed two runs in `ACTIVE_RUN_STATUSES` (queued, running, scheduled_retry). This includes a wake queued behind a running run that never starts. Steps to reproduce Daytona wave `w-20261107T013205-s11`, task t06 (opencode C1 K2): - Run b4b16a4d is running. - The C1 follow-up comment queues run c090799a. - c090799a is cancelled 1 s after b4b16a4d succeeds and never starts. - Before this change, R4 reported `concurrent_runs: c090799a(queued), b4b16a4d(running)`. Expected behavior R4 passes. It fails only when two runs execute at the same time, or when a side effect (a consumed comment) repeats. Refs #19544 ## What Changed - `tests/restart-chaos/grader.ts`: `gradeR4` fails when two runs are `running` in the same sample, or when the started→finished windows of two started runs overlap. A still-active run's window lasts until the last sample. Queued and scheduled-retry runs that never started do not count. The comment-consumed-twice check is unchanged. - `tests/restart-chaos/grader.test.ts`: a test with the t06 shape (a queued wake cancelled behind a running run) that passes; queued plus scheduled_retry without a start passes; overlapping windows fail even when no single poll saw both running. The existing two-running and comment tests are kept. - `tests/restart-chaos/README.md`: the R4 grading note. - R6 settle for Daytona (`tests/restart-chaos/daytona-sweep.ts`, `main.ts`). The product retries a `pending_cleanup` lease after `ENVIRONMENT_CLEANUP_RETRY_MS` (5 min), so a 2-min settle reported normal cleanup as a leak. R6 now waits `PENDING_CLEANUP_RETRY_MS + 90 s`, capped at 8 min; a test keeps the constant equal to the server's. An environment DELETE that returns 409 while a sandbox cleanup is pending is correct product behaviour. It is retried for the same window. If it is still blocked after that, it is reported as `environmentDeleteBlocked`, and it fails the wave and keeps the instance (whose database owns the cleanup work). It is not counted as a harness error. Tests cover the 409 retry and a real (500) delete failure. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 109/109 pass. `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - R6 evidence: in wave `w-20261107T015455-s12`, three leases were still `pending_cleanup` at the 2-min mark, and the environment DELETE returned 409 "Cannot delete this environment while a sandbox cleanup is pending". - The t06 sample sequence from `w-20261107T013205-s11` is the new test case. ## Risks - This change touches the test harness only. - R4 is now less strict about queued rows. If a queued duplicate starts executing, its start time makes its window overlap, so R4 still fails. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, GitHub CLI). Default context window, extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 21:31:29 -06:00
DottaandClaude Opus 5.5	aca908a165	fix(restart-chaos): stop the timeline echo loop when stdout breaks (#19695 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs waves of real agent tasks while it restarts the control plane on purpose > - A wave writes a timeline file and also echoes every timeline entry to stdout > - When the process that reads that stdout exits, every echo fails with EPIPE. The uncaught-exception handler then records the error with another echo, so the loop never ends > - In one Daytona wave this wrote 3.2 million entries (2.5 GB) in 30 minutes. It also stopped the wave from grading its tasks > - This pull request stops the echo after the first stdout error, and records the stdout loss once > - The benefit is that a wave keeps running and grading when its log reader goes away ## Linked Issues or Issue Description Bug description `Timeline.add` echoes with `console.info`. After stdout breaks, each echo raises EPIPE as an uncaught exception. The handler in `main.ts` calls `timeline.add("harness_uncaught_exception")`, and that call echoes again. Steps to reproduce Run a wave as a child process that reads its stdout, for example from a wave-loop script. Kill the parent while the wave runs. `timeline.jsonl` then grows without bound with `harness_uncaught_exception` / `write EPIPE` entries. Expected behavior The wave goes on writing its files (timeline, grades, ledger) and stops writing to the broken stdout. Refs #19544 ## What Changed - `tests/restart-chaos/timeline.ts`: a `process.stdout` error listener and a guard around `console.info` turn off the echo for good after a broken-pipe error. The timeline file is still written. On the first entry after the echo stops, the timeline writes one `harness_stdout_lost` entry (file only) with the error. The file adds `isBrokenStdoutError` and `markTimelineEchoBroken`. - `tests/restart-chaos/main.ts`: the uncaught-exception handler passes broken-stdout errors to `markTimelineEchoBroken`. It does not mark the wave as interrupted and does not set a harness error. - `tests/restart-chaos/timeline.test.ts` (new): the error classification, and the file-but-no-echo behaviour with exactly one `harness_stdout_lost` entry. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 102/102 pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - Incident: wave `w-20261107T001552-s13`. Its timeline had 76 real entries, then 3,242,762 `harness_uncaught_exception` entries. Those 3.2 million entries came from the loop this pull request removes. ## Risks - This change touches the test harness only. - After stdout breaks, the console shows no wave progress, but every result is still in the wave's files. A different uncaught exception still stops the wave as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, GitHub CLI). Default context window, extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 20:06:02 -06:00
DottaandClaude Opus 5.5	d26372dc8a	test(restart-chaos): add a result trigger and recovered-run comment-wake guards (#19689 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness kills the control plane at chosen moments and grades whether each task still finishes correctly > - One wave lost a user's follow-up comment after a kill that landed after the run accepted its result but before the run settled > - No existing trigger aims a fault at that moment, so the case was hard to reproduce on purpose > - The recovery path that settles such a run had no regression test that covers the saved comment > - This pull request adds a `result` trigger to the harness and a regression test for the recovery path > - The benefit is that the harness can aim faults at the result-accepted window, and a change that drops, repeats, or re-wakes a saved comment fails a test ## Linked Issues or Issue Description What happened? In a restart-chaos storm wave, a user posted a comment while a native run was active. A hard kill (K1) landed after the run accepted its result and before the run settled. The run settled `succeeded` and the task stayed `done`, but no follow-up run read the comment. Steps to reproduce The original failure did not reproduce. Three targeted waves ran 12 tasks with K1 at comment time or in the new result window. Every follow-up comment was consumed exactly once. The new regression test also passes on current `master`. The test and the trigger guard the path and do not claim a fix. Expected behavior A comment posted during a run is consumed exactly once after the run ends, including when restart recovery settles the run. ## What Changed - `tests/restart-chaos`: add `--triggers result` (C1 only). K1 fires 0–3 s after the run that received the follow-up proposes or accepts its result (`run.result.proposed` or `run.result.accepted`). The fault fires only while that run is still `running`: the observer checks the latest sample and rereads the runs right before dispatch. A run that settled during the delay, or a follow-up posted at rest, records a `result` miss. Add poll-driven unit tests for these cases and a README row. - `server/src/__tests__/native-run-projection-work.test.ts`: add embedded Postgres tests in which background recovery, not the executor, settles the run (projection, environment settlement, and deferred-receipt workers), with a deferred user comment and an active local lease left by the killed executor. The tests check: - The comment is promoted into exactly one follow-up run, and a second restart adds none. - A comment the run already received as a same-turn steer is not woken again. The other saved comment still is. - A queue whose only comment was steered stays cancelled, and no follow-up starts. - A Retry click does not consume the comment a second time. A click that observed the source is stale. A click that observed the follow-up joins it. ## Verification - `cd tests/restart-chaos && npx vitest run --config vitest.config.ts` (96 tests pass) - `cd server && npx vitest run src/__tests__/native-run-projection-work.test.ts` (54 tests pass) - Live, local, ACPX Claude, Codex and OpenCode: `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --workloads C1 --triggers comment --faults K1 --adapters codex,claude,opencode --tasks 4 --concurrency 2 --rc06-masked` passed 4 of 4. Two waves with `--triggers result` consumed each follow-up exactly once. One task in those waves found a separate defect: a kill 1.4 s after the result commit left the run `running` behind the dead controller's 20-minute finalization lease. A separate pull request fixes that defect. ## Risks - Low risk. These are test and harness changes only. No production code changes. - The `result` trigger depends on the observer's poll interval (about 3 s), so the exact kill point within the window varies. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 19:53:58 -06:00
DottaandClaude Opus 5.5	961455b9a0	test(restart-chaos): add the K13 restart storm mode (#19629 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task recovery restarts agent work after a control-plane restart or a failed run > - When automatic recovery runs out or is blocked, the user clicks Retry > - Retry then started a fresh isolated attempt cloned from committed HEAD, so the agent lost its uncommitted files, its task branch and the worktree Git link > - A restart must never lose the agent's work > - This pull request makes Retry keep the task's own workspace when that is safe, and tells the agent where the old files are when a fresh attempt is still required > - The benefit is that a Retry continues the real work instead of starting over ## Linked Issues or Issue Description What happened? After `native_restart_recovery_blocked` or an exhausted recovery generation, a Retry ran in `isolated-workspace-attempts/<company>/<issue>/<run>/workspace`. That directory was a detached clone of the base ref. It did not contain the uncommitted edits from the issue's git worktree. It was not on the task branch. The agent was not told where the old files were. A restart-chaos wave graded this as an R2 failure (`pre_fault_files_lost`). A second defect: when a restart hit the bootstrap of a fresh attempt, preparing the same run again failed with `isolated_task_directory_binding_mismatch`, so the Retry could not finish. What did you expect? Retry continues in the task's own workspace with its edits and `.git` intact when no other writer can hold it. When a fresh attempt is really needed, the old directory stays as it was and the agent gets its path and its changed files. Preparing the same attempt again after a restart is idempotent. Steps to reproduce 1. Start a native task in a git worktree. Let the agent write an uncommitted file. 2. Kill the control plane so recovery ends in `native_restart_recovery_blocked` or an exhausted generation. 3. Click Retry. 4. The successor runs in a fresh clone without the uncommitted file or the task branch. Root cause: - `server/src/services/fresh-workspace-continuation.ts` `canSelectFreshWorkspace` returns a fresh intent for every task Retry that has a system recovery record. - `server/src/services/explicit-native-continuation.ts` (fresh selection in the source loop) then always persisted `fresh_isolated_workspace`. - `server/src/services/fresh-workspace-seed.ts` `materializeFreshWorkspaceSeed` clones the project source at the base ref with `--no-local`, checks out a detached commit and removes `origin`. Uncommitted work and the branch are not carried over. - RC-35: `server/src/services/heartbeat/workspace-preparation.ts` rejected any reusable existing workspace while `fresh` was true, including the attempt directory that the same run had already created and bound. ## What Changed - New continuation mode `reuse_issue_workspace`. At Retry admission, after all stop proofs pass, `retainedIssueWorkspaceForRetry` checks that the task's bound workspace is the single local directory the failed run used, owned by the task, not a shared workspace, not under remote storage authority, with no other lease, failed cleanup or running workspace operation, a settled recovery generation, no workspace-export failure, and a readable Git state rooted at that directory. Then the Retry continues there with a fresh session. Otherwise the existing fresh attempt is kept. - The reuse decision consumes the generation's one alternative attempt (`freshAttemptRunId`), sets the Retry strategy to `continue`, and records `issue.retry_workspace_reused`. - Environment selection pins the reused workspace binding and fails with `retry_workspace_binding_mismatch` if the binding changed before dispatch. - Fresh fallback: the run's continuation context gets `priorWorkspace` (path, branch, bounded list of uncommitted files). Git status is read with optional locks disabled. The prompt tells the agent to inspect it and carry over the work. The old directory is not changed. - RC-35: preparing the same fresh attempt again after a restart reuses the attempt directory that this run already bound. - `inspectGitCloseReadiness` is exported for the reuse check. - `doc/task-recovery.md` and `doc/execution-semantics.md` describe the new contract. ## Verification - `server/src/services/retry-workspace-continuity.test.ts` (embedded Postgres, real git worktree with uncommitted edits): - exhausted-generation Retry and blocked-recovery Retry continue in the same worktree, `.git` file and branch intact. Both failed before the fix (readiness was `fresh_isolated_workspace`). - The fresh attempt is kept when the workspace is unsafe: live lease, running workspace operation, corrupt `.git`, missing directory, remote authority, shared workspace, running export generation. - `workspace-preparation.test.ts`: - RC-35 red/green: re-preparing the same fresh attempt reuses its bound directory and keeps files written before the restart. Another run still gets `isolated_task_directory_binding_mismatch`. - The fresh fallback sets `priorWorkspace` with the old path and `unpublished.txt`. - `environment-selection.test.ts`: the new mode pins the binding and rejects a changed binding or cwd. - `packages/adapter-utils/src/server-utils.test.ts`: the prompt names the retained directory and its files. - Regression files run locally: `workspace-preparation`, `environment-selection`, `task-retry`, `quarantined-task-retry`, `explicit-native-continuation`, `workspace-recovery`. - Restart-chaos wave: WAVE_RESULT_PLACEHOLDER ## Risks - A Retry now writes into the old directory instead of a new one. The checks above prevent a second writer. A stale binding fails the run with `retry_workspace_binding_mismatch`; it does not write to another directory. - If the cause of the failure was inside the workspace itself, reuse could repeat it. Workspace-export failures, restore failures and required workspace recovery keep the fresh attempt. - `priorWorkspace` is an optional new field in the continuation envelope. Older adapters ignore it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 18:01:42 -06:00
DottaandClaude Opus 5.5	8479929f62	test(restart-chaos): add an RC-06 mask, wave spend and replay masks (#19559 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) restarts the control plane during real agent tasks and grades each task on rules R1–R6 > - One known product bug (RC-06) leaves a completed run "awaiting accounting". The company hard stop then refuses all new work, so one stuck row fails every later task in the wave > - That masks every other restart bug in the wave. Waves also had no record of what they spent > - This pull request adds an opt-in RC-06 mask, records wave spend, and lets a replay keep or add the mask > - The benefit is that chaos waves can find new restart bugs while RC-06 is open, and stay inside a known spend ## Linked Issues or Issue Description Refs #19477 (the Phase 1 harness this extends). What existing behavior does this improve? Restart chaos waves. One stuck accounting row (RC-06) hard-stops the isolated Chaos Lab company. All later tasks then rest blocked "over budget", and a Retry is refused with "Company cannot start work while completed runs await accounting". Two waves went 2/6 and 0/6 for this one reason. Subsystem affected Test harness only: `tests/restart-chaos/` (config, seed, scheduler, ledger, supervisor, main, README). Current behavior - No way to see past RC-06 without disabling the budget. - Wave spend is not recorded. - `--replay` rejected any mask flag, and it was not documented that the saved plan carries the mask. Proposed behavior - `--rc06-masked` keeps the budget amount and turns off only the hard stop on the isolated company. Masked tasks are labeled in the ledger and summary. - Each wave records its spend. - A replay keeps the source plan's masks and may add the RC-06 mask. ## What Changed - `--rc06-masked` (wave mode): after the company is created, the seed upserts the company budget policy with the same amount and `hardStopEnabled: false`. It writes `seed_mask_rc06` to the timeline. The plan stores `options.masks = ["rc06"]`. - The accounting gate in `budgets.ts` (`observedBlocks`) applies only when `hardStopEnabled` is true, so the mask lifts it. No product code changes. - Ledger lines carry `masks`. `SUMMARY.md` labels the cell and the latest-wave row `rc06-masked`. - Wave spend: before the sweep, `main.ts` reads `/api/companies/:id/costs/summary?period=all` (each wave owns its company), writes `wave_spend` to the timeline and prints it. If the wave stops early, cleanup records it once, best effort, before the server stops. - Load: #19513 already records `loadAvg1m` on each fault receipt. This branch first added the same value under another name, and the rebase removed it. The README now documents `loadAvg1m` and the load-suspect threshold. - Replay: `--replay <wave>/<task>` keeps the masks saved in the source `plan.json`. `--replay … --rc06-masked` adds the RC-06 mask. A replay never drops a mask. - README documents the flag, spend, load and the replay rule. - New unit tests: replay keeps saved masks and the replay flag adds the mask; the masked seed posts the company policy with the same amount and `hardStopEnabled: false` and records `seed_mask_rc06`, and an unmasked seed makes no policy call. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 71/71 pass (after rebase onto master with #19513). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. - Live: 4 masked local waves ran with this code (seeds 3307, 4409, 5501, 6619; Codex, ACPX Claude and OpenCode). - `seed_mask_rc06` appeared and no task was blocked by the accounting gate, even though `pendingRunCount` was 1–8 at wave end. - `wave_spend` was recorded each time (38¢–127¢ per wave). - The 1-minute load average was recorded on every fault (15–58). - Before the mask, the same harness gave 2/6 and 0/6, every failure RC-06. With the mask, the waves found other restart failures, all since fixed on master: RC-14, RC-16, RC-20 and RC-21. ## Risks - Low risk. Test harness only, and the mask is opt-in. - A masked wave has no product hard stop. The harness still records spend for each wave, and the operator must watch it. The mask applies only to the isolated, temporary Chaos Lab company. It never touches another instance. - Masked results do not count as RC-06 coverage. The ledger labels them so they are not mixed with unmasked cells. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 15:27:12 -06:00
DottaandClaude Opus 5.5	1634f8b66b	test(restart-chaos): run chaos waves against Daytona sandboxes without Docker (#19544 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) restarts a local control plane on purpose while agent tasks run, and grades every task on rules R1 to R6 > - The plan targets agent runs in Daytona sandboxes, but the harness only supported `--env local` > - A local controller needs two inputs that match the sandbox image exactly: the image's Linux `paperclip-runnerd` and its provider pack. The documented way to get them uses Docker, and Docker is not available on the operator machine > - The Daytona org that chaos uses also runs the live Cloud fleet, so the harness must never change a sandbox it did not create through its own Paperclip environment > - This pull request adds `--env daytona`. It extracts both inputs from the image without Docker, and refuses a controller whose runner source differs from the image. It seeds a Daytona environment from the runner-e2e fixtures and adds a read-only Daytona R6 sweep > - The benefit is that chaos waves can now cross real Daytona sandboxes with a local, killable control plane ## Linked Issues or Issue Description Feature description Run restart chaos waves with a local control plane and agent runs in Daytona sandboxes (plan `doc/plans/2026-11-06-control-plane-restart-chaos.md`, Phase 1b). Problem it solves `--env daytona` threw "not wired yet". The controller-side runnerd and provider pack had to come from `docker cp`, which needs a Docker daemon. Proposed solution Use `skopeo copy` (no daemon) to fetch the linux/amd64 image into a directory, walk the OCI layers from top to bottom, and extract only the two paths. Before the wave starts, compare the image's content-ID label with this checkout. Refs #19477 ## What Changed - `tests/restart-chaos/daytona-image.ts`: a streaming tar reader and an OCI layer walk (top to bottom, `.wh.<name>` and `.wh..wh..opq` whiteouts, hard links, symlinks written last, no `..` paths). It extracts `/usr/local/bin/paperclip-runnerd` and `/opt/paperclip-runner/provider-pack/`. It checks the layer blob digests, the ELF header (x86-64), the provider-pack manifest schema, and its `runnerSourceRevision` against the image revision label. It records digests in `receipt.json`, caches the result by image digest under `results/daytona-images/`, and deletes the layer blobs. It also runs standalone. - Controller/image check: `assertControllerMatchesImage` compares the image's `io.paperclip.runner.content-id` label with `computeDaytonaImageContentId` for this checkout. On a mismatch it fails and prints both IDs and the image's source revision. - `seed.ts`: for `--env daytona`, it installs the bundled Daytona plugin (`installFixtureDaytonaProvider`) and waits for provider readiness. It creates the `daytona` environment fixture from `catalog.ts` with a `DAYTONA_API_KEY` company secret and runs the native Codex platform-qualification probe. Then it assigns the agents and the project to that environment. `--daytona-cpu` sets the sandbox vCPU (default 4, 4 GiB, 10 GiB). - `daytona-sweep.ts`: a read-only R6 for Daytona. The Daytona client is a frozen wrapper that exposes only a label-filtered `list`, and no stop, archive or delete exists. The sweep keeps only rows whose `paperclip-company-id` and `paperclip-environment-id` both equal the wave's own UUIDs, and it re-checks every row. It reconciles sandboxes with the environment's leases: - A sandbox archived after its lease was released is expected. - An orphaned sandbox or an `active`/`pending_cleanup` lease fails R6 for the owning task, or the wave. - `assertHarnessMayMutate` refuses `paperclip-create-` names and `fleet-sbx-` snapshots. - Cleanup only deletes the wave's environment through Paperclip and reports what is still running. - `config.ts`: `--daytona-image` (or `PAPERCLIP_E2E_DAYTONA_IMAGE`) needs an immutable digest. `--concurrency × --daytona-cpu` must stay within `RESTART_CHAOS_DAYTONA_VCPU_BUDGET` (default 40). B5 is local-only, because its oracle reads the live workspace. - `main.ts`: Daytona preflight (credentials, image extraction, content-ID check, Daytona plugin build as CI does). It passes `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` and `PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` to every server start and writes `sandbox-sweep.json`. Cleanup errors never skip stopping the controller. - README: a Daytona section (image matching, image-only CI dispatch, vCPU budget, fleet safety, R6). - Unit tests: synthetic tar layers for the tar reader, whiteouts, opaque directories, overrides, hard links, symlinks and parent replacement. Further tests cover the content-ID check, sweep reconciliation, exact wave selection (unlabelled rows and other labels are never selected), the read-only client, the fleet guard, and the CLI options. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 62/62 pass, credential-free. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - Real extraction without Docker, image `…@sha256:b0c71f4d…`: 42 layers, about 8 minutes, a 3 GB pack. - The content-ID check refused the current checkout. The image is `b8b16e15…` from `7e6d882558`; the checkout is `aec45589…`. - A checkout of `7e6d882558` computes `b8b16e15…`, so the check compares like with like. - Current image: `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:a84579522192de0393ae24d09e67c4e752df4c701bd315ff54ad9bbca42e2dc0` (source `5d25117ffa`, content `4e9a221f…`). It matches this branch, and the receipt shows the content ID match. - Paid waves, `--env daytona --adapters codex,claude --tasks 3 --concurrency 2 --faults K1 --seed 11`: - Wave 1 (10 vCPU org, 4 vCPU sandboxes): 0/3. Every create returned HTTP 400 "Total CPU limit exceeded. Maximum allowed: 10", reproduced directly with the SDK. This led to the vCPU budget, and the archived-sandbox fix in R6. - Wave 2 (same org, 2 vCPU sandboxes): 3/3 PASS, all restart-crossing. Codex completed through one working Retry; both Claude tasks continued automatically. R6: 4 sandboxes archived after release, 0 orphans, 0 held leases. - Wave 3 (company org, 4 vCPU sandboxes): 0/3. Two product failures were found and reported to the campaign ledger: an ACPX Claude recovery blocked on `ambiguous_remote_recovery_evidence` with no Retry, and a false budget hard stop at 10.75¢ of 3000¢ that paused the company. The harness changed no sandbox. R6 reported 2 held leases, and the environment delete returned 409 `pending_sandbox_cleanup` because of them. - Fleet-org key check: one sandbox with a unique non-`paperclip-create` name and label was created, then deleted by the exact ID that create returned. ## Risks - This change touches the test harness only, with no product code. - The extracted provider pack is about 3 GB per image in the gitignored results directory. Delete old `results/daytona-images/<digest>` directories by hand. - The harness never changes Daytona sandboxes. If Paperclip's environment cleanup is blocked, sandboxes stay until their 60-minute ttl or a person acts. The harness reports them in `sandbox-sweep.json` and fails the wave. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, GitHub CLI). Default context window, extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 14:26:45 -06:00
DottaandClaude Opus 5.5	e8f07b2a39	test(restart-chaos): fire faults relative to user actions and run state, add K6 crash loop (#19513 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane must keep tasks moving when the server restarts > - The restart chaos harness (#19477) kills the server only at a random time after a task starts > - Many restart bugs depend on what the user or the run was doing at that moment: a comment just sent, a Retry just clicked, a run still queued, a long tool call, or a second crash during startup recovery > - The harness cannot aim at these moments yet, so these cases occur only by chance > - This pull request adds fault triggers that fire relative to a user action or a run state, and the K6 crash loop > - The benefit is that chaos waves can target these boundaries on purpose and replay them ## Linked Issues or Issue Description Refs #19477 What is the problem? The restart chaos harness fires each fault at a seeded random offset from task launch. It cannot fire a fault right after the harness posts a comment to an active run, right after a Retry click, while a run is queued, during a long tool call, or a second time while the server is in the `recovering` startup phase. Why is it a problem? These are the boundaries where restart bugs hide (plan faults K6 and K8, boundaries B1, B6 and B8). Random timing seldom hits them, and a replay cannot repeat them. What is the proposed solution? Add a `--triggers` option. Each task samples a trigger with its cell. The observer arms the fault when the trigger condition holds, then fires it after a seeded delay. Add the K6 crash-loop fault to the supervisor. ## What Changed - `types.ts`, `config.ts`: add `TRIGGER_IDS` (`time`, `comment`, `retry`, `queued`, `tool`), the `--triggers` option, and the `K6` fault kind. - `scheduler.ts`: sample a trigger per task. `comment` pairs only with C1 and `tool` only with B5. Each trigger has a seeded delay range. `retry` adds an extra K1 after the planned fault. A replay re-arms the task's own triggered faults. - `observer.ts`: arm each triggered fault from the observed state: the follow-up reached a running run, an accepted Retry, the first run still queued, or the first `bench.sh` progress. A trigger that can no longer hold is recorded as `triggerMissed`. - `supervisor.ts`: K6 kills the replacement again as soon as `/api/ready` reports `recovering` (or `ready`, if recovery was too fast to see), then once more just after `ready`. The receipt records the phase that each kill hit. - `main.ts`, `ledger.ts`: pass the trigger to the fault receipt, the timeline and the ledger. The summary shows the trigger in each cell. - `triggers.test.ts`: unit tests for planning, firing, misses, replay and the crash-loop phase rule. - `README.md`: document the triggers and K6. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts` (52 tests pass). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live: `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude,opencode --faults K1,K6 --triggers comment,retry,queued,tool --concurrency 3 --tasks 6` runs waves with the new triggers. ## Risks - Low risk. The change is in the test harness only. The default trigger is `time`, so existing waves and replays keep their behavior. - A trigger can miss its window (for example, a run starts before the first poll). The ledger records the miss, so the task is not counted as a crossing of that boundary. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 13:49:52 -06:00
DottaandClaude Opus 5.5	7afb57c022	test(runner-e2e): repair restart-cell harness drift (#19538 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Product E2E suite proves that agent work survives a control-plane restart > - A restart chaos campaign found three restart cells that fail in the test harness, not in the product > - A harness failure hides the real restart result, so the campaign cannot classify these cells > - This pull request repairs the harness drift in those cells and names the one failure the harness cannot prevent > - The benefit is that these cells now test restart behavior, or fail with a clear cause ## Linked Issues or Issue Description Refs #14811 (connector setup became one screen). Refs #19475 (completion-update restart fixture). What happened? 1. `lifecycle-baseline.runner-codex.local.tool-review-restart` clicked a "Save and continue" button. #14811 removed that button. The cell failed during setup. After that fix, the cell also failed after the restart: the expanded review card does not survive the post-restart reload, so "Approve & run" was not visible. 2. `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` failed with `page.goto: net::ERR_ABORTED` after the controller restart. The old document's Vite client reloads when the replacement server starts, and this aborts the test navigation. The code tolerated only `TimeoutError`. 3. `completion-updates.runner-acpx-claude.local.handoff-completion-restart` waited for the source agent to run an instructed wait command. Claude replied without the command (2/2 attempts), so the restart never occurred. The cell then failed with a generic timeout. Expected behavior Each cell reaches its restart boundary and tests restart behavior. If the boundary cannot be reached, the cell names the cause. Steps to reproduce Run `pnpm test:e2e:runner -- --max-parallel 1 --max-automatic-retries 0 --id <cell>` for each cell above on `master`. ## What Changed - `tests/runner-e2e/connection-reviews.ts`: accept the one-screen "Check link" step and the older "Save and continue" step. This uses the same regex approach as `connection-flow.ts`. - `tests/runner-e2e/runner.spec.ts` (tool review): click "Review request" after the restart reload, on the document that acts on the card. - `tests/runner-e2e/chat-restart.ts`: add `gotoAfterServerRestart`. It retries only `net::ERR_ABORTED`, within the caller's deadline and at most 5 attempts. It still accepts `TimeoutError`. It rethrows all other errors. The structured-question restart uses it. The existing sentinel and durable-state assertions still prove the new document. - `tests/runner-e2e/completion-updates.ts` and `completion-update-flow.ts`: add `completionRestartGateSkipped` and `completionRestartGateTimeoutDetail`. The gate poll stops at once with `restart boundary not reached (agent skipped gate)` when no completion turn is active and the turn that carries this task's Done facts (`chatCompletionUpdates`, as in `completionDelivery`) succeeded or published its reply before the gate reports `waiting`. A failed or cancelled turn is not blamed on the agent, because delivery can retry it; the timeout detail names it instead. The post-restart chat navigation now uses `gotoAfterServerRestart`. - A fixture-level gate is not possible without a product hook. The native run finalizer publishes the completion reply inside its own transaction, through the issue service closure. The isolated-server entrypoint can patch only class prototypes and HTTP responses, and the reply uses no tool call or HTTP request. - `tests/runner-e2e/README.md`: document the skipped-gate failure. - Docs: use `pnpm --filter @paperclipai/paperclip-runner run build:runner-binaries` in `tests/runner-e2e/README.md`, `tests/restart-chaos/README.md`, `tests/restart-chaos/main.ts` and `doc/DEVELOPING.md`. - No product code changes. ## Verification - Unit: `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-restart.test.ts tests/runner-e2e/completion-updates.test.ts tests/runner-e2e/connection-reviews.test.ts` (36 passed). - Typecheck: `node scripts/typecheck-test-projects.mjs tests/runner-e2e` passed. - Live cells, local, `--max-parallel 1 --max-automatic-retries 0`, macOS, load average 35–65: - `lifecycle-baseline.runner-codex.local.tool-review-restart`: PASS (first run found the review-card reload drift, fixed in the second commit). - `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`: PASS. A first attempt failed before the restart with `PRP command session.open timed out` at load ~36 (environment). - `completion-updates.runner-acpx-claude.local.handoff-completion-restart`: FAIL as designed, twice (before and after the review fix), with `restart boundary not reached (agent skipped gate): completion turn … ended succeeded without the reference gate reporting waiting`. Claude skipped the wait both times, so this cell still does not exercise a restart. ## Risks - Low risk. Test harness and docs only. - The navigation retry could hide a real abort. It is limited to `net::ERR_ABORTED`, 5 attempts and the deadline, and the test still asserts that a new document loaded. - The ACPX Claude completion restart cell still does not exercise a restart when the agent skips the gate. It now reports this cause by name. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, browser test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 12:59:54 -06:00
DottaandClaude Opus 5.5	874fed09a4	test(restart-chaos): enable isolated workspaces for repo tasks (#19536 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos`) restarts the control plane while agents work, then grades the result > - Rule R2 says files an agent wrote before a restart must still be present afterwards > - Repo workloads failed R2 with `pre_fault_files_lost`. The first run worked in the shared project checkout, not in its own worktree > - The seed sets a `git_worktree` project policy, but the server applies that policy only when the instance `enableIsolatedWorkspaces` setting is on. The seed did not turn it on > - This pull request makes the seed turn on the setting (as users do on the Experimental page, and as `tests/runner-e2e` does) and fail if the server does not confirm it > - The benefit is that repo tasks get their own worktree, so one task's files cannot appear in another task's R2 snapshot ## Linked Issues or Issue Description Refs #19477 What happened? In a restart chaos wave, two repo tasks (B1 and B5) ran at the same time. Both ran in the project primary checkout (`strategyType: project_primary`). Task B1's `NOTES-.md` file and its `cli.py` edit appeared in the B5 task's workspace before any fault. After a Retry, the B5 task moved to a fresh attempt directory, and R2 reported the B1 file as lost from B5. Expected behavior* Each repo task gets its own git worktree, as the harness project policy asks. The R2 snapshot of a task contains only that task's files. Steps to reproduce 1. Run `tests/restart-chaos/main.ts --env local --adapters codex --workloads B1,B5 --concurrency 2`. 2. Read `currentExecutionWorkspace` for each task. It shows `project_primary` and the fixture checkout as `cwd`. 3. Read `GET /api/instance/settings/experimental`. `enableIsolatedWorkspaces` is `false`. Root cause `server/src/services/heartbeat.ts` reads `enableIsolatedWorkspaces` and `gateProjectExecutionWorkspacePolicy` (`server/src/services/execution-workspace-policy.ts`) returns `null` when the setting is off. The project policy is then ignored, and the task resolves to the shared primary checkout. `tests/restart-chaos/seed.ts` created the project policy but did not turn on the setting. ## What Changed - `tests/restart-chaos/seed.ts`: before it creates the project, the seed sends `PATCH /api/instance/settings/experimental` with `enableIsolatedWorkspaces: true`. If the response does not confirm the setting, the seed stops with an error. It records a `seed_isolated_workspaces` timeline entry. - `tests/restart-chaos/seed.test.ts`: new unit tests. One test checks that the seed turns on the setting before it creates the `git_worktree` project. One test checks that the seed refuses to continue when the server does not confirm the setting. - `tests/restart-chaos/README.md`: the `seed.ts` row says that the seed turns on the setting, and why. ## Verification - Red before the fix: `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts seed.test.ts` failed 2/2 (`expected -1 to be greater than or equal to 0`). - Green after the fix: the full restart-chaos unit suite passes, 5 files and 42 tests. - Live wave on this branch (`16f5c8c5d7`, local runner, native Codex, macOS, load average about 40): `tests/restart-chaos/main.ts --env local --adapters codex --workloads B1 --tasks 3 --concurrency 1 --faults K1 --seed 21` (wave `w-20261106T181436-s21`). The timeline shows `seed_isolated_workspaces`. Result: 3/3 pass. Each task crossed a SIGKILL restart. R1, R2, R4, R5, R6 and the workload oracle passed for each task, with no lost files. - Each task's pre-fault file list now has a `.git` file (a git worktree, not the primary checkout) and only its own `NOTES-.md` file. In the failing wave, the B5 task's list had the B1 task's `NOTES-.md` file. - Not exercised: no task needed a Retry in this wave, so the Retry-to-fresh-attempt path did not run. - Ledger preflight: install and runner builds passed. The provider-free restart suite had 2 failures in `runnerd-codex-transport-warm-attachment.test.ts` at load average 44; a rerun of that file passed 18/18. ## Risks - Low risk. The change is in the test harness only. It changes no product code. - The harness instance is temporary, so the experimental setting does not reach a real instance. - Repo tasks now run in git worktrees. Earlier waves ran in the shared checkout, so R2 results for repo workloads from earlier waves are not comparable with new waves. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). Standard context window; reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 12:33:13 -06:00
DottaandClaude Opus 5.5	871a9b3180	test(restart-chaos): add Phase 1 control-plane restart chaos harness (#19477 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane owns task execution, recovery and Retry for every agent run > - Server restarts were the largest remaining source of stalled tasks in the staging soak campaign > - The soak could only observe restarts that staging happened to perform, so a failure could not be repeated > - This pull request adds a script harness that causes the restarts on purpose and grades each task on fixed rules > - The benefit is a repeatable restart failure with a replay command and an evidence bundle for each fixer ## Linked Issues or Issue Description What existing behavior does this improve? The restart test coverage. Today the runner-e2e restart cells test one fixed restart point per cell. They cannot run many tasks at once, cannot choose a random fault time, and cannot replay a failure. Subsystem affected Test tooling (`tests/restart-chaos/`). It reuses the runner-e2e process code in `tests/runner-e2e/`. Product code does not change. Current behavior Restart robustness is checked through a small set of fixed cells. A random restart during a real task has no defined pass rule. Proposed behavior A harness starts an isolated control plane and seeds a "Chaos Lab" company. It launches seeded workloads, injects K1, K2 or K4 faults at seeded times, and grades each task on rules R1 to R6 plus a deterministic task oracle. `--replay <wave>/<task>` repeats one task. Reason and benefit Fixers get a replay command and an evidence bundle for each failure signature. The first smoke wave already found three product signatures (dead Retry after a restart, a false over-budget stop after restarts, and an OpenCode turn killed mid-turn that ends as a provider failure). Breaking changes None. The change adds new test files, one entry in the test-project typecheck list and a gitignore entry. ## What Changed - Add `tests/restart-chaos/`: - `supervisor.ts` owns one isolated server. It reuses `replaceController`, the process-tree stopper, the environment isolation helpers, the config seed, the port reservation, the wrapper-owned Postgres and `server-entry.ts` from `tests/runner-e2e/`. It does not copy them. Faults: K1 (SIGKILL, immediate restart), K2 (SIGTERM, restart) and K4 (SIGKILL, 2 to 5 minute hold). Postgres and the controller runtime cache stay across a fault. - `seed.ts` creates the company, encrypted provider secrets, native Codex, ACPX Claude and OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project. The project is a git repository with a Python CLI, one failing test and a four-minute `bench.sh`. - `scheduler.ts` makes a seeded plan of workload, adapter, fault, fire time, hold and nonce for each task. It saves the plan so that `--replay` can run one task again. - `observer.ts` polls each issue and its runs. When a task stops and offers Retry, it sends `POST /api/issues/:id/retry` one time, with the same body as `useTaskRetry.ts`. - `grader.ts` grades R1 to R6 as pure functions. `workloads.ts` defines workloads A1, B1, B5 and C1 and their oracles. - `evidence.ts` and `ledger.ts` write an evidence bundle for each failure, a JSONL ledger and `SUMMARY.md`. All results go to the gitignored `tests/restart-chaos/results/`. Known provider key values are redacted. - Unit tests: `scheduler.test.ts` (determinism, replay and CLI parsing) and `grader.test.ts` (R1 to R6 and the oracles on fixture timelines). These tests need no credentials. - Add `tests/restart-chaos` to the test-project typecheck list (`scripts/typecheck-test-projects.mjs`), with a `tsconfig.layers.json` like `tests/runner-e2e`. The harness runs through direct script invocations, not `package.json` scripts. - After the first Greptile review: - A failed fault or an unexpected server exit now gets one bounded recovery. If the recovery fails, the wave fails and does not hang. - Incomplete cleanup is now a harness error, and the harness keeps the instance. - After the C1 follow-up comment, the observer needs a fresh rest window before it grades the task. - `--replay` now fires the recorded faults that crossed the task, at the same offsets, including the faults that other tasks requested. - A failed workspace-operation read is now a sweep error. It cannot give a clean rest. - R2 now checks that workspace files from before the first fault still exist at rest. - B5 now needs the completion record that `bench.sh` writes for each task. - After the second round of reviews: - An idle `todo` or `in_progress` task with no live run and no named wait is no longer treated as resting. The observer keeps watching it until the task resumes or passes the R5 stall limit. - A failed injected fault now fails the wave with exit 2. - T-oracle commands run agent-editable code. They now run under a macOS Seatbelt sandbox with a scratch `HOME` and no network, and they cannot read the operator's home directory. On other platforms these commands are refused unless the operator explicitly allows them. - R4 now counts every active run. A wake comment may be served again only after a run that did not succeed. - Evidence redaction now covers every environment variable name that ends in `KEY`, `TOKEN`, `SECRET` or `PASSWORD`. - If the pre-fault snapshot fails, only R2 for that task is left unchecked. The wave continues. - B5 also needs bench progress that the harness observed while it polled. - Add `doc/plans/2026-11-06-control-plane-restart-chaos.md` and its campaign ledger. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 tests pass. The tests include observer tests with a fake API, a real `bench.sh` run, and a sandbox test that cannot read a canary secret. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - `server/scripts/typecheck-layers.test.mjs`: 45 tests pass, including "covers every project under tests/". - A supervisor check with no credentials passed. K1 was ready again in about 6 seconds. K2 with a 3-second hold was ready again in about 9 seconds. Company data stayed readable. - One paid smoke wave ran: `--env local --adapters codex,claude,opencode --workloads A1,B1,C1 --faults K1 --tasks 3 --concurrency 2 --seed 7`. The harness worked from start to end, and all three tasks crossed a restart. The 0/3 result comes from product failures, which the campaign ledger records. A scan found no provider key in the results. - To run a wave: ```sh pnpm --filter @paperclipai/paperclip-runner build:runner-binaries set -a; source ~/.secrets; set +a node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude,opencode --concurrency 4 --tasks 8 --faults K1,K2,K4 --seed 42 node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay <wave>/<task> ``` ## Risks - Low product risk. The change adds test tooling only and does not change product code. - A wave spends provider money. The company budget hard cap (`--budget-cents`, default 3000) is the only spend limit. The plan's per-wave and per-night refusal is not implemented yet. - Known gaps: - `--env daytona` is reserved. It fails with a TODO in `seed.ts`. The Daytona sandbox sweep for R6 is not implemented yet. - The project requests isolated git-worktree workspaces. A B1 task still ran in the shared fixture repository. Concurrent B1 and B5 tasks can share files. The oracles use marker files for each task. - Only K1, K2 and K4 at seeded random times are available. Boundary triggers (B1 to B16), other faults and the UI probe come in Phase 2. - The paid smoke did not exercise B5 or K4. - On macOS, B1 oracle commands need `python3` outside the operator's home directory, for example `/usr/bin` or Homebrew. On Linux they are refused until a sandbox is added. - The B5 proof is best-effort against a deliberately adversarial agent, because that agent can write to the workspace. - Startup phase timings come from polling `/api/ready`. The server records its own timings only as OpenTelemetry spans. - `pnpm verify` and the CI Vitest jobs do not select these unit tests. The root Vitest projects cover workspace packages only. CI typechecks the files, but the tests run only through the vitest command above. - After the smoke, rest detection and the review fixes changed. A recovery action that the board owns now counts as a rest, and R6 reports it. A paid wave did not run again after these changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code as an agent with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 10:45:52 -06:00