Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-12 14:19:00 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,118 Commits 1,880 Branches 2,440 Tags
42bc238aa5a1208cabecba76773e0500fcd33959
Commit Graph
6 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 0ddc9f1ef1 test(chaos): count an M1 subtask stall only when its recovery needs the board (#20178) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The M1 chaos oracle checks that a parent gets exactly one
`issue_blocker_stalled` wake per stalled subtask (#20147)
> - The product wakes a parent only for a stall that needs the board.
The harness recorded a stall for any blocked subtask, whatever its
latest recovery action
> - In the Daytona M1 wave, a subtask blocked twice on agent- and
system-owned recovery that resolved by itself. The harness counted two
stalls and failed the parent for getting no wake, a false failure
> - This pull request records a stall only when the subtask's latest
recovery action needs the board, matching the product's definition
> - The benefit is that `parent_woken_once_per_stall` fails only for a
wake the product owed

## Linked Issues or Issue Description

Refs #20147, #20098

**What happened?**
In wave `w-20261109T041248-s9501`, t10's subtask 195b5fd7 was blocked
twice. Action 56d036e0 was agent-owned (`native_session_interrupted`)
and was cancelled. Action 53eb91b0 was system-owned
(`native_continuation_requires_reconciliation`) and resolved with
outcome cancelled. Neither needed the board, so the product correctly
sent no stall wake. The harness still failed
`parent_woken_once_per_stall` with 2 stalls and 0 wakes.

**Expected behavior**
A stall counts when the subtask's latest recovery action is:
- open (active or escalated) and board-owned, or escalated; or
- board-owned and resolved with outcome blocked, with the subtask still
blocked.

That is the definition the product's blocker-stall wake uses.

## What Changed

- `observeChildStalls` reads the parent's `stalledDependencies` from
`GET /api/issues/:id`. That is the product's own stalled-dependency view
(`listStalledDependencies`, built with the same `readStalls` the
blocker-stall wake uses).
- Each `relation: "child"` entry is recorded as a stall
(`stalledIssueId`, `recoveryAction.id`). The harness no longer infers
stalls from `/recovery-actions`. That endpoint lists only open actions
and selected holds, and can omit a board action resolved with outcome
blocked.
- Clicking the board Retry on a blocked subtask is unchanged: it still
happens whenever the product offers Retry, and it still counts as a
needed Retry.
- Tests:
  - a stall the product lists is recorded;
- for t10's two non-board recovery states, the product lists nothing, so
no stall is recorded, and the subtask's Retry is still clicked.

## Verification

- `npx vitest run` in `tests/restart-chaos`: 212 passed.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.

## Risks

- The harness checks `stalledDependencies` every 10 s. A stall that
appears and clears inside one interval is not recorded, so its wake goes
unchecked. It is never reported as a false failure.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-08 23:36:40 -06:00
DottaandClaude Opus 5.5 61fa13c6d8 test(chaos): add S1, a board Stop across a restart then a continue comment (#20161) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness drives real user actions across
control-plane restarts: comments, Retry, approvals and budget resumes
> - The board's Stop button on a running task was not exercised. The
cancel route stamps the stop as operator-initiated so that recovery
stands down instead of re-waking the agent
> - A restart right after a Stop is where recovery could wrongly restart
work the operator stopped, or lose the operator's later "continue"
direction
> - This pull request adds S1: stop the running run, fire a fault right
after, then comment to continue
> - The benefit is a graded check that a Stop holds across a restart and
that a later continue comment finishes the task

## Linked Issues or Issue Description

Refs #20071

**What is missing?**
The campaign's scenario list includes mid-run stops. C1 covers a mid-run
comment (a steer), but no workload clicks Stop. Without one, a
regression where recovery restarts operator-stopped work, or drops the
continue comment, would go unseen.

**Expected behavior**
- After the board stops a run and a restart follows, the stopped run
stays cancelled.
- Nothing runs until the operator comments.
- The continue comment then finishes the task.

## What Changed

- S1 workload ("Stop while running, then continue"): a brief task. At
its follow-up time, while a run is running, the observer posts `POST
/api/heartbeat-runs/:id/cancel` (the board's Stop) and records `stop`.
- New `stop` trigger, for S1 only: the planned fault fires 0–2 s after
the stop.
- The continue comment is posted only after the stop has settled: at
least 20 s after the stop, with no run running or queued and no fault in
progress. It asks the agent to continue and add a `CONTINUE-<nonce>`
line.
- The oracle requires:
  - status done;
  - the brief with the chaos and continue markers;
  - `stop_took_effect`: the stopped run ended `cancelled`;
- `nothing_ran_after_stop`: no run was created between the stop and the
continue comment.
- S1 is not a default workload. There is a README section.

## Verification

- `npx vitest run` in `tests/restart-chaos`: 205 passed. New tests
cover:
- the S1 grade (pass, a run restarted after the stop, the stop not
taking effect, no stop);
  - the trigger-to-workload pairing;
- an observer test that clicks Stop once and posts the continue comment
only after the 20 s settle.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- A live S1 wave runs after the current chaos wave. Its result will be
posted here before merge.

## Risks

- If the product moves the stopped run to a status other than
`cancelled`, `stop_took_effect` fails. That check is intentional.
- The continue comment waits for the stop to settle, so a run stuck
`queued` delays it. R5 and the wave deadline still bound the wait.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-08 22:52:48 -06:00
DottaandClaude Opus 5.5 0502cc9fac test(chaos): retry a blocked M1 subtask and check the parent's stall wake (#20147) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness grades the M1 scenario: a parent task
delegates two subtasks across restarts
> - When a subtask's run failed recovery, the harness only clicked Retry
on the parent. The parent's Retry is correctly held behind the blocked
subtask (#20109), so the task waited until its deadline
> - A real board user would also click the subtask's Retry. #20098 also
added a wake for a parent whose subtask stalls, and the grade did not
check it
> - This pull request clicks the board Retry on a blocked subtask
(graded as a needed Retry) and checks that the parent got exactly one
stall wake per stall
> - The benefit is that M1 grades the full user path and proves the
stall wake from the product's own activity record

## Linked Issues or Issue Description

Refs #20098, #20109, #20094

**What happened?**
In wave `w-20261109T003126-s9701`, the M1 parent t05 rested blocked
behind a blocked codex subtask. The harness clicked Retry on the parent.
The product held that Retry ("waiting on a blocked task") as designed.
Nothing retried the subtask, and the task waited 38 minutes until the
wave deadline. The wave also could not show whether the parent got the
new `issue_blocker_stalled` wake.

**Expected behavior**
The harness acts like the board: it clicks Retry on a blocked subtask
that offers one, and it counts that as a needed Retry. The M1 oracle
checks the parent's activity for one `issue.blocker_stalled_wake` per
stalled subtask.

## What Changed

- `observer.ts`:
- For M1, the observer checks the parent's subtasks every 10 s while the
parent is open, independent of the parent's rest window.
- Each blocked subtask is recorded as a stall: the subtask plus its
latest recovery action, from `GET /api/issues/:id/recovery-actions`.
- When a blocked subtask's interruption offers Retry, the harness posts
the board Retry. It clicks once per subtask; a click that was not
accepted can be tried again on a later check.
- The parent's `issue.blocker_stalled_wake` activity entries are
collected as `stallWakes` through `GET /api/issues/:id/activity`.
- `grader.ts`: R3 fails with `retry_needed_on_subtask_<n> (<ids>)` when
a subtask needed a board Retry, unless the parent's own Retry already
failed R3.
- `workloads.ts`: the new M1 check `parent_woken_once_per_stall` is
added when a subtask stalled. Each (subtask, recovery action) stall must
have exactly one matching wake. A stall with an unknown action matches
any wake for that subtask.

## Verification

- `npx vitest run` in `tests/restart-chaos`: 203 passed. New tests
cover:
- an observer test: a blocked subtask gets one Retry click, and the
parent is still watched;
  - the R3 subtask-retry grade;
- the stall-wake check: woken once, not woken, woken twice, two stalls
of one subtask with only one woken, and no stall.
- G1, which also has subtasks, is unaffected: the existing governance
tests pass.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.

## Risks

- The harness now acts on subtasks, which it did not before. It clicks
once per subtask and only when the product offers Retry.
- The stall-wake check reads activity through the board API. If the API
omits these entries for a board user, the check fails visibly instead of
passing.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-08 20:12:03 -06:00
DottaandClaude Opus 5.5 88835e624a test(chaos): fail M1 fast on a blocked subtask and grade interrupted waves (#20094) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness proves that real agent tasks survive
control-plane restarts
> - In a K4/K6 wave with the M1 multi-agent scenario, one subtask
failed. Its parent waited on it, and the harness kept watching for over
50 minutes
> - When the operator stopped the wave, the harness wrote no grade
files, so the 17 finished tasks had no grade
> - Separately, master's harness test for time to first activity failed
after #20077 changed how time is measured from run creation
> - This pull request makes M1 fail fast on a blocked subtask, grades
the finished tasks of an interrupted wave, and aligns the speed grade
with #20077
> - The benefit is shorter waves with a grade for every task, and a
green harness suite on master

## Linked Issues or Issue Description

Refs #20071, #20077, #20088

**What happened?**
Wave `w-20261108T181313-s9701` (local, codex and claude, A1/C1/M1, K4
and K6) ran 71 minutes against a planned 30. Task t05 (M1) delegated
CHA-8 and CHA-9. The CHA-8 run failed after a K4 restart (the defect
that #20088 fixes), and CHA-8 went to `blocked`. The observer kept
waiting for CHA-8 to finish. After SIGINT, the harness threw before
grading, so no task had a grade file. Also, `wave-speed.test.ts` fails
on master: #20077 changed `runMetrics` to measure from the start of
`heartbeat.queue`.

**Steps to reproduce**
1. Run an M1 task whose subtask run fails and leaves the subtask
`blocked`.
2. See the observer wait until the parent deadline.
3. Send SIGINT to the harness and see that no grade files are written.

**Expected behavior**
A blocked subtask ends the wait, and the parent is graded as a failure
with a `task_child_blocked` timeline entry. An interrupted wave grades
every finished task and fails the wave.

## What Changed

- `observer.ts`: `childrenPending` treats a `blocked` subtask as
settled, so the parent rests and is graded. The timeline records
`task_child_blocked` with the subtask ids.
- `observer.ts`: `markUnfinished` closes an open task at interrupt as
unfinished (`timedOut`, last seen status) and records
`task_unfinished_at_interrupt`.
- `main.ts`: an interrupt now sets the harness error and grades the
wave, instead of throwing before the grade. The wave still fails.
- `wave-speed.ts`: `firstActivityMs` uses
`runMetrics().firstActivityAfterRunCreatedMs` from #20077, so there is
one definition of time from run creation.
- `wave-speed.test.ts`: the fixture starts `heartbeat.queue` when the
comment wait ends, as in real runs.

## Verification

- `npx vitest run` in `tests/restart-chaos`: 172 passed. Before this
change, the wave-speed test failed on master.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.

## Risks

- A subtask that is `blocked` only for a short time and then resumes
would end the parent's watch after the 60-second grace. The grade then
shows a failure that a longer wait could avoid. M1 subtasks have no
blockers, so `blocked` means a stuck subtask.
- An interrupted wave grades tasks while its server is still up. If the
server is already down, artifact collection fails, and those tasks are
graded as failures.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-08 14:20:57 -06:00
DottaandClaude Opus 5.5 1634f8b66b test(restart-chaos): run chaos waves against Daytona sandboxes without Docker (#19544) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) restarts a local
control plane on purpose while agent tasks run, and grades every task on
rules R1 to R6
> - The plan targets agent runs in Daytona sandboxes, but the harness
only supported `--env local`
> - A local controller needs two inputs that match the sandbox image
exactly: the image's Linux `paperclip-runnerd` and its provider pack.
The documented way to get them uses Docker, and Docker is not available
on the operator machine
> - The Daytona org that chaos uses also runs the live Cloud fleet, so
the harness must never change a sandbox it did not create through its
own Paperclip environment
> - This pull request adds `--env daytona`. It extracts both inputs from
the image without Docker, and refuses a controller whose runner source
differs from the image. It seeds a Daytona environment from the
runner-e2e fixtures and adds a read-only Daytona R6 sweep
> - The benefit is that chaos waves can now cross real Daytona sandboxes
with a local, killable control plane

## Linked Issues or Issue Description

**Feature description**
Run restart chaos waves with a local control plane and agent runs in
Daytona sandboxes (plan
`doc/plans/2026-11-06-control-plane-restart-chaos.md`, Phase 1b).

**Problem it solves**
`--env daytona` threw "not wired yet". The controller-side runnerd and
provider pack had to come from `docker cp`, which needs a Docker daemon.

**Proposed solution**
Use `skopeo copy` (no daemon) to fetch the linux/amd64 image into a
directory, walk the OCI layers from top to bottom, and extract only the
two paths. Before the wave starts, compare the image's content-ID label
with this checkout.

Refs #19477

## What Changed

- `tests/restart-chaos/daytona-image.ts`: a streaming tar reader and an
OCI layer walk (top to bottom, `.wh.<name>` and `.wh..wh..opq`
whiteouts, hard links, symlinks written last, no `..` paths). It
extracts `/usr/local/bin/paperclip-runnerd` and
`/opt/paperclip-runner/provider-pack/`. It checks the layer blob
digests, the ELF header (x86-64), the provider-pack manifest schema, and
its `runnerSourceRevision` against the image revision label. It records
digests in `receipt.json`, caches the result by image digest under
`results/daytona-images/`, and deletes the layer blobs. It also runs
standalone.
- Controller/image check: `assertControllerMatchesImage` compares the
image's `io.paperclip.runner.content-id` label with
`computeDaytonaImageContentId` for this checkout. On a mismatch it fails
and prints both IDs and the image's source revision.
- `seed.ts`: for `--env daytona`, it installs the bundled Daytona plugin
(`installFixtureDaytonaProvider`) and waits for provider readiness. It
creates the `daytona` environment fixture from `catalog.ts` with a
`DAYTONA_API_KEY` company secret and runs the native Codex
platform-qualification probe. Then it assigns the agents and the project
to that environment. `--daytona-cpu` sets the sandbox vCPU (default 4, 4
GiB, 10 GiB).
- `daytona-sweep.ts`: a read-only R6 for Daytona. The Daytona client is
a frozen wrapper that exposes only a label-filtered `list`, and no stop,
archive or delete exists. The sweep keeps only rows whose
`paperclip-company-id` and `paperclip-environment-id` both equal the
wave's own UUIDs, and it re-checks every row. It reconciles sandboxes
with the environment's leases:
  - A sandbox archived after its lease was released is expected.
- An orphaned sandbox or an `active`/`pending_cleanup` lease fails R6
for the owning task, or the wave.
- `assertHarnessMayMutate` refuses `paperclip-create-*` names and
`fleet-sbx-*` snapshots.
- Cleanup only deletes the wave's environment through Paperclip and
reports what is still running.
- `config.ts`: `--daytona-image` (or `PAPERCLIP_E2E_DAYTONA_IMAGE`)
needs an immutable digest. `--concurrency × --daytona-cpu` must stay
within `RESTART_CHAOS_DAYTONA_VCPU_BUDGET` (default 40). B5 is
local-only, because its oracle reads the live workspace.
- `main.ts`: Daytona preflight (credentials, image extraction,
content-ID check, Daytona plugin build as CI does). It passes
`PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` and
`PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` to every server start and
writes `sandbox-sweep.json`. Cleanup errors never skip stopping the
controller.
- README: a Daytona section (image matching, image-only CI dispatch,
vCPU budget, fleet safety, R6).
- Unit tests: synthetic tar layers for the tar reader, whiteouts, opaque
directories, overrides, hard links, symlinks and parent replacement.
Further tests cover the content-ID check, sweep reconciliation, exact
wave selection (unlabelled rows and other labels are never selected),
the read-only client, the fleet guard, and the CLI options.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
62/62 pass, credential-free.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- Real extraction without Docker, image `…@sha256:b0c71f4d…`: 42 layers,
about 8 minutes, a 3 GB pack.
- The content-ID check refused the current checkout. The image is
`b8b16e15…` from `7e6d882558`; the checkout is `aec45589…`.
- A checkout of `7e6d882558` computes `b8b16e15…`, so the check compares
like with like.
- Current image:
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:a84579522192de0393ae24d09e67c4e752df4c701bd315ff54ad9bbca42e2dc0`
(source `5d25117ffa`, content `4e9a221f…`). It matches this branch, and
the receipt shows the content ID match.
- Paid waves, `--env daytona --adapters codex,claude --tasks 3
--concurrency 2 --faults K1 --seed 11`:
- Wave 1 (10 vCPU org, 4 vCPU sandboxes): 0/3. Every create returned
HTTP 400 "Total CPU limit exceeded. Maximum allowed: 10", reproduced
directly with the SDK. This led to the vCPU budget, and the
archived-sandbox fix in R6.
- Wave 2 (same org, 2 vCPU sandboxes): **3/3 PASS**, all
restart-crossing. Codex completed through one working Retry; both Claude
tasks continued automatically. R6: 4 sandboxes archived after release, 0
orphans, 0 held leases.
- Wave 3 (company org, 4 vCPU sandboxes): 0/3. Two product failures were
found and reported to the campaign ledger: an ACPX Claude recovery
blocked on `ambiguous_remote_recovery_evidence` with no Retry, and a
false budget hard stop at 10.75¢ of 3000¢ that paused the company. The
harness changed no sandbox. R6 reported 2 held leases, and the
environment delete returned 409 `pending_sandbox_cleanup` because of
them.
- Fleet-org key check: one sandbox with a unique non-`paperclip-create`
name and label was created, then deleted by the exact ID that create
returned.

## Risks

- This change touches the test harness only, with no product code.
- The extracted provider pack is about 3 GB per image in the gitignored
results directory. Delete old `results/daytona-images/<digest>`
directories by hand.
- The harness never changes Daytona sandboxes. If Paperclip's
environment cleanup is blocked, sandboxes stay until their 60-minute ttl
or a person acts. The harness reports them in `sandbox-sweep.json` and
fails the wave.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, GitHub CLI). Default context window,
extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 14:26:45 -06:00
DottaandClaude Opus 5.5 871a9b3180 test(restart-chaos): add Phase 1 control-plane restart chaos harness (#19477) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane owns task execution, recovery and Retry for every
agent run
> - Server restarts were the largest remaining source of stalled tasks
in the staging soak campaign
> - The soak could only observe restarts that staging happened to
perform, so a failure could not be repeated
> - This pull request adds a script harness that causes the restarts on
purpose and grades each task on fixed rules
> - The benefit is a repeatable restart failure with a replay command
and an evidence bundle for each fixer

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The restart test coverage. Today the runner-e2e restart cells test one
fixed restart point per cell. They cannot run many tasks at once, cannot
choose a random fault time, and cannot replay a failure.

**Subsystem affected**
Test tooling (`tests/restart-chaos/`). It reuses the runner-e2e process
code in `tests/runner-e2e/`. Product code does not change.

**Current behavior**
Restart robustness is checked through a small set of fixed cells. A
random restart during a real task has no defined pass rule.

**Proposed behavior**
A harness starts an isolated control plane and seeds a "Chaos Lab"
company. It launches seeded workloads, injects K1, K2 or K4 faults at
seeded times, and grades each task on rules R1 to R6 plus a
deterministic task oracle. `--replay <wave>/<task>` repeats one task.

**Reason and benefit**
Fixers get a replay command and an evidence bundle for each failure
signature. The first smoke wave already found three product signatures
(dead Retry after a restart, a false over-budget stop after restarts,
and an OpenCode turn killed mid-turn that ends as a provider failure).

**Breaking changes**
None. The change adds new test files, one entry in the test-project
typecheck list and a gitignore entry.

## What Changed

- Add `tests/restart-chaos/`:
- `supervisor.ts` owns one isolated server. It reuses
`replaceController`, the process-tree stopper, the environment isolation
helpers, the config seed, the port reservation, the wrapper-owned
Postgres and `server-entry.ts` from `tests/runner-e2e/`. It does not
copy them. Faults: K1 (SIGKILL, immediate restart), K2 (SIGTERM,
restart) and K4 (SIGKILL, 2 to 5 minute hold). Postgres and the
controller runtime cache stay across a fault.
- `seed.ts` creates the company, encrypted provider secrets, native
Codex, ACPX Claude and OpenCode agents from the runner-e2e catalog, and
the `chaos-sandbox` project. The project is a git repository with a
Python CLI, one failing test and a four-minute `bench.sh`.
- `scheduler.ts` makes a seeded plan of workload, adapter, fault, fire
time, hold and nonce for each task. It saves the plan so that `--replay`
can run one task again.
- `observer.ts` polls each issue and its runs. When a task stops and
offers Retry, it sends `POST /api/issues/:id/retry` one time, with the
same body as `useTaskRetry.ts`.
- `grader.ts` grades R1 to R6 as pure functions. `workloads.ts` defines
workloads A1, B1, B5 and C1 and their oracles.
- `evidence.ts` and `ledger.ts` write an evidence bundle for each
failure, a JSONL ledger and `SUMMARY.md`. All results go to the
gitignored `tests/restart-chaos/results/`. Known provider key values are
redacted.
- Unit tests: `scheduler.test.ts` (determinism, replay and CLI parsing)
and `grader.test.ts` (R1 to R6 and the oracles on fixture timelines).
These tests need no credentials.
- Add `tests/restart-chaos` to the test-project typecheck list
(`scripts/typecheck-test-projects.mjs`), with a `tsconfig.layers.json`
like `tests/runner-e2e`. The harness runs through direct script
invocations, not `package.json` scripts.
- After the first Greptile review:
- A failed fault or an unexpected server exit now gets one bounded
recovery. If the recovery fails, the wave fails and does not hang.
- Incomplete cleanup is now a harness error, and the harness keeps the
instance.
- After the C1 follow-up comment, the observer needs a fresh rest window
before it grades the task.
- `--replay` now fires the recorded faults that crossed the task, at the
same offsets, including the faults that other tasks requested.
- A failed workspace-operation read is now a sweep error. It cannot give
a clean rest.
- R2 now checks that workspace files from before the first fault still
exist at rest.
- B5 now needs the completion record that `bench.sh` writes for each
task.
- After the second round of reviews:
- An idle `todo` or `in_progress` task with no live run and no named
wait is no longer treated as resting. The observer keeps watching it
until the task resumes or passes the R5 stall limit.
  - A failed injected fault now fails the wave with exit 2.
- T-oracle commands run agent-editable code. They now run under a macOS
Seatbelt sandbox with a scratch `HOME` and no network, and they cannot
read the operator's home directory. On other platforms these commands
are refused unless the operator explicitly allows them.
- R4 now counts every active run. A wake comment may be served again
only after a run that did not succeed.
- Evidence redaction now covers every environment variable name that
ends in `KEY`, `TOKEN`, `SECRET` or `PASSWORD`.
- If the pre-fault snapshot fails, only R2 for that task is left
unchecked. The wave continues.
- B5 also needs bench progress that the harness observed while it
polled.
- Add `doc/plans/2026-11-06-control-plane-restart-chaos.md` and its
campaign ledger.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
38 tests pass. The tests include observer tests with a fake API, a real
`bench.sh` run, and a sandbox test that cannot read a canary secret.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- `server/scripts/typecheck-layers.test.mjs`: 45 tests pass, including
"covers every project under tests/".
- A supervisor check with no credentials passed. K1 was ready again in
about 6 seconds. K2 with a 3-second hold was ready again in about 9
seconds. Company data stayed readable.
- One paid smoke wave ran: `--env local --adapters codex,claude,opencode
--workloads A1,B1,C1 --faults K1 --tasks 3 --concurrency 2 --seed 7`.
The harness worked from start to end, and all three tasks crossed a
restart. The 0/3 result comes from product failures, which the campaign
ledger records. A scan found no provider key in the results.
- To run a wave:

  ```sh
  pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
  set -a; source ~/.secrets; set +a
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env
local --adapters codex,claude,opencode --concurrency 4 --tasks 8
--faults K1,K2,K4 --seed 42
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--replay <wave>/<task>
  ```

## Risks

- Low product risk. The change adds test tooling only and does not
change product code.
- A wave spends provider money. The company budget hard cap
(`--budget-cents`, default 3000) is the only spend limit. The plan's
per-wave and per-night refusal is not implemented yet.
- Known gaps:
- `--env daytona` is reserved. It fails with a TODO in `seed.ts`. The
Daytona sandbox sweep for R6 is not implemented yet.
- The project requests isolated git-worktree workspaces. A B1 task still
ran in the shared fixture repository. Concurrent B1 and B5 tasks can
share files. The oracles use marker files for each task.
- Only K1, K2 and K4 at seeded random times are available. Boundary
triggers (B1 to B16), other faults and the UI probe come in Phase 2.
  - The paid smoke did not exercise B5 or K4.
- On macOS, B1 oracle commands need `python3` outside the operator's
home directory, for example `/usr/bin` or Homebrew. On Linux they are
refused until a sandbox is added.
- The B5 proof is best-effort against a deliberately adversarial agent,
because that agent can write to the workspace.
- Startup phase timings come from polling `/api/ready`. The server
records its own timings only as OpenTelemetry spans.
- `pnpm verify` and the CI Vitest jobs do not select these unit tests.
The root Vitest projects cover workspace packages only. CI typechecks
the files, but the tests run only through the vitest command above.
- After the smoke, rest detection and the review fixes changed. A
recovery action that the board owns now counts as a rest, and R6 reports
it. A paid wave did not run again after these changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code as an agent with tool use (shell, file edits) and extended
thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 10:45:52 -06:00