Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-10 12:37:55 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,734 Commits 1,705 Branches 2,405 Tags
c66b4a55efdc5675b5a5146197078b94de6407da
Commit Graph
4 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 8479929f62 test(restart-chaos): add an RC-06 mask, wave spend and replay masks (#19559) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) restarts the
control plane during real agent tasks and grades each task on rules
R1–R6
> - One known product bug (RC-06) leaves a completed run "awaiting
accounting". The company hard stop then refuses all new work, so one
stuck row fails every later task in the wave
> - That masks every other restart bug in the wave. Waves also had no
record of what they spent
> - This pull request adds an opt-in RC-06 mask, records wave spend, and
lets a replay keep or add the mask
> - The benefit is that chaos waves can find new restart bugs while
RC-06 is open, and stay inside a known spend

## Linked Issues or Issue Description

Refs #19477 (the Phase 1 harness this extends).

**What existing behavior does this improve?**
Restart chaos waves. One stuck accounting row (RC-06) hard-stops the
isolated Chaos Lab company. All later tasks then rest blocked "over
budget", and a Retry is refused with "Company cannot start work while
completed runs await accounting". Two waves went 2/6 and 0/6 for this
one reason.

**Subsystem affected**
Test harness only: `tests/restart-chaos/` (config, seed, scheduler,
ledger, supervisor, main, README).

**Current behavior**
- No way to see past RC-06 without disabling the budget.
- Wave spend is not recorded.
- `--replay` rejected any mask flag, and it was not documented that the
saved plan carries the mask.

**Proposed behavior**
- `--rc06-masked` keeps the budget amount and turns off only the hard
stop on the isolated company. Masked tasks are labeled in the ledger and
summary.
- Each wave records its spend.
- A replay keeps the source plan's masks and may add the RC-06 mask.

## What Changed

- `--rc06-masked` (wave mode): after the company is created, the seed
upserts the company budget policy with the same amount and
`hardStopEnabled: false`. It writes `seed_mask_rc06` to the timeline.
The plan stores `options.masks = ["rc06"]`.
- The accounting gate in `budgets.ts` (`observedBlocks`) applies only
when `hardStopEnabled` is true, so the mask lifts it. No product code
changes.
- Ledger lines carry `masks`. `SUMMARY.md` labels the cell and the
latest-wave row `rc06-masked`.
- Wave spend: before the sweep, `main.ts` reads
`/api/companies/:id/costs/summary?period=all` (each wave owns its
company), writes `wave_spend` to the timeline and prints it. If the wave
stops early, cleanup records it once, best effort, before the server
stops.
- Load: #19513 already records `loadAvg1m` on each fault receipt. This
branch first added the same value under another name, and the rebase
removed it. The README now documents `loadAvg1m` and the load-suspect
threshold.
- Replay: `--replay <wave>/<task>` keeps the masks saved in the source
`plan.json`. `--replay … --rc06-masked` adds the RC-06 mask. A replay
never drops a mask.
- README documents the flag, spend, load and the replay rule.
- New unit tests: replay keeps saved masks and the replay flag adds the
mask; the masked seed posts the company policy with the same amount and
`hardStopEnabled: false` and records `seed_mask_rc06`, and an unmasked
seed makes no policy call.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
71/71 pass (after rebase onto master with #19513).
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass.
- Live: 4 masked local waves ran with this code (seeds 3307, 4409, 5501,
6619; Codex, ACPX Claude and OpenCode).
- `seed_mask_rc06` appeared and no task was blocked by the accounting
gate, even though `pendingRunCount` was 1–8 at wave end.
  - `wave_spend` was recorded each time (38¢–127¢ per wave).
  - The 1-minute load average was recorded on every fault (15–58).
- Before the mask, the same harness gave 2/6 and 0/6, every failure
RC-06. With the mask, the waves found other restart failures, all since
fixed on master: RC-14, RC-16, RC-20 and RC-21.

## Risks

- Low risk. Test harness only, and the mask is opt-in.
- A masked wave has no product hard stop. The harness still records
spend for each wave, and the operator must watch it. The mask applies
only to the isolated, temporary Chaos Lab company. It never touches
another instance.
- Masked results do not count as RC-06 coverage. The ledger labels them
so they are not mixed with unmasked cells.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit). Extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 15:27:12 -06:00
DottaandClaude Opus 5.5 1634f8b66b test(restart-chaos): run chaos waves against Daytona sandboxes without Docker (#19544) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) restarts a local
control plane on purpose while agent tasks run, and grades every task on
rules R1 to R6
> - The plan targets agent runs in Daytona sandboxes, but the harness
only supported `--env local`
> - A local controller needs two inputs that match the sandbox image
exactly: the image's Linux `paperclip-runnerd` and its provider pack.
The documented way to get them uses Docker, and Docker is not available
on the operator machine
> - The Daytona org that chaos uses also runs the live Cloud fleet, so
the harness must never change a sandbox it did not create through its
own Paperclip environment
> - This pull request adds `--env daytona`. It extracts both inputs from
the image without Docker, and refuses a controller whose runner source
differs from the image. It seeds a Daytona environment from the
runner-e2e fixtures and adds a read-only Daytona R6 sweep
> - The benefit is that chaos waves can now cross real Daytona sandboxes
with a local, killable control plane

## Linked Issues or Issue Description

**Feature description**
Run restart chaos waves with a local control plane and agent runs in
Daytona sandboxes (plan
`doc/plans/2026-11-06-control-plane-restart-chaos.md`, Phase 1b).

**Problem it solves**
`--env daytona` threw "not wired yet". The controller-side runnerd and
provider pack had to come from `docker cp`, which needs a Docker daemon.

**Proposed solution**
Use `skopeo copy` (no daemon) to fetch the linux/amd64 image into a
directory, walk the OCI layers from top to bottom, and extract only the
two paths. Before the wave starts, compare the image's content-ID label
with this checkout.

Refs #19477

## What Changed

- `tests/restart-chaos/daytona-image.ts`: a streaming tar reader and an
OCI layer walk (top to bottom, `.wh.<name>` and `.wh..wh..opq`
whiteouts, hard links, symlinks written last, no `..` paths). It
extracts `/usr/local/bin/paperclip-runnerd` and
`/opt/paperclip-runner/provider-pack/`. It checks the layer blob
digests, the ELF header (x86-64), the provider-pack manifest schema, and
its `runnerSourceRevision` against the image revision label. It records
digests in `receipt.json`, caches the result by image digest under
`results/daytona-images/`, and deletes the layer blobs. It also runs
standalone.
- Controller/image check: `assertControllerMatchesImage` compares the
image's `io.paperclip.runner.content-id` label with
`computeDaytonaImageContentId` for this checkout. On a mismatch it fails
and prints both IDs and the image's source revision.
- `seed.ts`: for `--env daytona`, it installs the bundled Daytona plugin
(`installFixtureDaytonaProvider`) and waits for provider readiness. It
creates the `daytona` environment fixture from `catalog.ts` with a
`DAYTONA_API_KEY` company secret and runs the native Codex
platform-qualification probe. Then it assigns the agents and the project
to that environment. `--daytona-cpu` sets the sandbox vCPU (default 4, 4
GiB, 10 GiB).
- `daytona-sweep.ts`: a read-only R6 for Daytona. The Daytona client is
a frozen wrapper that exposes only a label-filtered `list`, and no stop,
archive or delete exists. The sweep keeps only rows whose
`paperclip-company-id` and `paperclip-environment-id` both equal the
wave's own UUIDs, and it re-checks every row. It reconciles sandboxes
with the environment's leases:
  - A sandbox archived after its lease was released is expected.
- An orphaned sandbox or an `active`/`pending_cleanup` lease fails R6
for the owning task, or the wave.
- `assertHarnessMayMutate` refuses `paperclip-create-*` names and
`fleet-sbx-*` snapshots.
- Cleanup only deletes the wave's environment through Paperclip and
reports what is still running.
- `config.ts`: `--daytona-image` (or `PAPERCLIP_E2E_DAYTONA_IMAGE`)
needs an immutable digest. `--concurrency × --daytona-cpu` must stay
within `RESTART_CHAOS_DAYTONA_VCPU_BUDGET` (default 40). B5 is
local-only, because its oracle reads the live workspace.
- `main.ts`: Daytona preflight (credentials, image extraction,
content-ID check, Daytona plugin build as CI does). It passes
`PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` and
`PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` to every server start and
writes `sandbox-sweep.json`. Cleanup errors never skip stopping the
controller.
- README: a Daytona section (image matching, image-only CI dispatch,
vCPU budget, fleet safety, R6).
- Unit tests: synthetic tar layers for the tar reader, whiteouts, opaque
directories, overrides, hard links, symlinks and parent replacement.
Further tests cover the content-ID check, sweep reconciliation, exact
wave selection (unlabelled rows and other labels are never selected),
the read-only client, the fleet guard, and the CLI options.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
62/62 pass, credential-free.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- Real extraction without Docker, image `…@sha256:b0c71f4d…`: 42 layers,
about 8 minutes, a 3 GB pack.
- The content-ID check refused the current checkout. The image is
`b8b16e15…` from `7e6d882558`; the checkout is `aec45589…`.
- A checkout of `7e6d882558` computes `b8b16e15…`, so the check compares
like with like.
- Current image:
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:a84579522192de0393ae24d09e67c4e752df4c701bd315ff54ad9bbca42e2dc0`
(source `5d25117ffa`, content `4e9a221f…`). It matches this branch, and
the receipt shows the content ID match.
- Paid waves, `--env daytona --adapters codex,claude --tasks 3
--concurrency 2 --faults K1 --seed 11`:
- Wave 1 (10 vCPU org, 4 vCPU sandboxes): 0/3. Every create returned
HTTP 400 "Total CPU limit exceeded. Maximum allowed: 10", reproduced
directly with the SDK. This led to the vCPU budget, and the
archived-sandbox fix in R6.
- Wave 2 (same org, 2 vCPU sandboxes): **3/3 PASS**, all
restart-crossing. Codex completed through one working Retry; both Claude
tasks continued automatically. R6: 4 sandboxes archived after release, 0
orphans, 0 held leases.
- Wave 3 (company org, 4 vCPU sandboxes): 0/3. Two product failures were
found and reported to the campaign ledger: an ACPX Claude recovery
blocked on `ambiguous_remote_recovery_evidence` with no Retry, and a
false budget hard stop at 10.75¢ of 3000¢ that paused the company. The
harness changed no sandbox. R6 reported 2 held leases, and the
environment delete returned 409 `pending_sandbox_cleanup` because of
them.
- Fleet-org key check: one sandbox with a unique non-`paperclip-create`
name and label was created, then deleted by the exact ID that create
returned.

## Risks

- This change touches the test harness only, with no product code.
- The extracted provider pack is about 3 GB per image in the gitignored
results directory. Delete old `results/daytona-images/<digest>`
directories by hand.
- The harness never changes Daytona sandboxes. If Paperclip's
environment cleanup is blocked, sandboxes stay until their 60-minute ttl
or a person acts. The harness reports them in `sandbox-sweep.json` and
fails the wave.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, GitHub CLI). Default context window,
extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 14:26:45 -06:00
DottaandClaude Opus 5.5 e8f07b2a39 test(restart-chaos): fire faults relative to user actions and run state, add K6 crash loop (#19513) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane must keep tasks moving when the server restarts
> - The restart chaos harness (#19477) kills the server only at a random
time after a task starts
> - Many restart bugs depend on what the user or the run was doing at
that moment: a comment just sent, a Retry just clicked, a run still
queued, a long tool call, or a second crash during startup recovery
> - The harness cannot aim at these moments yet, so these cases occur
only by chance
> - This pull request adds fault triggers that fire relative to a user
action or a run state, and the K6 crash loop
> - The benefit is that chaos waves can target these boundaries on
purpose and replay them

## Linked Issues or Issue Description

Refs #19477

**What is the problem?**
The restart chaos harness fires each fault at a seeded random offset
from task launch. It cannot fire a fault right after the harness posts a
comment to an active run, right after a Retry click, while a run is
queued, during a long tool call, or a second time while the server is in
the `recovering` startup phase.

**Why is it a problem?**
These are the boundaries where restart bugs hide (plan faults K6 and K8,
boundaries B1, B6 and B8). Random timing seldom hits them, and a replay
cannot repeat them.

**What is the proposed solution?**
Add a `--triggers` option. Each task samples a trigger with its cell.
The observer arms the fault when the trigger condition holds, then fires
it after a seeded delay. Add the K6 crash-loop fault to the supervisor.

## What Changed

- `types.ts`, `config.ts`: add `TRIGGER_IDS` (`time`, `comment`,
`retry`, `queued`, `tool`), the `--triggers` option, and the `K6` fault
kind.
- `scheduler.ts`: sample a trigger per task. `comment` pairs only with
C1 and `tool` only with B5. Each trigger has a seeded delay range.
`retry` adds an extra K1 after the planned fault. A replay re-arms the
task's own triggered faults.
- `observer.ts`: arm each triggered fault from the observed state: the
follow-up reached a running run, an accepted Retry, the first run still
queued, or the first `bench.sh` progress. A trigger that can no longer
hold is recorded as `triggerMissed`.
- `supervisor.ts`: K6 kills the replacement again as soon as
`/api/ready` reports `recovering` (or `ready`, if recovery was too fast
to see), then once more just after `ready`. The receipt records the
phase that each kill hit.
- `main.ts`, `ledger.ts`: pass the trigger to the fault receipt, the
timeline and the ledger. The summary shows the trigger in each cell.
- `triggers.test.ts`: unit tests for planning, firing, misses, replay
and the crash-loop phase rule.
- `README.md`: document the triggers and K6.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`
(52 tests pass).
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- Live: `node cli/node_modules/tsx/dist/cli.mjs
tests/restart-chaos/main.ts --env local --adapters codex,claude,opencode
--faults K1,K6 --triggers comment,retry,queued,tool --concurrency 3
--tasks 6` runs waves with the new triggers.

## Risks

- Low risk. The change is in the test harness only. The default trigger
is `time`, so existing waves and replays keep their behavior.
- A trigger can miss its window (for example, a run starts before the
first poll). The ledger records the miss, so the task is not counted as
a crossing of that boundary.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 13:49:52 -06:00
DottaandClaude Opus 5.5 871a9b3180 test(restart-chaos): add Phase 1 control-plane restart chaos harness (#19477) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane owns task execution, recovery and Retry for every
agent run
> - Server restarts were the largest remaining source of stalled tasks
in the staging soak campaign
> - The soak could only observe restarts that staging happened to
perform, so a failure could not be repeated
> - This pull request adds a script harness that causes the restarts on
purpose and grades each task on fixed rules
> - The benefit is a repeatable restart failure with a replay command
and an evidence bundle for each fixer

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The restart test coverage. Today the runner-e2e restart cells test one
fixed restart point per cell. They cannot run many tasks at once, cannot
choose a random fault time, and cannot replay a failure.

**Subsystem affected**
Test tooling (`tests/restart-chaos/`). It reuses the runner-e2e process
code in `tests/runner-e2e/`. Product code does not change.

**Current behavior**
Restart robustness is checked through a small set of fixed cells. A
random restart during a real task has no defined pass rule.

**Proposed behavior**
A harness starts an isolated control plane and seeds a "Chaos Lab"
company. It launches seeded workloads, injects K1, K2 or K4 faults at
seeded times, and grades each task on rules R1 to R6 plus a
deterministic task oracle. `--replay <wave>/<task>` repeats one task.

**Reason and benefit**
Fixers get a replay command and an evidence bundle for each failure
signature. The first smoke wave already found three product signatures
(dead Retry after a restart, a false over-budget stop after restarts,
and an OpenCode turn killed mid-turn that ends as a provider failure).

**Breaking changes**
None. The change adds new test files, one entry in the test-project
typecheck list and a gitignore entry.

## What Changed

- Add `tests/restart-chaos/`:
- `supervisor.ts` owns one isolated server. It reuses
`replaceController`, the process-tree stopper, the environment isolation
helpers, the config seed, the port reservation, the wrapper-owned
Postgres and `server-entry.ts` from `tests/runner-e2e/`. It does not
copy them. Faults: K1 (SIGKILL, immediate restart), K2 (SIGTERM,
restart) and K4 (SIGKILL, 2 to 5 minute hold). Postgres and the
controller runtime cache stay across a fault.
- `seed.ts` creates the company, encrypted provider secrets, native
Codex, ACPX Claude and OpenCode agents from the runner-e2e catalog, and
the `chaos-sandbox` project. The project is a git repository with a
Python CLI, one failing test and a four-minute `bench.sh`.
- `scheduler.ts` makes a seeded plan of workload, adapter, fault, fire
time, hold and nonce for each task. It saves the plan so that `--replay`
can run one task again.
- `observer.ts` polls each issue and its runs. When a task stops and
offers Retry, it sends `POST /api/issues/:id/retry` one time, with the
same body as `useTaskRetry.ts`.
- `grader.ts` grades R1 to R6 as pure functions. `workloads.ts` defines
workloads A1, B1, B5 and C1 and their oracles.
- `evidence.ts` and `ledger.ts` write an evidence bundle for each
failure, a JSONL ledger and `SUMMARY.md`. All results go to the
gitignored `tests/restart-chaos/results/`. Known provider key values are
redacted.
- Unit tests: `scheduler.test.ts` (determinism, replay and CLI parsing)
and `grader.test.ts` (R1 to R6 and the oracles on fixture timelines).
These tests need no credentials.
- Add `tests/restart-chaos` to the test-project typecheck list
(`scripts/typecheck-test-projects.mjs`), with a `tsconfig.layers.json`
like `tests/runner-e2e`. The harness runs through direct script
invocations, not `package.json` scripts.
- After the first Greptile review:
- A failed fault or an unexpected server exit now gets one bounded
recovery. If the recovery fails, the wave fails and does not hang.
- Incomplete cleanup is now a harness error, and the harness keeps the
instance.
- After the C1 follow-up comment, the observer needs a fresh rest window
before it grades the task.
- `--replay` now fires the recorded faults that crossed the task, at the
same offsets, including the faults that other tasks requested.
- A failed workspace-operation read is now a sweep error. It cannot give
a clean rest.
- R2 now checks that workspace files from before the first fault still
exist at rest.
- B5 now needs the completion record that `bench.sh` writes for each
task.
- After the second round of reviews:
- An idle `todo` or `in_progress` task with no live run and no named
wait is no longer treated as resting. The observer keeps watching it
until the task resumes or passes the R5 stall limit.
  - A failed injected fault now fails the wave with exit 2.
- T-oracle commands run agent-editable code. They now run under a macOS
Seatbelt sandbox with a scratch `HOME` and no network, and they cannot
read the operator's home directory. On other platforms these commands
are refused unless the operator explicitly allows them.
- R4 now counts every active run. A wake comment may be served again
only after a run that did not succeed.
- Evidence redaction now covers every environment variable name that
ends in `KEY`, `TOKEN`, `SECRET` or `PASSWORD`.
- If the pre-fault snapshot fails, only R2 for that task is left
unchecked. The wave continues.
- B5 also needs bench progress that the harness observed while it
polled.
- Add `doc/plans/2026-11-06-control-plane-restart-chaos.md` and its
campaign ledger.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
38 tests pass. The tests include observer tests with a fake API, a real
`bench.sh` run, and a sandbox test that cannot read a canary secret.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- `server/scripts/typecheck-layers.test.mjs`: 45 tests pass, including
"covers every project under tests/".
- A supervisor check with no credentials passed. K1 was ready again in
about 6 seconds. K2 with a 3-second hold was ready again in about 9
seconds. Company data stayed readable.
- One paid smoke wave ran: `--env local --adapters codex,claude,opencode
--workloads A1,B1,C1 --faults K1 --tasks 3 --concurrency 2 --seed 7`.
The harness worked from start to end, and all three tasks crossed a
restart. The 0/3 result comes from product failures, which the campaign
ledger records. A scan found no provider key in the results.
- To run a wave:

  ```sh
  pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
  set -a; source ~/.secrets; set +a
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env
local --adapters codex,claude,opencode --concurrency 4 --tasks 8
--faults K1,K2,K4 --seed 42
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--replay <wave>/<task>
  ```

## Risks

- Low product risk. The change adds test tooling only and does not
change product code.
- A wave spends provider money. The company budget hard cap
(`--budget-cents`, default 3000) is the only spend limit. The plan's
per-wave and per-night refusal is not implemented yet.
- Known gaps:
- `--env daytona` is reserved. It fails with a TODO in `seed.ts`. The
Daytona sandbox sweep for R6 is not implemented yet.
- The project requests isolated git-worktree workspaces. A B1 task still
ran in the shared fixture repository. Concurrent B1 and B5 tasks can
share files. The oracles use marker files for each task.
- Only K1, K2 and K4 at seeded random times are available. Boundary
triggers (B1 to B16), other faults and the UI probe come in Phase 2.
  - The paid smoke did not exercise B5 or K4.
- On macOS, B1 oracle commands need `python3` outside the operator's
home directory, for example `/usr/bin` or Homebrew. On Linux they are
refused until a sandbox is added.
- The B5 proof is best-effort against a deliberately adversarial agent,
because that agent can write to the workspace.
- Startup phase timings come from polling `/api/ready`. The server
records its own timings only as OpenTelemetry spans.
- `pnpm verify` and the CI Vitest jobs do not select these unit tests.
The root Vitest projects cover workspace packages only. CI typechecks
the files, but the tests run only through the vitest command above.
- After the smoke, rest detection and the review fixes changed. A
recovery action that the board owns now counts as a rest, and R6 reports
it. A paid wave did not run again after these changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code as an agent with tool use (shell, file edits) and extended
thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 10:45:52 -06:00