test(restart-chaos): add config fault waves (CF1-CF3) and the CF4 design (#20446)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) starts an
isolated control plane, runs real agent tasks and restarts the server
during them
> - Every wave starts from a new instance with valid default settings
and one valid provider
> - On 2026-11-09 production failed on state that a new instance never
has: a setup command that named a renamed script (#20269), and an
account default that did not exist (#20361)
> - This pull request adds a "config fault" family (CF1–CF3): the
harness breaks one saved reference on a task's own agent or project,
restarts the server while the task is parked, repairs the setting and
grades the result
> - The benefit is that the harness now catches "stale saved setting"
failures, including a run that retries forever or a task that stays
Queued with no explanation

## Linked Issues or Issue Description

Refs #20269
Refs #20361

**What existing behavior does this improve?**
The restart chaos harness only tests new instances with valid settings.
It cannot see failures that come from saved settings that point at a
missing object.

**Subsystem affected**
`tests/restart-chaos/` (test harness only). No product code changes.

**Current behavior**
No wave breaks a saved setting. The CF2 failure (#20361) and the CF1
failure (#20269) were found in production, not by the harness.

**Proposed behavior**
`config-wave.ts` runs CF1–CF3. The oracle is not "the task completes". A
task passes when all of these are true:

- it completes, or it rests on a message that names the broken setting
on the task page within a bound;
- no run retries unchanged configuration forever;
- no task stays Queued with no owner;
- the rest survives a K1 or K2 restart;
- after repair and the offered action (connection card, Retry, or agent
setup Retry), the task completes.

## What Changed

- `tests/restart-chaos/config-faults.ts` (new, pure): the catalog of
broken references, the seeded plan, `parkState`, the task-level message
check and `gradeConfigTask`. CF1 is a missing setup script. CF2 is a
missing personal OpenRouter default. CF3 is a seeded walk over a deleted
secret, a missing workspace path, a missing base ref, an unknown model,
plus CF1 and CF2. `CONFIG_REFERENCES_NOT_BROKEN` lists the settings CF3
cannot break, and why.
- `tests/restart-chaos/config-wave.ts` (new): the wave runner. Each task
gets its own agent, credential secret and project. The runner breaks the
setting through board routes, launches an A1 brief task, polls, fires K1
or K2 while the task is parked, repairs the setting, uses the offered
action and grades the task. It reuses the existing supervisor, seed and
A1 oracle. Waves are limited to 4 tasks and a concurrency of 2. The
instance is deleted at the end.
- `tests/restart-chaos/config-faults.test.ts` (new): unit tests for the
plan, the arguments, `parkState` and the grader. These include the CF2
bug signature and the CF1 "message only on the run page" signature.
- `tests/restart-chaos/README.md`: a new "Config faults" section and a
row in the pieces table.
- `doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md`: the CF4
design (an instance aged by an older release, built on the existing K14
`--upgrade-from` machinery). CF4 is not built in this pull request.
- There are no edits to `main.ts`, `types.ts`, `workloads.ts`,
`invariants.ts` or `grader.ts`. Other agents work on those files in
parallel.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
24 files, 291 tests pass (before the last unit test was added). The
config-fault suite has 13 tests, and all pass.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Live waves (local, isolated instance, one wave at a time):
  - **CF2 red before #20361** (base `8004ad069e`, before #20361 merged):
- Waves `w-20261110T000602-s201-cf` and `w-20261110T004818-s202-cf` fail
with
`CF2/ai_default:bounded_rest+actionable_message+lifecycle_retrying_unchanged_config+queued_without_live_owner`.
- The user sees `Queued`, a `resume` user action, and "Execution
configuration check failed. Paperclip could not complete this step. It
will retry automatically." There is no card.
- A debug run with logging added to the lifecycle failure log (not
committed) showed that the underlying error is
`ai_connection_default_missing`. Verification retried it every 60 s,
more than 20 times.
- After the harness connects an account that is installed for the agent,
the task completes 35 s later on its own.
- The CF2 run after #20361, and the CF1 and CF3 waves on current master,
are added below as they finish.

## Risks

- Low risk. The change is test harness code and docs only.
- A wave costs provider tokens only when the repaired task runs (one A1
brief).
- The CF2 repair creates a real OpenRouter account in the isolated
instance. The server validates the key live.
- The oracle reads the task page through the API (interruption,
execution status, cards, agent setup status and notices). It does not
render the board UI. A message that appears only in the run's own error
text is graded `message_only_in_run_detail`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, subagents). Standard context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>

doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md

@@ -0,0 +1,86 @@
+# Restart chaos: config and aged-state faults (CF1–CF4)
+
+Status: CF1–CF3 are implemented in `tests/restart-chaos/config-faults.ts` and `config-wave.ts`. CF4 is designed here but not built.
+
+## Why
+
+Every chaos wave starts from a new instance that has default, valid settings. On
+2026-11-09, production failures came from state that a new instance never has:
+
+- **#20269.** A project's explicit setup command named `scripts/provision-worktree.sh`,
+  a script that had been renamed. Workspace preparation failed before the provider ran.
+- **#20361.** An OpenRouter binding selected the responsible user's personal account
+  default, but that default did not exist. Verification retried forever, and the task
+  stayed Queued.
+
+## Oracle (CF1–CF4)
+
+"The task completes" is not the oracle. A task passes when all of these hold:
+
+- **Bounded rest.** Within a bound, the task either completes or rests where the task
+  page names the broken setting.
+- **No silent waiting.** No run retries unchanged configuration forever. Nothing stays
+  Queued without a live owner or a named wait.
+- **Restart safety.** The rest survives a K1 or K2 restart.
+- **Repair.** After the harness repairs the setting and uses the action the task offers
+  (card, Retry, or agent setup Retry), the task completes.
+
+See the README section "Config faults" for the exact checks.
+
+## CF4: aged instance (design)
+
+**Goal.** Start the wave on an instance that an older release created. That release saves
+tasks and settings, including CF1–CF3 breaks and parked tasks. Then switch the same
+database and instance directory to the current build. Grade with the CF oracle.
+
+**Reuse.** K14 already has the switching machinery:
+
+- `ChaosSupervisor.create({ upgradeFrom })` starts the server from a prepared older
+  checkout.
+- `supervisor.fault("K14")` stops that server gracefully and starts the current checkout
+  against the same database, port, secrets and instance home.
+- The prepared checkout is made with `git worktree add <dir> <commit>` and
+  `node scripts/agent-preflight.mjs`, as in the K14 section of the README.
+
+Build CF4 on these. Do not add a second upgrade path.
+
+**Changes to `config-wave.ts`** (about half a day):
+
+1. **New flag.** Add `--upgrade-from <dir>`. Check that the directory is prepared, the
+   same way `main.ts preflight` does.
+2. **Seed and break on the older build.** The older build runs `seedChaosLab`, creates
+   each task's resources, applies the break, and launches the task. Use only routes that
+   both builds have. Where a route differs, use the older build's route.
+3. **Barrier.** Wait until every task has finished its broken phase, so it has rested or
+   reached its bound. The parked state is then saved by the older build.
+4. **One switch.** Fire `K14` once for the wave. This replaces each task's own K1/K2.
+   Re-check the rest on the current build, using the same check as after a restart.
+5. **Repair on the current build.** Run repair and the offered action on the current
+   build, then grade.
+6. **Record both commits.** Write both commits into the summary. The K14 receipt already
+   includes them.
+
+**What it covers that CF1–CF3 do not:**
+
+- Older-release run state is resumed by the current build:
+  - setup cards created by the older build
+  - lifecycle operations whose old retry policy was "retry every 60 s"
+  - interruptions projected by older code
+- Settings written by older validators that newer validators would reject. A
+  `refused_on_write` setting can still exist in an aged database. CF4 is the only way to
+  reach that case without writing directly to the database.
+
+**Commit choice.** Use the commit one or two releases back. Also use the commit just
+before a known fix. For example, the merge base of #20361 makes the old build save a
+CF2 queued-without-card state, which the fixed build must then recover.
+
+**Cost and limits.**
+
+- A prepared older checkout takes about 3 minutes with the shared preflight cache, plus
+  about 2 GB of disk.
+- Waves stay at 4 tasks or fewer and a concurrency of 2 or less.
+- It runs locally only, as K14 does.
+
+**Not designed here.** Months-old data shapes, such as rows from migrations long since
+squashed. A seeded fixture database taken from an older release would cover them.
+Capture one only when an upgrade incident shows the need.