refactor: split the runner-e2e and restart-chaos READMEs into a runbook and a catalog (#21743)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `tests/runner-e2e` is the paid Product E2E harness and
`tests/restart-chaos` is the restart chaos harness
> - A README is the first file a person or an agent opens in a directory
> - These two READMEs grew to 3,805 and 2,277 lines. Most of each is a
catalog with one section per suite or per fault scenario, and almost
none of it is ever trimmed
> - This pull request splits each README into a runbook and a catalog
file
> - The benefit is a README that can be read in full, and a clear place
for the next suite or scenario note

## Linked Issues or Issue Description

Refs #21670 and #21700 (the same cleanup for `doc/DEVELOPING.md` and
`doc/DATABASE.md`).

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`tests/runner-e2e/README.md` and `tests/restart-chaos/README.md`

**What's wrong?**

In the last 60 days, `tests/runner-e2e/README.md` gained 3,476 lines and
lost 182. `tests/restart-chaos/README.md` gained 2,411 and lost 134. The
setup, evidence, and CI sections sit between suite notes. The first 225
lines of the runner-e2e README are suite notes with no heading.

**Suggested fix**

Keep how to run the harness in the README. Move the per-suite and
per-scenario sections to a catalog file in the same directory.

## What Changed

- `tests/runner-e2e/README.md` is now 767 lines: credentials, local
commands, the Daytona image, evidence and cleanup, billing, the
dashboard, GitHub Actions, and the report preview.
- New `tests/runner-e2e/SUITES.md` (3,055 lines) holds every per-suite
section. The two blocks of suite notes that had no heading now have one.
- `tests/restart-chaos/README.md` is now 592 lines: run, fault timing,
Daytona mode, hosted mode, universal checks, grading notes, pieces,
output, and the crash-point suite.
- New `tests/restart-chaos/SCENARIOS.md` (1,694 lines) holds every
workload and fault scenario section.
- All text is moved without wording changes. Nothing is deleted. The
only edits are three "(below)" references in the restart-chaos README,
which are now links.
- Repointed 22 anchor links and 3 plain links in `doc/evals.md` and
`maps/feature-map/cases.md`.
- Updated seven code comments in `tests/restart-chaos/` that name a
moved section, and two skills that list the runner-e2e docs to read.

## Verification

- The split script asserts that each source line is kept or moved
exactly one time, and checks the first line of each range.
- A link check on the four files and on every file that links into them
found 0 missing files and 0 missing anchors.
- `tests/runner-e2e/daytona-image.test.ts` reads the README for a Docker
build recipe. The recipe is still in the README. I did not run that test
locally.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `pnpm verify` was not run. This worktree has no installed
dependencies.

## Risks

- Low risk. The change is documentation and comments only.
- A link to a moved section from outside this repository lands on the
top of the README.
- The catalog files are still long. This change moves them off the
first-read path. It does not prune them.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool
use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Fable 5.2 authored and GitHub committed 2026-11-15 06:01:06 -06:00
1 parent 066c8244b5
commit fb3b09aaef
15 files changed
+4976 -4946

No files matched your search

+2 -1
View File
@@ -17,7 +17,8 @@ workspace roots such as `~/paperclipai/paperclip`; verify the selected root
contains `tests/runner-e2e` and `packages/paperclip-runner`. Run commands from
that repository root. The copied skill may live outside the checkout.
Read `doc/evals.md`, then the authoritative
`tests/runner-e2e/README.md`, `FIXTURES.md`, `SECURITY.md`, and
`tests/runner-e2e/README.md`, the suite's section in `SUITES.md`,
`FIXTURES.md`, `SECURITY.md`, and
`EVERYDAY-WORKFLOWS.md` for the selected area. Inspect the nearest existing
catalog entry, case, harness flow, matcher, evidence writer, and report test
before changing anything. Keep the user journey on production browser/API
+2 -1
View File
@@ -28,7 +28,8 @@ mock control plane. Authoritative details are in
Choose **Product E2E Evals** for real browser/server/database/runner/provider
workflows, including local and Daytona environments. Read
`tests/runner-e2e/README.md`, then `FIXTURES.md`, `SECURITY.md`, or
`tests/runner-e2e/README.md`, then the suite's section in `SUITES.md`,
`FIXTURES.md`, `SECURITY.md`, or
`EVERYDAY-WORKFLOWS.md` as relevant. Everyday Workflows remain Product E2E
even when imported into Evalbook. “Headless” is a browser mode, not a family.
+24 -24
View File
@@ -26,14 +26,14 @@ Selection fails closed until TestOps-assigned identities are committed; see the
The explicit deployment-targeted [saved agent instructions journey](../tests/runner-e2e/product-journeys/AGENT-SETTINGS.md) checks an owned agent’s ordinary UI save, persisted instruction bytes, one current-run output, and restoration. The task request omits its instruction-only reference. It reuses the existing Product E2E target and report path; local controls do not establish a staging pass.
The explicit-only [verified API artifact transfer](../tests/runner-e2e/README.md#verified-api-artifact-transfer)
The explicit-only [verified API artifact transfer](../tests/runner-e2e/SUITES.md#verified-api-artifact-transfer)
cell checks a native Codex agent on Daytona installing an authorized saved API
response and delivering unchanged bytes. The oracle binds the source capture,
tool completion and independent attachment download, including the complete
388131-byte synthetic description. Registration and oracle calibration are
separate from live provider qualification.
The explicit-only [Kimi Code qualification suite](../tests/runner-e2e/README.md#kimi-code-qualification)
The explicit-only [Kimi Code qualification suite](../tests/runner-e2e/SUITES.md#kimi-code-qualification)
registers twenty candidate cells across local and Daytona. It reuses the
browser, semantic interaction, file/artifact, Stop and continuation oracles with
scoped candidate admission and an encrypted Code credential. Registration does
@@ -77,7 +77,7 @@ existing campaigns. Private Cloud bundles are rejected. The output contains
only index HTML and history/pointer JSON; it does not upload evidence or change
retained campaign bundles.
The explicit-only [Cursor Cloud native cells](../tests/runner-e2e/README.md)
The explicit-only [Cursor Cloud native cells](../tests/runner-e2e/SUITES.md)
cover task/artifact delivery, same-agent follow-up, active-run UI Stop and
semantic Plan revision/approval from a local controller. They require explicit
model/repository/key and a callback-only
@@ -100,7 +100,7 @@ from master. Registration, known gaps and private prerequisites remain distinct
from measured results. This lane does not qualify the current master app or reset
earlier campaign accounting.
The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/README.md)
The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/SUITES.md)
extends the existing native restart oracle to a 300-second controller outage
and another replacement after completion, using the recorded Claude and Codex
models in disposable development fixtures.
@@ -125,7 +125,7 @@ native API mutation receipt. It does not qualify named-tool selection. Its provi
usage and cleanup. Registration and local calibration do not establish a fresh
Cloud pass or inclusion in a recurring selection.
The explicit-only [necessary free-text question journey](../tests/runner-e2e/README.md#necessary-free-text-question)
The explicit-only [necessary free-text question journey](../tests/runner-e2e/SUITES.md#necessary-free-text-question)
checks one essential input, the rendered prompt and context, the real answer,
one bound continuation, and an exact saved confirmation. Its four development
cells use the recorded Codex and Claude models. Registration and calibration do
@@ -149,7 +149,7 @@ It requires a fresh owned agent with no prior work runs and uses three provider
Cloud fleet operations and trusted CI remain private. Local eval gateway qualification
does not qualify Cloud routes, and these selections stay outside `--all`.
The explicit [onboarding approval-history journey](../tests/runner-e2e/README.md#first-task-onboarding)
The explicit [onboarding approval-history journey](../tests/runner-e2e/SUITES.md#first-task-onboarding)
continues an interview through partial plan acceptance, approved hiring, and a
separate execution confirmation. It requires the assigned child's real execution,
saved output, unchanged approved scope, and reload persistence. Deterministic
@@ -158,7 +158,7 @@ require an explicit paid selection and a recorded live result.
The deployment-targeted [saved skill journey](../tests/runner-e2e/product-journeys/saved-skill-use.md) checks one company skill assigned by key, its exact admitted immutable version, a bound native source read/load, and a run-authored saved result from skill-only inputs. It uses the reusable Product journey target and remains unqualified until a real target run passes.
The explicit-only [case handoff workflow](../tests/runner-e2e/README.md#persistent-case-handoffs-explicit-only)
The explicit-only [case handoff workflow](../tests/runner-e2e/SUITES.md#persistent-case-handoffs-explicit-only)
contains arithmetic and receipt smoke cells for a same-task round trip among three native agents using real case/task
APIs, saved documents, handoff receipts, and browser readback. Receipt smoke has
an explicit terminal handoff to `done` on its fourth run, proving case closure
@@ -241,15 +241,15 @@ not an eval category.
The explicit Product E2E `completion-updates` suite compares onboarding and
idle, busy, multiple-task, and restart Agent Chat handoffs on native Claude/Codex. It separates mechanical
completion delivery/result access from semantic review of the retained answer;
see the [probe contract](../tests/runner-e2e/README.md#completion-update-probes-explicit-only).
see the [probe contract](../tests/runner-e2e/SUITES.md#completion-update-probes-explicit-only).
The explicit-only [task-titles suite](../tests/runner-e2e/README.md#automatic-task-titles)
The explicit-only [task-titles suite](../tests/runner-e2e/SUITES.md#automatic-task-titles)
checks that production guidance causes a real native agent to name prompt-only
standard/Ask tasks early, while preserving user-supplied titles. Its oracle
correlates browser creation, native tool receipts, durable titles, audit ownership,
and the reloaded task UI; fixture prompts contain no naming instructions.
The explicit-only [native connection guidance suite](../tests/runner-e2e/README.md#native-connection-guidance-explicit-only)
The explicit-only [native connection guidance suite](../tests/runner-e2e/SUITES.md#native-connection-guidance-explicit-only)
adds neutral decline prompts, same-task run-attributed explanations, and measured
no-use controls across three native local profiles. Its fifteen configured cells
are preparation for future matched instruction comparisons, not a live result.
@@ -311,11 +311,11 @@ real APIs, drives Chromium, and invokes the selected local or Daytona runner.
The explicit-only `agent-chat-hardening` Product E2E suite covers native chat
recovery, hiring, status evidence, and review handoff on local and selected warm
Daytona paths. Its [fixture contract](../tests/runner-e2e/README.md) distinguishes
Daytona paths. Its [fixture contract](../tests/runner-e2e/SUITES.md) distinguishes
startup cancellation from active response cancellation and HTTP send replay
from ambiguous provider action recovery. Select it explicitly; `--all` excludes it.
The explicit-only [production hiring templates suite](../tests/runner-e2e/README.md#production-hiring-templates)
The explicit-only [production hiring templates suite](../tests/runner-e2e/SUITES.md#production-hiring-templates)
adds two local native Codex/Claude cells. It exercises API-created production
CEO defaults, an explicitly requested hiring skill/reference read, a permanent
coder hire, independently computed saved JSON fixtures and worker reuse.
@@ -351,9 +351,9 @@ The explicit-only `grok-qualification` and `grok-subscription-qualification`
Product suites exercise Grok Build with API and company subscription
authentication respectively. Keep their results separate; the subscription
fixture seeds an explicitly supplied login and does not qualify interactive
login. See the [Grok fixture contract](../tests/runner-e2e/README.md#grok-build-qualification).
login. See the [Grok fixture contract](../tests/runner-e2e/SUITES.md#grok-build-qualification).
The explicit [Direct blocker guidance suite](../tests/runner-e2e/README.md#direct-blocker-guidance)
The explicit [Direct blocker guidance suite](../tests/runner-e2e/SUITES.md#direct-blocker-guidance)
checks the legacy coordination skill against human authority, missing hiring
permission, and requester scope decisions through saved browser interactions.
@@ -439,7 +439,7 @@ analytical label.
## Evidence, provenance, and history
Polling latency is an interval rather than an exact first-detection time. The
[Product E2E timing contract](../tests/runner-e2e/README.md#latency-observation-bounds)
[Product E2E timing contract](../tests/runner-e2e/SUITES.md#latency-observation-bounds)
keeps last-absent/first-present bounds, resolution, strict or inclusive targets,
and pass/fail/inconclusive timing verdicts alongside the unchanged workflow grade.
@@ -627,15 +627,15 @@ a newer rendering of the same campaign.
Remaining native chat boundaries are in the explicit-only
`agent-chat-qualification` suite: active task reassignment, user Retry after
verified worker process loss, and multi-turn answers grounded in actual task
records. See the [workflow and qualification limits](../tests/runner-e2e/README.md#remaining-native-agent-chat-qualification).
records. See the [workflow and qualification limits](../tests/runner-e2e/SUITES.md#remaining-native-agent-chat-qualification).
The 26 native `first-task` cells inspect the production wizard's saved native
default on qualified targets, without an API runner switch. The 26 Legacy cells
use an API-configured inactive setup record and resume it in the wizard, as
described in the [first-task setup contract](../tests/runner-e2e/README.md#first-task-onboarding).
described in the [first-task setup contract](../tests/runner-e2e/SUITES.md#first-task-onboarding).
Live results and semantic answer reviews must accompany
any qualification claim; catalog presence alone is not a pass.
The explicit-only [native question/resume qualification](../tests/runner-e2e/README.md#task-continuation)
The explicit-only [native question/resume qualification](../tests/runner-e2e/SUITES.md#task-continuation)
separates a completed two-answer user journey from semantic-tool documentation
qualification. It verifies the exact provider-pause or native response-wake
binding for each answer and preserves prior grades when the definition changes.
@@ -664,11 +664,11 @@ downloaded proof on local native/legacy Codex and native Daytona. See the
The explicit-only Product E2E `api-response-reading` suite verifies retrieval of
large saved API responses on local and Daytona native Codex runs. See the
[Runner E2E guide](../tests/runner-e2e/README.md#bounded-api-response-reading).
[Runner E2E guide](../tests/runner-e2e/SUITES.md#bounded-api-response-reading).
The explicit-only Product E2E `extended-harnesses` suite covers pending Cursor,
Copilot and Pi ACP profiles on local and Daytona. See the
[fixture admission, credentials and budget contract](../tests/runner-e2e/README.md#extended-acp-harnesses-explicit-only).
[fixture admission, credentials and budget contract](../tests/runner-e2e/SUITES.md#extended-acp-harnesses-explicit-only).
The private Runner Evals campaign of the same name provides complementary
semantic protocol cases; catalog membership is not live qualification.
@@ -679,19 +679,19 @@ source/pack provenance and complete persisted evidence are required.
The explicit-only Product E2E `confirmation-replies` suite tests conversational
approval and rejection, persisted message provenance, approval before execution,
ambiguous proposals, and the existing card-click path with native Claude/Codex.
See the [suite contract](../tests/runner-e2e/README.md#conversational-confirmation-replies-explicit-only).
See the [suite contract](../tests/runner-e2e/SUITES.md#conversational-confirmation-replies-explicit-only).
Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately.
The explicit-only [planning guidance utility comparison](../tests/runner-e2e/PLAN-TASK-GUIDANCE.md) measures task decomposition and handoffs with current, short, and disabled skills.
The explicit-only [task-workspaces Product E2E suite](../tests/runner-e2e/README.md#projectless-task-workspaces-explicit-only)
The explicit-only [task-workspaces Product E2E suite](../tests/runner-e2e/SUITES.md#projectless-task-workspaces-explicit-only)
covers projectless task files, deferred repository acquisition, controller restart
and nested Git/dirty-file persistence on legacy local, native local and native
Daytona OpenCode. Its three configured cells use independent filesystem, public
API and download oracles; no live qualification is implied by fixture registration.
The explicit-only [feedback persistence case](../tests/runner-e2e/README.md#feedback-persistence-explicit-only)
The explicit-only [feedback persistence case](../tests/runner-e2e/SUITES.md#feedback-persistence-explicit-only)
checks real native complaint/suggestion submissions and retained bodies with
run-attributed tool receipts on local and Daytona execution. Its four native
Claude/Codex cells are explicit-only. Remote registrations remain unqualified
@@ -699,7 +699,7 @@ until retained live attempts prove them. It uses the normal Product E2E report
and keeps synthetic feedback distinct from actionable product findings.
# Legacy workspace preservation
The explicit [legacy workspace migration Product E2E suites](../tests/runner-e2e/README.md#legacy-workspace-migration)
The explicit [legacy workspace migration Product E2E suites](../tests/runner-e2e/SUITES.md#legacy-workspace-migration)
verify a retained session through the production offline migration and an empty
replacement controller. They complement the rolling deployment suites and do
not claim uninterrupted service during initial enrollment.
+1 -1
View File
@@ -117,7 +117,7 @@ Three agents use four bounded runs to save, review, and correct a tiny case rece
on the same task. Two API-seeded passive cases exercise read-only list filters;
the browser reads the saved document, its revisions and handoff history. The
independent oracle checks actual native document reads and revision attribution.
See [fixture and limits](../../tests/runner-e2e/README.md#persistent-case-handoffs-explicit-only).
See [fixture and limits](../../tests/runner-e2e/SUITES.md#persistent-case-handoffs-explicit-only).
## Gotchas
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -1,6 +1,6 @@
/**
* Config and aged-state fault waves (CF1–CF3). See config-faults.ts for the
* oracle and README.md ("Config faults") for usage.
* oracle and SCENARIOS.md ("Config faults") for usage.
*
* node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts \
* --faults CF1,CF3 --adapters codex --tasks 4 --concurrency 1 --restarts K1,K2 --seed 61
+1 -1
View File
@@ -1,6 +1,6 @@
/**
* Environment-provider fault waves (EP1). See env-provider-faults.ts for the
* oracle and README.md ("Environment-provider faults") for usage.
* oracle and SCENARIOS.md ("Environment-provider faults") for usage.
*
* node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/env-provider-wave.ts \
* --fault EP1 --adapters codex --tasks 2 --seed 71
+1 -1
View File
@@ -1,6 +1,6 @@
/**
* `--long-turn-minutes <n>` (15–60): every task's first turn does n minutes of real tool work before its
* workload, for any cell (README "Long turn").
* workload, for any cell (SCENARIOS.md "Long turn").
*
* The staging failures of 2026-11-11 followed turns of 14–43 minutes, while every chaos workload finishes in
* a few minutes. A long turn grows what the run holds when its fault or next wake arrives: the provider
+1 -1
View File
@@ -1,5 +1,5 @@
/**
* MS1–MS3: two control planes on one database (README "Two servers, one database").
* MS1–MS3: two control planes on one database (SCENARIOS.md "Two servers, one database").
*
* Hosted Cloud deploys by rolling overlap: a new server B starts on the same database while the old
* server A still serves, then A drains (doc/plans/2026-10-31-zero-downtime-web-deploys.md: 5 s overlap,
+1 -1
View File
@@ -1,5 +1,5 @@
/**
* `--fail-recovery-step <step>[:<nth>]`: fail the Nth attempt of one recovery step once (README "Fail the Nth
* `--fail-recovery-step <step>[:<nth>]`: fail the Nth attempt of one recovery step once (SCENARIOS.md "Fail the Nth
* recovery step").
*
* Every other fault hits the control plane once and then lets recovery run. On staging the first fault was
+1 -1
View File
@@ -54,7 +54,7 @@ export function parseK15HoldSeconds(options: { faults: readonly string[] }, raw:
export const K15_OVERLAP_SECONDS_RANGE: readonly [number, number] = [5, 600];
/**
* The staging deploy shape (README "Redeploy without a volume, with overlap"):
* The staging deploy shape (SCENARIOS.md "Redeploy without a volume, with overlap"):
*
* - `--k15-overlap-seconds <n>` or `<min>-<max>`: a K15 boots the next deploy generation beside the old server,
* switches traffic (the wave's stable port) to it once it is ready, keeps the old server running for the seeded
+1 -1
View File
@@ -1,5 +1,5 @@
/**
* W2, W2Q, W2C: the same task woken again right after a run of it finished (README "Re-wake after success").
* W2, W2Q, W2C: the same task woken again right after a run of it finished (SCENARIOS.md "Re-wake after success").
*
* On staging, 20 of 25 failures in one 8-hour window happened with no restart at all. Ten or more runs of
* one agent ran at once on reused (warm) sandboxes, and 30–60 s after a long run of a task succeeded, an
+22 -3060
View File
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff