diff --git a/.agents/skills/add-product-e2e-eval/SKILL.md b/.agents/skills/add-product-e2e-eval/SKILL.md index 686fa44f136..2eba0f56114 100644 --- a/.agents/skills/add-product-e2e-eval/SKILL.md +++ b/.agents/skills/add-product-e2e-eval/SKILL.md @@ -17,7 +17,8 @@ workspace roots such as `~/paperclipai/paperclip`; verify the selected root contains `tests/runner-e2e` and `packages/paperclip-runner`. Run commands from that repository root. The copied skill may live outside the checkout. Read `doc/evals.md`, then the authoritative -`tests/runner-e2e/README.md`, `FIXTURES.md`, `SECURITY.md`, and +`tests/runner-e2e/README.md`, the suite's section in `SUITES.md`, +`FIXTURES.md`, `SECURITY.md`, and `EVERYDAY-WORKFLOWS.md` for the selected area. Inspect the nearest existing catalog entry, case, harness flow, matcher, evidence writer, and report test before changing anything. Keep the user journey on production browser/API diff --git a/.agents/skills/paperclip-evals/SKILL.md b/.agents/skills/paperclip-evals/SKILL.md index 9735805c806..4c65f470206 100644 --- a/.agents/skills/paperclip-evals/SKILL.md +++ b/.agents/skills/paperclip-evals/SKILL.md @@ -28,7 +28,8 @@ mock control plane. Authoritative details are in Choose **Product E2E Evals** for real browser/server/database/runner/provider workflows, including local and Daytona environments. Read -`tests/runner-e2e/README.md`, then `FIXTURES.md`, `SECURITY.md`, or +`tests/runner-e2e/README.md`, then the suite's section in `SUITES.md`, +`FIXTURES.md`, `SECURITY.md`, or `EVERYDAY-WORKFLOWS.md` as relevant. Everyday Workflows remain Product E2E even when imported into Evalbook. “Headless” is a browser mode, not a family. diff --git a/doc/evals.md b/doc/evals.md index 3ca152b664b..34eceed497d 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -26,14 +26,14 @@ Selection fails closed until TestOps-assigned identities are committed; see the The explicit deployment-targeted [saved agent instructions journey](../tests/runner-e2e/product-journeys/AGENT-SETTINGS.md) checks an owned agent’s ordinary UI save, persisted instruction bytes, one current-run output, and restoration. The task request omits its instruction-only reference. It reuses the existing Product E2E target and report path; local controls do not establish a staging pass. -The explicit-only [verified API artifact transfer](../tests/runner-e2e/README.md#verified-api-artifact-transfer) +The explicit-only [verified API artifact transfer](../tests/runner-e2e/SUITES.md#verified-api-artifact-transfer) cell checks a native Codex agent on Daytona installing an authorized saved API response and delivering unchanged bytes. The oracle binds the source capture, tool completion and independent attachment download, including the complete 388131-byte synthetic description. Registration and oracle calibration are separate from live provider qualification. -The explicit-only [Kimi Code qualification suite](../tests/runner-e2e/README.md#kimi-code-qualification) +The explicit-only [Kimi Code qualification suite](../tests/runner-e2e/SUITES.md#kimi-code-qualification) registers twenty candidate cells across local and Daytona. It reuses the browser, semantic interaction, file/artifact, Stop and continuation oracles with scoped candidate admission and an encrypted Code credential. Registration does @@ -77,7 +77,7 @@ existing campaigns. Private Cloud bundles are rejected. The output contains only index HTML and history/pointer JSON; it does not upload evidence or change retained campaign bundles. -The explicit-only [Cursor Cloud native cells](../tests/runner-e2e/README.md) +The explicit-only [Cursor Cloud native cells](../tests/runner-e2e/SUITES.md) cover task/artifact delivery, same-agent follow-up, active-run UI Stop and semantic Plan revision/approval from a local controller. They require explicit model/repository/key and a callback-only @@ -100,7 +100,7 @@ from master. Registration, known gaps and private prerequisites remain distinct from measured results. This lane does not qualify the current master app or reset earlier campaign accounting. -The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/README.md) +The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/SUITES.md) extends the existing native restart oracle to a 300-second controller outage and another replacement after completion, using the recorded Claude and Codex models in disposable development fixtures. @@ -125,7 +125,7 @@ native API mutation receipt. It does not qualify named-tool selection. Its provi usage and cleanup. Registration and local calibration do not establish a fresh Cloud pass or inclusion in a recurring selection. -The explicit-only [necessary free-text question journey](../tests/runner-e2e/README.md#necessary-free-text-question) +The explicit-only [necessary free-text question journey](../tests/runner-e2e/SUITES.md#necessary-free-text-question) checks one essential input, the rendered prompt and context, the real answer, one bound continuation, and an exact saved confirmation. Its four development cells use the recorded Codex and Claude models. Registration and calibration do @@ -149,7 +149,7 @@ It requires a fresh owned agent with no prior work runs and uses three provider Cloud fleet operations and trusted CI remain private. Local eval gateway qualification does not qualify Cloud routes, and these selections stay outside `--all`. -The explicit [onboarding approval-history journey](../tests/runner-e2e/README.md#first-task-onboarding) +The explicit [onboarding approval-history journey](../tests/runner-e2e/SUITES.md#first-task-onboarding) continues an interview through partial plan acceptance, approved hiring, and a separate execution confirmation. It requires the assigned child's real execution, saved output, unchanged approved scope, and reload persistence. Deterministic @@ -158,7 +158,7 @@ require an explicit paid selection and a recorded live result. The deployment-targeted [saved skill journey](../tests/runner-e2e/product-journeys/saved-skill-use.md) checks one company skill assigned by key, its exact admitted immutable version, a bound native source read/load, and a run-authored saved result from skill-only inputs. It uses the reusable Product journey target and remains unqualified until a real target run passes. -The explicit-only [case handoff workflow](../tests/runner-e2e/README.md#persistent-case-handoffs-explicit-only) +The explicit-only [case handoff workflow](../tests/runner-e2e/SUITES.md#persistent-case-handoffs-explicit-only) contains arithmetic and receipt smoke cells for a same-task round trip among three native agents using real case/task APIs, saved documents, handoff receipts, and browser readback. Receipt smoke has an explicit terminal handoff to `done` on its fourth run, proving case closure @@ -241,15 +241,15 @@ not an eval category. The explicit Product E2E `completion-updates` suite compares onboarding and idle, busy, multiple-task, and restart Agent Chat handoffs on native Claude/Codex. It separates mechanical completion delivery/result access from semantic review of the retained answer; -see the [probe contract](../tests/runner-e2e/README.md#completion-update-probes-explicit-only). +see the [probe contract](../tests/runner-e2e/SUITES.md#completion-update-probes-explicit-only). -The explicit-only [task-titles suite](../tests/runner-e2e/README.md#automatic-task-titles) +The explicit-only [task-titles suite](../tests/runner-e2e/SUITES.md#automatic-task-titles) checks that production guidance causes a real native agent to name prompt-only standard/Ask tasks early, while preserving user-supplied titles. Its oracle correlates browser creation, native tool receipts, durable titles, audit ownership, and the reloaded task UI; fixture prompts contain no naming instructions. -The explicit-only [native connection guidance suite](../tests/runner-e2e/README.md#native-connection-guidance-explicit-only) +The explicit-only [native connection guidance suite](../tests/runner-e2e/SUITES.md#native-connection-guidance-explicit-only) adds neutral decline prompts, same-task run-attributed explanations, and measured no-use controls across three native local profiles. Its fifteen configured cells are preparation for future matched instruction comparisons, not a live result. @@ -311,11 +311,11 @@ real APIs, drives Chromium, and invokes the selected local or Daytona runner. The explicit-only `agent-chat-hardening` Product E2E suite covers native chat recovery, hiring, status evidence, and review handoff on local and selected warm -Daytona paths. Its [fixture contract](../tests/runner-e2e/README.md) distinguishes +Daytona paths. Its [fixture contract](../tests/runner-e2e/SUITES.md) distinguishes startup cancellation from active response cancellation and HTTP send replay from ambiguous provider action recovery. Select it explicitly; `--all` excludes it. -The explicit-only [production hiring templates suite](../tests/runner-e2e/README.md#production-hiring-templates) +The explicit-only [production hiring templates suite](../tests/runner-e2e/SUITES.md#production-hiring-templates) adds two local native Codex/Claude cells. It exercises API-created production CEO defaults, an explicitly requested hiring skill/reference read, a permanent coder hire, independently computed saved JSON fixtures and worker reuse. @@ -351,9 +351,9 @@ The explicit-only `grok-qualification` and `grok-subscription-qualification` Product suites exercise Grok Build with API and company subscription authentication respectively. Keep their results separate; the subscription fixture seeds an explicitly supplied login and does not qualify interactive -login. See the [Grok fixture contract](../tests/runner-e2e/README.md#grok-build-qualification). +login. See the [Grok fixture contract](../tests/runner-e2e/SUITES.md#grok-build-qualification). -The explicit [Direct blocker guidance suite](../tests/runner-e2e/README.md#direct-blocker-guidance) +The explicit [Direct blocker guidance suite](../tests/runner-e2e/SUITES.md#direct-blocker-guidance) checks the legacy coordination skill against human authority, missing hiring permission, and requester scope decisions through saved browser interactions. @@ -439,7 +439,7 @@ analytical label. ## Evidence, provenance, and history Polling latency is an interval rather than an exact first-detection time. The -[Product E2E timing contract](../tests/runner-e2e/README.md#latency-observation-bounds) +[Product E2E timing contract](../tests/runner-e2e/SUITES.md#latency-observation-bounds) keeps last-absent/first-present bounds, resolution, strict or inclusive targets, and pass/fail/inconclusive timing verdicts alongside the unchanged workflow grade. @@ -627,15 +627,15 @@ a newer rendering of the same campaign. Remaining native chat boundaries are in the explicit-only `agent-chat-qualification` suite: active task reassignment, user Retry after verified worker process loss, and multi-turn answers grounded in actual task -records. See the [workflow and qualification limits](../tests/runner-e2e/README.md#remaining-native-agent-chat-qualification). +records. See the [workflow and qualification limits](../tests/runner-e2e/SUITES.md#remaining-native-agent-chat-qualification). The 26 native `first-task` cells inspect the production wizard's saved native default on qualified targets, without an API runner switch. The 26 Legacy cells use an API-configured inactive setup record and resume it in the wizard, as -described in the [first-task setup contract](../tests/runner-e2e/README.md#first-task-onboarding). +described in the [first-task setup contract](../tests/runner-e2e/SUITES.md#first-task-onboarding). Live results and semantic answer reviews must accompany any qualification claim; catalog presence alone is not a pass. -The explicit-only [native question/resume qualification](../tests/runner-e2e/README.md#task-continuation) +The explicit-only [native question/resume qualification](../tests/runner-e2e/SUITES.md#task-continuation) separates a completed two-answer user journey from semantic-tool documentation qualification. It verifies the exact provider-pause or native response-wake binding for each answer and preserves prior grades when the definition changes. @@ -664,11 +664,11 @@ downloaded proof on local native/legacy Codex and native Daytona. See the The explicit-only Product E2E `api-response-reading` suite verifies retrieval of large saved API responses on local and Daytona native Codex runs. See the -[Runner E2E guide](../tests/runner-e2e/README.md#bounded-api-response-reading). +[Runner E2E guide](../tests/runner-e2e/SUITES.md#bounded-api-response-reading). The explicit-only Product E2E `extended-harnesses` suite covers pending Cursor, Copilot and Pi ACP profiles on local and Daytona. See the -[fixture admission, credentials and budget contract](../tests/runner-e2e/README.md#extended-acp-harnesses-explicit-only). +[fixture admission, credentials and budget contract](../tests/runner-e2e/SUITES.md#extended-acp-harnesses-explicit-only). The private Runner Evals campaign of the same name provides complementary semantic protocol cases; catalog membership is not live qualification. @@ -679,19 +679,19 @@ source/pack provenance and complete persisted evidence are required. The explicit-only Product E2E `confirmation-replies` suite tests conversational approval and rejection, persisted message provenance, approval before execution, ambiguous proposals, and the existing card-click path with native Claude/Codex. -See the [suite contract](../tests/runner-e2e/README.md#conversational-confirmation-replies-explicit-only). +See the [suite contract](../tests/runner-e2e/SUITES.md#conversational-confirmation-replies-explicit-only). Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately. The explicit-only [planning guidance utility comparison](../tests/runner-e2e/PLAN-TASK-GUIDANCE.md) measures task decomposition and handoffs with current, short, and disabled skills. -The explicit-only [task-workspaces Product E2E suite](../tests/runner-e2e/README.md#projectless-task-workspaces-explicit-only) +The explicit-only [task-workspaces Product E2E suite](../tests/runner-e2e/SUITES.md#projectless-task-workspaces-explicit-only) covers projectless task files, deferred repository acquisition, controller restart and nested Git/dirty-file persistence on legacy local, native local and native Daytona OpenCode. Its three configured cells use independent filesystem, public API and download oracles; no live qualification is implied by fixture registration. -The explicit-only [feedback persistence case](../tests/runner-e2e/README.md#feedback-persistence-explicit-only) +The explicit-only [feedback persistence case](../tests/runner-e2e/SUITES.md#feedback-persistence-explicit-only) checks real native complaint/suggestion submissions and retained bodies with run-attributed tool receipts on local and Daytona execution. Its four native Claude/Codex cells are explicit-only. Remote registrations remain unqualified @@ -699,7 +699,7 @@ until retained live attempts prove them. It uses the normal Product E2E report and keeps synthetic feedback distinct from actionable product findings. # Legacy workspace preservation -The explicit [legacy workspace migration Product E2E suites](../tests/runner-e2e/README.md#legacy-workspace-migration) +The explicit [legacy workspace migration Product E2E suites](../tests/runner-e2e/SUITES.md#legacy-workspace-migration) verify a retained session through the production offline migration and an empty replacement controller. They complement the rolling deployment suites and do not claim uninterrupted service during initial enrollment. diff --git a/maps/feature-map/cases.md b/maps/feature-map/cases.md index 07b801c2dc9..4ba3bdadf29 100644 --- a/maps/feature-map/cases.md +++ b/maps/feature-map/cases.md @@ -117,7 +117,7 @@ Three agents use four bounded runs to save, review, and correct a tiny case rece on the same task. Two API-seeded passive cases exercise read-only list filters; the browser reads the saved document, its revisions and handoff history. The independent oracle checks actual native document reads and revision attribution. -See [fixture and limits](../../tests/runner-e2e/README.md#persistent-case-handoffs-explicit-only). +See [fixture and limits](../../tests/runner-e2e/SUITES.md#persistent-case-handoffs-explicit-only). ## Gotchas diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index aba9ef36c23..53d63488edf 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -8,6 +8,10 @@ A script, not an LLM. It starts an isolated Paperclip control plane (temporary plane during those tasks, and grades every task on R1–R6 plus a deterministic T-oracle. +This file covers running the harness, its modes, how tasks are graded, and its +output. Each workload and fault scenario, with its command and oracle, is in +[SCENARIOS.md](SCENARIOS.md). Add a note about one scenario there, not here. + ## Run ```sh @@ -22,14 +26,14 @@ node scripts/typecheck-test-projects.mjs tests/restart-chaos # typecheck ``` Options: `--triggers time,comment,retry,queued,tool,result` (fault timing, below), -`--workloads A1,B1,B5,C1` (opt-in: G1, H1, L1, R1, S1, W2, W2Q, W2C), `--db-pressure ` (below), `--budget-cents` (company hard cap, +`--workloads A1,B1,B5,C1` (opt-in: G1, H1, L1, R1, S1, W2, W2Q, W2C), `--db-pressure ` ([SCENARIOS.md](SCENARIOS.md#database-contention-l1---db-pressure)), `--budget-cents` (company hard cap, default 3000), `--poll-ms` (2000–5000), `--keep` (keep the temporary instance), `--k4-hold-max-seconds` (raise the seeded K4 hold upper end from 300 s, up to the 600 s outage cap). -`--faults K0` plans cells with no fault, for workloads whose shape is the fault (W2, below). `--fail-recovery-step`, +`--faults K0` plans cells with no fault, for workloads whose shape is the fault (W2, in [SCENARIOS.md](SCENARIOS.md#re-wake-after-success-w2-w2q-w2c)). `--fail-recovery-step`, `--long-turn-minutes`, `--k15-overlap-seconds`, `--k15-fresh-home`, `--ephemeral-server`, `--runner-bind-fault`, `--workload staging-mix` and `--release-swap` are described in their own sections. `--fault-after-seconds ` fires every planned fault `n` seconds after its task launches (5–3600; with `--triggers time`), keeping the rest of the seeded plan, for example to restart a Daytona R1 task after its workspace preparation. -`--matrix` crosses every selected workload with every fault kind (below), and `--server-from ` +`--matrix` crosses every selected workload with every fault kind ([SCENARIOS.md](SCENARIOS.md#journey--fault-matrix---matrix)), and `--server-from ` runs every server of the wave from another prepared checkout, for example a commit before a fix. Each wave's temporary instance (embedded Postgres, workspaces, often 5–13 GB) is @@ -51,1018 +55,29 @@ A replay inherits the masks saved in the source wave's `plan.json`; Each fault receipt records the host 1-minute load average (`loadAvg1m`); treat timeout failures above about 16 as load-suspect. -## Restart storm (K13, `--storm-minutes`) +## Fault timing (`--triggers`) -```sh -node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ - --adapters codex,claude,opencode --concurrency 6 --storm-minutes 60 --seed 7 -``` +Each task samples one trigger with its cell. `time` (default) fires at the +seeded offset. The others fire relative to a user action or run state seen by +the observer, after a seeded delay: -A storm wave fires faults on its own seeded clock instead of one fault per -task. Each fault is drawn from `--faults` (default K1, K2, K4, K6) and fires -2–4 minutes after the previous one settled (server ready), so a K4 hold never -eats the next gap. Each fault hits every task in flight. While the storm lasts -the harness launches a new task whenever one rests, keeping `--concurrency` -in flight; `--tasks` (default 200) is only the pool. A pool that runs dry -before the storm ends fails the wave as a harness error (`storm_pool_exhausted`). When the storm ends no -new task or fault starts, the in-flight tasks drain to rest or their deadline, -and only launched tasks are graded. Storm tasks show fault `K13` in the ledger; -`storm_fault`, `storm_started` and `storm_ended` are in the timeline and the -receipts are in `faults.json`. A replay of a storm task fires the faults that -crossed it, without a new storm. `--triggers time,result` may join a storm: a `result` task (C1) also fires its own K1 or K2 (from `--faults`) just after its run accepts a result, the next storm gap runs from its settle, and no triggered fault fires during the drain. Other triggers do not combine with a storm. - -## Governance across restarts (G1, H1) - -```sh -# Approval-gated action: the fault fires 0-3 s after the approval appears pending. -node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ - --adapters codex,claude --workloads G1 --faults K1,K2,K4 --triggers approval --concurrency 4 --tasks 6 --seed 31 -# Budget hard stop: its own wave, one task at a time. -node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ - --adapters codex,claude --workloads H1 --faults K1,K2,K4 --triggers paused --concurrency 1 --tasks 3 --seed 32 -``` - -**G1.** The agent asks for board approval through its `call_api` tool -(`POST /api/companies/{companyId}/approvals`, type `request_board_approval`, -linked to its task), blocks the task and ends its turn. The `approval` trigger -fires the planned fault once the observer sees the approval pending. - -After the restart, with the server ready, the harness acts as the board through -the board's own routes. It either approves (`POST /api/approvals/:id/approve`), -or rejects (`POST /api/approvals/:id/reject`) and cancels the task. The -decision is fixed by the task's nonce, so about half approve and half reject. - -On approval, the woken agent creates exactly one backlog subtask named after the -nonce and finishes. The oracle checks these points: - -- exactly one approval was requested (a restart that made the agent ask twice fails); -- it was still pending when the board decided after the restart; -- it reached the board's decision; -- the subtask exists exactly once on approval, and never after a rejection. - -A rejected task rests `cancelled`; R1 and R3 grade against that status. - -**H1.** Before launch, the harness sets the company budget to its observed spend -plus one cent through the board's budget route. The task's first priced usage -then crosses the hard stop mid-run, which pauses the company. The `paused` -trigger fires the fault once the observer sees `paused` with `pauseReason` -`budget`. - -After the restart, the pause must hold for 30 s with the server ready. The -harness records every run that started while paused (there must be none), then -resolves the incident with `raise_budget_and_resume` (spend plus 3000 cents). -The task must then finish its brief on its own. H1 pauses the whole company, so -it runs alone (`--workloads H1 --concurrency 1`) and never with `--rc06-masked`. - -Neither G1 nor H1 is in the default workloads. The board's actions and what the -observer saw are in the timeline (`governance_*`) and in each task's grade -(`governance`). - -## Stop while running, then continue (S1, `--triggers stop`) - -```sh -node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ - --adapters codex,claude --workloads S1 --faults K1,K2,K4 --triggers stop --concurrency 4 --tasks 6 --seed 23 -``` - -S1 is the board's Stop button. While the task's run is running, the harness cancels it through -`POST /api/heartbeat-runs/:id/cancel`. The `stop` trigger fires the fault 0–2 s later. At least -20 s after the stop, once no run is running or queued and no fault is in progress, the harness -comments that it stopped the run on purpose and asks the agent to continue with an extra marker -line. The oracle checks that the stopped run ended `cancelled`, that nothing started between the -stop and the continue comment (recovery must not restart work the operator stopped), and that the -finished brief has both markers. The Stop waits for a ready server with no fault in progress. If the -run already finished when the stop window opens, the harness records a missed Stop -(`task_stop_missed`), skips the stop-triggered fault, still posts the continue comment, and the -oracle checks only the continue. S1 is not a default workload. - -## Human-in-the-loop journeys (Q1, Q2, Q3) - -```sh -node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ - --adapters codex,claude --workloads Q1,Q2,Q3 --faults K1,K2,K4 --triggers asked,answered --concurrency 2 --tasks 4 --seed 41 -``` - -Each journey starts from a user entry point in -[`maps/feature-map/questions-and-approvals.md`](../../maps/feature-map/questions-and-approvals.md). -The harness acts as the board through the routes the board UI uses. It acts only -when the server is ready, no fault is in progress, and every fault the task planned -before the response has fired. Two triggers fire at the journey's own boundaries: - -- `asked`: 0–3 s after the observer first sees the saved request pending. -- `answered`: 0–3 s after the board's response returned. - -Until the board acts, the task waits on the board, as G1 does: this is not a rest, -not a stall, and gets no Retry. The code lives in `journeys.ts` (definitions and -oracles of Q2 and Q3), `journey-driver.ts` (their board) and `journey-oauth-app.ts` (Q3's app). - -**Q1, a question** (`questions`, `continuation`) is the saved question described -under "Provider message faults" below (`question-handoff.ts`). With restart faults -it takes the same two triggers: `asked` arms on its first pending poll and -`answered` on the board's answer. The board answers before an `answered` fault -fires and after every other planned fault. - -**Q2, a question on a blocked task** (#20363). The agent asks one single-select -question with its native `request_human_input` tool and ends its turn. The board -reads the card and answers with the option named by the task's nonce (`amber` or -`cobalt`), on the task thread -(`POST /api/issues/:id/interactions/:interactionId/respond`). Before it answers, the board -adds an unassigned `backlog` prerequisite. It sets the task `blocked` by it -(`blockedByIssueIds`), and then answers. The answer must reach the agent in the -bounded answer mode, while the task stays blocked. The agent replies with the -answer marker, in a comment or in that turn's accepted run result (a bounded turn -may report through either). The board waits up to 6 minutes of ready -server for that reply. Then it marks the prerequisite `done`. The -blockers-resolved wake lets the agent write a brief that carries -`ANSWER--` and finish. The oracle checks these points: - -- exactly one question was saved (a restart that made the agent ask twice fails); -- the question was still pending when the board answered; -- with `asked`, the board answered after the restart; -- the brief carries the answer marker and the task rests `done`; -- the reply came while the task was `blocked`; -- the prerequisite relation was kept through the reply; -- the reply came before the prerequisite was finished; -- no run of the task carries `issue_dependencies_blocked`; -- no stale Retry notice at rest. - -**Q3, an app action reviewed late** (`tool-review`, #20340). The harness starts its -own OAuth 2.1 + MCP app on loopback. The app has discovery, dynamic client -registration, PKCE, rotating refresh tokens, one-hour access tokens and one write -tool, `create_page`. No third-party account is used. Before launch, the board -connects the app (`POST /api/companies/:id/tools/apps/connect`). It follows the -app's authorize redirect back to `/api/tools/oauth/callback`. Then it finishes setup -for one scripted `process` agent with `create_page` set to Ask first. That agent -is adapted from `tests/e2e/connection-reviews.spec.ts`. Q3 grades the review path, -not a model, so `--adapters` only labels the cell. A process run records usage with no price, so Q3 setup also sets the company budget policy to allow unpriced usage (the amount and hard stop still apply). - -The agent's call waits for review. Then the board does three things: - -1. It moves that review's saved timestamps and the grant's access-token expiry back - by 61 minutes in the wave's own database (`shiftReviewClock`); a shared connection keeps that expiry on the connection. This is the - `GRACE_SHIFT_MS` approach used by `invariants.ts`. -2. It runs **Check connection** (`POST /api/tool-connections/:id/health-check`). - That check renews the now-expiring token through the app. -3. It approves (`POST /api/tool-gateway/action-requests/:id/approve`). - -The oracle checks these points: - -- exactly one review was requested; -- the review aged more than an hour; -- the app saw a refresh between the request and the decision; -- the approval was accepted; -- the app saw exactly one `tools/call` with the reviewed title; -- the request ended `executed`; -- the agent reported the provider's one-time receipt, which only the tool result carries; -- the task rests `done`. - -## Journey × fault matrix (`--matrix`) - -```sh -node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ - --adapters codex --workloads Q1,Q2,Q3 --faults K1,K2,K4 --matrix --concurrency 2 --seed 43 -``` - -`--matrix` plans one task per cell, with a fixed order and a seeded nonce. -The cells are workload × adapter × one of these: - -- `K0`: the same journey with no fault, as a baseline; -- each restart fault in `--faults` at each of the workload's own triggers; -- each provider message fault in `--faults` (PV1–PV3), armed once per workload. - -For a journey workload, those triggers are `asked` and `answered`. The example -plans 3 × (1 + 3 × 2) = 21 tasks. Any workload can join the matrix, with its own -boundary triggers: G1 at `approval`, H1 at `paused`, S1 at `stop`, C1 at -`comment` and `result`, B5 at `tool`, and the rest at `time` (`WORKLOAD_TRIGGERS` -in `journeys.ts`). G1 and H1 get no baseline, because their oracles grade a board -decision made after a restart. `--fault-after-seconds` pins the time-fired cells. -The matrix sets `--tasks` itself. It does not combine with `--tasks`, `--triggers`, -`--storm-minutes`, `--crash-point` or K14. - -A restart hits every task in flight. So the baselines come first, and the wave -never runs a baseline beside a fault cell: fault cells start only after the last -baseline rests. - -To prove that a journey fails before a fix, prepare a checkout of the fix's parent. -Then run the same wave against it with `--server-from`. The harness code stays the -same; every server start and restart runs the older build: - -```sh -git worktree add --detach ../paperclip-pre20363 3cc1032f6a^ && (cd ../paperclip-pre20363 && node scripts/agent-preflight.mjs) -node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads Q2 \ - --faults K1 --triggers answered --concurrency 1 --tasks 1 --seed 44 --server-from ../paperclip-pre20363 -``` - -## Code change in a real repository (R1, `--workspace-repo`) - -```sh -node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ - --adapters codex,claude --workloads R1,C1 --workspace-repo /path/to/paperclip \ - --faults K1,K2,K4 --concurrency 4 --tasks 8 --seed 22 -``` - -The harness clones the given git repository (shallow, from its committed HEAD) -as the project's workspace, in place of the chaoscli fixture. R1 asks the agent -to add a TypeScript module and a `node:test` file under -`packages/shared/src/chaos/`, named with the task's nonce so concurrent tasks -share the repository without colliding. The oracle runs the agent's test and a -hidden case against its module. B1 and B5 need the chaoscli fixture, so they do -not combine with `--workspace-repo`. Local only. - -## Upgrade mid-run (K14, `--upgrade-from`) - -```sh -git worktree add ../paperclip-838e9bf3 838e9bf3d2 # an older master build -(cd ../paperclip-838e9bf3 && node scripts/agent-preflight.mjs) -node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ - --adapters codex,claude --workloads A1,C1,M1 --faults K14 --upgrade-from ../paperclip-838e9bf3 \ - --concurrency 4 --tasks 8 --seed 21 -``` - -The wave's server starts from the older checkout. The first K14 stops it -gracefully (SIGTERM, as a deploy does) and starts this checkout against the same -database, instance directory, port and secrets, so the new build applies its -migrations to the old build's data and resumes what the old build saved. Each -task is graded as usual. Pass means: - -- no task needed Retry (R3); -- C1's follow-up comment reached the work (T); -- the wave-end sweep finds no stuck records (R6). - -A later K14 in the same wave restarts the current build, like K2. Every K14 -receipt names both commits (`upgrade` in `faults.json`, `upgrade_switched` in the -timeline). K14 is local-only, because a Daytona runner must match its image, and -it is never part of a storm's default faults. - -## Redeploy (K15, `redeploy.ts`) - -```sh -node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \ - --adapters opencode,claude --workloads A1,C1 --faults K15 --daytona-reuse-lease --concurrency 3 --tasks 6 --seed 9301 -``` - -The hosted staging stack deploys about every 40 minutes. Each deploy replaces -the server's container. The replacement has the same database and volume, but it -runs in another process space (host name, kernel boot and PID namespace) and -has a new Railway deployment id. The server's PID-based proof that an earlier -owner exited (`controller-process-identity.ts`, `owner-proof.ts`) never applies -across a deploy. Recovery must use the deployment proof or the lease deadlines -instead. K1 and K2 restart on the same machine, so every PID proof succeeds and -that path stays untested. See `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`. - -K15 emulates the container replacement without a container: - -- Every server in a wave that has a K15 is launched with a `--import` preload - (`redeploy-preload.mjs`). The preload makes `os.hostname()` return a - per-generation name, which changes the process-space hash. -- Every server gets Railway-shaped variables: `RAILWAY_PROJECT_ID`, - `RAILWAY_ENVIRONMENT_ID`, `RAILWAY_SERVICE_ID` and `RAILWAY_DEPLOYMENT_ID`, - plus `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH`, which is the - instance home. -- A K15 stops the server with SIGTERM, like K2. The replacement starts as the - next generation, with a new host name and deployment id, on the same project, - service and volume. -- Other faults in the same wave restart within the current generation, like a - platform restart of the same deployment. -- Identities derive from the wave id; a replay uses the source wave id, so it sees the same ones. -- Each K15 receipt records `redeploy` (both generations) in `faults.json`, and - `redeploy_switched` in the timeline. - -Limits: - -- The Daytona sandboxes and remote runners survive, as they do on staging. -- Local runners are stopped with the server's process tree. -- The cloud stop-proof service (`PAPERCLIP_CLOUD_*`) is not emulated, so stop - proofs that need it stay unsupported, as on any self-hosted Railway deploy. -- K15 does not combine with MS1–MS3, because a second server would run in the - real host's process space. -- Combine K15 with `--adapters opencode` and `--daytona-reuse-lease` or - `--daytona-warm` to match staging's workload mix. - -### Long mid-tool redeploy (`--k15-hold-seconds`, T1, Runner logs) - -```sh -node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \ - --adapters opencode --workloads T1 --faults K15 --k15-hold-seconds 240-480 --concurrency 2 --tasks 4 --seed 9310 -``` - -A staging deploy usually lands mid-turn and keeps the server down for 4–7 -minutes. In staging class R, recovery then found the remote Runner gone -(`runner_remote_process_identity_unavailable`). - -- **`--k15-hold-seconds ` or `-`** (0–600 s) seeds each K15's - outage. The server stops with SIGTERM and stays down for the hold. Then the - next deploy generation starts. Without the option, a K15 restarts at once. - The range is drawn only for K15 cells, so other seeds keep their plans. -- **H2** (opt-in workload, staging class H) has the agent save `brief`, then hand - the task to the person who created it with status `in_review` as its last - step. The task must rest `in_review` with that person, no run of the task may - end `issue_reassigned`, and the task must have no execution blocker or active - recovery action. Run it with a native adapter, for example - `--adapters claude --workloads H2`. -- **T1** (opt-in workload) has the agent run one shell command in the - foreground for about 100 s (50 ticks of 2 s). The command prints the loop's - own clock window. The agent then saves `brief`, whose first line is - `TICKS-=50 start= end=`. Its fault window (35–80 s - after launch) aims to land while the tool runs in the sandbox, and stays short - of provider shell-tool timeouts. - - The `fault_crossed_tool` check (skipped for `K0` baselines and provider faults; - a storm task counts the storm's faults) fails the task unless one of its own faults - started inside the reported window, with 5 s of clock slack. A fault that hit - preparation, with the loop run afterwards, tested nothing mid-tool, as a - provider-fault task whose fault never applied. -- **Runner evidence:** on a Daytona K15 wave, the harness captures every - started sandbox of the wave three times: - - right after the server stopped (`-stopped`); - - after the hold, just before the next deploy starts (`-before_start`); - - at wave end (`wave-end`). - - Each capture is written to `results//daytona-runner/