mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
test(restart-chaos): K15 redeploy, F1 workspace symlink, and the staging failure census (#20679)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The restart-chaos harness (`tests/restart-chaos/`) restarts the
control plane while real agent tasks run. It grades whether every task
finishes without a human Retry.
> - The hosted staging stack deploys about every 40 minutes. In 48
hours, 276 runs failed and 63 tasks stayed blocked. About a quarter of
the failures came within 10 minutes of a deploy. The harness passed the
same areas.
> - A hosted deploy replaces the server container. The new server runs
in another process space and has a new deployment id. The PID-based "the
earlier owner exited" proofs never apply across that boundary. A harness
restart on one machine keeps the process space, so every PID proof
succeeds and the hosted recovery path is never tested.
> - This pull request adds K15, a redeploy fault. It also adds the
staging census that shows the gap.
> - The benefit: chaos waves can now exercise the recovery path that
staging runs after every deploy, on local and Daytona.
## Linked Issues or Issue Description
**What happened?**
Many staging tasks that a deploy crossed ended blocked. Examples are
`native_remote_recovery_lease_mismatch`,
`runner_remote_process_identity_unavailable`,
`prior_owner_terminal_state_indeterminate` and `PRP provider resume
state is unavailable`. The restart-chaos Daytona waves passed for the
same adapters.
**What did you expect?**
The chaos harness should reproduce the deploy conditions that staging
has, so that these failure classes show up in a wave first.
**Steps to reproduce**
1. Compare the staging run records' `controllerProcess.hostHash` and
`deployment.deploymentId` across a deploy. Both change on every deploy.
2. Compare the same fields across a harness K2 restart. Both stay the
same.
3. Read `controllerProcessExited` and `sameProcessSpace` in
`server/src/services/controller-process-identity.ts`. A different
`hostHash` means that no PID proof applies.
**Additional context**
The full census, the failure classes and the per-class harness gaps are
in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`. The
task identifiers in it are neutral labels.
## What Changed
- New fault `K15` (redeploy), in `tests/restart-chaos/redeploy.ts` and
`redeploy-preload.mjs`. A K15 wave gives every server an emulated host
name through a `--import` preload, which changes the process-space hash.
It also gives Railway-shaped deployment variables with the instance home
as the exclusive volume. A K15 stops the server with SIGTERM and starts
the next generation, with a new host name and deployment id.
- `supervisor.ts`, `main.ts`, `config.ts` and `types.ts`: select K15
with `--faults K15`. It works on `--env local` and `--env daytona`. The
receipt records `redeploy` and the timeline records `redeploy_switched`.
Config refuses K15 together with MS1–MS3.
- New opt-in workload `F1` (`workloads.ts`): the agent keeps an absolute
symlink in its repository workspace. A staging save-back refused such a
link and blocked the task. The oracle needs `done` and the saved notes
file, with no Retry. `f1-symlink.test.ts` covers the prompt, the grading
and the option planning.
- `redeploy.test.ts`: identity derivation, the environment, the argument
order, a real child process that proves the preload changes both CJS and
ESM `hostname()`, option parsing and plan detection.
- `tests/restart-chaos/README.md`: a K15 section and a row in the
"Pieces" table.
- `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`: the
48-hour staging census, 12 failure classes, the deploy timeline, the
reason the harness missed each class, and proposed scenarios.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/redeploy.test.ts
tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/repeated-restarts.test.ts`: 68 tests pass. The full
restart-chaos unit suite passes: 43 files, 516 tests.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Live Daytona waves on 8f6ca25976, image
`paperclip-daytona-runner@sha256:bc7ca92d…`:
| Wave | Faults and cells | Result |
| --- | --- | --- |
| `w-20261111T002313-s9301` | K15, opencode and claude, A1 and C1, reuse
lease, 6 tasks | 5 of 6 pass. t05 (OpenCode, C1) failed R5
`silent_dead_run`: 183 s without evidence after the redeploy, then
succeeded. |
| `w-20261111T004241-s9301` | K2 control, same seed | 6 of 6 pass |
| `w-20261111T005008-s9302` | K15, warm sandbox, comment trigger, 4
tasks | 4 of 4 pass |
| `w-20261111T010000-s9303` | K15, opencode only, 8 tasks | 8 of 8 pass
|
| `w-20261111T011044-s9303` | K2 control, same seed | 8 of 8 pass |
| `w-20261111T012523-s9304` | F1 with K2, codex and opencode, `--storage
s3` | 2 of 2 pass |
- Every K15 fault switched the generation, with a new host name and
deployment id on the same volume. The server was ready again in 14–24 s.
- K15 does not yet reproduce the staging refusal codes on current
master. The census document explains the open conditions, such as a code
change at the deploy.
## Risks
- Low risk. The change is in the test harness and docs only. No product
code changes.
- K15 is an emulation. It changes the host name only in the server
process (and forked Node children). It does not emulate a new kernel
boot or PID namespace. The cloud stop-proof service is not emulated, so
stop proofs that need it stay unsupported, as on a self-hosted Railway
deploy. The README lists these limits.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, GitHub CLI).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed
1 parent
7435abcc63
commit
52160eba35
12 files changed
+723
-13
No files matched your search
@@ -0,0 +1,309 @@
|
||||
# Staging failures vs chaos-harness coverage (2026-11-09 00:00Z to 2026-11-11 00:05Z)
|
||||
|
||||
Question from Dotta: what failed on `paperclip.staging.paperclip.app` in the last 48 hours, why
|
||||
did the restart-chaos harness (`tests/restart-chaos/`) not find it first, and what must the harness
|
||||
add so it would.
|
||||
|
||||
## Answer first
|
||||
|
||||
1. **276 runs failed, were cancelled or were interrupted in 48 hours, across 114 tasks.** At census
|
||||
time, 63 of those tasks (55%) were still `blocked`.
|
||||
2. **Staging is a continuous deploy target.** There were 47 server process starts in 48 hours. 45 of
|
||||
them were new Railway deployments: about one deploy every 40 minutes in working hours. Each deploy
|
||||
replaces the container, so the new server runs in a **different process space** (host, kernel boot
|
||||
and PID namespace) with **newer code**. Remote Daytona sandboxes keep the previous runner.
|
||||
3. **The harness restarts the same build on the same Mac, at once.** A harness restart keeps the
|
||||
process space and has no deployment identity. It is back in seconds. A staging deploy has three
|
||||
differences:
|
||||
- it sends a graceful SIGTERM, often 1–3 s after a native coordinator claim;
|
||||
- the server stays down for 4–7 minutes;
|
||||
- it has a new deployment identity.
|
||||
|
||||
Native recovery uses the deployment proof (`proveOwnerGone`, `deployment_replaced`), not PID
|
||||
proofs. So the process space matters less than first assumed. This was corrected after review.
|
||||
The open gap is the long, mid-turn outage, during which the remote Runner is lost (class R
|
||||
below). The harness passed D1 8/8 and D5 6/6 on master while staging blocked tasks.
|
||||
4. **The Daytona matrix does not use the staging workload mix.** Staging runs OpenCode in 66% of
|
||||
native runs (1266 of 1917), uses reusable sandboxes (saved epochs up to 5), runs 25 agents and has
|
||||
long-lived parent/child task trees. The Daytona matrix runs only `codex,claude` on fresh
|
||||
per-run sandboxes, one task tree deep, with waves shorter than one hour.
|
||||
5. **The harness never builds old saved state.** In the largest class by count, automatic wakes
|
||||
failed 99 times in a row on one parent task over 8.5 hours. That needed a workspace shared under a
|
||||
policy retired on 2026-11-07, and every wave starts from an empty database. The existing U2 check
|
||||
would have failed the task, so grading was not the gap.
|
||||
|
||||
## Method and sources (read-only)
|
||||
|
||||
Task labels `T01`–`T60` stand in for staging task identifiers, which are instance-local. The census
|
||||
owner keeps the mapping outside the repository.
|
||||
|
||||
- Staging API, GET only, with the operator's existing CLI token loaded into a child process:
|
||||
- `/api/health`;
|
||||
- `/api/companies/:id/heartbeat-runs` (company and per agent);
|
||||
- `/api/heartbeat-runs/:id`;
|
||||
- `/api/heartbeat-runs/:id/events`;
|
||||
- `/api/issues/:id`, `/runs`, `/activity` and `/recovery-actions`.
|
||||
|
||||
One company exists. There were 1917 runs in the window.
|
||||
- Deploy and restart times: every run record carries `controllerProcess`, with `processStartedAt`,
|
||||
`bootId` and the Railway `deploymentId`. The timeline below is built from those fields.
|
||||
GitHub has no staging deployment records, because staging deploys from Railway.
|
||||
- `git log origin/master` for fixes that touch each failure code.
|
||||
- Window: a failure counts when its run finished at or after 2026-11-09T00:00Z.
|
||||
|
||||
### Data the census could not read
|
||||
|
||||
- **Sentry and Honeycomb:** no read credentials for staging were available. The only keys are
|
||||
ingest/test-config keys. Sentry triage was not run.
|
||||
- **a PR-review agent runs:** runs from 2026-11-09 00:00–14:52Z are missing. That agent made more than
|
||||
1000 runs in the window, and the run list caps at 1000 with no cursor. Other agents are complete.
|
||||
- **Commit per deploy:** Railway deployment IDs do not map to git SHAs through the API. Only the
|
||||
current SHA (`358977454b`, #20649) is visible. "Fixed on master" therefore means "a merged PR
|
||||
changes this path". It does not mean the fix was proven live on staging.
|
||||
- **Brief staging outage:** staging was in a maintenance window, `tenant_app_unavailable`, at about
|
||||
00:11Z on 2026-11-11, during the census. That was another deploy. Data read before the window is
|
||||
complete.
|
||||
|
||||
## Class table
|
||||
|
||||
The share column is the share of the 276 failed runs. The "Blocked now" column counts tasks still
|
||||
blocked at census time.
|
||||
|
||||
| Class | Runs | Share | Tasks | Blocked now | Mechanism (root) | Fixed on master? | Why the harness missed it | Proposed scenario |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| W: wake loop on a held shared workspace | 103 | 37% | 4 | 1 | A parent shares a legacy execution workspace with a child that stranded a writer hold on Nov 5. The new stall-wake feature (#20098) woke the parent every 10 min. Each wake failed setup with no backoff. T01 had 99 runs from 07:27 to 15:53Z. | **Fixed:** #20273, #20295, #20314 | Fresh database every wave, so there was no legacy shared-workspace state. The feature landed between waves. U2 would have caught the loop. | None (fixed). If needed: K14 from a pre-2026-11-07 build with M1. |
|
||||
| R: remote recovery refuses after a deploy | 34 | 12% | 32 | **24** | Recovery after a deploy refuses to continue. Codes include `runner_remote_process_identity_unavailable` (9), `native_remote_recovery_lease_mismatch …identity_run_id` (8), `prior_owner_terminal_state_indeterminate` (6), `cleanup_quarantined` (4) and `missing_evidence` (3). 27 of 34 failed within 10 min of a process start. | **Mostly fixed:** `identity_run_id` by #20642, `prior_owner…` by #20677, others by #20639, #20631, #20666, #20662 and #20650. **Open:** `runner_remote_process_identity_unavailable`, where the remote Runner is gone after a 4–7 min mid-turn outage. | Harness restarts come back at once, with no long mid-turn outage. There were no OpenCode cells on Daytona (19 of 34 are OpenCode). No Daytona workload keeps a tool running across the fault. | **K15 redeploy** (implemented). A long-hold and mid-tool variant with sandbox-side Runner log capture is the follow-up. |
|
||||
| F: finalization and workspace save-back | 24 | 9% | 15 | 8 | A saved result cannot be copied back or restored. Causes: sandbox disk full (ENOSPC in 2 runs), unsafe symlink in the archive, `recovery_receipt_invalid` (7), a stuck "Recovering saved work…" phase (8) and an exhausted finalization budget (3). | **Partly fixed:** #20666 (receipt), #20578 (re-finalize). Nothing for disk full or symlinks. | Workloads write small files. They never fill the sandbox disk and never create symlinks. ST faults hit S3, not the sandbox disk. | **F1 symlink** workload (built; passes on master). **DF1 sandbox disk full** (proposed). |
|
||||
| E: Daytona capacity outage | 18 | 7% | 13 | 4 | Daytona returned HTTP 502 and the worker hit SIGKILL from 16:36 to 17:02Z on Nov 10. Reusable sandbox resume failed, plus a git clone network failure. | **Mostly recovers:** 8 tasks recovered automatically and bounded retry works. 4 tasks still blocked. | The harness has no environment-provider fault. All faults hit the control plane or S3. | **EP1 environment provider 5xx window** (proposed, not built). |
|
||||
| U: provider usage limit | 17 | 6% | 12 | 8 | `provider_limit_wait` waits as designed. The ACPX "terminal limit failure" (7) fails the run, where it should wait. | **Open:** ACPX limit classification. | Not a restart class. Provider shims have no usage-limit message. | Add a usage-limit message to PV faults (proposed). |
|
||||
| P: provider session resume after a deploy | 16 | 6% | 16 | **11** | The runner or provider cannot resume its session after the controller moved. Codes: `PRP provider resume state is unavailable` (5), `runner_harness_state_mismatch` (8) with sub-codes `provider_identity_incomplete`, `resumed_unavailable` and `backup_without_reusable_lease`, plus `ingress_auth_failed` and `run.attach provider_side_effect_pending`. 8 of the 16 are ACPX and 5 are OpenCode. | **Partly fixed:** #20627 (warm backup). `resume state unavailable` is open. | Same as R: same process space, no OpenCode, no reusable warm sandboxes across deploys. | K15 with `--daytona-warm` and `--adapters claude,opencode`. |
|
||||
| O: other or unclassified native failures | 13 | 5% | 13 | 5 | `native_run_failed` with no classified error (3), ACPX `session.open` rejected (2), `local_runtime_api_listener_unavailable` (2), `tool.resolve provider_tool_call_retired`, a DB query error, and `Object not found`. | **Open:** classification. | Not deploy-correlated (0 of 13 within 10 min). | None. These are product triage items. |
|
||||
| B: authorization or binding readiness | 13 | 5% | 9 | 3 | `reviewed_chat_execution_binding_not_authorized` (7) and `not_ready` (2), a credential not shared with the responsible user (2), and a skill package still uploading (2). | **Open** | The harness uses one operator. There are no chat bindings, no shared credentials and no skill uploads. | Out of scope for restart chaos. Route to product. |
|
||||
| L: legacy runtime lost across a deploy | 12 | 4% | 12 | 5 | Legacy (non-native) runs: `process_lost` (5), `Legacy controller lease lost` (4), graceful SIGTERM (1) and `lease_released_before_terminal` (2). | **Recovers in most cases:** 7 of 12 tasks recovered. | The matrix runs native adapters only. | K15 with a legacy adapter cell (proposed). |
|
||||
| X: expected cancellations | 11 | 4% | 9 | 2 | Reassignment, ownership changes, dependency blocks and operator interrupts. | No fix needed | n/a | none |
|
||||
| A: runner or provider-pack artifact incompatible after an upgrade | 8 | 3% | 8 | 4 | New code meets a sandbox prepared by older code. Codes: `OpenCode executable is missing from the provider pack`, `ACPX launch profile omitted its provider-pack authority`, `native_provider_artifacts_admission_unverified` (3), `runner_remote_artifact_platform_mismatch` and `runner_revision_mismatch`. | **Open** | Daytona waves require the runner content to match the image and refuse mismatches. That is exactly the mixed-version state staging lives in. K14 is local only. | **K15 with `--upgrade-from` on Daytona** (allows mixed runner/image versions; proposed follow-up). |
|
||||
| V: runner event commit stall | 7 | 3% | 5 | 2 | `native_event_replay_conflict: runner_event_commit_stalled`, Codex and ACPX, 20:49–23:55Z on Nov 9. No deploy correlation. | **Likely fixed:** #20385 (usage receipt stalls). No occurrence after 23:55Z on Nov 9. | Database contention (L1) used one-row pressure, not the receipt-update pattern. | none (fixed) |
|
||||
|
||||
Totals: 276 runs and 114 tasks. Classes R, P, A and L together are the deploy-crossing failures:
|
||||
70 runs (25%) and **44 currently blocked tasks**, which is most of the user-visible harm. Class W is
|
||||
the largest by count but is fixed.
|
||||
|
||||
## Deploy timeline (Railway, from run records)
|
||||
|
||||
47 process starts between 2026-11-08T22:03Z and 2026-11-10T23:23Z. All but one are new
|
||||
`deploymentId`s. The exception is a same-deployment restart at 2026-11-09T14:28Z. The median gap in
|
||||
working hours is about 40 minutes. Every native run longer than about 40 minutes crosses a deploy.
|
||||
|
||||
67 of 276 failures (24%) finished within 10 minutes after a process start. In classes R and P, 30 of
|
||||
50 did. In classes E, O and V, almost none did.
|
||||
|
||||
## Per-class detail
|
||||
|
||||
### W: wake loop on a held shared workspace (103 runs, fixed)
|
||||
|
||||
- Examples: T01 (99 runs), T02, T03 and T04.
|
||||
- Timeline:
|
||||
1. On Nov 5, child T05 failed setup (`workspace_busy`, then `setup_failed`). Its parent
|
||||
T01 shared the workspace, a pre-2026-11-07 policy.
|
||||
2. On Nov 9, #20098 ("wake the parent once when a child stalls") reached staging. At 07:27Z the
|
||||
parent began to be woken about every 10 min by `issue.blocker_stalled_wake`.
|
||||
3. Every wake failed with `The workspace still has an unfinished writer or save operation.`
|
||||
4. At 15:53Z the parent became `blocked`.
|
||||
- Fixed by:
|
||||
- #20273, which stops stall wakes from failing setup on a held workspace;
|
||||
- #20295, which re-proves an exhausted save's agent stop;
|
||||
- #20314, which moves stranded legacy subtasks off a parent's held workspace.
|
||||
- Why the harness missed it:
|
||||
- **Old saved state:** every wave starts from an empty database, so a parent/child pair sharing a
|
||||
workspace (the legacy policy) never exists.
|
||||
- **Missing feature in the tested build:** #20098 landed at 00:26Z on Nov 9, after the last wave
|
||||
that had a stalled child.
|
||||
- **Timing window:** the loop starts only after a child stalls for longer than the stall threshold.
|
||||
- **Grading is not the gap.** The enforced U2 check already fails a task after more than three
|
||||
failures with the same code and no status change. The parent would have failed U2 on its fourth
|
||||
wake, if a wave had created the legacy state.
|
||||
- Scenario: none is built in this pass, because the class is fixed and the shared-workspace policy
|
||||
is retired for new subtasks. A K14 wave from a pre-2026-11-07 build with M1 would recreate the
|
||||
legacy state, if the fix needs regression cover.
|
||||
|
||||
### R: remote native recovery refuses after a deploy (34 runs, 24 tasks blocked)
|
||||
|
||||
- Examples:
|
||||
- T06, T07, T08 and T09 (`lease_mismatch identity_run_id`);
|
||||
- T10, T11, T12 and T13 (`identity_unavailable`);
|
||||
- T14, T15, T16 and T17 (`prior_owner_terminal_state_indeterminate`, with
|
||||
`retirement_refused=journal_commands_unsettled(run.attach.pending | semantic_tool.result.pending)`);
|
||||
- T18 and T19 (`cleanup_quarantined`).
|
||||
- Mechanism, per sub-code. This was corrected after review by the class R fix owner.
|
||||
- Native recovery does **not** depend on PID proofs. It uses `proveOwnerGone`, which accepts a
|
||||
replaced Railway deployment on an exclusive volume (`deployment_replaced`) or a stopped or fenced
|
||||
boot row. The only PID-only caller left is the runtime skill cache lock (#20688).
|
||||
- `native_remote_recovery_lease_mismatch … identity_run_id`: the leases were ephemeral and the
|
||||
sandboxes fresh, so a reused sandbox was not the cause. This is the shape that #20642 fixed. The
|
||||
last staging occurrence was 14:43Z on Nov 10.
|
||||
- `prior_owner_terminal_state_indeterminate` with unsettled `run.attach` or `semantic_tool.result`:
|
||||
fixed by #20677. Retirement is no longer held when the prior ephemeral sandbox is provably
|
||||
destroyed.
|
||||
- `runner_remote_process_identity_unavailable` is **still open**:
|
||||
1. The controller dies mid-turn while tools run.
|
||||
2. Recovery starts 4–7 minutes later.
|
||||
3. By then the remote Runner process is gone, or its `session.snapshot` times out.
|
||||
|
||||
Why the Runner exits despite its 60-minute connection lease is not yet known.
|
||||
- Last occurrences: `identity_unavailable` 2026-11-10T08:10Z, `lease_mismatch` 14:43Z, `prior_owner…`
|
||||
18:56Z and `cleanup_quarantined` 17:48Z.
|
||||
- Code pointers:
|
||||
- `server/src/services/native-runtime/native-session-executor.ts:14247` (identity wait);
|
||||
- `server/src/services/native-runtime/remote-recovery-lease-mismatch.ts`;
|
||||
- `server/src/services/heartbeat/restart-interrupted-turn.ts`.
|
||||
- Why the harness missed it:
|
||||
1. **Deploy timing:** a staging deploy sends a graceful SIGTERM, often 1–3 s after
|
||||
`native.coordinator.claim`, and the server stays down for 4–7 minutes. K2 restarts at once,
|
||||
and K4's long hold uses SIGKILL.
|
||||
2. **Missing adapter:** OpenCode (19 of 34) never ran in a Daytona cell.
|
||||
3. **Mid-turn Runner loss:** no Daytona workload keeps a tool running across the fault. The
|
||||
`tool` trigger covers only the local B5 shell bench, so the remote Runner is never under a
|
||||
running tool when the controller goes away.
|
||||
4. **Missing environment shape:** `--daytona-reuse-lease` and `--daytona-warm` are not in the
|
||||
matrix.
|
||||
- Scenario: **K15 redeploy** (implemented). The fault does a graceful SIGTERM like K2. The
|
||||
replacement then starts in a new emulated process space (a new hostname through a preload, so
|
||||
`processSpaceHash` changes) and with a new Railway-shaped deployment identity on the same volume.
|
||||
Selected with `--faults K15`. RED wave: see "Harness expansion" below.
|
||||
|
||||
### F: finalization and workspace save-back (24 runs, 8 tasks blocked)
|
||||
|
||||
- Examples:
|
||||
- T20 (`sync_out_retry_exhausted` from disk full, then the finalization budget, three times);
|
||||
- T21 (`remote_workspace_archive_unsafe_symlink`);
|
||||
- T22, T23 and T24 (`remote_workspace_recovery_receipt_invalid`);
|
||||
- T25 and T10 ("Recovering saved work…" phase cancelled with
|
||||
`execution_reconciliation_required`);
|
||||
- T26, T27 and T28 (07:26Z on Nov 9, right after a deploy).
|
||||
- Fixed: #20666 (skip unverifiable recovery sources) covers `receipt_invalid`, which last occurred
|
||||
at 22:35Z, before that fix was deployed. Disk-full and symlink save-back have no fix and no
|
||||
scenario.
|
||||
- Why the harness missed it: **missing workload.** The workloads write a few KB, never fill the
|
||||
sandbox disk and never create symlinks. ST faults break S3, not the sandbox file system.
|
||||
- Built: the **F1** workload (`--workloads F1`). The agent keeps an absolute symlink to `/etc/hosts`
|
||||
in its repository workspace. Wave `w-20261111T012523-s9304` (Daytona, `--storage s3`, K2, Codex
|
||||
and OpenCode) passed 2 of 2 on master. The save-back did not block the task. The harness keeps no
|
||||
evidence for passing tasks, so it is not verified that the link existed at save-back time.
|
||||
- Proposed (not built): `DF1`, which fills the sandbox disk to ENOSPC before result copy-back.
|
||||
|
||||
### E: Daytona capacity outage (18 runs, mostly self-healing)
|
||||
|
||||
- 2026-11-10 16:36–17:02Z:
|
||||
- `Daytona sandbox creation failed (HTTP 502)` (10);
|
||||
- `Worker process exited (code=null, signal=SIGKILL)` (3);
|
||||
- `Reusable sandbox lease … could not be resumed (cause=provider_error | transport_timeout)` (3).
|
||||
- Plus one `git=network` clone failure and one Daytona rate-limit failure.
|
||||
- Bounded retry recovered 8 of 13 tasks. 4 remain blocked (T29, T30, T31 and
|
||||
T32 are examples).
|
||||
- Fix #20626 (retry and replace a reusable sandbox that will not resume) shipped before the outage.
|
||||
Its effect at this scale is not verified.
|
||||
- Why the harness missed it: **missing fault type.** There is no environment-provider fault. All
|
||||
harness faults hit the control plane, S3 or the provider message stream.
|
||||
- Proposed: `EP1`, a proxy in front of the Daytona API that returns 502 or times out for a window.
|
||||
|
||||
### U: provider usage limit (17 runs)
|
||||
|
||||
- `provider_limit_wait` (10) is the designed wait.
|
||||
- The ACPX "terminal limit failure" (7, an error-triage agent) fails the run. It should wait. This is a product
|
||||
classification gap in the ACPX limit mapping.
|
||||
|
||||
### P: provider session resume after a deploy (16 runs, 11 tasks blocked)
|
||||
|
||||
- Examples:
|
||||
- T33, T34, T35 and T25 (`PRP provider resume state is unavailable`);
|
||||
- T36, T37, T38 and T39 (`runner_harness_state_mismatch …`);
|
||||
- T40 (`run.attach provider_side_effect_pending`).
|
||||
- Last occurrence: 2026-11-10T21:06Z.
|
||||
- #20627 (restore a warm backup when its source sandbox is idle) covers `backup_without_reusable_lease`
|
||||
and `resumed_unavailable`. `resume state is unavailable` and `provider_identity_incomplete` have no
|
||||
fix.
|
||||
- Why the harness missed it: the same reasons as R, plus ACPX (8) and OpenCode (5) on warm reusable
|
||||
sandboxes, which the matrix does not run.
|
||||
|
||||
### O: other (13 runs)
|
||||
|
||||
- Not deploy-correlated.
|
||||
- `native_run_failed` without a classified provider error appeared on T41, T42 and
|
||||
T43. Those runs reached `run.result.accepted` after two `turn.failed` events.
|
||||
- ACPX `session.open` was rejected on T44 and T45.
|
||||
- These are product triage items, not chaos gaps.
|
||||
|
||||
### B: authorization or binding readiness (13 runs)
|
||||
|
||||
- `reviewed_chat_execution_binding_not_authorized`: 7 runs on a PR-review agent (T30 and T46).
|
||||
- A credential not shared with the responsible user.
|
||||
- Skill package uploads still in flight (T47).
|
||||
- These are product authorization and readiness checks. They are outside the restart-chaos domain.
|
||||
|
||||
### L: legacy runtime lost across a deploy (12 runs)
|
||||
|
||||
- Examples:
|
||||
- `process_lost` on T26, T27 and T48 (07:11Z on Nov 9, three minutes after a
|
||||
deploy);
|
||||
- `Legacy controller lease lost` on T49, T50, T51 and T52 (08:14–08:19Z on
|
||||
Nov 10, 20 minutes after a deploy).
|
||||
- 7 of 12 tasks recovered.
|
||||
- The matrix has no legacy adapter cell. K15 could run one, but this was not done in this pass.
|
||||
|
||||
### A: artifact incompatible after an upgrade (8 runs)
|
||||
|
||||
- Examples: T53 and T54 (`OpenCode executable is missing from the provider pack`), T55
|
||||
(`ACPX launch profile omitted …`), T56, T57 and T58
|
||||
(`native_provider_artifacts_admission_unverified`), and T59 (`runner_revision_mismatch`).
|
||||
- Mechanism: a sandbox prepared and admitted by older code is resumed by newer code that expects
|
||||
new launch-profile fields or provider-pack contents.
|
||||
- Why the harness missed it: Daytona waves refuse to run unless the runner content ID equals the
|
||||
image (`daytona-image.ts`). This is deliberate, and it excludes exactly this mixed-version state.
|
||||
- Follow-up: K15 with `--upgrade-from` on Daytona, with the image of the older checkout.
|
||||
|
||||
### V: runner event commit stall (7 runs, likely fixed)
|
||||
|
||||
- Examples: T60 and T22 (Nov 9, 20:49–23:55Z).
|
||||
- #20385 (usage receipt stalls during streamed run updates) merged at 23:35Z, and the class has not
|
||||
occurred since.
|
||||
|
||||
## Harness expansion (this pass)
|
||||
|
||||
| Scenario | Classes | Selector | Status |
|
||||
| --- | --- | --- | --- |
|
||||
| K15 redeploy into a new process space and deployment identity | R, P, L (and A with `--upgrade-from`) | `--faults K15` | Implemented, unit-tested. Five Daytona waves are recorded below. |
|
||||
| EP1 environment-provider 5xx window | E | — | Proposed |
|
||||
| F1 absolute symlink kept in the repository workspace | F (unsafe-symlink save-back) | `--workloads F1` | Implemented, unit-tested. Daytona wave: 2 of 2 pass on master. |
|
||||
| DF1 sandbox disk full | F (ENOSPC save-back) | — | Proposed |
|
||||
|
||||
### Waves on master 8f6ca25976
|
||||
|
||||
All waves ran on Daytona image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:bc7ca92d…`
|
||||
(runner content `d7e4d82a…`).
|
||||
|
||||
| Wave | Args | Result |
|
||||
| --- | --- | --- |
|
||||
| `w-20261111T002313-s9301` | `--faults K15 --adapters opencode,claude --workloads A1,C1 --daytona-reuse-lease --concurrency 3 --tasks 6` | 5 of 6 pass. t05 (OpenCode, C1) failed R5 `silent_dead_run`: its run stayed `running` with no event, activity or start evidence for 183 s after the redeploy, then succeeded. No Retry was needed. |
|
||||
| `w-20261111T004241-s9301` (control) | the same seed and cells with `--faults K2` | 6 of 6 pass |
|
||||
| `w-20261111T005008-s9302` | `--faults K15 --triggers time,comment --adapters opencode,claude --workloads C1 --daytona-warm --concurrency 4 --tasks 4` | 4 of 4 pass |
|
||||
| `w-20261111T010000-s9303` | `--faults K15 --adapters opencode --workloads A1,C1 --daytona-reuse-lease --concurrency 4 --tasks 8` | 8 of 8 pass |
|
||||
| `w-20261111T011044-s9303` (control) | the same seed and cells with `--faults K2` | 8 of 8 pass |
|
||||
|
||||
What this shows:
|
||||
|
||||
- Under K15, 17 of 18 tasks passed. Under K2 with the same seeds, 14 of 14 passed. The one failure,
|
||||
a 183 s silent OpenCode run after a redeploy, occurred only under K15. One failure in 18 is not a
|
||||
stable reproduction.
|
||||
- The staging refusal codes of classes R and P did **not** reproduce on current master with K15
|
||||
alone. Examples are `identity_run_id` lease mismatch, `runner_remote_process_identity_unavailable`
|
||||
and `prior_owner_terminal_state_indeterminate`. Two explanations remain open:
|
||||
- The fixes merged during the window (#20625, #20631, #20639, #20650, #20662, #20666) cover them.
|
||||
Staging ran older revisions for most of the window.
|
||||
- They need conditions that K15 does not emulate yet:
|
||||
- a code change at the deploy, where new server code meets saved state and a sandbox runner from
|
||||
the older build (class A);
|
||||
- one sandbox reused across different tasks of one agent;
|
||||
- many agents and concurrent tasks under load.
|
||||
- The next scenario to build is **K15 with `--upgrade-from` on Daytona**: start with the older
|
||||
checkout and its image, redeploy to the current checkout, and keep the older runner in the
|
||||
retained sandboxes. Today the Daytona image check refuses that mixed state on purpose.
|
||||
@@ -292,6 +292,76 @@ receipt names both commits (`upgrade` in `faults.json`, `upgrade_switched` in th
|
||||
timeline). K14 is local-only, because a Daytona runner must match its image, and
|
||||
it is never part of a storm's default faults.
|
||||
|
||||
## Redeploy (K15, `redeploy.ts`)
|
||||
|
||||
```sh
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \
|
||||
--adapters opencode,claude --workloads A1,C1 --faults K15 --daytona-reuse-lease --concurrency 3 --tasks 6 --seed 9301
|
||||
```
|
||||
|
||||
The hosted staging stack deploys about every 40 minutes. Each deploy replaces
|
||||
the server's container. The replacement has the same database and volume, but it
|
||||
runs in another process space (host name, kernel boot and PID namespace) and
|
||||
has a new Railway deployment id. The server's PID-based proof that an earlier
|
||||
owner exited (`controller-process-identity.ts`, `owner-proof.ts`) never applies
|
||||
across a deploy. Recovery must use the deployment proof or the lease deadlines
|
||||
instead. K1 and K2 restart on the same machine, so every PID proof succeeds and
|
||||
that path stays untested. See `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`.
|
||||
|
||||
K15 emulates the container replacement without a container:
|
||||
|
||||
- Every server in a wave that has a K15 is launched with a `--import` preload
|
||||
(`redeploy-preload.mjs`). The preload makes `os.hostname()` return a
|
||||
per-generation name, which changes the process-space hash.
|
||||
- Every server gets Railway-shaped variables: `RAILWAY_PROJECT_ID`,
|
||||
`RAILWAY_ENVIRONMENT_ID`, `RAILWAY_SERVICE_ID` and `RAILWAY_DEPLOYMENT_ID`,
|
||||
plus `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH`, which is the
|
||||
instance home.
|
||||
- A K15 stops the server with SIGTERM, like K2. The replacement starts as the
|
||||
next generation, with a new host name and deployment id, on the same project,
|
||||
service and volume.
|
||||
- Other faults in the same wave restart within the current generation, like a
|
||||
platform restart of the same deployment.
|
||||
- Identities derive from the wave id; a replay uses the source wave id, so it sees the same ones.
|
||||
- Each K15 receipt records `redeploy` (both generations) in `faults.json`, and
|
||||
`redeploy_switched` in the timeline.
|
||||
|
||||
Limits:
|
||||
|
||||
- The Daytona sandboxes and remote runners survive, as they do on staging.
|
||||
- Local runners are stopped with the server's process tree.
|
||||
- The cloud stop-proof service (`PAPERCLIP_CLOUD_*`) is not emulated, so stop
|
||||
proofs that need it stay unsupported, as on any self-hosted Railway deploy.
|
||||
- K15 does not combine with MS1–MS3, because a second server would run in the
|
||||
real host's process space.
|
||||
- Combine K15 with `--adapters opencode` and `--daytona-reuse-lease` or
|
||||
`--daytona-warm` to match staging's workload mix.
|
||||
|
||||
## Absolute symlink in the workspace (F1)
|
||||
|
||||
```sh
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \
|
||||
--adapters codex,opencode --workloads F1 --faults K2 --storage s3 --concurrency 2 --tasks 2 --seed 9304
|
||||
```
|
||||
|
||||
F1 is opt-in. The agent creates `hosts-link-<nonce>`, which is an absolute
|
||||
symlink to `/etc/hosts`, in its repository workspace and keeps it. It also
|
||||
writes a notes file, and then marks the task done.
|
||||
|
||||
On staging, a remote save-back refused such a link
|
||||
(`remote_workspace_archive_unsafe_symlink`). The save-back then used up its
|
||||
retries and left the task blocked. Agents create absolute links in ordinary
|
||||
work, for example in tool caches or `node_modules/.bin`.
|
||||
|
||||
The oracle passes when:
|
||||
|
||||
- the task is `done`;
|
||||
- the notes file is saved, without a Retry.
|
||||
|
||||
How the product saves the link itself (skip it, keep it, or record it) is not
|
||||
graded. F1 is meaningful with a remote workspace (`--env daytona`, with
|
||||
`--storage s3` as on staging). Locally it is a plain repository task.
|
||||
|
||||
## Config faults (CF1–CF3, `config-wave.ts`)
|
||||
|
||||
```sh
|
||||
@@ -991,7 +1061,7 @@ re-arms the task's own triggered faults instead of using their offsets.
|
||||
| `grader.ts` | Pure R1–R6 grading over the samples, plus the universal checks. |
|
||||
| `universal.ts`, `universal-audit.ts` | U1–U4 on every task, and their audit over saved waves. |
|
||||
| `contention.ts` | L1 workload, its accounting oracle and the `--db-pressure` writers. |
|
||||
| `workloads.ts` | A1 brief document, B1 python CLI with tests, B5 long bench, C1 comment while running, M1 delegation, G1 approval across a restart, H1 budget hard stop across a restart, with their T-oracles. |
|
||||
| `workloads.ts` | A1 brief document, B1 python CLI with tests, B5 long bench, C1 comment while running, M1 delegation, G1 approval across a restart, H1 budget hard stop across a restart, F1 absolute symlink kept in the workspace, with their T-oracles. |
|
||||
| `journeys.ts`, `journey-driver.ts`, `journey-oauth-app.ts`, `journey-matrix.ts` | Q2 and Q3 definitions and oracles, the board's side of each, Q3's loopback OAuth/MCP app, and the `--matrix` planner. |
|
||||
| `evidence.ts`, `ledger.ts` | Evidence bundle for each failure and the JSONL ledger plus `SUMMARY.md`. |
|
||||
| `daytona-image.ts` | Docker-free extraction of the image's runnerd and provider pack, cached per digest, and the controller/image content-ID check. |
|
||||
@@ -1001,6 +1071,7 @@ re-arms the task's own triggered faults instead of using their offsets.
|
||||
| `time-faults.ts`, `time-faults-config.ts` | TM1–TM3: process freezing, database aging with recorded rows, and the time-corrected API the grader reads through. |
|
||||
| `multi-server.ts` | MS1–MS3: the second server's steps, the seeded holds, and the cross-server invariant over the wave database. |
|
||||
| `repeated-restarts.ts` | RR1–RR3: seeded restart steps, `--rr-gap-seconds`, and the stop-and-replace loop the supervisor runs. |
|
||||
| `redeploy.ts`, `redeploy-preload.mjs` | K15: per-generation emulated host name and Railway deployment identity, and the preload that applies the host name. |
|
||||
| `daytona-sweep.ts` | Daytona R6: read-only reconciliation of this wave's labelled sandboxes with leases; cleanup only through Paperclip. |
|
||||
|
||||
## Output (gitignored)
|
||||
|
||||
@@ -4,7 +4,7 @@ import { parseArgs } from "node:util";
|
||||
import { applyJourneyOptions } from "./journey-matrix.js";
|
||||
import { takesJourneyTriggers } from "./journeys.js";
|
||||
import { STORAGE_FAULT_KINDS, type StorageFaultKind } from "./storage-chaos.js";
|
||||
import { validateMultiServerOptions } from "./multi-server.js";
|
||||
import { isMultiServerFault, validateMultiServerOptions } from "./multi-server.js";
|
||||
import { parseTimeFaultOptions } from "./time-faults-config.js";
|
||||
import { parseStorageTrigger } from "./storage-trigger.js";
|
||||
import { parseNativeTrigger, parseNativeTriggerOutage } from "./native-trigger.js";
|
||||
@@ -17,7 +17,10 @@ import {
|
||||
type AdapterId, type EnvironmentKind, type FaultKind, type TriggerId, type WaveOptions, type WorkloadId,
|
||||
} from "./types.js";
|
||||
|
||||
/** A storm's default faults: every restart kind except K14, which needs an older build (`--upgrade-from`). */
|
||||
/**
|
||||
* A storm's default faults: every restart kind except K14, which needs an older build (`--upgrade-from`),
|
||||
* and K15, which changes the deploy identity of every later server (pass `--faults` to include it).
|
||||
*/
|
||||
export const STORM_DEFAULT_FAULTS: FaultKind[] = ["K1", "K2", "K4", "K6"];
|
||||
|
||||
export const CREDENTIAL_FOR_ADAPTER: Record<AdapterId, "OPENAI_API_KEY" | "ANTHROPIC_API_KEY" | "OPENROUTER_API_KEY"> = {
|
||||
@@ -176,7 +179,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
env,
|
||||
adapters,
|
||||
// The governance scenarios (G1, H1), the journeys (Q1–Q3) and the contention run (L1) are opt-in: H1 needs a wave of its own.
|
||||
workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1"
|
||||
workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1" && id !== "F1"
|
||||
&& !takesJourneyTriggers(id) && (id !== "M1" || adapters.length >= 2))),
|
||||
// A crash-point wave schedules no fault; K1 is the restart its self-kill gets.
|
||||
faults: allFaults.filter((fault): fault is FaultKind => !isProviderFault(fault)),
|
||||
@@ -233,6 +236,10 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults");
|
||||
}
|
||||
validateMultiServerOptions(options);
|
||||
// K15 emulates one deploy identity per generation; a second server beside it would run in the real host's process space.
|
||||
if (options.faults.includes("K15") && options.faults.some(isMultiServerFault)) {
|
||||
throw new Error("K15 (redeploy) does not combine with MS1-MS3: the second server would not share the emulated deploy identity");
|
||||
}
|
||||
Object.assign(options, parseTimeFaultOptions(options, values["time-age-hours"]));
|
||||
Object.assign(options, parseRepeatedRestartOptions(options, values["rr-gap-seconds"], values["rr-stop-on"]));
|
||||
const upgradeFrom = values["upgrade-from"];
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { parseCommand } from "./config.js";
|
||||
import { WORKLOAD_TRIGGERS } from "./journeys.js";
|
||||
import { planTasks } from "./scheduler.js";
|
||||
import type { OracleArtifacts } from "./types.js";
|
||||
import { f1LinkName, workloadDefinitions } from "./workloads.js";
|
||||
|
||||
const NONCE = "abc123de";
|
||||
const LISTING = `lrwxrwxrwx 1 daytona daytona 10 Nov 11 01:30 hosts-link-${NONCE} -> /etc/hosts`;
|
||||
const notes = (text: string) => ({ [`NOTES-${NONCE}.md`]: text });
|
||||
const artifacts = (overrides: Partial<OracleArtifacts> = {}): OracleArtifacts => ({
|
||||
issueStatus: "done", documents: {}, agentComments: [], files: notes(`F1-${NONCE}\n${LISTING}\n`), commands: {},
|
||||
workspacePath: "/workspace/chaos", ...overrides,
|
||||
}) as OracleArtifacts;
|
||||
|
||||
describe("F1: absolute symlink kept in the repository workspace", () => {
|
||||
const f1 = workloadDefinitions.F1;
|
||||
|
||||
it("asks for the absolute link and the notes file in a repository workspace", () => {
|
||||
expect(f1.usesRepository).toBe(true);
|
||||
const text = f1.description(NONCE);
|
||||
expect(text).toContain(`ln -s /etc/hosts ${f1LinkName(NONCE)}`);
|
||||
expect(text).toContain(`NOTES-${NONCE}.md`);
|
||||
expect(f1.collect.files(NONCE)).toEqual([`NOTES-${NONCE}.md`]);
|
||||
});
|
||||
|
||||
it("passes only when the task is done and its saved notes file is in the workspace", () => {
|
||||
expect(f1.grade(NONCE, artifacts()).verdict).toBe("pass");
|
||||
expect(f1.grade(NONCE, artifacts({ issueStatus: "blocked" })).verdict).toBe("fail");
|
||||
expect(f1.grade(NONCE, artifacts({ files: {} })).verdict).toBe("fail");
|
||||
expect(f1.grade(NONCE, artifacts({ workspacePath: null })).verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("needs the agent's ls -l line proving the absolute link existed before the save-back", () => {
|
||||
expect(f1.description(NONCE)).toContain(`ls -l ${f1LinkName(NONCE)}`);
|
||||
// Skipped link: notes only.
|
||||
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n`) })).verdict).toBe("fail");
|
||||
// A regular file of the same name, or a link elsewhere, is not the link under test.
|
||||
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n-rw-r--r-- 1 d d 3 Nov 11 01:30 hosts-link-${NONCE}\n`) })).verdict).toBe("fail");
|
||||
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n${LISTING.replace("/etc/hosts", "etc/hosts")}\n`) })).verdict).toBe("fail");
|
||||
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n${LISTING.replace(NONCE, "ffffffff")}\n`) })).verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("is opt-in, time-triggered, and plans with restart faults on Daytona", () => {
|
||||
expect(WORKLOAD_TRIGGERS.F1).toEqual(["time"]);
|
||||
const defaults = parseCommand(["--seed", "1"]);
|
||||
expect(defaults.mode === "wave" && defaults.options.workloads).not.toContain("F1");
|
||||
const parsed = parseCommand(["--env", "daytona", "--daytona-image", `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"b".repeat(64)}`,
|
||||
"--workloads", "F1", "--faults", "K2,K15", "--adapters", "codex", "--tasks", "2", "--seed", "3"]);
|
||||
if (parsed.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(planTasks(parsed.options).map(task => task.workload)).toEqual(["F1", "F1"]);
|
||||
});
|
||||
});
|
||||
@@ -35,7 +35,7 @@ export type JourneyTrigger = (typeof JOURNEY_TRIGGERS)[number];
|
||||
/** Each workload's own fault boundaries, for the journey × fault matrix. */
|
||||
export const WORKLOAD_TRIGGERS: Record<WorkloadId, readonly TriggerId[]> = {
|
||||
A1: ["time"], B1: ["time"], B5: ["tool"], C1: ["comment", "result"], M1: ["time"], G1: ["approval"],
|
||||
H1: ["paused"], R1: ["time"], S1: ["stop"], L1: ["time"], Q1: JOURNEY_TRIGGERS, Q2: JOURNEY_TRIGGERS, Q3: JOURNEY_TRIGGERS,
|
||||
H1: ["paused"], R1: ["time"], S1: ["stop"], L1: ["time"], F1: ["time"], Q1: JOURNEY_TRIGGERS, Q2: JOURNEY_TRIGGERS, Q3: JOURNEY_TRIGGERS,
|
||||
};
|
||||
|
||||
/** Q3 reviews must survive a delay longer than the implicit deadline #20340 removed. */
|
||||
|
||||
@@ -38,6 +38,7 @@ import { collectWaveSpeed, speedMarkdown } from "./wave-speed.js";
|
||||
import { TaskObserver, childSpeedTasksInTurn } from "./observer.js";
|
||||
import { closeJourneyApps } from "./journey-driver.js";
|
||||
import { matrixMayLaunch, planMatrixWave } from "./journey-matrix.js";
|
||||
import { redeployWave } from "./redeploy.js";
|
||||
import { StormRun, TASK_FAULT_LOG, observedFaultSchedule, planWave, replayOnCurrentBuild, replayRecordFor, replayWave, type TaskFaultLog } from "./scheduler.js";
|
||||
import { chaosLifecycleOverrides, seedChaosLab } from "./seed.js";
|
||||
import { ChaosSupervisor } from "./supervisor.js";
|
||||
@@ -200,7 +201,9 @@ async function main(): Promise<number> {
|
||||
const batchSize = crashPoints.length ? Math.max(1, Math.floor(plan.tasks.length / crashPoints.length)) : 0;
|
||||
if (crashPoint && plan.tasks.length !== batchSize * crashPoints.length) throw new Error(`--tasks must be a multiple of the ${crashPoints.length} crash point(s)`);
|
||||
const supervisor = await ChaosSupervisor.create({ repositoryRoot, waveDir, timeline,
|
||||
...(plan.options.upgradeFrom ?? plan.options.serverFrom ? { upgradeFrom: plan.options.upgradeFrom ?? plan.options.serverFrom } : {}) });
|
||||
...(plan.options.upgradeFrom ?? plan.options.serverFrom ? { upgradeFrom: plan.options.upgradeFrom ?? plan.options.serverFrom } : {}),
|
||||
// K15: every server of the wave runs under an emulated deploy identity; a replay derives the source wave's ones.
|
||||
...(redeployWave(plan) ? { redeployWaveId: plan.replayOf?.waveId ?? plan.waveId } : {}) });
|
||||
// Crash-point suite: one booted server, one point at a time. Each point gets
|
||||
// its own batch of tasks; the point is armed once the batch launched and the
|
||||
// server kills itself at its nth hit (restarted like K1). Invariants are
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
// K15 (redeploy.ts): give the server process an emulated host name, so its process-space hash
|
||||
// differs from the previous deploy's, as a replaced container's does. Loaded with `--import`
|
||||
// before the server. Node children forked with the server's execArgv inherit it, as they share its container.
|
||||
import os from "node:os";
|
||||
import { syncBuiltinESMExports } from "node:module";
|
||||
|
||||
const hostname = process.env.RESTART_CHAOS_REDEPLOY_HOSTNAME?.trim();
|
||||
if (hostname) {
|
||||
os.hostname = () => hostname;
|
||||
// `import { hostname } from "node:os"` reads the ESM export, which only follows the CJS object after a sync.
|
||||
syncBuiltinESMExports();
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { parseCommand } from "./config.js";
|
||||
import {
|
||||
REDEPLOY_HOSTNAME_ENV, REDEPLOY_PRELOAD_PATH, nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs,
|
||||
redeployReceipt, redeployWave,
|
||||
} from "./redeploy.js";
|
||||
import { planTasks } from "./scheduler.js";
|
||||
import { FAULT_SIGNAL } from "./supervisor.js";
|
||||
import type { WavePlan } from "./types.js";
|
||||
|
||||
const UUID = /^[a-f0-9]{8}-[a-f0-9]{4}-5[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}$/;
|
||||
const IMAGE = `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"a".repeat(64)}`;
|
||||
|
||||
describe("K15 redeploy identity", () => {
|
||||
it("keeps the service and volume and changes the host and deployment on every generation", () => {
|
||||
const first = redeployIdentity("w-1", 0);
|
||||
const second = nextRedeployIdentity("w-1", first);
|
||||
expect(second.generation).toBe(1);
|
||||
expect(second.hostname).not.toBe(first.hostname);
|
||||
expect(second.deploymentId).not.toBe(first.deploymentId);
|
||||
for (const key of ["projectId", "environmentId", "serviceId", "volumeId"] as const) expect(second[key]).toBe(first[key]);
|
||||
for (const id of [first.projectId, first.environmentId, first.serviceId, first.deploymentId, first.volumeId]) expect(id).toMatch(UUID);
|
||||
});
|
||||
|
||||
it("is deterministic per wave, so a replay derives the same identities", () => {
|
||||
expect(redeployIdentity("w-1", 3)).toEqual(redeployIdentity("w-1", 3));
|
||||
expect(redeployIdentity("w-2", 0).deploymentId).not.toBe(redeployIdentity("w-1", 0).deploymentId);
|
||||
expect(() => redeployIdentity("w-1", -1)).toThrow(/non-negative/);
|
||||
});
|
||||
|
||||
it("names the Railway variables the server reads as its deployment, with the instance home as the exclusive volume", () => {
|
||||
const identity = redeployIdentity("w-1", 2);
|
||||
expect(redeployEnvironment(identity, "/tmp/chaos/home")).toEqual({
|
||||
[REDEPLOY_HOSTNAME_ENV]: identity.hostname,
|
||||
RAILWAY_PROJECT_ID: identity.projectId, RAILWAY_ENVIRONMENT_ID: identity.environmentId, RAILWAY_SERVICE_ID: identity.serviceId,
|
||||
RAILWAY_DEPLOYMENT_ID: identity.deploymentId, RAILWAY_VOLUME_ID: identity.volumeId, RAILWAY_VOLUME_MOUNT_PATH: "/tmp/chaos/home",
|
||||
});
|
||||
expect(redeployReceipt(redeployIdentity("w-1", 0), identity)).toMatchObject({ fromGeneration: 0, toGeneration: 2,
|
||||
toDeploymentId: identity.deploymentId, volumeId: identity.volumeId });
|
||||
});
|
||||
|
||||
it("loads the preload ahead of the server's own --import loader", () => {
|
||||
expect(redeployNodeArgs(["--import", "/x/tsx/loader.mjs", "entry.ts"])).toEqual(["--import", REDEPLOY_PRELOAD_PATH, "--import", "/x/tsx/loader.mjs", "entry.ts"]);
|
||||
});
|
||||
|
||||
it("changes os.hostname for both the CommonJS object and ESM named imports, as processSpaceHash reads it", () => {
|
||||
const script = "import os from 'node:os'; import { hostname } from 'node:os'; console.log(JSON.stringify([os.hostname(), hostname()]));";
|
||||
const run = (env: NodeJS.ProcessEnv) => spawnSync(process.execPath, ["--import", REDEPLOY_PRELOAD_PATH, "--input-type=module", "-e", script],
|
||||
{ env: { PATH: process.env.PATH, ...env }, encoding: "utf8" });
|
||||
const emulated = run({ [REDEPLOY_HOSTNAME_ENV]: "chaos-deploy-test" });
|
||||
expect(emulated.status).toBe(0);
|
||||
expect(JSON.parse(emulated.stdout)).toEqual(["chaos-deploy-test", "chaos-deploy-test"]);
|
||||
// Without the variable the preload changes nothing.
|
||||
const real = run({});
|
||||
expect(real.status).toBe(0);
|
||||
const [cjs, esm] = JSON.parse(real.stdout) as string[];
|
||||
expect(cjs).toBe(esm);
|
||||
expect(cjs).not.toBe("chaos-deploy-test");
|
||||
});
|
||||
});
|
||||
|
||||
describe("K15 wave options", () => {
|
||||
it("is a graceful stop, selectable on local and Daytona like K2", () => {
|
||||
expect(FAULT_SIGNAL.K15).toBe("SIGTERM");
|
||||
const local = parseCommand(["--faults", "K15", "--adapters", "opencode", "--workloads", "A1,C1", "--seed", "1"]);
|
||||
expect(local.mode === "wave" && local.options.faults).toEqual(["K15"]);
|
||||
const daytona = parseCommand(["--env", "daytona", "--daytona-image", IMAGE, "--faults", "K2,K15", "--adapters", "opencode",
|
||||
"--workloads", "C1", "--daytona-reuse-lease", "--seed", "1"]);
|
||||
if (daytona.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(daytona.options.faults).toEqual(["K2", "K15"]);
|
||||
expect(daytona.options.daytonaReuseLease).toBe(true);
|
||||
});
|
||||
|
||||
it("refuses a second server beside an emulated deploy", () => {
|
||||
expect(() => parseCommand(["--faults", "K15,MS1", "--workloads", "B5,Q1", "--concurrency", "1", "--seed", "1"])).toThrow(/K15 \(redeploy\) does not combine with MS1-MS3/);
|
||||
});
|
||||
|
||||
it("plans K15 cells with no hold and marks the wave as a redeploy wave", () => {
|
||||
const parsed = parseCommand(["--faults", "K15", "--adapters", "codex,opencode", "--workloads", "A1,C1", "--tasks", "4", "--seed", "7"]);
|
||||
if (parsed.mode !== "wave") throw new Error("expected a wave");
|
||||
const tasks = planTasks(parsed.options);
|
||||
expect(tasks.every(task => task.fault === "K15" && task.holdMs === 0)).toBe(true);
|
||||
const plan = { options: parsed.options, tasks } as Pick<WavePlan, "options" | "tasks" | "storm">;
|
||||
expect(redeployWave(plan)).toBe(true);
|
||||
const k2 = parseCommand(["--faults", "K2", "--workloads", "A1", "--tasks", "2", "--seed", "7"]);
|
||||
if (k2.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(redeployWave({ options: k2.options, tasks: planTasks(k2.options) })).toBe(false);
|
||||
// A replay of a K2 task that a K15 crossed still needs the emulated identity.
|
||||
expect(redeployWave({ options: k2.options, tasks: [{ ...planTasks(k2.options)[0]!,
|
||||
replayFaults: [{ kind: "K15", holdMs: 0, atMs: 1_000 }] }] })).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,96 @@
|
||||
/**
|
||||
* K15 (redeploy): what a hosted deploy does to the control plane, beyond K2's graceful stop.
|
||||
*
|
||||
* On the hosted staging stack every deploy replaces the server's container. The replacement runs
|
||||
* in another process space (host name, kernel boot and PID namespace), so the server's PID-based
|
||||
* "the previous owner has exited" proofs (`server/src/services/controller-process-identity.ts`,
|
||||
* `owner-proof.ts`) never apply. It also carries a new Railway deployment id on the same exclusive
|
||||
* volume, which is the proof hosted recovery uses instead. A K1/K2 restart on one machine keeps the
|
||||
* process space, so every PID proof succeeds there and this path is never exercised.
|
||||
*
|
||||
* K15 emulates both changes without a container: each server of a K15 wave sees an emulated host
|
||||
* name (a `--import` preload replaces `os.hostname`, which changes the process-space hash) and
|
||||
* Railway-shaped deployment variables naming one stable project, environment, service and volume
|
||||
* and a deployment id per generation. A K15 stops the server with SIGTERM like K2 and starts the
|
||||
* replacement as the next generation. Other faults in the same wave restart within a generation,
|
||||
* like a platform restart of the same deployment.
|
||||
*/
|
||||
import { createHash } from "node:crypto";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import type { WavePlan } from "./types.js";
|
||||
|
||||
/** Whether any server of this wave can be redeployed: a planned, storm or replayed K15. */
|
||||
export function redeployWave(plan: Pick<WavePlan, "options" | "tasks" | "storm">): boolean {
|
||||
return plan.options.faults.includes("K15") || (plan.storm?.faults.some(fault => fault.kind === "K15") ?? false)
|
||||
|| plan.tasks.some(task => task.fault === "K15" || (task.replayFaults?.some(fault => fault.kind === "K15") ?? false));
|
||||
}
|
||||
|
||||
/** The preload that applies the emulated host name before the server loads. */
|
||||
export const REDEPLOY_PRELOAD_PATH = fileURLToPath(new URL("./redeploy-preload.mjs", import.meta.url));
|
||||
/** Read by the preload: the host name `os.hostname()` returns in the server process. */
|
||||
export const REDEPLOY_HOSTNAME_ENV = "RESTART_CHAOS_REDEPLOY_HOSTNAME";
|
||||
|
||||
export interface RedeployIdentity {
|
||||
/** 0 for the wave's first server; each K15 adds one. */
|
||||
generation: number;
|
||||
hostname: string;
|
||||
projectId: string;
|
||||
environmentId: string;
|
||||
serviceId: string;
|
||||
deploymentId: string;
|
||||
volumeId: string;
|
||||
}
|
||||
|
||||
/** A deterministic UUID-shaped id for one wave, role and generation (no randomness: replays match). */
|
||||
function uuidFor(waveId: string, role: string, generation = 0): string {
|
||||
const hex = createHash("sha256").update(`restart-chaos-redeploy:${waveId}:${role}:${generation}`).digest("hex");
|
||||
// Version 5-style layout so validators that require a UUID shape accept it.
|
||||
const variant = ((parseInt(hex[16]!, 16) & 0x3) | 0x8).toString(16);
|
||||
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-5${hex.slice(13, 16)}-${variant}${hex.slice(17, 20)}-${hex.slice(20, 32)}`;
|
||||
}
|
||||
|
||||
/** The identity of the server that serves generation `generation` of this wave. */
|
||||
export function redeployIdentity(waveId: string, generation: number): RedeployIdentity {
|
||||
if (!Number.isInteger(generation) || generation < 0) throw new Error(`Redeploy generation must be a non-negative integer, got ${generation}`);
|
||||
return {
|
||||
generation,
|
||||
hostname: `chaos-deploy-${createHash("sha256").update(`${waveId}:${generation}`).digest("hex").slice(0, 10)}`,
|
||||
projectId: uuidFor(waveId, "project"),
|
||||
environmentId: uuidFor(waveId, "environment"),
|
||||
serviceId: uuidFor(waveId, "service"),
|
||||
deploymentId: uuidFor(waveId, "deployment", generation),
|
||||
volumeId: uuidFor(waveId, "volume"),
|
||||
};
|
||||
}
|
||||
|
||||
/** The next deploy of the same service: same project, environment, service and volume. */
|
||||
export function nextRedeployIdentity(waveId: string, current: RedeployIdentity): RedeployIdentity {
|
||||
return redeployIdentity(waveId, current.generation + 1);
|
||||
}
|
||||
|
||||
/**
|
||||
* Server environment for one generation. The volume mount is the instance home: on staging the
|
||||
* instance directory lives on the exclusive volume that survives each deploy.
|
||||
*/
|
||||
export function redeployEnvironment(identity: RedeployIdentity, volumeMountPath: string): Record<string, string> {
|
||||
return {
|
||||
[REDEPLOY_HOSTNAME_ENV]: identity.hostname,
|
||||
RAILWAY_PROJECT_ID: identity.projectId,
|
||||
RAILWAY_ENVIRONMENT_ID: identity.environmentId,
|
||||
RAILWAY_SERVICE_ID: identity.serviceId,
|
||||
RAILWAY_DEPLOYMENT_ID: identity.deploymentId,
|
||||
RAILWAY_VOLUME_ID: identity.volumeId,
|
||||
RAILWAY_VOLUME_MOUNT_PATH: volumeMountPath,
|
||||
};
|
||||
}
|
||||
|
||||
/** Node arguments that load the preload ahead of the server entry's own `--import` loader. */
|
||||
export function redeployNodeArgs(args: readonly string[]): string[] {
|
||||
return ["--import", REDEPLOY_PRELOAD_PATH, ...args];
|
||||
}
|
||||
|
||||
/** What a K15 receipt records: both generations, so a replay and the evidence name them. */
|
||||
export function redeployReceipt(from: RedeployIdentity, to: RedeployIdentity) {
|
||||
return { fromGeneration: from.generation, toGeneration: to.generation, fromHostname: from.hostname, toHostname: to.hostname,
|
||||
fromDeploymentId: from.deploymentId, toDeploymentId: to.deploymentId, volumeId: to.volumeId };
|
||||
}
|
||||
@@ -39,6 +39,7 @@ import { isRepeatedRestartFault, newRepeatedRestartRecord, recordFirstRestart, r
|
||||
type RepeatedRestartRecord, type RestartStep } from "./repeated-restarts.js";
|
||||
import type { FaultKind, FaultReceipt, TriggerId } from "./types.js";
|
||||
import { isMultiServerFault, runMultiServerFault, type MultiServerKind, type MultiServerRecord, type ServerProcessWindow } from "./multi-server.js";
|
||||
import { nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs, redeployReceipt, type RedeployIdentity } from "./redeploy.js";
|
||||
|
||||
/** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */
|
||||
const HEALTH_TIMEOUT_MS = (() => {
|
||||
@@ -48,7 +49,7 @@ const HEALTH_TIMEOUT_MS = (() => {
|
||||
const READY_TIMEOUT_MS = 300_000;
|
||||
const STOP_TIMEOUT_MS = 30_000;
|
||||
|
||||
export const FAULT_SIGNAL: Record<FaultKind, FaultReceipt["signal"]> = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM",
|
||||
export const FAULT_SIGNAL: Record<FaultKind, FaultReceipt["signal"]> = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM", K15: "SIGTERM",
|
||||
TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL", RR1: "SIGKILL", RR2: "SIGKILL", RR3: "SIGKILL",
|
||||
// MS1 leaves the primary alone; MS2 freezes it; MS3 stops it gracefully (multi-server.ts).
|
||||
MS1: "none", MS2: "SIGSTOP", MS3: "SIGTERM" };
|
||||
@@ -154,13 +155,16 @@ export class ChaosSupervisor {
|
||||
private serverRoot: string;
|
||||
/** The environment overrides the wave passed to `start`, kept for every restart and for the K14 switch. */
|
||||
private extraEnvironment: Record<string, string> = {};
|
||||
/** K15 waves: the wave id the emulated deploy identities derive from, and the generation serving now. */
|
||||
private redeploy: { waveId: string; identity: RedeployIdentity } | null = null;
|
||||
/** Wave secrets, kept across a K14 switch as an in-place upgrade keeps its configuration. */
|
||||
private readonly secrets = { PAPERCLIP_AGENT_JWT_SECRET: secret(), PAPERCLIP_DECISION_SIGNING_SECRET: secret(),
|
||||
PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: secret(), BETTER_AUTH_SECRET: secret() };
|
||||
|
||||
private constructor(readonly repositoryRoot: string, readonly port: number, readonly paths: SupervisorPaths,
|
||||
private readonly timeline: Timeline, upgradeFrom: string | null) {
|
||||
private readonly timeline: Timeline, upgradeFrom: string | null, redeployWaveId: string | null) {
|
||||
this.serverRoot = upgradeFrom ?? repositoryRoot;
|
||||
if (redeployWaveId) this.redeploy = { waveId: redeployWaveId, identity: redeployIdentity(redeployWaveId, 0) };
|
||||
this.baseUrl = `http://127.0.0.1:${port}`;
|
||||
this.log = createWriteStream(paths.serverLogPath, { flags: "a", mode: 0o600 });
|
||||
this.stopper = createRunnerE2EServerStopper({
|
||||
@@ -171,13 +175,16 @@ export class ChaosSupervisor {
|
||||
});
|
||||
}
|
||||
|
||||
static async create(input: { repositoryRoot: string; waveDir: string; timeline: Timeline; upgradeFrom?: string }): Promise<ChaosSupervisor> {
|
||||
/** `redeployWaveId`: a K15 wave; every server gets the emulated deploy identity of its generation (redeploy.ts). */
|
||||
static async create(input: { repositoryRoot: string; waveDir: string; timeline: Timeline; upgradeFrom?: string;
|
||||
redeployWaveId?: string }): Promise<ChaosSupervisor> {
|
||||
const temporaryRoot = await createRunnerE2ETemporaryRoot();
|
||||
const home = path.join(temporaryRoot, "home");
|
||||
const paths: SupervisorPaths = { temporaryRoot, home, configPath: path.join(home, "instances", "default", "config.json"),
|
||||
serverLogPath: path.join(input.waveDir, "server.log") };
|
||||
const port = await reserveRunnerE2EServerPort();
|
||||
const supervisor = new ChaosSupervisor(input.repositoryRoot, port, paths, input.timeline, input.upgradeFrom ?? null);
|
||||
const supervisor = new ChaosSupervisor(input.repositoryRoot, port, paths, input.timeline, input.upgradeFrom ?? null,
|
||||
input.redeployWaveId ?? null);
|
||||
input.timeline.add("supervisor_created", { temporaryRoot, port, ...(input.upgradeFrom
|
||||
? { upgradeFrom: input.upgradeFrom, upgradeFromSha: checkoutSha(input.upgradeFrom) } : {}) });
|
||||
return supervisor;
|
||||
@@ -241,6 +248,8 @@ export class ChaosSupervisor {
|
||||
PAPERCLIP_IN_WORKTREE: "false",
|
||||
...waveServerUrlEnvironment(this.port),
|
||||
...extraEnvironment,
|
||||
// K15: the generation's emulated host name and Railway deployment identity; the volume is the instance home.
|
||||
...(this.redeploy ? redeployEnvironment(this.redeploy.identity, home) : {}),
|
||||
};
|
||||
const environment = buildPaperclipServerEnvironment(source, overrides);
|
||||
const workspaceStorage = extraEnvironment.PAPERCLIP_STORAGE_PROVIDER === "s3"
|
||||
@@ -257,7 +266,9 @@ export class ChaosSupervisor {
|
||||
const control = this.crashPointControl;
|
||||
const env = control ? { ...this.environment, PAPERCLIP_CRASH_POINT_CONTROL: control.controlPath,
|
||||
PAPERCLIP_CRASH_POINT_RECEIPT: control.receiptPath } : this.environment;
|
||||
const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]),
|
||||
const args = runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]);
|
||||
// K15 waves: the emulated host name of the serving deploy generation (redeploy.ts).
|
||||
const child = spawn(process.execPath, this.redeploy ? redeployNodeArgs(args) : args,
|
||||
{ cwd: root, env, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached });
|
||||
this.child = child;
|
||||
const window = this.trackWindow(child, "primary");
|
||||
@@ -484,6 +495,8 @@ export class ChaosSupervisor {
|
||||
receipt.restartedAt = Date.now();
|
||||
// K14: the replacement is the current build, on the same database and instance directory.
|
||||
if (kind === "K14") receipt.upgrade = await this.switchToCurrentBuild(receipt.faultId);
|
||||
// K15: the replacement is the next deploy: a new process space and deployment id on the same volume.
|
||||
if (kind === "K15") receipt.redeploy = this.advanceDeploy(receipt.faultId);
|
||||
this.timeline.add("fault_restarting", { faultId: receipt.faultId });
|
||||
return this.spawnServer();
|
||||
},
|
||||
@@ -668,6 +681,19 @@ export class ChaosSupervisor {
|
||||
throw new Error(`Second server did not report ready within ${READY_TIMEOUT_MS}ms`);
|
||||
}
|
||||
|
||||
/** K15: move the next server to the next deploy generation (redeploy.ts). */
|
||||
private advanceDeploy(faultId: string): NonNullable<FaultReceipt["redeploy"]> {
|
||||
if (!this.redeploy || !this.environment) throw new Error("K15 needs a redeploy wave (ChaosSupervisor.create redeployWaveId)");
|
||||
const from = this.redeploy.identity;
|
||||
const to = nextRedeployIdentity(this.redeploy.waveId, from);
|
||||
this.redeploy.identity = to;
|
||||
Object.assign(this.environment, redeployEnvironment(to, this.paths.home));
|
||||
const receipt = redeployReceipt(from, to);
|
||||
this.timeline.add("redeploy_switched", { faultId, ...receipt });
|
||||
this.log.write(`\n[restart-chaos] ${faultId} K15 redeploy ${from.hostname}/${from.deploymentId} -> ${to.hostname}/${to.deploymentId}\n`);
|
||||
return receipt;
|
||||
}
|
||||
|
||||
/**
|
||||
* K14: move the server from the older build to this checkout. Only the first K14 switches; a later
|
||||
* one restarts the current build (recorded with `switched: false`).
|
||||
|
||||
@@ -6,11 +6,13 @@ import type { TimeFaultRecord } from "./time-faults.js";
|
||||
|
||||
/**
|
||||
* K14 (upgrade mid-run): a graceful stop, then the replacement starts from the current build (`--upgrade-from`).
|
||||
* K15 (redeploy, redeploy.ts): a graceful stop, then the replacement starts in a new emulated process space with a new
|
||||
* Railway-shaped deployment id on the same volume, as a hosted deploy replaces the server's container.
|
||||
* TM1–TM3 (time faults, time-faults.ts): freeze the server, sleep with aged rows, or an aged outage.
|
||||
* RR1–RR3 (repeated restarts, repeated-restarts.ts): a K1, then more restarts while the replacement recovers.
|
||||
* MS1–MS3 (two servers, one database, multi-server.ts): a second server beside the first.
|
||||
*/
|
||||
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3"] as const;
|
||||
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "K15", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3"] as const;
|
||||
export type FaultKind = (typeof FAULT_KINDS)[number];
|
||||
/** The shortest seeded K4 outage (scheduler.ts K4_HOLD_RANGE_MS), also the K4 hold after a provider fault. */
|
||||
export const K4_MIN_HOLD_MS = 120_000;
|
||||
@@ -25,8 +27,9 @@ export type TaskFaultKind = FaultKind | "K13" | "K0" | ProviderFaultKind;
|
||||
* G1: an approval-gated action across a restart. H1: a budget hard stop crossed mid-run with a restart (its own wave).
|
||||
* Q1: a saved question (question-handoff.ts). Q2, Q3: journeys.ts.
|
||||
* L1: high-output, high-accounting run for database contention (contention.ts).
|
||||
* F1: an absolute symlink pointing out of the repository workspace, kept at rest (staging save-back class F).
|
||||
*/
|
||||
export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3"] as const;
|
||||
export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3", "F1"] as const;
|
||||
export type WorkloadId = (typeof WORKLOAD_IDS)[number];
|
||||
|
||||
export const ADAPTER_IDS = ["codex", "claude", "opencode"] as const;
|
||||
@@ -281,6 +284,8 @@ export interface FaultReceipt {
|
||||
crashPoint?: { name: string; hit: number; pid: number; token?: string };
|
||||
/** K14: the build the stopped server ran and the build its replacement runs. */
|
||||
upgrade?: { fromRoot: string; fromSha: string | null; toRoot: string; toSha: string | null; switched: boolean };
|
||||
/** K15: the deploy generation, emulated host name and deployment id before and after. */
|
||||
redeploy?: ReturnType<typeof import("./redeploy.js").redeployReceipt>;
|
||||
/** TM1–TM3: what was frozen and which rows were aged. */
|
||||
timeFault?: TimeFaultRecord;
|
||||
/** RR1–RR3: every stop and replacement of the sequence, the first kill included. */
|
||||
|
||||
@@ -41,6 +41,10 @@ const briefMarker = (nonce: string) => `CHAOS-${nonce}`;
|
||||
const followUpMarker = (nonce: string) => `FOLLOWUP-${nonce}`;
|
||||
const continueMarker = (nonce: string) => `CONTINUE-${nonce}`;
|
||||
const notesFile = (nonce: string) => `NOTES-${nonce}.md`;
|
||||
/** F1: the absolute symlink the agent creates and keeps in the repository root. */
|
||||
export const f1LinkName = (nonce: string) => `hosts-link-${nonce}`;
|
||||
/** F1: an `ls -l` line for that link: a symlink (`l` mode) naming the link and its absolute target. */
|
||||
export const f1LinkListing = (nonce: string) => new RegExp(`^l\\S*\\s.*\\s${f1LinkName(nonce)} -> /etc/hosts$`);
|
||||
const benchFile = (nonce: string) => `bench-result-${nonce}.txt`;
|
||||
const benchRecord = (nonce: string) => `.bench/run-${nonce}.txt`;
|
||||
const r1Module = (nonce: string) => `packages/shared/src/chaos/slug-${nonce}.ts`;
|
||||
@@ -319,6 +323,37 @@ export const workloadDefinitions: Record<WorkloadId, WorkloadDefinition> = {
|
||||
]);
|
||||
},
|
||||
},
|
||||
F1: {
|
||||
// Staging class F (doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md): a remote save-back refused
|
||||
// an absolute symlink (`remote_workspace_archive_unsafe_symlink`), exhausted its retries and left the task
|
||||
// blocked. Agents create such links in ordinary work (tool caches, `node_modules/.bin`). The oracle expects
|
||||
// the task to finish with its result saved, with no Retry; how the link itself is saved is the product's choice.
|
||||
id: "F1",
|
||||
label: "Absolute symlink kept in the repository workspace",
|
||||
usesRepository: true,
|
||||
fireWindowMs: [20_000, 90_000],
|
||||
budgetMs: 12 * 60_000,
|
||||
title: nonce => `Chaos F1 workspace link ${nonce}`,
|
||||
description: nonce => [
|
||||
"Work in this task's repository workspace (a small Python CLI called chaoscli).",
|
||||
`Run \`ln -s /etc/hosts ${f1LinkName(nonce)}\` in the repository root to create a symbolic link, and keep it.`,
|
||||
`Create \`${notesFile(nonce)}\` in the repository root. Its first line is exactly \`F1-${nonce}\`.`,
|
||||
`Its second line is the exact output of \`ls -l ${f1LinkName(nonce)}\`, run after you created the link.`,
|
||||
"Do not change any other file and do not run ./bench.sh. Then mark this task done.",
|
||||
].join("\n"),
|
||||
collect: { documents: [], files: nonce => [notesFile(nonce)], commands: [] },
|
||||
grade(nonce, artifacts) {
|
||||
const lines = (artifacts.files[notesFile(nonce)] ?? "").split(/\r?\n/).map(line => line.trim()).filter(Boolean);
|
||||
return result([
|
||||
statusCheck(artifacts),
|
||||
{ name: "workspace_located", ok: artifacts.workspacePath !== null },
|
||||
{ name: "notes_file", ok: lines[0] === `F1-${nonce}` },
|
||||
// Proof the absolute link was in the workspace before the save-back: the agent's own `ls -l` output, saved
|
||||
// with the notes file. Best-effort against an agent that invents it; how the link itself is saved is not graded.
|
||||
{ name: "link_listed", ok: f1LinkListing(nonce).test(lines[1] ?? ""), detail: JSON.stringify((lines[1] ?? "").slice(0, 160)) },
|
||||
]);
|
||||
},
|
||||
},
|
||||
R1: {
|
||||
id: "R1",
|
||||
label: "Code change in a Paperclip clone",
|
||||
|
||||
Reference in new issue
Block a user