diff --git a/doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md b/doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md new file mode 100644 index 00000000000..2550b6ff024 --- /dev/null +++ b/doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md @@ -0,0 +1,309 @@ +# Staging failures vs chaos-harness coverage (2026-11-09 00:00Z to 2026-11-11 00:05Z) + +Question from Dotta: what failed on `paperclip.staging.paperclip.app` in the last 48 hours, why +did the restart-chaos harness (`tests/restart-chaos/`) not find it first, and what must the harness +add so it would. + +## Answer first + +1. **276 runs failed, were cancelled or were interrupted in 48 hours, across 114 tasks.** At census + time, 63 of those tasks (55%) were still `blocked`. +2. **Staging is a continuous deploy target.** There were 47 server process starts in 48 hours. 45 of + them were new Railway deployments: about one deploy every 40 minutes in working hours. Each deploy + replaces the container, so the new server runs in a **different process space** (host, kernel boot + and PID namespace) with **newer code**. Remote Daytona sandboxes keep the previous runner. +3. **The harness restarts the same build on the same Mac, at once.** A harness restart keeps the + process space and has no deployment identity. It is back in seconds. A staging deploy has three + differences: + - it sends a graceful SIGTERM, often 1–3 s after a native coordinator claim; + - the server stays down for 4–7 minutes; + - it has a new deployment identity. + + Native recovery uses the deployment proof (`proveOwnerGone`, `deployment_replaced`), not PID + proofs. So the process space matters less than first assumed. This was corrected after review. + The open gap is the long, mid-turn outage, during which the remote Runner is lost (class R + below). The harness passed D1 8/8 and D5 6/6 on master while staging blocked tasks. +4. **The Daytona matrix does not use the staging workload mix.** Staging runs OpenCode in 66% of + native runs (1266 of 1917), uses reusable sandboxes (saved epochs up to 5), runs 25 agents and has + long-lived parent/child task trees. The Daytona matrix runs only `codex,claude` on fresh + per-run sandboxes, one task tree deep, with waves shorter than one hour. +5. **The harness never builds old saved state.** In the largest class by count, automatic wakes + failed 99 times in a row on one parent task over 8.5 hours. That needed a workspace shared under a + policy retired on 2026-11-07, and every wave starts from an empty database. The existing U2 check + would have failed the task, so grading was not the gap. + +## Method and sources (read-only) + +Task labels `T01`–`T60` stand in for staging task identifiers, which are instance-local. The census +owner keeps the mapping outside the repository. + +- Staging API, GET only, with the operator's existing CLI token loaded into a child process: + - `/api/health`; + - `/api/companies/:id/heartbeat-runs` (company and per agent); + - `/api/heartbeat-runs/:id`; + - `/api/heartbeat-runs/:id/events`; + - `/api/issues/:id`, `/runs`, `/activity` and `/recovery-actions`. + + One company exists. There were 1917 runs in the window. +- Deploy and restart times: every run record carries `controllerProcess`, with `processStartedAt`, + `bootId` and the Railway `deploymentId`. The timeline below is built from those fields. + GitHub has no staging deployment records, because staging deploys from Railway. +- `git log origin/master` for fixes that touch each failure code. +- Window: a failure counts when its run finished at or after 2026-11-09T00:00Z. + +### Data the census could not read + +- **Sentry and Honeycomb:** no read credentials for staging were available. The only keys are + ingest/test-config keys. Sentry triage was not run. +- **a PR-review agent runs:** runs from 2026-11-09 00:00–14:52Z are missing. That agent made more than + 1000 runs in the window, and the run list caps at 1000 with no cursor. Other agents are complete. +- **Commit per deploy:** Railway deployment IDs do not map to git SHAs through the API. Only the + current SHA (`358977454b`, #20649) is visible. "Fixed on master" therefore means "a merged PR + changes this path". It does not mean the fix was proven live on staging. +- **Brief staging outage:** staging was in a maintenance window, `tenant_app_unavailable`, at about + 00:11Z on 2026-11-11, during the census. That was another deploy. Data read before the window is + complete. + +## Class table + +The share column is the share of the 276 failed runs. The "Blocked now" column counts tasks still +blocked at census time. + +| Class | Runs | Share | Tasks | Blocked now | Mechanism (root) | Fixed on master? | Why the harness missed it | Proposed scenario | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| W: wake loop on a held shared workspace | 103 | 37% | 4 | 1 | A parent shares a legacy execution workspace with a child that stranded a writer hold on Nov 5. The new stall-wake feature (#20098) woke the parent every 10 min. Each wake failed setup with no backoff. T01 had 99 runs from 07:27 to 15:53Z. | **Fixed:** #20273, #20295, #20314 | Fresh database every wave, so there was no legacy shared-workspace state. The feature landed between waves. U2 would have caught the loop. | None (fixed). If needed: K14 from a pre-2026-11-07 build with M1. | +| R: remote recovery refuses after a deploy | 34 | 12% | 32 | **24** | Recovery after a deploy refuses to continue. Codes include `runner_remote_process_identity_unavailable` (9), `native_remote_recovery_lease_mismatch …identity_run_id` (8), `prior_owner_terminal_state_indeterminate` (6), `cleanup_quarantined` (4) and `missing_evidence` (3). 27 of 34 failed within 10 min of a process start. | **Mostly fixed:** `identity_run_id` by #20642, `prior_owner…` by #20677, others by #20639, #20631, #20666, #20662 and #20650. **Open:** `runner_remote_process_identity_unavailable`, where the remote Runner is gone after a 4–7 min mid-turn outage. | Harness restarts come back at once, with no long mid-turn outage. There were no OpenCode cells on Daytona (19 of 34 are OpenCode). No Daytona workload keeps a tool running across the fault. | **K15 redeploy** (implemented). A long-hold and mid-tool variant with sandbox-side Runner log capture is the follow-up. | +| F: finalization and workspace save-back | 24 | 9% | 15 | 8 | A saved result cannot be copied back or restored. Causes: sandbox disk full (ENOSPC in 2 runs), unsafe symlink in the archive, `recovery_receipt_invalid` (7), a stuck "Recovering saved work…" phase (8) and an exhausted finalization budget (3). | **Partly fixed:** #20666 (receipt), #20578 (re-finalize). Nothing for disk full or symlinks. | Workloads write small files. They never fill the sandbox disk and never create symlinks. ST faults hit S3, not the sandbox disk. | **F1 symlink** workload (built; passes on master). **DF1 sandbox disk full** (proposed). | +| E: Daytona capacity outage | 18 | 7% | 13 | 4 | Daytona returned HTTP 502 and the worker hit SIGKILL from 16:36 to 17:02Z on Nov 10. Reusable sandbox resume failed, plus a git clone network failure. | **Mostly recovers:** 8 tasks recovered automatically and bounded retry works. 4 tasks still blocked. | The harness has no environment-provider fault. All faults hit the control plane or S3. | **EP1 environment provider 5xx window** (proposed, not built). | +| U: provider usage limit | 17 | 6% | 12 | 8 | `provider_limit_wait` waits as designed. The ACPX "terminal limit failure" (7) fails the run, where it should wait. | **Open:** ACPX limit classification. | Not a restart class. Provider shims have no usage-limit message. | Add a usage-limit message to PV faults (proposed). | +| P: provider session resume after a deploy | 16 | 6% | 16 | **11** | The runner or provider cannot resume its session after the controller moved. Codes: `PRP provider resume state is unavailable` (5), `runner_harness_state_mismatch` (8) with sub-codes `provider_identity_incomplete`, `resumed_unavailable` and `backup_without_reusable_lease`, plus `ingress_auth_failed` and `run.attach provider_side_effect_pending`. 8 of the 16 are ACPX and 5 are OpenCode. | **Partly fixed:** #20627 (warm backup). `resume state unavailable` is open. | Same as R: same process space, no OpenCode, no reusable warm sandboxes across deploys. | K15 with `--daytona-warm` and `--adapters claude,opencode`. | +| O: other or unclassified native failures | 13 | 5% | 13 | 5 | `native_run_failed` with no classified error (3), ACPX `session.open` rejected (2), `local_runtime_api_listener_unavailable` (2), `tool.resolve provider_tool_call_retired`, a DB query error, and `Object not found`. | **Open:** classification. | Not deploy-correlated (0 of 13 within 10 min). | None. These are product triage items. | +| B: authorization or binding readiness | 13 | 5% | 9 | 3 | `reviewed_chat_execution_binding_not_authorized` (7) and `not_ready` (2), a credential not shared with the responsible user (2), and a skill package still uploading (2). | **Open** | The harness uses one operator. There are no chat bindings, no shared credentials and no skill uploads. | Out of scope for restart chaos. Route to product. | +| L: legacy runtime lost across a deploy | 12 | 4% | 12 | 5 | Legacy (non-native) runs: `process_lost` (5), `Legacy controller lease lost` (4), graceful SIGTERM (1) and `lease_released_before_terminal` (2). | **Recovers in most cases:** 7 of 12 tasks recovered. | The matrix runs native adapters only. | K15 with a legacy adapter cell (proposed). | +| X: expected cancellations | 11 | 4% | 9 | 2 | Reassignment, ownership changes, dependency blocks and operator interrupts. | No fix needed | n/a | none | +| A: runner or provider-pack artifact incompatible after an upgrade | 8 | 3% | 8 | 4 | New code meets a sandbox prepared by older code. Codes: `OpenCode executable is missing from the provider pack`, `ACPX launch profile omitted its provider-pack authority`, `native_provider_artifacts_admission_unverified` (3), `runner_remote_artifact_platform_mismatch` and `runner_revision_mismatch`. | **Open** | Daytona waves require the runner content to match the image and refuse mismatches. That is exactly the mixed-version state staging lives in. K14 is local only. | **K15 with `--upgrade-from` on Daytona** (allows mixed runner/image versions; proposed follow-up). | +| V: runner event commit stall | 7 | 3% | 5 | 2 | `native_event_replay_conflict: runner_event_commit_stalled`, Codex and ACPX, 20:49–23:55Z on Nov 9. No deploy correlation. | **Likely fixed:** #20385 (usage receipt stalls). No occurrence after 23:55Z on Nov 9. | Database contention (L1) used one-row pressure, not the receipt-update pattern. | none (fixed) | + +Totals: 276 runs and 114 tasks. Classes R, P, A and L together are the deploy-crossing failures: +70 runs (25%) and **44 currently blocked tasks**, which is most of the user-visible harm. Class W is +the largest by count but is fixed. + +## Deploy timeline (Railway, from run records) + +47 process starts between 2026-11-08T22:03Z and 2026-11-10T23:23Z. All but one are new +`deploymentId`s. The exception is a same-deployment restart at 2026-11-09T14:28Z. The median gap in +working hours is about 40 minutes. Every native run longer than about 40 minutes crosses a deploy. + +67 of 276 failures (24%) finished within 10 minutes after a process start. In classes R and P, 30 of +50 did. In classes E, O and V, almost none did. + +## Per-class detail + +### W: wake loop on a held shared workspace (103 runs, fixed) + +- Examples: T01 (99 runs), T02, T03 and T04. +- Timeline: + 1. On Nov 5, child T05 failed setup (`workspace_busy`, then `setup_failed`). Its parent + T01 shared the workspace, a pre-2026-11-07 policy. + 2. On Nov 9, #20098 ("wake the parent once when a child stalls") reached staging. At 07:27Z the + parent began to be woken about every 10 min by `issue.blocker_stalled_wake`. + 3. Every wake failed with `The workspace still has an unfinished writer or save operation.` + 4. At 15:53Z the parent became `blocked`. +- Fixed by: + - #20273, which stops stall wakes from failing setup on a held workspace; + - #20295, which re-proves an exhausted save's agent stop; + - #20314, which moves stranded legacy subtasks off a parent's held workspace. +- Why the harness missed it: + - **Old saved state:** every wave starts from an empty database, so a parent/child pair sharing a + workspace (the legacy policy) never exists. + - **Missing feature in the tested build:** #20098 landed at 00:26Z on Nov 9, after the last wave + that had a stalled child. + - **Timing window:** the loop starts only after a child stalls for longer than the stall threshold. + - **Grading is not the gap.** The enforced U2 check already fails a task after more than three + failures with the same code and no status change. The parent would have failed U2 on its fourth + wake, if a wave had created the legacy state. +- Scenario: none is built in this pass, because the class is fixed and the shared-workspace policy + is retired for new subtasks. A K14 wave from a pre-2026-11-07 build with M1 would recreate the + legacy state, if the fix needs regression cover. + +### R: remote native recovery refuses after a deploy (34 runs, 24 tasks blocked) + +- Examples: + - T06, T07, T08 and T09 (`lease_mismatch identity_run_id`); + - T10, T11, T12 and T13 (`identity_unavailable`); + - T14, T15, T16 and T17 (`prior_owner_terminal_state_indeterminate`, with + `retirement_refused=journal_commands_unsettled(run.attach.pending | semantic_tool.result.pending)`); + - T18 and T19 (`cleanup_quarantined`). +- Mechanism, per sub-code. This was corrected after review by the class R fix owner. + - Native recovery does **not** depend on PID proofs. It uses `proveOwnerGone`, which accepts a + replaced Railway deployment on an exclusive volume (`deployment_replaced`) or a stopped or fenced + boot row. The only PID-only caller left is the runtime skill cache lock (#20688). + - `native_remote_recovery_lease_mismatch … identity_run_id`: the leases were ephemeral and the + sandboxes fresh, so a reused sandbox was not the cause. This is the shape that #20642 fixed. The + last staging occurrence was 14:43Z on Nov 10. + - `prior_owner_terminal_state_indeterminate` with unsettled `run.attach` or `semantic_tool.result`: + fixed by #20677. Retirement is no longer held when the prior ephemeral sandbox is provably + destroyed. + - `runner_remote_process_identity_unavailable` is **still open**: + 1. The controller dies mid-turn while tools run. + 2. Recovery starts 4–7 minutes later. + 3. By then the remote Runner process is gone, or its `session.snapshot` times out. + + Why the Runner exits despite its 60-minute connection lease is not yet known. +- Last occurrences: `identity_unavailable` 2026-11-10T08:10Z, `lease_mismatch` 14:43Z, `prior_owner…` + 18:56Z and `cleanup_quarantined` 17:48Z. +- Code pointers: + - `server/src/services/native-runtime/native-session-executor.ts:14247` (identity wait); + - `server/src/services/native-runtime/remote-recovery-lease-mismatch.ts`; + - `server/src/services/heartbeat/restart-interrupted-turn.ts`. +- Why the harness missed it: + 1. **Deploy timing:** a staging deploy sends a graceful SIGTERM, often 1–3 s after + `native.coordinator.claim`, and the server stays down for 4–7 minutes. K2 restarts at once, + and K4's long hold uses SIGKILL. + 2. **Missing adapter:** OpenCode (19 of 34) never ran in a Daytona cell. + 3. **Mid-turn Runner loss:** no Daytona workload keeps a tool running across the fault. The + `tool` trigger covers only the local B5 shell bench, so the remote Runner is never under a + running tool when the controller goes away. + 4. **Missing environment shape:** `--daytona-reuse-lease` and `--daytona-warm` are not in the + matrix. +- Scenario: **K15 redeploy** (implemented). The fault does a graceful SIGTERM like K2. The + replacement then starts in a new emulated process space (a new hostname through a preload, so + `processSpaceHash` changes) and with a new Railway-shaped deployment identity on the same volume. + Selected with `--faults K15`. RED wave: see "Harness expansion" below. + +### F: finalization and workspace save-back (24 runs, 8 tasks blocked) + +- Examples: + - T20 (`sync_out_retry_exhausted` from disk full, then the finalization budget, three times); + - T21 (`remote_workspace_archive_unsafe_symlink`); + - T22, T23 and T24 (`remote_workspace_recovery_receipt_invalid`); + - T25 and T10 ("Recovering saved work…" phase cancelled with + `execution_reconciliation_required`); + - T26, T27 and T28 (07:26Z on Nov 9, right after a deploy). +- Fixed: #20666 (skip unverifiable recovery sources) covers `receipt_invalid`, which last occurred + at 22:35Z, before that fix was deployed. Disk-full and symlink save-back have no fix and no + scenario. +- Why the harness missed it: **missing workload.** The workloads write a few KB, never fill the + sandbox disk and never create symlinks. ST faults break S3, not the sandbox file system. +- Built: the **F1** workload (`--workloads F1`). The agent keeps an absolute symlink to `/etc/hosts` + in its repository workspace. Wave `w-20261111T012523-s9304` (Daytona, `--storage s3`, K2, Codex + and OpenCode) passed 2 of 2 on master. The save-back did not block the task. The harness keeps no + evidence for passing tasks, so it is not verified that the link existed at save-back time. +- Proposed (not built): `DF1`, which fills the sandbox disk to ENOSPC before result copy-back. + +### E: Daytona capacity outage (18 runs, mostly self-healing) + +- 2026-11-10 16:36–17:02Z: + - `Daytona sandbox creation failed (HTTP 502)` (10); + - `Worker process exited (code=null, signal=SIGKILL)` (3); + - `Reusable sandbox lease … could not be resumed (cause=provider_error | transport_timeout)` (3). +- Plus one `git=network` clone failure and one Daytona rate-limit failure. +- Bounded retry recovered 8 of 13 tasks. 4 remain blocked (T29, T30, T31 and + T32 are examples). +- Fix #20626 (retry and replace a reusable sandbox that will not resume) shipped before the outage. + Its effect at this scale is not verified. +- Why the harness missed it: **missing fault type.** There is no environment-provider fault. All + harness faults hit the control plane, S3 or the provider message stream. +- Proposed: `EP1`, a proxy in front of the Daytona API that returns 502 or times out for a window. + +### U: provider usage limit (17 runs) + +- `provider_limit_wait` (10) is the designed wait. +- The ACPX "terminal limit failure" (7, an error-triage agent) fails the run. It should wait. This is a product + classification gap in the ACPX limit mapping. + +### P: provider session resume after a deploy (16 runs, 11 tasks blocked) + +- Examples: + - T33, T34, T35 and T25 (`PRP provider resume state is unavailable`); + - T36, T37, T38 and T39 (`runner_harness_state_mismatch …`); + - T40 (`run.attach provider_side_effect_pending`). +- Last occurrence: 2026-11-10T21:06Z. +- #20627 (restore a warm backup when its source sandbox is idle) covers `backup_without_reusable_lease` + and `resumed_unavailable`. `resume state is unavailable` and `provider_identity_incomplete` have no + fix. +- Why the harness missed it: the same reasons as R, plus ACPX (8) and OpenCode (5) on warm reusable + sandboxes, which the matrix does not run. + +### O: other (13 runs) + +- Not deploy-correlated. +- `native_run_failed` without a classified provider error appeared on T41, T42 and + T43. Those runs reached `run.result.accepted` after two `turn.failed` events. +- ACPX `session.open` was rejected on T44 and T45. +- These are product triage items, not chaos gaps. + +### B: authorization or binding readiness (13 runs) + +- `reviewed_chat_execution_binding_not_authorized`: 7 runs on a PR-review agent (T30 and T46). +- A credential not shared with the responsible user. +- Skill package uploads still in flight (T47). +- These are product authorization and readiness checks. They are outside the restart-chaos domain. + +### L: legacy runtime lost across a deploy (12 runs) + +- Examples: + - `process_lost` on T26, T27 and T48 (07:11Z on Nov 9, three minutes after a + deploy); + - `Legacy controller lease lost` on T49, T50, T51 and T52 (08:14–08:19Z on + Nov 10, 20 minutes after a deploy). +- 7 of 12 tasks recovered. +- The matrix has no legacy adapter cell. K15 could run one, but this was not done in this pass. + +### A: artifact incompatible after an upgrade (8 runs) + +- Examples: T53 and T54 (`OpenCode executable is missing from the provider pack`), T55 + (`ACPX launch profile omitted …`), T56, T57 and T58 + (`native_provider_artifacts_admission_unverified`), and T59 (`runner_revision_mismatch`). +- Mechanism: a sandbox prepared and admitted by older code is resumed by newer code that expects + new launch-profile fields or provider-pack contents. +- Why the harness missed it: Daytona waves refuse to run unless the runner content ID equals the + image (`daytona-image.ts`). This is deliberate, and it excludes exactly this mixed-version state. +- Follow-up: K15 with `--upgrade-from` on Daytona, with the image of the older checkout. + +### V: runner event commit stall (7 runs, likely fixed) + +- Examples: T60 and T22 (Nov 9, 20:49–23:55Z). +- #20385 (usage receipt stalls during streamed run updates) merged at 23:35Z, and the class has not + occurred since. + +## Harness expansion (this pass) + +| Scenario | Classes | Selector | Status | +| --- | --- | --- | --- | +| K15 redeploy into a new process space and deployment identity | R, P, L (and A with `--upgrade-from`) | `--faults K15` | Implemented, unit-tested. Five Daytona waves are recorded below. | +| EP1 environment-provider 5xx window | E | — | Proposed | +| F1 absolute symlink kept in the repository workspace | F (unsafe-symlink save-back) | `--workloads F1` | Implemented, unit-tested. Daytona wave: 2 of 2 pass on master. | +| DF1 sandbox disk full | F (ENOSPC save-back) | — | Proposed | + +### Waves on master 8f6ca25976 + +All waves ran on Daytona image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:bc7ca92d…` +(runner content `d7e4d82a…`). + +| Wave | Args | Result | +| --- | --- | --- | +| `w-20261111T002313-s9301` | `--faults K15 --adapters opencode,claude --workloads A1,C1 --daytona-reuse-lease --concurrency 3 --tasks 6` | 5 of 6 pass. t05 (OpenCode, C1) failed R5 `silent_dead_run`: its run stayed `running` with no event, activity or start evidence for 183 s after the redeploy, then succeeded. No Retry was needed. | +| `w-20261111T004241-s9301` (control) | the same seed and cells with `--faults K2` | 6 of 6 pass | +| `w-20261111T005008-s9302` | `--faults K15 --triggers time,comment --adapters opencode,claude --workloads C1 --daytona-warm --concurrency 4 --tasks 4` | 4 of 4 pass | +| `w-20261111T010000-s9303` | `--faults K15 --adapters opencode --workloads A1,C1 --daytona-reuse-lease --concurrency 4 --tasks 8` | 8 of 8 pass | +| `w-20261111T011044-s9303` (control) | the same seed and cells with `--faults K2` | 8 of 8 pass | + +What this shows: + +- Under K15, 17 of 18 tasks passed. Under K2 with the same seeds, 14 of 14 passed. The one failure, + a 183 s silent OpenCode run after a redeploy, occurred only under K15. One failure in 18 is not a + stable reproduction. +- The staging refusal codes of classes R and P did **not** reproduce on current master with K15 + alone. Examples are `identity_run_id` lease mismatch, `runner_remote_process_identity_unavailable` + and `prior_owner_terminal_state_indeterminate`. Two explanations remain open: + - The fixes merged during the window (#20625, #20631, #20639, #20650, #20662, #20666) cover them. + Staging ran older revisions for most of the window. + - They need conditions that K15 does not emulate yet: + - a code change at the deploy, where new server code meets saved state and a sandbox runner from + the older build (class A); + - one sandbox reused across different tasks of one agent; + - many agents and concurrent tasks under load. +- The next scenario to build is **K15 with `--upgrade-from` on Daytona**: start with the older + checkout and its image, redeploy to the current checkout, and keep the older runner in the + retained sandboxes. Today the Daytona image check refuses that mixed state on purpose. diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 672f94618d4..b457d35b03f 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -292,6 +292,76 @@ receipt names both commits (`upgrade` in `faults.json`, `upgrade_switched` in th timeline). K14 is local-only, because a Daytona runner must match its image, and it is never part of a storm's default faults. +## Redeploy (K15, `redeploy.ts`) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \ + --adapters opencode,claude --workloads A1,C1 --faults K15 --daytona-reuse-lease --concurrency 3 --tasks 6 --seed 9301 +``` + +The hosted staging stack deploys about every 40 minutes. Each deploy replaces +the server's container. The replacement has the same database and volume, but it +runs in another process space (host name, kernel boot and PID namespace) and +has a new Railway deployment id. The server's PID-based proof that an earlier +owner exited (`controller-process-identity.ts`, `owner-proof.ts`) never applies +across a deploy. Recovery must use the deployment proof or the lease deadlines +instead. K1 and K2 restart on the same machine, so every PID proof succeeds and +that path stays untested. See `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`. + +K15 emulates the container replacement without a container: + +- Every server in a wave that has a K15 is launched with a `--import` preload + (`redeploy-preload.mjs`). The preload makes `os.hostname()` return a + per-generation name, which changes the process-space hash. +- Every server gets Railway-shaped variables: `RAILWAY_PROJECT_ID`, + `RAILWAY_ENVIRONMENT_ID`, `RAILWAY_SERVICE_ID` and `RAILWAY_DEPLOYMENT_ID`, + plus `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH`, which is the + instance home. +- A K15 stops the server with SIGTERM, like K2. The replacement starts as the + next generation, with a new host name and deployment id, on the same project, + service and volume. +- Other faults in the same wave restart within the current generation, like a + platform restart of the same deployment. +- Identities derive from the wave id; a replay uses the source wave id, so it sees the same ones. +- Each K15 receipt records `redeploy` (both generations) in `faults.json`, and + `redeploy_switched` in the timeline. + +Limits: + +- The Daytona sandboxes and remote runners survive, as they do on staging. +- Local runners are stopped with the server's process tree. +- The cloud stop-proof service (`PAPERCLIP_CLOUD_*`) is not emulated, so stop + proofs that need it stay unsupported, as on any self-hosted Railway deploy. +- K15 does not combine with MS1–MS3, because a second server would run in the + real host's process space. +- Combine K15 with `--adapters opencode` and `--daytona-reuse-lease` or + `--daytona-warm` to match staging's workload mix. + +## Absolute symlink in the workspace (F1) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \ + --adapters codex,opencode --workloads F1 --faults K2 --storage s3 --concurrency 2 --tasks 2 --seed 9304 +``` + +F1 is opt-in. The agent creates `hosts-link-`, which is an absolute +symlink to `/etc/hosts`, in its repository workspace and keeps it. It also +writes a notes file, and then marks the task done. + +On staging, a remote save-back refused such a link +(`remote_workspace_archive_unsafe_symlink`). The save-back then used up its +retries and left the task blocked. Agents create absolute links in ordinary +work, for example in tool caches or `node_modules/.bin`. + +The oracle passes when: + +- the task is `done`; +- the notes file is saved, without a Retry. + +How the product saves the link itself (skip it, keep it, or record it) is not +graded. F1 is meaningful with a remote workspace (`--env daytona`, with +`--storage s3` as on staging). Locally it is a plain repository task. + ## Config faults (CF1–CF3, `config-wave.ts`) ```sh @@ -991,7 +1061,7 @@ re-arms the task's own triggered faults instead of using their offsets. | `grader.ts` | Pure R1–R6 grading over the samples, plus the universal checks. | | `universal.ts`, `universal-audit.ts` | U1–U4 on every task, and their audit over saved waves. | | `contention.ts` | L1 workload, its accounting oracle and the `--db-pressure` writers. | -| `workloads.ts` | A1 brief document, B1 python CLI with tests, B5 long bench, C1 comment while running, M1 delegation, G1 approval across a restart, H1 budget hard stop across a restart, with their T-oracles. | +| `workloads.ts` | A1 brief document, B1 python CLI with tests, B5 long bench, C1 comment while running, M1 delegation, G1 approval across a restart, H1 budget hard stop across a restart, F1 absolute symlink kept in the workspace, with their T-oracles. | | `journeys.ts`, `journey-driver.ts`, `journey-oauth-app.ts`, `journey-matrix.ts` | Q2 and Q3 definitions and oracles, the board's side of each, Q3's loopback OAuth/MCP app, and the `--matrix` planner. | | `evidence.ts`, `ledger.ts` | Evidence bundle for each failure and the JSONL ledger plus `SUMMARY.md`. | | `daytona-image.ts` | Docker-free extraction of the image's runnerd and provider pack, cached per digest, and the controller/image content-ID check. | @@ -1001,6 +1071,7 @@ re-arms the task's own triggered faults instead of using their offsets. | `time-faults.ts`, `time-faults-config.ts` | TM1–TM3: process freezing, database aging with recorded rows, and the time-corrected API the grader reads through. | | `multi-server.ts` | MS1–MS3: the second server's steps, the seeded holds, and the cross-server invariant over the wave database. | | `repeated-restarts.ts` | RR1–RR3: seeded restart steps, `--rr-gap-seconds`, and the stop-and-replace loop the supervisor runs. | +| `redeploy.ts`, `redeploy-preload.mjs` | K15: per-generation emulated host name and Railway deployment identity, and the preload that applies the host name. | | `daytona-sweep.ts` | Daytona R6: read-only reconciliation of this wave's labelled sandboxes with leases; cleanup only through Paperclip. | ## Output (gitignored) diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts index 8c6baac226e..32a4439b5b6 100644 --- a/tests/restart-chaos/config.ts +++ b/tests/restart-chaos/config.ts @@ -4,7 +4,7 @@ import { parseArgs } from "node:util"; import { applyJourneyOptions } from "./journey-matrix.js"; import { takesJourneyTriggers } from "./journeys.js"; import { STORAGE_FAULT_KINDS, type StorageFaultKind } from "./storage-chaos.js"; -import { validateMultiServerOptions } from "./multi-server.js"; +import { isMultiServerFault, validateMultiServerOptions } from "./multi-server.js"; import { parseTimeFaultOptions } from "./time-faults-config.js"; import { parseStorageTrigger } from "./storage-trigger.js"; import { parseNativeTrigger, parseNativeTriggerOutage } from "./native-trigger.js"; @@ -17,7 +17,10 @@ import { type AdapterId, type EnvironmentKind, type FaultKind, type TriggerId, type WaveOptions, type WorkloadId, } from "./types.js"; -/** A storm's default faults: every restart kind except K14, which needs an older build (`--upgrade-from`). */ +/** + * A storm's default faults: every restart kind except K14, which needs an older build (`--upgrade-from`), + * and K15, which changes the deploy identity of every later server (pass `--faults` to include it). + */ export const STORM_DEFAULT_FAULTS: FaultKind[] = ["K1", "K2", "K4", "K6"]; export const CREDENTIAL_FOR_ADAPTER: Record = { @@ -176,7 +179,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC env, adapters, // The governance scenarios (G1, H1), the journeys (Q1–Q3) and the contention run (L1) are opt-in: H1 needs a wave of its own. - workloads: list(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1" + workloads: list(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1" && id !== "F1" && !takesJourneyTriggers(id) && (id !== "M1" || adapters.length >= 2))), // A crash-point wave schedules no fault; K1 is the restart its self-kill gets. faults: allFaults.filter((fault): fault is FaultKind => !isProviderFault(fault)), @@ -233,6 +236,10 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults"); } validateMultiServerOptions(options); + // K15 emulates one deploy identity per generation; a second server beside it would run in the real host's process space. + if (options.faults.includes("K15") && options.faults.some(isMultiServerFault)) { + throw new Error("K15 (redeploy) does not combine with MS1-MS3: the second server would not share the emulated deploy identity"); + } Object.assign(options, parseTimeFaultOptions(options, values["time-age-hours"])); Object.assign(options, parseRepeatedRestartOptions(options, values["rr-gap-seconds"], values["rr-stop-on"])); const upgradeFrom = values["upgrade-from"]; diff --git a/tests/restart-chaos/f1-symlink.test.ts b/tests/restart-chaos/f1-symlink.test.ts new file mode 100644 index 00000000000..39af1900dd6 --- /dev/null +++ b/tests/restart-chaos/f1-symlink.test.ts @@ -0,0 +1,53 @@ +import { describe, expect, it } from "vitest"; +import { parseCommand } from "./config.js"; +import { WORKLOAD_TRIGGERS } from "./journeys.js"; +import { planTasks } from "./scheduler.js"; +import type { OracleArtifacts } from "./types.js"; +import { f1LinkName, workloadDefinitions } from "./workloads.js"; + +const NONCE = "abc123de"; +const LISTING = `lrwxrwxrwx 1 daytona daytona 10 Nov 11 01:30 hosts-link-${NONCE} -> /etc/hosts`; +const notes = (text: string) => ({ [`NOTES-${NONCE}.md`]: text }); +const artifacts = (overrides: Partial = {}): OracleArtifacts => ({ + issueStatus: "done", documents: {}, agentComments: [], files: notes(`F1-${NONCE}\n${LISTING}\n`), commands: {}, + workspacePath: "/workspace/chaos", ...overrides, +}) as OracleArtifacts; + +describe("F1: absolute symlink kept in the repository workspace", () => { + const f1 = workloadDefinitions.F1; + + it("asks for the absolute link and the notes file in a repository workspace", () => { + expect(f1.usesRepository).toBe(true); + const text = f1.description(NONCE); + expect(text).toContain(`ln -s /etc/hosts ${f1LinkName(NONCE)}`); + expect(text).toContain(`NOTES-${NONCE}.md`); + expect(f1.collect.files(NONCE)).toEqual([`NOTES-${NONCE}.md`]); + }); + + it("passes only when the task is done and its saved notes file is in the workspace", () => { + expect(f1.grade(NONCE, artifacts()).verdict).toBe("pass"); + expect(f1.grade(NONCE, artifacts({ issueStatus: "blocked" })).verdict).toBe("fail"); + expect(f1.grade(NONCE, artifacts({ files: {} })).verdict).toBe("fail"); + expect(f1.grade(NONCE, artifacts({ workspacePath: null })).verdict).toBe("fail"); + }); + + it("needs the agent's ls -l line proving the absolute link existed before the save-back", () => { + expect(f1.description(NONCE)).toContain(`ls -l ${f1LinkName(NONCE)}`); + // Skipped link: notes only. + expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n`) })).verdict).toBe("fail"); + // A regular file of the same name, or a link elsewhere, is not the link under test. + expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n-rw-r--r-- 1 d d 3 Nov 11 01:30 hosts-link-${NONCE}\n`) })).verdict).toBe("fail"); + expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n${LISTING.replace("/etc/hosts", "etc/hosts")}\n`) })).verdict).toBe("fail"); + expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n${LISTING.replace(NONCE, "ffffffff")}\n`) })).verdict).toBe("fail"); + }); + + it("is opt-in, time-triggered, and plans with restart faults on Daytona", () => { + expect(WORKLOAD_TRIGGERS.F1).toEqual(["time"]); + const defaults = parseCommand(["--seed", "1"]); + expect(defaults.mode === "wave" && defaults.options.workloads).not.toContain("F1"); + const parsed = parseCommand(["--env", "daytona", "--daytona-image", `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"b".repeat(64)}`, + "--workloads", "F1", "--faults", "K2,K15", "--adapters", "codex", "--tasks", "2", "--seed", "3"]); + if (parsed.mode !== "wave") throw new Error("expected a wave"); + expect(planTasks(parsed.options).map(task => task.workload)).toEqual(["F1", "F1"]); + }); +}); diff --git a/tests/restart-chaos/journeys.ts b/tests/restart-chaos/journeys.ts index e2b0199ec0c..50c4d3a0c10 100644 --- a/tests/restart-chaos/journeys.ts +++ b/tests/restart-chaos/journeys.ts @@ -35,7 +35,7 @@ export type JourneyTrigger = (typeof JOURNEY_TRIGGERS)[number]; /** Each workload's own fault boundaries, for the journey × fault matrix. */ export const WORKLOAD_TRIGGERS: Record = { A1: ["time"], B1: ["time"], B5: ["tool"], C1: ["comment", "result"], M1: ["time"], G1: ["approval"], - H1: ["paused"], R1: ["time"], S1: ["stop"], L1: ["time"], Q1: JOURNEY_TRIGGERS, Q2: JOURNEY_TRIGGERS, Q3: JOURNEY_TRIGGERS, + H1: ["paused"], R1: ["time"], S1: ["stop"], L1: ["time"], F1: ["time"], Q1: JOURNEY_TRIGGERS, Q2: JOURNEY_TRIGGERS, Q3: JOURNEY_TRIGGERS, }; /** Q3 reviews must survive a delay longer than the implicit deadline #20340 removed. */ diff --git a/tests/restart-chaos/main.ts b/tests/restart-chaos/main.ts index d31cdd29190..2d205493d6e 100644 --- a/tests/restart-chaos/main.ts +++ b/tests/restart-chaos/main.ts @@ -38,6 +38,7 @@ import { collectWaveSpeed, speedMarkdown } from "./wave-speed.js"; import { TaskObserver, childSpeedTasksInTurn } from "./observer.js"; import { closeJourneyApps } from "./journey-driver.js"; import { matrixMayLaunch, planMatrixWave } from "./journey-matrix.js"; +import { redeployWave } from "./redeploy.js"; import { StormRun, TASK_FAULT_LOG, observedFaultSchedule, planWave, replayOnCurrentBuild, replayRecordFor, replayWave, type TaskFaultLog } from "./scheduler.js"; import { chaosLifecycleOverrides, seedChaosLab } from "./seed.js"; import { ChaosSupervisor } from "./supervisor.js"; @@ -200,7 +201,9 @@ async function main(): Promise { const batchSize = crashPoints.length ? Math.max(1, Math.floor(plan.tasks.length / crashPoints.length)) : 0; if (crashPoint && plan.tasks.length !== batchSize * crashPoints.length) throw new Error(`--tasks must be a multiple of the ${crashPoints.length} crash point(s)`); const supervisor = await ChaosSupervisor.create({ repositoryRoot, waveDir, timeline, - ...(plan.options.upgradeFrom ?? plan.options.serverFrom ? { upgradeFrom: plan.options.upgradeFrom ?? plan.options.serverFrom } : {}) }); + ...(plan.options.upgradeFrom ?? plan.options.serverFrom ? { upgradeFrom: plan.options.upgradeFrom ?? plan.options.serverFrom } : {}), + // K15: every server of the wave runs under an emulated deploy identity; a replay derives the source wave's ones. + ...(redeployWave(plan) ? { redeployWaveId: plan.replayOf?.waveId ?? plan.waveId } : {}) }); // Crash-point suite: one booted server, one point at a time. Each point gets // its own batch of tasks; the point is armed once the batch launched and the // server kills itself at its nth hit (restarted like K1). Invariants are diff --git a/tests/restart-chaos/redeploy-preload.mjs b/tests/restart-chaos/redeploy-preload.mjs new file mode 100644 index 00000000000..7767bfc0b44 --- /dev/null +++ b/tests/restart-chaos/redeploy-preload.mjs @@ -0,0 +1,12 @@ +// K15 (redeploy.ts): give the server process an emulated host name, so its process-space hash +// differs from the previous deploy's, as a replaced container's does. Loaded with `--import` +// before the server. Node children forked with the server's execArgv inherit it, as they share its container. +import os from "node:os"; +import { syncBuiltinESMExports } from "node:module"; + +const hostname = process.env.RESTART_CHAOS_REDEPLOY_HOSTNAME?.trim(); +if (hostname) { + os.hostname = () => hostname; + // `import { hostname } from "node:os"` reads the ESM export, which only follows the CJS object after a sync. + syncBuiltinESMExports(); +} diff --git a/tests/restart-chaos/redeploy.test.ts b/tests/restart-chaos/redeploy.test.ts new file mode 100644 index 00000000000..dd23abd74b3 --- /dev/null +++ b/tests/restart-chaos/redeploy.test.ts @@ -0,0 +1,93 @@ +import { spawnSync } from "node:child_process"; +import { describe, expect, it } from "vitest"; +import { parseCommand } from "./config.js"; +import { + REDEPLOY_HOSTNAME_ENV, REDEPLOY_PRELOAD_PATH, nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs, + redeployReceipt, redeployWave, +} from "./redeploy.js"; +import { planTasks } from "./scheduler.js"; +import { FAULT_SIGNAL } from "./supervisor.js"; +import type { WavePlan } from "./types.js"; + +const UUID = /^[a-f0-9]{8}-[a-f0-9]{4}-5[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}$/; +const IMAGE = `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"a".repeat(64)}`; + +describe("K15 redeploy identity", () => { + it("keeps the service and volume and changes the host and deployment on every generation", () => { + const first = redeployIdentity("w-1", 0); + const second = nextRedeployIdentity("w-1", first); + expect(second.generation).toBe(1); + expect(second.hostname).not.toBe(first.hostname); + expect(second.deploymentId).not.toBe(first.deploymentId); + for (const key of ["projectId", "environmentId", "serviceId", "volumeId"] as const) expect(second[key]).toBe(first[key]); + for (const id of [first.projectId, first.environmentId, first.serviceId, first.deploymentId, first.volumeId]) expect(id).toMatch(UUID); + }); + + it("is deterministic per wave, so a replay derives the same identities", () => { + expect(redeployIdentity("w-1", 3)).toEqual(redeployIdentity("w-1", 3)); + expect(redeployIdentity("w-2", 0).deploymentId).not.toBe(redeployIdentity("w-1", 0).deploymentId); + expect(() => redeployIdentity("w-1", -1)).toThrow(/non-negative/); + }); + + it("names the Railway variables the server reads as its deployment, with the instance home as the exclusive volume", () => { + const identity = redeployIdentity("w-1", 2); + expect(redeployEnvironment(identity, "/tmp/chaos/home")).toEqual({ + [REDEPLOY_HOSTNAME_ENV]: identity.hostname, + RAILWAY_PROJECT_ID: identity.projectId, RAILWAY_ENVIRONMENT_ID: identity.environmentId, RAILWAY_SERVICE_ID: identity.serviceId, + RAILWAY_DEPLOYMENT_ID: identity.deploymentId, RAILWAY_VOLUME_ID: identity.volumeId, RAILWAY_VOLUME_MOUNT_PATH: "/tmp/chaos/home", + }); + expect(redeployReceipt(redeployIdentity("w-1", 0), identity)).toMatchObject({ fromGeneration: 0, toGeneration: 2, + toDeploymentId: identity.deploymentId, volumeId: identity.volumeId }); + }); + + it("loads the preload ahead of the server's own --import loader", () => { + expect(redeployNodeArgs(["--import", "/x/tsx/loader.mjs", "entry.ts"])).toEqual(["--import", REDEPLOY_PRELOAD_PATH, "--import", "/x/tsx/loader.mjs", "entry.ts"]); + }); + + it("changes os.hostname for both the CommonJS object and ESM named imports, as processSpaceHash reads it", () => { + const script = "import os from 'node:os'; import { hostname } from 'node:os'; console.log(JSON.stringify([os.hostname(), hostname()]));"; + const run = (env: NodeJS.ProcessEnv) => spawnSync(process.execPath, ["--import", REDEPLOY_PRELOAD_PATH, "--input-type=module", "-e", script], + { env: { PATH: process.env.PATH, ...env }, encoding: "utf8" }); + const emulated = run({ [REDEPLOY_HOSTNAME_ENV]: "chaos-deploy-test" }); + expect(emulated.status).toBe(0); + expect(JSON.parse(emulated.stdout)).toEqual(["chaos-deploy-test", "chaos-deploy-test"]); + // Without the variable the preload changes nothing. + const real = run({}); + expect(real.status).toBe(0); + const [cjs, esm] = JSON.parse(real.stdout) as string[]; + expect(cjs).toBe(esm); + expect(cjs).not.toBe("chaos-deploy-test"); + }); +}); + +describe("K15 wave options", () => { + it("is a graceful stop, selectable on local and Daytona like K2", () => { + expect(FAULT_SIGNAL.K15).toBe("SIGTERM"); + const local = parseCommand(["--faults", "K15", "--adapters", "opencode", "--workloads", "A1,C1", "--seed", "1"]); + expect(local.mode === "wave" && local.options.faults).toEqual(["K15"]); + const daytona = parseCommand(["--env", "daytona", "--daytona-image", IMAGE, "--faults", "K2,K15", "--adapters", "opencode", + "--workloads", "C1", "--daytona-reuse-lease", "--seed", "1"]); + if (daytona.mode !== "wave") throw new Error("expected a wave"); + expect(daytona.options.faults).toEqual(["K2", "K15"]); + expect(daytona.options.daytonaReuseLease).toBe(true); + }); + + it("refuses a second server beside an emulated deploy", () => { + expect(() => parseCommand(["--faults", "K15,MS1", "--workloads", "B5,Q1", "--concurrency", "1", "--seed", "1"])).toThrow(/K15 \(redeploy\) does not combine with MS1-MS3/); + }); + + it("plans K15 cells with no hold and marks the wave as a redeploy wave", () => { + const parsed = parseCommand(["--faults", "K15", "--adapters", "codex,opencode", "--workloads", "A1,C1", "--tasks", "4", "--seed", "7"]); + if (parsed.mode !== "wave") throw new Error("expected a wave"); + const tasks = planTasks(parsed.options); + expect(tasks.every(task => task.fault === "K15" && task.holdMs === 0)).toBe(true); + const plan = { options: parsed.options, tasks } as Pick; + expect(redeployWave(plan)).toBe(true); + const k2 = parseCommand(["--faults", "K2", "--workloads", "A1", "--tasks", "2", "--seed", "7"]); + if (k2.mode !== "wave") throw new Error("expected a wave"); + expect(redeployWave({ options: k2.options, tasks: planTasks(k2.options) })).toBe(false); + // A replay of a K2 task that a K15 crossed still needs the emulated identity. + expect(redeployWave({ options: k2.options, tasks: [{ ...planTasks(k2.options)[0]!, + replayFaults: [{ kind: "K15", holdMs: 0, atMs: 1_000 }] }] })).toBe(true); + }); +}); diff --git a/tests/restart-chaos/redeploy.ts b/tests/restart-chaos/redeploy.ts new file mode 100644 index 00000000000..7be1e02ca65 --- /dev/null +++ b/tests/restart-chaos/redeploy.ts @@ -0,0 +1,96 @@ +/** + * K15 (redeploy): what a hosted deploy does to the control plane, beyond K2's graceful stop. + * + * On the hosted staging stack every deploy replaces the server's container. The replacement runs + * in another process space (host name, kernel boot and PID namespace), so the server's PID-based + * "the previous owner has exited" proofs (`server/src/services/controller-process-identity.ts`, + * `owner-proof.ts`) never apply. It also carries a new Railway deployment id on the same exclusive + * volume, which is the proof hosted recovery uses instead. A K1/K2 restart on one machine keeps the + * process space, so every PID proof succeeds there and this path is never exercised. + * + * K15 emulates both changes without a container: each server of a K15 wave sees an emulated host + * name (a `--import` preload replaces `os.hostname`, which changes the process-space hash) and + * Railway-shaped deployment variables naming one stable project, environment, service and volume + * and a deployment id per generation. A K15 stops the server with SIGTERM like K2 and starts the + * replacement as the next generation. Other faults in the same wave restart within a generation, + * like a platform restart of the same deployment. + */ +import { createHash } from "node:crypto"; +import { fileURLToPath } from "node:url"; +import type { WavePlan } from "./types.js"; + +/** Whether any server of this wave can be redeployed: a planned, storm or replayed K15. */ +export function redeployWave(plan: Pick): boolean { + return plan.options.faults.includes("K15") || (plan.storm?.faults.some(fault => fault.kind === "K15") ?? false) + || plan.tasks.some(task => task.fault === "K15" || (task.replayFaults?.some(fault => fault.kind === "K15") ?? false)); +} + +/** The preload that applies the emulated host name before the server loads. */ +export const REDEPLOY_PRELOAD_PATH = fileURLToPath(new URL("./redeploy-preload.mjs", import.meta.url)); +/** Read by the preload: the host name `os.hostname()` returns in the server process. */ +export const REDEPLOY_HOSTNAME_ENV = "RESTART_CHAOS_REDEPLOY_HOSTNAME"; + +export interface RedeployIdentity { + /** 0 for the wave's first server; each K15 adds one. */ + generation: number; + hostname: string; + projectId: string; + environmentId: string; + serviceId: string; + deploymentId: string; + volumeId: string; +} + +/** A deterministic UUID-shaped id for one wave, role and generation (no randomness: replays match). */ +function uuidFor(waveId: string, role: string, generation = 0): string { + const hex = createHash("sha256").update(`restart-chaos-redeploy:${waveId}:${role}:${generation}`).digest("hex"); + // Version 5-style layout so validators that require a UUID shape accept it. + const variant = ((parseInt(hex[16]!, 16) & 0x3) | 0x8).toString(16); + return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-5${hex.slice(13, 16)}-${variant}${hex.slice(17, 20)}-${hex.slice(20, 32)}`; +} + +/** The identity of the server that serves generation `generation` of this wave. */ +export function redeployIdentity(waveId: string, generation: number): RedeployIdentity { + if (!Number.isInteger(generation) || generation < 0) throw new Error(`Redeploy generation must be a non-negative integer, got ${generation}`); + return { + generation, + hostname: `chaos-deploy-${createHash("sha256").update(`${waveId}:${generation}`).digest("hex").slice(0, 10)}`, + projectId: uuidFor(waveId, "project"), + environmentId: uuidFor(waveId, "environment"), + serviceId: uuidFor(waveId, "service"), + deploymentId: uuidFor(waveId, "deployment", generation), + volumeId: uuidFor(waveId, "volume"), + }; +} + +/** The next deploy of the same service: same project, environment, service and volume. */ +export function nextRedeployIdentity(waveId: string, current: RedeployIdentity): RedeployIdentity { + return redeployIdentity(waveId, current.generation + 1); +} + +/** + * Server environment for one generation. The volume mount is the instance home: on staging the + * instance directory lives on the exclusive volume that survives each deploy. + */ +export function redeployEnvironment(identity: RedeployIdentity, volumeMountPath: string): Record { + return { + [REDEPLOY_HOSTNAME_ENV]: identity.hostname, + RAILWAY_PROJECT_ID: identity.projectId, + RAILWAY_ENVIRONMENT_ID: identity.environmentId, + RAILWAY_SERVICE_ID: identity.serviceId, + RAILWAY_DEPLOYMENT_ID: identity.deploymentId, + RAILWAY_VOLUME_ID: identity.volumeId, + RAILWAY_VOLUME_MOUNT_PATH: volumeMountPath, + }; +} + +/** Node arguments that load the preload ahead of the server entry's own `--import` loader. */ +export function redeployNodeArgs(args: readonly string[]): string[] { + return ["--import", REDEPLOY_PRELOAD_PATH, ...args]; +} + +/** What a K15 receipt records: both generations, so a replay and the evidence name them. */ +export function redeployReceipt(from: RedeployIdentity, to: RedeployIdentity) { + return { fromGeneration: from.generation, toGeneration: to.generation, fromHostname: from.hostname, toHostname: to.hostname, + fromDeploymentId: from.deploymentId, toDeploymentId: to.deploymentId, volumeId: to.volumeId }; +} diff --git a/tests/restart-chaos/supervisor.ts b/tests/restart-chaos/supervisor.ts index 86e35504c40..6ae0399120a 100644 --- a/tests/restart-chaos/supervisor.ts +++ b/tests/restart-chaos/supervisor.ts @@ -39,6 +39,7 @@ import { isRepeatedRestartFault, newRepeatedRestartRecord, recordFirstRestart, r type RepeatedRestartRecord, type RestartStep } from "./repeated-restarts.js"; import type { FaultKind, FaultReceipt, TriggerId } from "./types.js"; import { isMultiServerFault, runMultiServerFault, type MultiServerKind, type MultiServerRecord, type ServerProcessWindow } from "./multi-server.js"; +import { nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs, redeployReceipt, type RedeployIdentity } from "./redeploy.js"; /** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */ const HEALTH_TIMEOUT_MS = (() => { @@ -48,7 +49,7 @@ const HEALTH_TIMEOUT_MS = (() => { const READY_TIMEOUT_MS = 300_000; const STOP_TIMEOUT_MS = 30_000; -export const FAULT_SIGNAL: Record = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM", +export const FAULT_SIGNAL: Record = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM", K15: "SIGTERM", TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL", RR1: "SIGKILL", RR2: "SIGKILL", RR3: "SIGKILL", // MS1 leaves the primary alone; MS2 freezes it; MS3 stops it gracefully (multi-server.ts). MS1: "none", MS2: "SIGSTOP", MS3: "SIGTERM" }; @@ -154,13 +155,16 @@ export class ChaosSupervisor { private serverRoot: string; /** The environment overrides the wave passed to `start`, kept for every restart and for the K14 switch. */ private extraEnvironment: Record = {}; + /** K15 waves: the wave id the emulated deploy identities derive from, and the generation serving now. */ + private redeploy: { waveId: string; identity: RedeployIdentity } | null = null; /** Wave secrets, kept across a K14 switch as an in-place upgrade keeps its configuration. */ private readonly secrets = { PAPERCLIP_AGENT_JWT_SECRET: secret(), PAPERCLIP_DECISION_SIGNING_SECRET: secret(), PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: secret(), BETTER_AUTH_SECRET: secret() }; private constructor(readonly repositoryRoot: string, readonly port: number, readonly paths: SupervisorPaths, - private readonly timeline: Timeline, upgradeFrom: string | null) { + private readonly timeline: Timeline, upgradeFrom: string | null, redeployWaveId: string | null) { this.serverRoot = upgradeFrom ?? repositoryRoot; + if (redeployWaveId) this.redeploy = { waveId: redeployWaveId, identity: redeployIdentity(redeployWaveId, 0) }; this.baseUrl = `http://127.0.0.1:${port}`; this.log = createWriteStream(paths.serverLogPath, { flags: "a", mode: 0o600 }); this.stopper = createRunnerE2EServerStopper({ @@ -171,13 +175,16 @@ export class ChaosSupervisor { }); } - static async create(input: { repositoryRoot: string; waveDir: string; timeline: Timeline; upgradeFrom?: string }): Promise { + /** `redeployWaveId`: a K15 wave; every server gets the emulated deploy identity of its generation (redeploy.ts). */ + static async create(input: { repositoryRoot: string; waveDir: string; timeline: Timeline; upgradeFrom?: string; + redeployWaveId?: string }): Promise { const temporaryRoot = await createRunnerE2ETemporaryRoot(); const home = path.join(temporaryRoot, "home"); const paths: SupervisorPaths = { temporaryRoot, home, configPath: path.join(home, "instances", "default", "config.json"), serverLogPath: path.join(input.waveDir, "server.log") }; const port = await reserveRunnerE2EServerPort(); - const supervisor = new ChaosSupervisor(input.repositoryRoot, port, paths, input.timeline, input.upgradeFrom ?? null); + const supervisor = new ChaosSupervisor(input.repositoryRoot, port, paths, input.timeline, input.upgradeFrom ?? null, + input.redeployWaveId ?? null); input.timeline.add("supervisor_created", { temporaryRoot, port, ...(input.upgradeFrom ? { upgradeFrom: input.upgradeFrom, upgradeFromSha: checkoutSha(input.upgradeFrom) } : {}) }); return supervisor; @@ -241,6 +248,8 @@ export class ChaosSupervisor { PAPERCLIP_IN_WORKTREE: "false", ...waveServerUrlEnvironment(this.port), ...extraEnvironment, + // K15: the generation's emulated host name and Railway deployment identity; the volume is the instance home. + ...(this.redeploy ? redeployEnvironment(this.redeploy.identity, home) : {}), }; const environment = buildPaperclipServerEnvironment(source, overrides); const workspaceStorage = extraEnvironment.PAPERCLIP_STORAGE_PROVIDER === "s3" @@ -257,7 +266,9 @@ export class ChaosSupervisor { const control = this.crashPointControl; const env = control ? { ...this.environment, PAPERCLIP_CRASH_POINT_CONTROL: control.controlPath, PAPERCLIP_CRASH_POINT_RECEIPT: control.receiptPath } : this.environment; - const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]), + const args = runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]); + // K15 waves: the emulated host name of the serving deploy generation (redeploy.ts). + const child = spawn(process.execPath, this.redeploy ? redeployNodeArgs(args) : args, { cwd: root, env, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached }); this.child = child; const window = this.trackWindow(child, "primary"); @@ -484,6 +495,8 @@ export class ChaosSupervisor { receipt.restartedAt = Date.now(); // K14: the replacement is the current build, on the same database and instance directory. if (kind === "K14") receipt.upgrade = await this.switchToCurrentBuild(receipt.faultId); + // K15: the replacement is the next deploy: a new process space and deployment id on the same volume. + if (kind === "K15") receipt.redeploy = this.advanceDeploy(receipt.faultId); this.timeline.add("fault_restarting", { faultId: receipt.faultId }); return this.spawnServer(); }, @@ -668,6 +681,19 @@ export class ChaosSupervisor { throw new Error(`Second server did not report ready within ${READY_TIMEOUT_MS}ms`); } + /** K15: move the next server to the next deploy generation (redeploy.ts). */ + private advanceDeploy(faultId: string): NonNullable { + if (!this.redeploy || !this.environment) throw new Error("K15 needs a redeploy wave (ChaosSupervisor.create redeployWaveId)"); + const from = this.redeploy.identity; + const to = nextRedeployIdentity(this.redeploy.waveId, from); + this.redeploy.identity = to; + Object.assign(this.environment, redeployEnvironment(to, this.paths.home)); + const receipt = redeployReceipt(from, to); + this.timeline.add("redeploy_switched", { faultId, ...receipt }); + this.log.write(`\n[restart-chaos] ${faultId} K15 redeploy ${from.hostname}/${from.deploymentId} -> ${to.hostname}/${to.deploymentId}\n`); + return receipt; + } + /** * K14: move the server from the older build to this checkout. Only the first K14 switches; a later * one restarts the current build (recorded with `switched: false`). diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts index 58d82c4fd65..0dee982800e 100644 --- a/tests/restart-chaos/types.ts +++ b/tests/restart-chaos/types.ts @@ -6,11 +6,13 @@ import type { TimeFaultRecord } from "./time-faults.js"; /** * K14 (upgrade mid-run): a graceful stop, then the replacement starts from the current build (`--upgrade-from`). + * K15 (redeploy, redeploy.ts): a graceful stop, then the replacement starts in a new emulated process space with a new + * Railway-shaped deployment id on the same volume, as a hosted deploy replaces the server's container. * TM1–TM3 (time faults, time-faults.ts): freeze the server, sleep with aged rows, or an aged outage. * RR1–RR3 (repeated restarts, repeated-restarts.ts): a K1, then more restarts while the replacement recovers. * MS1–MS3 (two servers, one database, multi-server.ts): a second server beside the first. */ -export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3"] as const; +export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "K15", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3"] as const; export type FaultKind = (typeof FAULT_KINDS)[number]; /** The shortest seeded K4 outage (scheduler.ts K4_HOLD_RANGE_MS), also the K4 hold after a provider fault. */ export const K4_MIN_HOLD_MS = 120_000; @@ -25,8 +27,9 @@ export type TaskFaultKind = FaultKind | "K13" | "K0" | ProviderFaultKind; * G1: an approval-gated action across a restart. H1: a budget hard stop crossed mid-run with a restart (its own wave). * Q1: a saved question (question-handoff.ts). Q2, Q3: journeys.ts. * L1: high-output, high-accounting run for database contention (contention.ts). + * F1: an absolute symlink pointing out of the repository workspace, kept at rest (staging save-back class F). */ -export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3"] as const; +export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3", "F1"] as const; export type WorkloadId = (typeof WORKLOAD_IDS)[number]; export const ADAPTER_IDS = ["codex", "claude", "opencode"] as const; @@ -281,6 +284,8 @@ export interface FaultReceipt { crashPoint?: { name: string; hit: number; pid: number; token?: string }; /** K14: the build the stopped server ran and the build its replacement runs. */ upgrade?: { fromRoot: string; fromSha: string | null; toRoot: string; toSha: string | null; switched: boolean }; + /** K15: the deploy generation, emulated host name and deployment id before and after. */ + redeploy?: ReturnType; /** TM1–TM3: what was frozen and which rows were aged. */ timeFault?: TimeFaultRecord; /** RR1–RR3: every stop and replacement of the sequence, the first kill included. */ diff --git a/tests/restart-chaos/workloads.ts b/tests/restart-chaos/workloads.ts index 4e01eb612bb..fd622ca82a8 100644 --- a/tests/restart-chaos/workloads.ts +++ b/tests/restart-chaos/workloads.ts @@ -41,6 +41,10 @@ const briefMarker = (nonce: string) => `CHAOS-${nonce}`; const followUpMarker = (nonce: string) => `FOLLOWUP-${nonce}`; const continueMarker = (nonce: string) => `CONTINUE-${nonce}`; const notesFile = (nonce: string) => `NOTES-${nonce}.md`; +/** F1: the absolute symlink the agent creates and keeps in the repository root. */ +export const f1LinkName = (nonce: string) => `hosts-link-${nonce}`; +/** F1: an `ls -l` line for that link: a symlink (`l` mode) naming the link and its absolute target. */ +export const f1LinkListing = (nonce: string) => new RegExp(`^l\\S*\\s.*\\s${f1LinkName(nonce)} -> /etc/hosts$`); const benchFile = (nonce: string) => `bench-result-${nonce}.txt`; const benchRecord = (nonce: string) => `.bench/run-${nonce}.txt`; const r1Module = (nonce: string) => `packages/shared/src/chaos/slug-${nonce}.ts`; @@ -319,6 +323,37 @@ export const workloadDefinitions: Record = { ]); }, }, + F1: { + // Staging class F (doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md): a remote save-back refused + // an absolute symlink (`remote_workspace_archive_unsafe_symlink`), exhausted its retries and left the task + // blocked. Agents create such links in ordinary work (tool caches, `node_modules/.bin`). The oracle expects + // the task to finish with its result saved, with no Retry; how the link itself is saved is the product's choice. + id: "F1", + label: "Absolute symlink kept in the repository workspace", + usesRepository: true, + fireWindowMs: [20_000, 90_000], + budgetMs: 12 * 60_000, + title: nonce => `Chaos F1 workspace link ${nonce}`, + description: nonce => [ + "Work in this task's repository workspace (a small Python CLI called chaoscli).", + `Run \`ln -s /etc/hosts ${f1LinkName(nonce)}\` in the repository root to create a symbolic link, and keep it.`, + `Create \`${notesFile(nonce)}\` in the repository root. Its first line is exactly \`F1-${nonce}\`.`, + `Its second line is the exact output of \`ls -l ${f1LinkName(nonce)}\`, run after you created the link.`, + "Do not change any other file and do not run ./bench.sh. Then mark this task done.", + ].join("\n"), + collect: { documents: [], files: nonce => [notesFile(nonce)], commands: [] }, + grade(nonce, artifacts) { + const lines = (artifacts.files[notesFile(nonce)] ?? "").split(/\r?\n/).map(line => line.trim()).filter(Boolean); + return result([ + statusCheck(artifacts), + { name: "workspace_located", ok: artifacts.workspacePath !== null }, + { name: "notes_file", ok: lines[0] === `F1-${nonce}` }, + // Proof the absolute link was in the workspace before the save-back: the agent's own `ls -l` output, saved + // with the notes file. Best-effort against an agent that invents it; how the link itself is saved is not graded. + { name: "link_listed", ok: f1LinkListing(nonce).test(lines[1] ?? ""), detail: JSON.stringify((lines[1] ?? "").slice(0, 160)) }, + ]); + }, + }, R1: { id: "R1", label: "Code change in a Paperclip clone",