test(restart-chaos): K15 redeploy, F1 workspace symlink, and the staging failure census (#20679)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The restart-chaos harness (`tests/restart-chaos/`) restarts the
control plane while real agent tasks run. It grades whether every task
finishes without a human Retry.
> - The hosted staging stack deploys about every 40 minutes. In 48
hours, 276 runs failed and 63 tasks stayed blocked. About a quarter of
the failures came within 10 minutes of a deploy. The harness passed the
same areas.
> - A hosted deploy replaces the server container. The new server runs
in another process space and has a new deployment id. The PID-based "the
earlier owner exited" proofs never apply across that boundary. A harness
restart on one machine keeps the process space, so every PID proof
succeeds and the hosted recovery path is never tested.
> - This pull request adds K15, a redeploy fault. It also adds the
staging census that shows the gap.
> - The benefit: chaos waves can now exercise the recovery path that
staging runs after every deploy, on local and Daytona.

## Linked Issues or Issue Description

**What happened?**
Many staging tasks that a deploy crossed ended blocked. Examples are
`native_remote_recovery_lease_mismatch`,
`runner_remote_process_identity_unavailable`,
`prior_owner_terminal_state_indeterminate` and `PRP provider resume
state is unavailable`. The restart-chaos Daytona waves passed for the
same adapters.

**What did you expect?**
The chaos harness should reproduce the deploy conditions that staging
has, so that these failure classes show up in a wave first.

**Steps to reproduce**
1. Compare the staging run records' `controllerProcess.hostHash` and
`deployment.deploymentId` across a deploy. Both change on every deploy.
2. Compare the same fields across a harness K2 restart. Both stay the
same.
3. Read `controllerProcessExited` and `sameProcessSpace` in
`server/src/services/controller-process-identity.ts`. A different
`hostHash` means that no PID proof applies.

**Additional context**
The full census, the failure classes and the per-class harness gaps are
in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`. The
task identifiers in it are neutral labels.

## What Changed

- New fault `K15` (redeploy), in `tests/restart-chaos/redeploy.ts` and
`redeploy-preload.mjs`. A K15 wave gives every server an emulated host
name through a `--import` preload, which changes the process-space hash.
It also gives Railway-shaped deployment variables with the instance home
as the exclusive volume. A K15 stops the server with SIGTERM and starts
the next generation, with a new host name and deployment id.
- `supervisor.ts`, `main.ts`, `config.ts` and `types.ts`: select K15
with `--faults K15`. It works on `--env local` and `--env daytona`. The
receipt records `redeploy` and the timeline records `redeploy_switched`.
Config refuses K15 together with MS1–MS3.
- New opt-in workload `F1` (`workloads.ts`): the agent keeps an absolute
symlink in its repository workspace. A staging save-back refused such a
link and blocked the task. The oracle needs `done` and the saved notes
file, with no Retry. `f1-symlink.test.ts` covers the prompt, the grading
and the option planning.
- `redeploy.test.ts`: identity derivation, the environment, the argument
order, a real child process that proves the preload changes both CJS and
ESM `hostname()`, option parsing and plan detection.
- `tests/restart-chaos/README.md`: a K15 section and a row in the
"Pieces" table.
- `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`: the
48-hour staging census, 12 failure classes, the deploy timeline, the
reason the harness missed each class, and proposed scenarios.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/redeploy.test.ts
tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/repeated-restarts.test.ts`: 68 tests pass. The full
restart-chaos unit suite passes: 43 files, 516 tests.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Live Daytona waves on 8f6ca25976, image
`paperclip-daytona-runner@sha256:bc7ca92d…`:

  | Wave | Faults and cells | Result |
  | --- | --- | --- |
| `w-20261111T002313-s9301` | K15, opencode and claude, A1 and C1, reuse
lease, 6 tasks | 5 of 6 pass. t05 (OpenCode, C1) failed R5
`silent_dead_run`: 183 s without evidence after the redeploy, then
succeeded. |
  | `w-20261111T004241-s9301` | K2 control, same seed | 6 of 6 pass |
| `w-20261111T005008-s9302` | K15, warm sandbox, comment trigger, 4
tasks | 4 of 4 pass |
| `w-20261111T010000-s9303` | K15, opencode only, 8 tasks | 8 of 8 pass
|
  | `w-20261111T011044-s9303` | K2 control, same seed | 8 of 8 pass |
| `w-20261111T012523-s9304` | F1 with K2, codex and opencode, `--storage
s3` | 2 of 2 pass |

- Every K15 fault switched the generation, with a new host name and
deployment id on the same volume. The server was ready again in 14–24 s.
- K15 does not yet reproduce the staging refusal codes on current
master. The census document explains the open conditions, such as a code
change at the deploy.

## Risks

- Low risk. The change is in the test harness and docs only. No product
code changes.
- K15 is an emulation. It changes the host name only in the server
process (and forked Node children). It does not emulate a new kernel
boot or PID namespace. The cloud stop-proof service is not emulated, so
stop proofs that need it stay unsupported, as on a self-hosted Railway
deploy. The README lists these limits.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed 2026-11-10 20:05:17 -06:00
1 parent 7435abcc63
commit 52160eba35
12 files changed
+723 -13

No files matched your search

@@ -0,0 +1,309 @@
# Staging failures vs chaos-harness coverage (2026-11-09 00:00Z to 2026-11-11 00:05Z)
Question from Dotta: what failed on `paperclip.staging.paperclip.app` in the last 48 hours, why
did the restart-chaos harness (`tests/restart-chaos/`) not find it first, and what must the harness
add so it would.
## Answer first
1. **276 runs failed, were cancelled or were interrupted in 48 hours, across 114 tasks.** At census
time, 63 of those tasks (55%) were still `blocked`.
2. **Staging is a continuous deploy target.** There were 47 server process starts in 48 hours. 45 of
them were new Railway deployments: about one deploy every 40 minutes in working hours. Each deploy
replaces the container, so the new server runs in a **different process space** (host, kernel boot
and PID namespace) with **newer code**. Remote Daytona sandboxes keep the previous runner.
3. **The harness restarts the same build on the same Mac, at once.** A harness restart keeps the
process space and has no deployment identity. It is back in seconds. A staging deploy has three
differences:
- it sends a graceful SIGTERM, often 1–3 s after a native coordinator claim;
- the server stays down for 4–7 minutes;
- it has a new deployment identity.
Native recovery uses the deployment proof (`proveOwnerGone`, `deployment_replaced`), not PID
proofs. So the process space matters less than first assumed. This was corrected after review.
The open gap is the long, mid-turn outage, during which the remote Runner is lost (class R
below). The harness passed D1 8/8 and D5 6/6 on master while staging blocked tasks.
4. **The Daytona matrix does not use the staging workload mix.** Staging runs OpenCode in 66% of
native runs (1266 of 1917), uses reusable sandboxes (saved epochs up to 5), runs 25 agents and has
long-lived parent/child task trees. The Daytona matrix runs only `codex,claude` on fresh
per-run sandboxes, one task tree deep, with waves shorter than one hour.
5. **The harness never builds old saved state.** In the largest class by count, automatic wakes
failed 99 times in a row on one parent task over 8.5 hours. That needed a workspace shared under a
policy retired on 2026-11-07, and every wave starts from an empty database. The existing U2 check
would have failed the task, so grading was not the gap.
## Method and sources (read-only)
Task labels `T01`–`T60` stand in for staging task identifiers, which are instance-local. The census
owner keeps the mapping outside the repository.
- Staging API, GET only, with the operator's existing CLI token loaded into a child process:
- `/api/health`;
- `/api/companies/:id/heartbeat-runs` (company and per agent);
- `/api/heartbeat-runs/:id`;
- `/api/heartbeat-runs/:id/events`;
- `/api/issues/:id`, `/runs`, `/activity` and `/recovery-actions`.
One company exists. There were 1917 runs in the window.
- Deploy and restart times: every run record carries `controllerProcess`, with `processStartedAt`,
`bootId` and the Railway `deploymentId`. The timeline below is built from those fields.
GitHub has no staging deployment records, because staging deploys from Railway.
- `git log origin/master` for fixes that touch each failure code.
- Window: a failure counts when its run finished at or after 2026-11-09T00:00Z.
### Data the census could not read
- **Sentry and Honeycomb:** no read credentials for staging were available. The only keys are
ingest/test-config keys. Sentry triage was not run.
- **a PR-review agent runs:** runs from 2026-11-09 00:00–14:52Z are missing. That agent made more than
1000 runs in the window, and the run list caps at 1000 with no cursor. Other agents are complete.
- **Commit per deploy:** Railway deployment IDs do not map to git SHAs through the API. Only the
current SHA (`358977454b`, #20649) is visible. "Fixed on master" therefore means "a merged PR
changes this path". It does not mean the fix was proven live on staging.
- **Brief staging outage:** staging was in a maintenance window, `tenant_app_unavailable`, at about
00:11Z on 2026-11-11, during the census. That was another deploy. Data read before the window is
complete.
## Class table
The share column is the share of the 276 failed runs. The "Blocked now" column counts tasks still
blocked at census time.
| Class | Runs | Share | Tasks | Blocked now | Mechanism (root) | Fixed on master? | Why the harness missed it | Proposed scenario |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| W: wake loop on a held shared workspace | 103 | 37% | 4 | 1 | A parent shares a legacy execution workspace with a child that stranded a writer hold on Nov 5. The new stall-wake feature (#20098) woke the parent every 10 min. Each wake failed setup with no backoff. T01 had 99 runs from 07:27 to 15:53Z. | **Fixed:** #20273, #20295, #20314 | Fresh database every wave, so there was no legacy shared-workspace state. The feature landed between waves. U2 would have caught the loop. | None (fixed). If needed: K14 from a pre-2026-11-07 build with M1. |
| R: remote recovery refuses after a deploy | 34 | 12% | 32 | **24** | Recovery after a deploy refuses to continue. Codes include `runner_remote_process_identity_unavailable` (9), `native_remote_recovery_lease_mismatch …identity_run_id` (8), `prior_owner_terminal_state_indeterminate` (6), `cleanup_quarantined` (4) and `missing_evidence` (3). 27 of 34 failed within 10 min of a process start. | **Mostly fixed:** `identity_run_id` by #20642, `prior_owner…` by #20677, others by #20639, #20631, #20666, #20662 and #20650. **Open:** `runner_remote_process_identity_unavailable`, where the remote Runner is gone after a 4–7 min mid-turn outage. | Harness restarts come back at once, with no long mid-turn outage. There were no OpenCode cells on Daytona (19 of 34 are OpenCode). No Daytona workload keeps a tool running across the fault. | **K15 redeploy** (implemented). A long-hold and mid-tool variant with sandbox-side Runner log capture is the follow-up. |
| F: finalization and workspace save-back | 24 | 9% | 15 | 8 | A saved result cannot be copied back or restored. Causes: sandbox disk full (ENOSPC in 2 runs), unsafe symlink in the archive, `recovery_receipt_invalid` (7), a stuck "Recovering saved work…" phase (8) and an exhausted finalization budget (3). | **Partly fixed:** #20666 (receipt), #20578 (re-finalize). Nothing for disk full or symlinks. | Workloads write small files. They never fill the sandbox disk and never create symlinks. ST faults hit S3, not the sandbox disk. | **F1 symlink** workload (built; passes on master). **DF1 sandbox disk full** (proposed). |
| E: Daytona capacity outage | 18 | 7% | 13 | 4 | Daytona returned HTTP 502 and the worker hit SIGKILL from 16:36 to 17:02Z on Nov 10. Reusable sandbox resume failed, plus a git clone network failure. | **Mostly recovers:** 8 tasks recovered automatically and bounded retry works. 4 tasks still blocked. | The harness has no environment-provider fault. All faults hit the control plane or S3. | **EP1 environment provider 5xx window** (proposed, not built). |
| U: provider usage limit | 17 | 6% | 12 | 8 | `provider_limit_wait` waits as designed. The ACPX "terminal limit failure" (7) fails the run, where it should wait. | **Open:** ACPX limit classification. | Not a restart class. Provider shims have no usage-limit message. | Add a usage-limit message to PV faults (proposed). |
| P: provider session resume after a deploy | 16 | 6% | 16 | **11** | The runner or provider cannot resume its session after the controller moved. Codes: `PRP provider resume state is unavailable` (5), `runner_harness_state_mismatch` (8) with sub-codes `provider_identity_incomplete`, `resumed_unavailable` and `backup_without_reusable_lease`, plus `ingress_auth_failed` and `run.attach provider_side_effect_pending`. 8 of the 16 are ACPX and 5 are OpenCode. | **Partly fixed:** #20627 (warm backup). `resume state unavailable` is open. | Same as R: same process space, no OpenCode, no reusable warm sandboxes across deploys. | K15 with `--daytona-warm` and `--adapters claude,opencode`. |
| O: other or unclassified native failures | 13 | 5% | 13 | 5 | `native_run_failed` with no classified error (3), ACPX `session.open` rejected (2), `local_runtime_api_listener_unavailable` (2), `tool.resolve provider_tool_call_retired`, a DB query error, and `Object not found`. | **Open:** classification. | Not deploy-correlated (0 of 13 within 10 min). | None. These are product triage items. |
| B: authorization or binding readiness | 13 | 5% | 9 | 3 | `reviewed_chat_execution_binding_not_authorized` (7) and `not_ready` (2), a credential not shared with the responsible user (2), and a skill package still uploading (2). | **Open** | The harness uses one operator. There are no chat bindings, no shared credentials and no skill uploads. | Out of scope for restart chaos. Route to product. |
| L: legacy runtime lost across a deploy | 12 | 4% | 12 | 5 | Legacy (non-native) runs: `process_lost` (5), `Legacy controller lease lost` (4), graceful SIGTERM (1) and `lease_released_before_terminal` (2). | **Recovers in most cases:** 7 of 12 tasks recovered. | The matrix runs native adapters only. | K15 with a legacy adapter cell (proposed). |
| X: expected cancellations | 11 | 4% | 9 | 2 | Reassignment, ownership changes, dependency blocks and operator interrupts. | No fix needed | n/a | none |
| A: runner or provider-pack artifact incompatible after an upgrade | 8 | 3% | 8 | 4 | New code meets a sandbox prepared by older code. Codes: `OpenCode executable is missing from the provider pack`, `ACPX launch profile omitted its provider-pack authority`, `native_provider_artifacts_admission_unverified` (3), `runner_remote_artifact_platform_mismatch` and `runner_revision_mismatch`. | **Open** | Daytona waves require the runner content to match the image and refuse mismatches. That is exactly the mixed-version state staging lives in. K14 is local only. | **K15 with `--upgrade-from` on Daytona** (allows mixed runner/image versions; proposed follow-up). |
| V: runner event commit stall | 7 | 3% | 5 | 2 | `native_event_replay_conflict: runner_event_commit_stalled`, Codex and ACPX, 20:49–23:55Z on Nov 9. No deploy correlation. | **Likely fixed:** #20385 (usage receipt stalls). No occurrence after 23:55Z on Nov 9. | Database contention (L1) used one-row pressure, not the receipt-update pattern. | none (fixed) |
Totals: 276 runs and 114 tasks. Classes R, P, A and L together are the deploy-crossing failures:
70 runs (25%) and **44 currently blocked tasks**, which is most of the user-visible harm. Class W is
the largest by count but is fixed.
## Deploy timeline (Railway, from run records)
47 process starts between 2026-11-08T22:03Z and 2026-11-10T23:23Z. All but one are new
`deploymentId`s. The exception is a same-deployment restart at 2026-11-09T14:28Z. The median gap in
working hours is about 40 minutes. Every native run longer than about 40 minutes crosses a deploy.
67 of 276 failures (24%) finished within 10 minutes after a process start. In classes R and P, 30 of
50 did. In classes E, O and V, almost none did.
## Per-class detail
### W: wake loop on a held shared workspace (103 runs, fixed)
- Examples: T01 (99 runs), T02, T03 and T04.
- Timeline:
1. On Nov 5, child T05 failed setup (`workspace_busy`, then `setup_failed`). Its parent
T01 shared the workspace, a pre-2026-11-07 policy.
2. On Nov 9, #20098 ("wake the parent once when a child stalls") reached staging. At 07:27Z the
parent began to be woken about every 10 min by `issue.blocker_stalled_wake`.
3. Every wake failed with `The workspace still has an unfinished writer or save operation.`
4. At 15:53Z the parent became `blocked`.
- Fixed by:
- #20273, which stops stall wakes from failing setup on a held workspace;
- #20295, which re-proves an exhausted save's agent stop;
- #20314, which moves stranded legacy subtasks off a parent's held workspace.
- Why the harness missed it:
- **Old saved state:** every wave starts from an empty database, so a parent/child pair sharing a
workspace (the legacy policy) never exists.
- **Missing feature in the tested build:** #20098 landed at 00:26Z on Nov 9, after the last wave
that had a stalled child.
- **Timing window:** the loop starts only after a child stalls for longer than the stall threshold.
- **Grading is not the gap.** The enforced U2 check already fails a task after more than three
failures with the same code and no status change. The parent would have failed U2 on its fourth
wake, if a wave had created the legacy state.
- Scenario: none is built in this pass, because the class is fixed and the shared-workspace policy
is retired for new subtasks. A K14 wave from a pre-2026-11-07 build with M1 would recreate the
legacy state, if the fix needs regression cover.
### R: remote native recovery refuses after a deploy (34 runs, 24 tasks blocked)
- Examples:
- T06, T07, T08 and T09 (`lease_mismatch identity_run_id`);
- T10, T11, T12 and T13 (`identity_unavailable`);
- T14, T15, T16 and T17 (`prior_owner_terminal_state_indeterminate`, with
`retirement_refused=journal_commands_unsettled(run.attach.pending | semantic_tool.result.pending)`);
- T18 and T19 (`cleanup_quarantined`).
- Mechanism, per sub-code. This was corrected after review by the class R fix owner.
- Native recovery does **not** depend on PID proofs. It uses `proveOwnerGone`, which accepts a
replaced Railway deployment on an exclusive volume (`deployment_replaced`) or a stopped or fenced
boot row. The only PID-only caller left is the runtime skill cache lock (#20688).
- `native_remote_recovery_lease_mismatch … identity_run_id`: the leases were ephemeral and the
sandboxes fresh, so a reused sandbox was not the cause. This is the shape that #20642 fixed. The
last staging occurrence was 14:43Z on Nov 10.
- `prior_owner_terminal_state_indeterminate` with unsettled `run.attach` or `semantic_tool.result`:
fixed by #20677. Retirement is no longer held when the prior ephemeral sandbox is provably
destroyed.
- `runner_remote_process_identity_unavailable` is **still open**:
1. The controller dies mid-turn while tools run.
2. Recovery starts 4–7 minutes later.
3. By then the remote Runner process is gone, or its `session.snapshot` times out.
Why the Runner exits despite its 60-minute connection lease is not yet known.
- Last occurrences: `identity_unavailable` 2026-11-10T08:10Z, `lease_mismatch` 14:43Z, `prior_owner…`
18:56Z and `cleanup_quarantined` 17:48Z.
- Code pointers:
- `server/src/services/native-runtime/native-session-executor.ts:14247` (identity wait);
- `server/src/services/native-runtime/remote-recovery-lease-mismatch.ts`;
- `server/src/services/heartbeat/restart-interrupted-turn.ts`.
- Why the harness missed it:
1. **Deploy timing:** a staging deploy sends a graceful SIGTERM, often 1–3 s after
`native.coordinator.claim`, and the server stays down for 4–7 minutes. K2 restarts at once,
and K4's long hold uses SIGKILL.
2. **Missing adapter:** OpenCode (19 of 34) never ran in a Daytona cell.
3. **Mid-turn Runner loss:** no Daytona workload keeps a tool running across the fault. The
`tool` trigger covers only the local B5 shell bench, so the remote Runner is never under a
running tool when the controller goes away.
4. **Missing environment shape:** `--daytona-reuse-lease` and `--daytona-warm` are not in the
matrix.
- Scenario: **K15 redeploy** (implemented). The fault does a graceful SIGTERM like K2. The
replacement then starts in a new emulated process space (a new hostname through a preload, so
`processSpaceHash` changes) and with a new Railway-shaped deployment identity on the same volume.
Selected with `--faults K15`. RED wave: see "Harness expansion" below.
### F: finalization and workspace save-back (24 runs, 8 tasks blocked)
- Examples:
- T20 (`sync_out_retry_exhausted` from disk full, then the finalization budget, three times);
- T21 (`remote_workspace_archive_unsafe_symlink`);
- T22, T23 and T24 (`remote_workspace_recovery_receipt_invalid`);
- T25 and T10 ("Recovering saved work…" phase cancelled with
`execution_reconciliation_required`);
- T26, T27 and T28 (07:26Z on Nov 9, right after a deploy).
- Fixed: #20666 (skip unverifiable recovery sources) covers `receipt_invalid`, which last occurred
at 22:35Z, before that fix was deployed. Disk-full and symlink save-back have no fix and no
scenario.
- Why the harness missed it: **missing workload.** The workloads write a few KB, never fill the
sandbox disk and never create symlinks. ST faults break S3, not the sandbox file system.
- Built: the **F1** workload (`--workloads F1`). The agent keeps an absolute symlink to `/etc/hosts`
in its repository workspace. Wave `w-20261111T012523-s9304` (Daytona, `--storage s3`, K2, Codex
and OpenCode) passed 2 of 2 on master. The save-back did not block the task. The harness keeps no
evidence for passing tasks, so it is not verified that the link existed at save-back time.
- Proposed (not built): `DF1`, which fills the sandbox disk to ENOSPC before result copy-back.
### E: Daytona capacity outage (18 runs, mostly self-healing)
- 2026-11-10 16:36–17:02Z:
- `Daytona sandbox creation failed (HTTP 502)` (10);
- `Worker process exited (code=null, signal=SIGKILL)` (3);
- `Reusable sandbox lease … could not be resumed (cause=provider_error | transport_timeout)` (3).
- Plus one `git=network` clone failure and one Daytona rate-limit failure.
- Bounded retry recovered 8 of 13 tasks. 4 remain blocked (T29, T30, T31 and
T32 are examples).
- Fix #20626 (retry and replace a reusable sandbox that will not resume) shipped before the outage.
Its effect at this scale is not verified.
- Why the harness missed it: **missing fault type.** There is no environment-provider fault. All
harness faults hit the control plane, S3 or the provider message stream.
- Proposed: `EP1`, a proxy in front of the Daytona API that returns 502 or times out for a window.
### U: provider usage limit (17 runs)
- `provider_limit_wait` (10) is the designed wait.
- The ACPX "terminal limit failure" (7, an error-triage agent) fails the run. It should wait. This is a product
classification gap in the ACPX limit mapping.
### P: provider session resume after a deploy (16 runs, 11 tasks blocked)
- Examples:
- T33, T34, T35 and T25 (`PRP provider resume state is unavailable`);
- T36, T37, T38 and T39 (`runner_harness_state_mismatch …`);
- T40 (`run.attach provider_side_effect_pending`).
- Last occurrence: 2026-11-10T21:06Z.
- #20627 (restore a warm backup when its source sandbox is idle) covers `backup_without_reusable_lease`
and `resumed_unavailable`. `resume state is unavailable` and `provider_identity_incomplete` have no
fix.
- Why the harness missed it: the same reasons as R, plus ACPX (8) and OpenCode (5) on warm reusable
sandboxes, which the matrix does not run.
### O: other (13 runs)
- Not deploy-correlated.
- `native_run_failed` without a classified provider error appeared on T41, T42 and
T43. Those runs reached `run.result.accepted` after two `turn.failed` events.
- ACPX `session.open` was rejected on T44 and T45.
- These are product triage items, not chaos gaps.
### B: authorization or binding readiness (13 runs)
- `reviewed_chat_execution_binding_not_authorized`: 7 runs on a PR-review agent (T30 and T46).
- A credential not shared with the responsible user.
- Skill package uploads still in flight (T47).
- These are product authorization and readiness checks. They are outside the restart-chaos domain.
### L: legacy runtime lost across a deploy (12 runs)
- Examples:
- `process_lost` on T26, T27 and T48 (07:11Z on Nov 9, three minutes after a
deploy);
- `Legacy controller lease lost` on T49, T50, T51 and T52 (08:14–08:19Z on
Nov 10, 20 minutes after a deploy).
- 7 of 12 tasks recovered.
- The matrix has no legacy adapter cell. K15 could run one, but this was not done in this pass.
### A: artifact incompatible after an upgrade (8 runs)
- Examples: T53 and T54 (`OpenCode executable is missing from the provider pack`), T55
(`ACPX launch profile omitted …`), T56, T57 and T58
(`native_provider_artifacts_admission_unverified`), and T59 (`runner_revision_mismatch`).
- Mechanism: a sandbox prepared and admitted by older code is resumed by newer code that expects
new launch-profile fields or provider-pack contents.
- Why the harness missed it: Daytona waves refuse to run unless the runner content ID equals the
image (`daytona-image.ts`). This is deliberate, and it excludes exactly this mixed-version state.
- Follow-up: K15 with `--upgrade-from` on Daytona, with the image of the older checkout.
### V: runner event commit stall (7 runs, likely fixed)
- Examples: T60 and T22 (Nov 9, 20:49–23:55Z).
- #20385 (usage receipt stalls during streamed run updates) merged at 23:35Z, and the class has not
occurred since.
## Harness expansion (this pass)
| Scenario | Classes | Selector | Status |
| --- | --- | --- | --- |
| K15 redeploy into a new process space and deployment identity | R, P, L (and A with `--upgrade-from`) | `--faults K15` | Implemented, unit-tested. Five Daytona waves are recorded below. |
| EP1 environment-provider 5xx window | E | — | Proposed |
| F1 absolute symlink kept in the repository workspace | F (unsafe-symlink save-back) | `--workloads F1` | Implemented, unit-tested. Daytona wave: 2 of 2 pass on master. |
| DF1 sandbox disk full | F (ENOSPC save-back) | — | Proposed |
### Waves on master 8f6ca25976
All waves ran on Daytona image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:bc7ca92d…`
(runner content `d7e4d82a…`).
| Wave | Args | Result |
| --- | --- | --- |
| `w-20261111T002313-s9301` | `--faults K15 --adapters opencode,claude --workloads A1,C1 --daytona-reuse-lease --concurrency 3 --tasks 6` | 5 of 6 pass. t05 (OpenCode, C1) failed R5 `silent_dead_run`: its run stayed `running` with no event, activity or start evidence for 183 s after the redeploy, then succeeded. No Retry was needed. |
| `w-20261111T004241-s9301` (control) | the same seed and cells with `--faults K2` | 6 of 6 pass |
| `w-20261111T005008-s9302` | `--faults K15 --triggers time,comment --adapters opencode,claude --workloads C1 --daytona-warm --concurrency 4 --tasks 4` | 4 of 4 pass |
| `w-20261111T010000-s9303` | `--faults K15 --adapters opencode --workloads A1,C1 --daytona-reuse-lease --concurrency 4 --tasks 8` | 8 of 8 pass |
| `w-20261111T011044-s9303` (control) | the same seed and cells with `--faults K2` | 8 of 8 pass |
What this shows:
- Under K15, 17 of 18 tasks passed. Under K2 with the same seeds, 14 of 14 passed. The one failure,
a 183 s silent OpenCode run after a redeploy, occurred only under K15. One failure in 18 is not a
stable reproduction.
- The staging refusal codes of classes R and P did **not** reproduce on current master with K15
alone. Examples are `identity_run_id` lease mismatch, `runner_remote_process_identity_unavailable`
and `prior_owner_terminal_state_indeterminate`. Two explanations remain open:
- The fixes merged during the window (#20625, #20631, #20639, #20650, #20662, #20666) cover them.
Staging ran older revisions for most of the window.
- They need conditions that K15 does not emulate yet:
- a code change at the deploy, where new server code meets saved state and a sandbox runner from
the older build (class A);
- one sandbox reused across different tasks of one agent;
- many agents and concurrent tasks under load.
- The next scenario to build is **K15 with `--upgrade-from` on Daytona**: start with the older
checkout and its image, redeploy to the current checkout, and keep the older runner in the
retained sandboxes. Today the Daytona image check refuses that mixed state on purpose.