docs(plans): Phase 2 Daytona wave results (#19947)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Phase 2 of the durable-ownership plan moves in-progress record kinds
to one shared ownership path, guided by chaos data.
> - The first data doc (#19927) covered local waves only. Remote
runners, leases and sandboxes need a Daytona wave.
> - This pull request adds the results of one Daytona wave to the data
doc.
> - The benefit is that Phase 2 and Phase 3 prioritize remote kinds from
measured data.

## Linked Issues or Issue Description

Refs #19927 (data doc), #19934 (claimed-wake settle), #19863 (speed
budgets).

**What is the problem?**
The Phase 2 data doc had no remote data.

**Where is it documented?**
`doc/plans/2026-11-07-phase2-reconciler-data.md`.

**What should change?**
Add the Daytona wave's per-kind violations, settle latency, failures and
speed numbers.

## What Changed

- Added a "Daytona wave" section: 6 tasks with K1, K2 and two K4 outages
of about 575 s. 5 of 6 passed, and R6 was clean.
- Recorded the t03 failure: a Retry was cancelled by the company-wide
accounting gate (Phase 3 scope).
- Recorded the claimed-wake leak again (#19934 fixes it) and remote
settle times of 76–107 s.
- Corrected the earlier note about the image content-ID mismatch.

## Verification

- Wave `w-20261108T012041-s9301`; the numbers come from a copy of the
kept database and the read-only ownership sampler.
- This is a docs-only change.

## Risks

- Low. Docs only. The samples are small and the doc says so.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/plans/2026-11-07-durable-ownership-and-reconciliation.md

@@ -126,7 +126,7 @@ What the data does and does not show:
 
 ### Phase 2 data: harsh local waves (2026-11-07)
 
-Storms, crash loops and outages up to 588 s, all unmasked, are in [`2026-11-07-phase2-reconciler-data.md`](2026-11-07-phase2-reconciler-data.md). Existing paths settle most run-lifecycle records within 45 s of `ready`. This is not a bound: one environment lease took 119 s, and some native claims stayed proven gone during a load spike. One leak was found: a claimed wake whose run ended without settling it. The unowned pending-cleanup queue and unstamped accounting rows are the main I1 findings. Daytona was not run in this pass (see the data doc).
+Storms, crash loops and outages up to 588 s, all unmasked, are in [`2026-11-07-phase2-reconciler-data.md`](2026-11-07-phase2-reconciler-data.md). Existing paths settle most run-lifecycle records within 45 s of `ready`. This is not a bound: one environment lease took 119 s, and some native claims stayed proven gone during a load spike. One leak was found: a claimed wake whose run ended without settling it. The unowned pending-cleanup queue and unstamped accounting rows are the main I1 findings. One Daytona wave found the same claimed-wake leak, remote settling of 76–107 s, and a Retry blocked by the company-wide accounting gate (Phase 3).
 
 ## Relationship to the zero-downtime upgrade work
 

doc/plans/2026-11-07-phase2-reconciler-data.md

@@ -24,7 +24,7 @@ Long holds (K4 up to 579 s, so outages up to 588 s, past the Runner's 7-minute r
 
 TTFT, boot and restart recovery stay within the p50 budgets for cold starts and for recovery. The exceptions are claude cold TTFT, at the budget (16.3–17.5 s against 16.5 s), and warm follow-ups on claude (about 3×) and codex (slightly over). See "Speed".
 
-Daytona did not run in this pass. A follow-up adds one Daytona wave. See "Not covered".
+One Daytona wave ran afterwards. It found the claimed-wake leak again, slower remote settling (76–107 s), and a Retry blocked by the company-wide accounting gate. See "Daytona wave".
 
 ## Waves
 
@@ -127,6 +127,52 @@ Findings:
 - Recovery after an outage past the reconnect grace needs its own metric: from `ready` to the first provider event of the run that continues the task (the retry), not of the interrupted run.
 - No change in this phase adds work before `turn.start`. The settle paths run in restart recovery and in the background reconciler.
 
+## Daytona wave (2026-11-08)
+
+One wave ran on Daytona with the same reconciler and budget instrumentation: `w-20261108T012041-s9301`. Its setup:
+- the controller ran at docs head `197e5165a4` (runner content ID `0a65c1e5…`);
+- the signed `image_only` runner image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:35823e49…` was built from master `1de794c` with the same content ID;
+- 6 tasks on codex, claude and opencode;
+- faults: K2 ×2, K1 ×2, and two long K4 holds (498 s and 399 s, so outages of 575 s and 573 s, past both the 5-minute Daytona auto-stop and the 7-minute reconnect grace);
+- load 10.8–15.5, except the last fault at 60, caused by other sessions' processes.
+
+All 6 tasks crossed a fault and 5 passed. R6 (sandbox leaks) was clean.
+
+- **t03 (claude A1 K2) failed R3 `retry_no_new_run` (P, Phase 3).**
+  - The previous run failed `process_lost`, and the task offered Retry.
+  - The Retry run `7a93bd1e` was cancelled `budget_blocked` by the company-wide accounting gate: "New work must wait while a native run recovers its unfinished accounting."
+  - The same gate skipped assignment wake `c0b2a035` terminally, which is a real I4 (the other I4 in this wave was a delivered retry, the known false positive).
+  - Spend was far below the 3,000-cent cap.
+  - This is the gate that decision 1 removes. The evidence went to the Phase 3 session.
+- **Two runs failed `native_restart_recovery_blocked` (`ambiguous_remote_recovery_evidence`) after the long outages.** Both tasks still finished. Each run left its assignment wake `claimed` until the wave ended (37+ minutes). This is the same leak the K4 local rerun found, and #19934 settles it.
+
+Violations recorded by the reconciler (distinct records):
+
+| Rule | Kind | Records | Settled after `ready` |
+|---|---|---|---|
+| I2 | `wake_request_claimed` | 2 | never (stuck until the end: the #19934 leak) |
+| I2 | `environment_run_lease_active` | 2 | 76–107 s |
+| I2 | `issue_execution_lock`, `issue_checkout_lock` | 2 each | 76–107 s |
+| I2 | `native_pending_workspace_export` | 2 | 98–107 s |
+| I2 | `native_awaiting_evidence` | 2 (`pid_dead`) | 26.5 s |
+| I1 | `environment_pending_cleanup` | 3 | queue item, no owner by design |
+| I1 | `environment_teardown_claim` | 1 | claim written without an owner stamp |
+| I1 | `environment_login_probe_lease` | 1 | no owner stamp |
+| I4 | `wake_request` | 2 | 1 real (accounting gate), 1 false positive |
+
+Remote recovery settles these kinds more slowly than local recovery: 76–107 s after `ready` against under 45 s locally. Gone owners seen while the server was down or recovering (record-passes): `native_executor_claim` 57, `wake_request_claimed` 56, `sandbox_creation_intent` 29.
+
+Speed: remote environments have no budget yet (#19863). These are the first remote numbers, and they are small samples:
+
+| Measure | Value |
+|---|---|
+| Uncrossed runs: opencode cold TTFT | 15.2 s, boot 12.5 s |
+| Uncrossed runs: codex warm boot | 39.8 s (n 1) |
+| Restart recovery from `ready` to the next provider event | claude 37.8 / 42.2 s (n 2), codex 80.3 s (n 1) |
+| Runs with no provider event before the next fault | 4 |
+
+A Daytona budget needs a dedicated remote baseline. Restart recovery is 5–10× the local values.
+
 ## Step 2: kinds moved to the shared path
 
 Order of the moves, highest risk first:
@@ -149,10 +195,7 @@ Phase 3 is not started. It needs a separate go-ahead.
 
 ## Not covered
 
-- **Daytona.** No signed runner image existed for the tested commit.
-  - An apparent content-ID mismatch with CI run 42422571919 was a misreading: that run was dispatched from master but built target commit `a37a2e4`, and its Daytona image step was skipped.
-  - Local content IDs reproduce CI's for the same commit; for example, `35d41f11` gives `11475796…` on both.
-  - A follow-up builds an `image_only` image and adds one Daytona wave.
+- **Daytona** is covered by one wave only (see "Daytona wave"). The first apparent content-ID mismatch was a misreading: CI run 42422571919 built target commit `a37a2e4`, not master. Local content IDs reproduce CI's for the same commit.
 - The remote, workspace-operation, sandbox and accounting kinds (see "Per-kind data").
 - Crash points that sit between two durable writes. Random waves hit 20 ms–1 s windows only by luck. The crash-point suite (Phase 5) is the tool for those.