diff --git a/doc/plans/2026-10-10-boat-validation.md b/doc/plans/2026-10-10-boat-validation.md index 7d5763c4847..1aa0435a7f3 100644 --- a/doc/plans/2026-10-10-boat-validation.md +++ b/doc/plans/2026-10-10-boat-validation.md @@ -2,16 +2,19 @@ **Experimental and unmerged. Local journeys and partial staging qualification passed on the sources below; final-source staging acceptance remains incomplete.** A successful build, probe, or earlier-source journey does not establish the remaining product conditions. -## Source boundary +## Current source boundary (2026-10-11 03:29 UTC) | Surface | Source and evidence | | --- | --- | -| Canonical combined branch | `21deac0abfd2919525114f8cdc4a0d49f139665a`: exact Boat cleanup scope, verified retained ACPX provider packages, and fixed-field cancellation diagnostics. All identity and settlement gates remain. | -| Core branch | `66ac692e446bd2ce0175613787f82eeb048f2749`; optional cancellation diagnostics are in the acceptance PR. | -| Serving staging | `610c78a5ad97534380c61fa376c96c1b7a73a86a`, deployment 13 verified 2026-10-11 at 01:05:49 UTC; official receipt checked at 01:05:58.740. Runtime-equivalent to d50268c932 with compatible migration metadata. | -| Deployment / retry boundary | Deployment 12 had verified 1830beb serving on 2026-10-10 at 22:14:02.236; its failed retries remain recorded below. One ordinary Retry on deployment 13 started run `5957cc86` at 01:06:33, requesting a fresh session; fresh run succeeded at 01:09:46. It cannot diagnose the old saved-session failure. Its ordinary warm follow-up `1b882646` passed. Later `2ac97bd2` was unexpectedly cancelled, and two repair attempts failed before provider work; idle resume remains unverified. Earlier deployment 11 legacy passes remain attributed to 2df40da. | +| Combined branch | Runtime `3bd413a9b3`; corrects both pending-wait lifecycle defects. | +| Core branch | `11ccf7b1a2d8ad2921cff2a41ae2265e0896640b`; 338 actual-entrypoint tests, server TypeScript and independent review passed. Fresh CI remains pending. | +| Serving staging | Deployment 16 `2e8859ab762e5b2ce97f2ea774e385fd333f9cee` verified at 02:59:15.776 UTC. | +| Cancelled candidate | Deployment 17 `ca8ab88c78556c4711d59f71c6293619543a66a1` was cancelled during build before check/deploy. Fresh review found that Stop could leave a retry authorized and that premature terminalization could release the task lock. Corrections passed real cancellation/lifecycle-path tests and independent review; deployment18 of `bd86e22e72` is in progress. | +| External blocker | At 03:23:54 UTC, staging still had a failing stop and no successful snapshot after 02:19:59 UTC. Local recovered: snapshot verified at 03:16:03.591 and stop completed at 03:16:03.734; provider state was archived. No force, resume, or state reset was used. | -The [consolidated retry receipt](assets/2026-10-10-boat/evidence/retry-qualification-summary.json) records deployment 12 and 13 verification, exact retry outcomes, checks and original-receipt hashes. Deployment proof does not establish a product journey. Historical [candidate provenance](assets/2026-10-10-boat/evidence/final-candidate-provenance.json) and [deployment 11](assets/2026-10-10-boat/evidence/staging-deploy-11-verified.json) retain their original cutoff. The experimental staging database retains its previously applied migration prefix; future staging deployments must preserve it. The main PR retains canonical history. Deployment 9 stopped before provider deployment mutation; subsequent old-database upgrade, schema and forward-manifest checks passed without reset, history rewrite or guard bypass. +Latest source-specific live successes are staging warm HMR and corrected native computer use, local detach/reconnect retention, and local saved-session continuation after controller restart with an unchanged one-line marker. Latest native Claude saved-session recovery, active sibling cancellation isolation, and the new pending-wait UI journey remain unverified. A local cancellation during a **failing** stop is narrower evidence. + +The [consolidated retry receipt](assets/2026-10-10-boat/evidence/retry-qualification-summary.json) preserves exact sources, timestamps, original failures, and receipt hashes. A deployment proves serving source, not product acceptance. Staging retains its previously applied migration prefix; no database reset, history rewrite, or guard bypass was used. Historical observations below retain their original cutoffs. ## Product outcomes @@ -41,7 +44,7 @@ These outcomes do not erase earlier source-specific passes. The [retry receipt]( | --- | --- | | Staging 1830 native Codex | Run `20c69872` failed after three same-run attempts with `runner_remote_retirement_checkpoint_unsettled`, before a new runner launched. Remote checkpoint lifecycle was `ready`, runner outbox was empty, and pending terminal delivery was null; the controller proof rejection is not yet identified. Diagnostics now serve on deployment 13; UI retry `5957cc86` requested a fresh session and skipped saved-session resume. It succeeded, but cannot diagnose this old recovery failure. | | Staging 610 fresh → warm | Fresh `5957cc86` succeeded at 01:09:46 (3m13). Ordinary follow-up `1b882646` completed around 01:12:14. Independent receipts preserved normalized session, provider thread, runner PID 30466/start ticks 30188, Codex PID 30570 and Vite PID 31093, with unchanged proof hash. Later configuration turn `2ac97bd2` was cancelled; it is excluded from the successful warm claim. Saved-session continuation after natural idle remains unverified. | -| Staging 610 preview / HMR | The normal generated link rendered, with an unsaved draft and page-session baseline. The reported edit never appeared despite Vite reporting connected. Published and Vite-cwd files had identical bytes but different device/inode identities in the same mount namespace; this does not establish the cause. Polling configuration existed on disk, but loading by the running server was not proved. The fixture lacked a React HMR plugin; proposed repair never executed. Preview later became upstream-unavailable after owner retirement. **HMR has not passed.** | +| Staging 610 preview / HMR | The normal generated link rendered, with an unsaved draft and page-session baseline. The reported edit never appeared despite Vite reporting connected. Published and Vite-cwd files had identical bytes but different device/inode identities in the same mount namespace; this does not establish the cause. Polling configuration existed on disk, but loading by the running server was not proved. The fixture lacked a React HMR plugin; proposed repair never executed. Preview later became upstream-unavailable after owner retirement. **HMR did not pass on this source.** | | Staging 610 unexpected cancellation / quarantine | `2ac97bd2` ended **cancelled**, exit 0, at 01:19:01.499, with reason “Execution stopped without a recorded cancellation request”, source unknown and expected false. It is not a successful run. Follow-ups `139092c0` and `d33c8290` failed with `native_session_cleanup_quarantined` before provider work; no fixture repair ran and quarantine was preserved. | | Staging 610 retirement / persistent proof | Exact owner `f298be2e` retirement marker dates to 01:19:05.647, generation 3. At 01:28:20–01:28:58, no owned processes or listeners remained and the original proof hash was intact. The saved `2ac97bd2` checkpoint remained ready with empty outbox/no pending terminal and authoritative inactive provider turn; that does not override the controller cancellation. Boat was healthy and hydrated at 01:27:43 with no newer provider stop. This is post-cancellation cleanup evidence, not a natural idle-resume pass. | | Staging 610 native Claude | `a38070b3` failed `setup_failed` after 12m42 (01:18:44→01:31:25). The expanded log identifies a durable ACPX launch-profile digest mismatch after approximately 990MiB provider-pack upload. A later lease-acquisition attempt reported “Computer is connecting or settling a previous stop”; a lease-deadline cause is not proved. Recovery remained bootstrap with provider work not started. Stage sync took 473,763ms; transport connected at 01:27:47. An independent 01:29:48 checkpoint read found suspended state, empty outbox and pending terminal delivery. The reviewed correction on 21deac0abf reuses the package pinned by bound controller state. It passed 33 focused tests and server TypeScript; deployment 14 is pending, with no live recovery pass yet. Earlier Claude passes remain source-specific. | @@ -51,7 +54,7 @@ These outcomes do not erase earlier source-specific passes. The [retry receipt]( | Staging 1830 native Claude / sibling test | Run `3a34` failed during setup before the planned idle/sibling experiment. No sibling isolation pass or root cause is claimed. The fixture later naturally archived. | | Staging 1830 desktop / close | Stream visible at 00:43 and 00:50 on 2026-10-11. Input attempts produced no observable result, and one timed out before sending input: controls remain unverified on this retry. Normal Close Computer preceded natural archive; stop completed 00:50:43.150 and archived/snapshots-preserved state was observed 00:51:49. No manual provider stop. | | Local 817 controller restart | First run `3681473c` succeeded. Orderly shutdown retired the old owner; the remote checkpoint was suspended, old descendants absent, and the one-line marker retained. Continuation `f5f361a7` failed with cleanup quarantine after host sleep spanning 22:31–22:48 on 2026-10-10. **Inconclusive** for successful restart continuation or active-crash recovery; quarantine was preserved. | -| Local 817 fresh task | Initial setup attempt `de63e6ba` was also confounded by host sleep. Ordinary retry `e8c52459` succeeded 00:42:00.094→00:45:27.883 on 2026-10-11; independent read found exactly one marker line and its matching hash. Its exact owner later retired and physical archive completed at 00:52:14.179; no post-archive checkpoint read was attempted. No restart/continuation has yet been attempted for this task. | +| Local 817 fresh task | Initial setup attempt `de63e6ba` was also confounded by host sleep. Ordinary retry `e8c52459` succeeded 00:42:00.094→00:45:27.883 on 2026-10-11; independent read found exactly one marker line and its matching hash. Its exact owner later retired and physical archive completed at 00:52:14.179; no post-archive checkpoint read was attempted. At that cutoff no restart continuation had been attempted; local 36 subsequently passed, as recorded below. | | Local instruction save / detach | Two saves returned HTTP 200, but the subsequent fresh read returned 502 `provider_error` during the host-wake failure. Durable bytes were not independently verified; this is not evidence of a missing file. No detach was performed. | | Local authorization boundary | Authenticated foreign-company issue read and board foreign-task computer connect returned 404; no owner ledger change or foreign computer row was observed. Temporary credentials were revoked. This is a live local boundary pass, not a staging journey. | @@ -80,9 +83,13 @@ Local 36 restart continuation `6f31e383` succeeded 02:40:41.830–02:43:37.252. Deployment 16 **2e8859ab762e5b2ce97f2ea774e385fd333f9cee** verified serving at 02:59:15.776 through the official workflow. It carries the reviewed saved-package authority fix. Live saved-session verification remains blocked: at 02:56:29 Boat reported stop `804de979` as **failing**, refusing archive because no successful snapshot existed within 30 minutes. The latest snapshot attempt remained in progress. Provider `idle` is not a completed-stop receipt; no resume, forced stop, or bypass was issued. -The separate pending-stop correction is committed as core `a4c864a062`, combined `04ce7eb2a8`, and staging `ca8ab88c78`. It reuses durable scheduled retries before provider work and preserves cancellation, task ownership, explicit message authorization, and saved-session identity. It passed 437 focused tests with no skips using isolated real PostgreSQL databases; the final wording change passed 84 affected tests, server TypeScript passed, and independent review found no remaining issue. [Verification receipt](assets/2026-10-10-boat/evidence/computer-admission-wait-tests.json). The native test-cluster bootstrap failed, so tests used unique disposable databases on the task-owned server; none remain and the product database was untouched. Explicit one-run Retry/Interrupt intents retain their previous manual-retry behavior. Deployment 17 is in progress; this correction is not part of deployment 16. +The separate pending-stop correction is committed as core `a4c864a062`, combined `04ce7eb2a8`, and staging `ca8ab88c78`. It is intended to reuse durable scheduled retries before provider work while preserving cancellation, task ownership, explicit message authorization, and saved-session identity. Later integration review found that the current implementation does not yet satisfy the cancellation and task-lock conditions. It passed 437 focused tests with no skips using isolated real PostgreSQL databases; the final wording change passed 84 affected tests, server TypeScript passed, and initial independent review found no remaining issue, before the later CI findings exposed missing integration coverage. [Verification receipt](assets/2026-10-10-boat/evidence/computer-admission-wait-tests.json). The native test-cluster bootstrap failed, so tests used unique disposable databases on the task-owned server; none remain and the product database was untouched. Explicit one-run Retry/Interrupt intents retain their previous manual-retry behavior. Deployment 17 was cancelled before rollout after fresh CI review found a Stop race; this correction is not part of deployment 16. -The attempted local 37 UI check encountered a **failing** stop instead of a pending one: Boat reported that the final snapshot failed and kept the machine alive to protect unsaved data. Normal UI Stop cancelled read-only attempt `b380b52e` at 03:09:07.157, before any environment lease, computer owner, native runner, or provider work. This does not count as a live pending-wait or successful continuation pass; final cleanup/cancellation verification is recorded separately when complete. +The attempted local 37 UI check encountered a **failing** stop instead of a pending one: Boat reported that the final snapshot failed and kept the machine alive to protect unsaved data. Normal UI Stop cancelled read-only attempt `b380b52e` at 03:09:07.157, before any environment lease, computer owner, native runner, or provider work. This does not count as a live pending-wait or successful continuation pass. A 03:10:53 check found no owner, lease, provider dispatch, scheduled retry, or successor. That first sample was still preparing. A separate 03:13:38 DB-only check confirmed executor cleanup settled at 03:11:16.987, controller lease cleared, and still no successor or retry; no guest read or wake was used. + +The follow-up correction, core `11ccf7b1a2` / combined `3bd413a9b3` / staging `bd86e22e72`, separates pre-provider ownership retention from cleanup-gated scheduling. Actual Stop persists suppression under scheduling locks and cancels only the exact retry chain. It passed 338 tests across four suites with no skips, server TypeScript and independent review. Tests now exercise real lifecycle and cancellation entry points, including queued messages and unrelated/reassigned work. Deployment18 is in progress; this is not yet a live pending-wait pass. + +Local37 then attempted an ordinary saved Claude continuation after provider recovery. Run `7b6baf95` retained the original normalized session, provider thread and logical runner, but failed at 03:28:55.832 with `runner_remote_provider_cache_corrupt`. Lease cleanup succeeded and the new owner retired without a process binding. No proof file was read; the browser showed [Run failed and Blocked](assets/2026-10-10-boat/local-37-claude-resume-failed.png). Cache diagnosis is in progress; no cache reset or mutation was used. ## Checks and their limits diff --git a/doc/plans/assets/2026-10-10-boat/evidence/README.md b/doc/plans/assets/2026-10-10-boat/evidence/README.md index b96f29a545f..82568397948 100644 --- a/doc/plans/assets/2026-10-10-boat/evidence/README.md +++ b/doc/plans/assets/2026-10-10-boat/evidence/README.md @@ -2,13 +2,18 @@ These allowlisted summaries support the [validation matrix](../../../2026-10-10-boat-validation.md). They record observed results on named sources, not a blanket final-revision pass. Originals remain separately preserved; `provenance.json` records earlier originals; the consolidated retry summary carries its own original filenames and SHA-256 hashes. -## Current boundary +## Current boundary (2026-10-11 03:23 UTC) -Canonical candidate is `21deac0abfd2919525114f8cdc4a0d49f139665a`; core is `66ac692e446bd2ce0175613787f82eeb048f2749`. The reviewed fixes scope cleanup to its Boat owner and retain verified ACPX packages across deployment. Fixed-field cancellation diagnostics preserve outcome precedence. Deployment 14 candidate `5cf388845902c88d0406116ed9b2c44f335d8bb2` is pending; the serving evidence below remains deployment 13. Deployment 12 verified serving `1830beb0c26a7ba9d5e21460426936936ffc8db4` at 2026-10-10 22:14:02.236 UTC, runtime-equivalent to earlier 2ef1772. Deployment 13 verified 610c78a5 serving at 2026-10-11 01:05:49 UTC, runtime-equivalent to d502; UI Retry 5957cc86 started a fresh session and succeeded; the ordinary warm follow-up preserved runner/provider/Vite identity. Preview rendered, but HMR and saved-session idle recovery remain unverified. Later `2ac97bd2` was unexpectedly cancelled (not succeeded), its owner retired, and both repair attempts were quarantined before work. Native Claude `a38070b3` failed during setup; the first observed failure was an ACPX durable launch-profile digest mismatch after the pack upload, not a proved lease-deadline failure. The provider naturally archived at 01:32:48.421, confirmed at 01:33:52. A later normal Computer Connect resumed it for a desktop comparison: direct browser mouse and keyboard passed; embedded input remains unverified because automation targeted the outer page. Both viewers closed normally, but exact close time was not captured. At 01:47:06 the provider was archived with snapshots retained and stop completed at 01:42:11.287; precise close-to-archive causality is not claimed. The new retry summary records staging failures, local host-sleep limitations, a local authorization boundary pass, and natural viewer-close archive. Earlier receipts retain their exact sources and original cutoffs; the candidate/CI files below are historical, not current-status assertions. +Runtime sources are combined `3bd413a9b3` and core `11ccf7b1a2`. Deployment 16 verified `2e8859ab76` serving; deployment 17 for `ca8ab88c78` was cancelled during build before rollout. Later review found two admission lifecycle defects: Stop could leave a retry authorized, and premature terminalization could release the task lock. Both corrections passed 338 actual-entrypoint tests, server TypeScript and independent review; deployment18 is in progress. At 03:23:54 UTC, local Boat had recovered and archived with a verified snapshot; staging still had a failing snapshot-preserving stop, blocking staging saved-session acceptance. No forced stop, resume, or state reset was used. + +Recent source-specific passes: staging warm HMR preserving draft/page/process identity, corrected native computer use and persisted screenshot, local detach/reconnect retention, and local controller-restart saved-session continuation with unchanged one-line proof. Native Claude recovery on the latest package-authority fix, active sibling isolation, latest embedded input, and live pending-wait acceptance remain incomplete. Historical failures and earlier source boundaries remain in the validation matrix and receipts. | Receipt | Source / narrow claim | | --- | --- | -| [Latest retry qualification](retry-qualification-summary.json) | Deployments 12 and 13 verified; native failures on 1830 are preserved, and fresh and warm turns on 610 passed but cannot diagnose the old failed session. Preview rendered; HMR remains unverified and on-disk polling was not proved loaded. Later Codex cancellation, quarantine, owner retirement with intact proof, and Claude setup failure are preserved. Direct browser desktop mouse and keyboard passed; embedded input has an automation-targeting limit. Local restart/save observations are inconclusive; ordinary fresh-task retry, local authorization denial and viewer-close archive have narrower proof. Includes exact diagnostic-head CI: d502 with 52 successes/fresh 5/5/zero unresolved, and ee2 with 52 successes/2 skips/fresh review, plus original-receipt hashes. | +| [Latest retry qualification](retry-qualification-summary.json) | Deployments 12–16, exact failures and later HMR/computer-use/detach/restart successes; provider snapshot failure blocks latest staging sessions. Original hashes and narrower cancellation observations remain intact. | +| [Pending-stop correction tests](computer-admission-wait-tests.json) | 437 focused tests, 84 final affected tests and TypeScript initially passed, but later review found two integration defects, then the correction passed 338 actual-entrypoint tests and review; isolated disposable PostgreSQL databases, no product DB mutations. Live pending-wait remains unverified. | +| [Local restart retained proof](local-36-restart-retained.json) | Same saved session on a replacement machine, exactly one unchanged marker line; not active-crash recovery. | +| [CI on ea1272 / 44b1](ci-ea1272-44b1.json) | Historical exact heads: 52 successes each, 2 core skips, 5/5 reviews. New runtime-head review found the admission defects described above; those heads are not approved. | | [Staging 11 corrected legacy runs](staging-11-legacy-regression.json) | Both runs succeeded 21:23:29, real Claude Skill launched, both persistent entries/proofs read, helper disposition passed without recovery. Includes the Claude activity-display inconsistency, Codex corrected arguments, and natural archive at 21:23:43.156. | | [Deployment 11 serving receipt](staging-deploy-11-verified.json) | Official verification of 2df40da serving; no product pass implied. | | [Candidate provenance](final-candidate-provenance.json) | Exact canonical/candidate comparison. Historical cutoff: 2df40da built while 1830beb was pending. Later deployment 12 verification is in the retry summary. | @@ -27,7 +32,7 @@ Canonical candidate is `21deac0abfd2919525114f8cdc4a0d49f139665a`; core is `66ac | [Focused checks](focused-checks.json) | Earlier checks retain individual source attribution. | | [Clean file-resource CI](file-resource-clean-ci-proof.json) | Core 5e5fe7a7: 44 passed, 0 skipped, including 7 regressions. | | [Earlier core CI](core-5e5-final-ci-review.json) | Historical exact-head receipt; does not describe current core. | -| [Full-suite limits](full-suite-limits.json) | Frozen 2cbe65e9:34,914 passed, 24 failed, 338 skipped; original failures and separate retries preserved. | +| [Full-suite limits](full-suite-limits.json) | Frozen 2cbe65e9:34,914 passed, 24 failed, 338 skipped; original failures and separate retries preserved. | | [Compatibility](staging-compat-final-proof.json), [manifest](staging-compat-manifest-proof.json), [schema](staging-compat-schema-proof.json) | Source-specific runtime/schema equality and strict migration-prefix evidence. Future staging deployments must retain the compatible prefix. | | [Deployment 9 preflight](staging-deploy-9-assessment.json) | Stopped before provider deployment mutation; no reset/history rewrite/guard bypass. | | [Latest resource inventory](local-final-resource-inventory.json) | Local ledgers at 21:14:08: zero live owners/pending actions. Provider snapshots at 20:38/20:53: both local fixtures, builder and staging archived. Later qualification may resume a fixture. | @@ -36,7 +41,7 @@ The older `local-final-state.json` is a historical snapshot, superseded for clea ## Limits -The feature remains experimental and unmerged. Unexpected cancellation/quarantine, current-source Claude setup, saved-session continuation, staging HMR, embedded desktop input, sibling cancellation isolation, detach retention, live feature-off cleanup, staging company denial, and successful controller restart continuation remain incomplete. Local authorization denial, earlier viewer-close archive, and direct browser desktop keyboard/mouse passed. The latest two-viewer comparison showed stream interference and did not validate embedded input. Neither establishes the remaining journeys. Shared personal folders use one Unix UID; they are not an OS security boundary. Shell writers can race editor writes. Cold startup can be expensive; retained private staging/snapshot files may need operator cleanup. +The feature remains experimental and unmerged. Current-source Claude saved-session recovery, embedded desktop input, active sibling cancellation isolation, live pending-wait continuation, feature-off cleanup, staging company denial, and active-crash recovery remain incomplete. Local orderly controller restart, detach retention, staging HMR, and corrected native computer use subsequently passed on their recorded sources. Shared Unix UID is not an OS security boundary; shell writers can race editor writes. Cold startup and provider archival can be slow. Staging snapshot failure still prevents safe archival; local recovered and archived at 03:16:03 UTC; onboarding and builder were already archived. The frozen full-suite result is not green. Later fixes/retries do not erase it: the isolated Git streaming retry exceeded its unchanged 300-second timeout, and CLI/persona cases remained blocked by native PostgreSQL bootstrap/shared-memory exhaustion. Earlier local file-resource skips remain recorded despite subsequent clean-CI success. diff --git a/doc/plans/assets/2026-10-10-boat/evidence/computer-admission-wait-tests.json b/doc/plans/assets/2026-10-10-boat/evidence/computer-admission-wait-tests.json index 23b01d359fb..a6a1d9dbc6c 100644 --- a/doc/plans/assets/2026-10-10-boat/evidence/computer-admission-wait-tests.json +++ b/doc/plans/assets/2026-10-10-boat/evidence/computer-admission-wait-tests.json @@ -13,7 +13,7 @@ "skipped": 0 }, "typecheck": "server passed", - "independentReview": "No remaining actionable findings", + "independentReview": "Initial review found no actionable findings; later CI review found two P1 integration defects.", "databaseMethod": "Unique disposable databases with actual migrations on the task-owned local PostgreSQL server via a temporary test harness override; product database untouched; all disposable databases removed. Native isolated-cluster bootstrap failed before this fallback.", "automaticScope": "Verified pending provider stop, ordinary pre-provider turn or unchanged explicit user message; task lock, original session, cancellation and continuation authorization retained.", "excluded": "Failing/superseded stops, existing owner conflicts, provider work already started, one-run Retry/Interrupt intents retain existing behavior.", @@ -30,5 +30,61 @@ "file": "computer-wait-tsc.log", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" } - ] + ], + "laterReview": { + "result": "P1 found after initial independent review", + "url": "https://github.com/paperclipai/paperclip/pull/15813#discussion_r4240083053", + "finding": "Actual Stop returns early for an already-cancelled admission-wait parent, so the accepted wait may still schedule a successor. Initial tests did not cover this real cancellation entry point.", + "deployment": "Candidate17 cancelled during build, before check/deploy.", + "fix": "Addressed by correction11ccf7b1a2; original findings and pre-fix CI retained.", + "secondFinding": { + "url": "https://github.com/paperclipai/paperclip/pull/15804#discussion_r4240095623", + "finding": "Actual run lifecycle terminalizes the wait before cleanup; the retry predicate then rejects the live control, releasing the task lock and creating a recovery hold. Initial dependency-stubbed tests missed this integration path." + }, + "ci": { + "observedAt": "2026-10-11T03:25:56.089584+00:00", + "core": { + "head": "a4c864a062a685e40d92fb015496dad3b1ce0f1e", + "passed": 51, + "skipped": 2, + "failed": 1 + }, + "ui": { + "head": "f63eccc9ce6ef7440df30fcfe3c5ab79cded415a", + "passed": 51, + "failed": 1 + }, + "failure": "Each head has one Greptile review failure and one distinct unresolved P1; no checks pending." + } + }, + "correction": { + "coreCommit": "11ccf7b1a2d8ad2921cff2a41ae2265e0896640b", + "combinedCommit": "3bd413a9b3", + "stagingCommit": "bd86e22e72578fa8a240160b6001802eb2cd40f1", + "behavior": [ + "Retain task ownership on exact pre-provider wait evidence while requiring settled cleanup before scheduling.", + "Actual Stop suppresses retries under the scheduling locks and cancels only the exact successor chain.", + "Acknowledge cleanup before releasing ownership; retain old queued input and protect unrelated or reassigned work." + ], + "tests": { + "files": 4, + "passed": 338, + "skipped": 0, + "scope": "Actual heartbeatService Stop and real run-lifecycle finalization, scheduling races, queued/running successors, repeated Stop, remote unexpired cleanup, queued input and unrelated/reassigned work." + }, + "typecheck": "server passed", + "independentReview": "No remaining actionable findings", + "databaseMethod": "Unique disposable real PostgreSQL databases; all removed. Product database and running controller untouched.", + "provenance": [ + { + "file": "computer-wait-stop-final-tests.log", + "sha256": "e3b2c29d2bef7697546633ca7cb12db6413ad865a06b1464940d4cb77defd58f" + }, + { + "file": "computer-wait-stop-tsc.log", + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + } + ], + "liveAcceptance": "Not yet verified; deployment18 in progress." + } } diff --git a/doc/plans/assets/2026-10-10-boat/evidence/retry-qualification-summary.json b/doc/plans/assets/2026-10-10-boat/evidence/retry-qualification-summary.json index e071f8af7fd..00d2af1fb22 100644 --- a/doc/plans/assets/2026-10-10-boat/evidence/retry-qualification-summary.json +++ b/doc/plans/assets/2026-10-10-boat/evidence/retry-qualification-summary.json @@ -1103,5 +1103,491 @@ "providerErrorPresent": false } ] + }, + "local37FailingStopCancellation": { + "classification": "operator cancellation while provider stop was failing; not a pending-wait qualification", + "checkedAt": "2026-10-11T03:10:53.835Z", + "run": { + "id": "b380b52e-12df-4231-9f79-7e38539050f1", + "status": "cancelled", + "created_at": "2026-10-11T03:08:39.968Z", + "started_at": "2026-10-11T03:08:40.002Z", + "finished_at": "2026-10-11T03:09:07.157Z", + "error_code": "cancelled", + "error": "Cancelled by a board operator", + "native_session_id": null, + "runner_instance_id": null, + "session_id_before": "01a1286b-9338-7101-8ced-733e9b28ae99", + "session_id_after": null, + "retry_of_run_id": null, + "scheduled_retry_at": null, + "scheduled_retry_attempt": 0, + "scheduled_retry_reason": null, + "execution_stage": "preparing", + "execution_recovery": null, + "computer_admission": null, + "cancellation": { + "reason": "Cancelled by a board operator", + "source": "operator", + "expected": true, + "initiator": { + "id": "local-board", + "type": "user" + }, + "recordedAt": "2026-10-11T03:09:07.156Z" + } + }, + "newOwnerIdsAfterFirstSample": [], + "ownersBoundToRun": [], + "leasesBoundToRun": [], + "subsequentTaskRuns": [], + "wakeRequestsAfter0306": [ + { + "id": "dc6f6f64-fa09-4757-954f-dac920ffe185", + "run_id": "b380b52e-12df-4231-9f79-7e38539050f1", + "status": "cancelled", + "reason": "issue_reopened_via_comment", + "requested_at": "2026-10-11T03:08:39.968Z", + "claimed_at": "2026-10-11T03:08:40.002Z", + "finished_at": "2026-10-11T03:09:07.157Z" + } + ], + "runEventTypes": [ + { + "event_type": "lifecycle", + "count": 2, + "first_at": "2026-10-11T03:09:07.165Z", + "last_at": "2026-10-11T03:09:07.694Z" + } + ], + "providerFailureBaseline": { + "recordedAt": "2026-10-11T03:06:37.292098+00:00", + "scope": "Read-only precondition for pending-wait cancellation; no guest reads or lifecycle actions", + "fixtures": [ + { + "fixture": "local-main", + "sandboxId": "bx_ndnggjad", + "observedAt": "2026-10-11T03:06:37.280972+00:00", + "state": "archiving", + "hydrated": true, + "health": "ok", + "snapshots": true, + "lastSnapshotAttemptAt": "2026-10-11T03:02:42.113Z", + "lastSnapshotStatus": "in_progress", + "snapshotCompletedAt": "2026-10-11T02:41:45.984Z", + "snapshotVerifiedAt": "2026-10-11T02:41:45.984Z", + "stop": { + "id": "stop_e6158ef3a5ab49f1", + "status": "failing", + "requestedAt": "2026-10-11T02:49:01.985Z", + "lastAttemptAt": "2026-10-11T03:02:40.643Z", + "endedAt": null, + "error": "Final snapshot failed; stop aborted to protect unsaved data: archival snapshot 52656953-0970-44b4-b373-f9f29e5f3d50 did not complete (status=failed); machine kept alive" + }, + "providerErrorPresent": false + }, + { + "fixture": "staging", + "sandboxId": "bx_b8sawfyp", + "observedAt": "2026-10-11T03:06:37.291945+00:00", + "state": "idle", + "hydrated": true, + "health": "ok", + "snapshots": true, + "lastSnapshotAttemptAt": "2026-10-11T03:02:40.556Z", + "lastSnapshotStatus": "in_progress", + "snapshotCompletedAt": "2026-10-11T02:19:59.759Z", + "snapshotVerifiedAt": "2026-10-11T02:19:59.759Z", + "stop": { + "id": "stop_804de979b357c47a", + "status": "failing", + "requestedAt": "2026-10-11T02:34:21.063Z", + "lastAttemptAt": "2026-10-11T03:06:20.495Z", + "endedAt": null, + "error": "Refusing archive/stop: Sandbox machine [sandbox machine] has no successful snapshot in the last 30 minutes (latest completed at 2026-10-11T02:19:59.759Z)." + }, + "providerErrorPresent": false + } + ] + }, + "evidence": [ + { + "file": "retry-local37-admission-first.json", + "sha256": "5cd9c10c0dc66d82e994e12fdfcbe80c818f17419da2ad7481c4de60379e2662" + }, + { + "file": "retry-local37-admission-settled.json", + "sha256": "818e5e94330d2d80d0714522ea4ba2c3d4904a6fbd02b5ea5e8cd1f9dd6a9ce0" + }, + { + "file": "retry-pending-wait-provider-baseline.json", + "sha256": "f575cd34f84655a99b35c2094ab9cb5e1fc0cdb5bb3e797f709cf8bd39c34879" + }, + { + "file": "retry-local37-admission-cleanup0313.json", + "sha256": "d4e010b16335d01c4d8167e52e9c3cd62b1cf180e3ebc38150a24cc848e511bd" + } + ], + "limits": [ + "No guest inspection was performed while the provider was archiving with a failing stop.", + "Absence of provider dispatch is supported by null runtime binding, preparing stage, no owner/lease, and lifecycle-only run events.", + "No durable computer_admission_wait record was produced in this run.", + "The 03:10:53 sample still showed preparing; executor cleanup is proven only by the 03:13:38 sample and its 03:11:16.987 settlement receipt." + ], + "cleanupVerifiedAt": "2026-10-11T03:13:38.297Z", + "finalCleanup": { + "checkedAt": "2026-10-11T03:13:38.297Z", + "run": { + "id": "b380b52e-12df-4231-9f79-7e38539050f1", + "status": "cancelled", + "created_at": "2026-10-11T03:08:39.968Z", + "started_at": "2026-10-11T03:08:40.002Z", + "finished_at": "2026-10-11T03:09:07.157Z", + "error_code": "cancelled", + "error": "Cancelled by a board operator", + "native_session_id": null, + "runner_instance_id": null, + "session_id_before": "01a1286b-9338-7101-8ced-733e9b28ae99", + "session_id_after": null, + "retry_of_run_id": null, + "scheduled_retry_at": null, + "scheduled_retry_attempt": 0, + "scheduled_retry_reason": null, + "execution_stage": "settled", + "runtime_mode": "legacy", + "controller_lease_expires_at": null, + "startup_preparation_settled_at": "2026-10-11T03:11:16.987Z", + "computer_admission_preparation_settled_at": null, + "computer_admission_retry_outcome": null, + "execution_recovery": null, + "computer_admission": null, + "cancellation": { + "reason": "Cancelled by a board operator", + "source": "operator", + "expected": true, + "initiator": { + "id": "local-board", + "type": "user" + }, + "recordedAt": "2026-10-11T03:09:07.156Z" + } + }, + "noOwnerOrLeaseBoundToRun": true, + "noScheduledRetryOrSuccessor": true, + "sourceFile": "retry-local37-admission-cleanup0313.json", + "sourceSha256": "d4e010b16335d01c4d8167e52e9c3cd62b1cf180e3ebc38150a24cc848e511bd" + }, + "finalSuccessorRunIds": [] + }, + "providerSnapshotInvestigation": { + "recordedAt": "2026-10-11T03:11:36.970048+00:00", + "scope": "GET sandbox, GET exact stop, GET latest five completed snapshots only; no guest contact or mutations", + "fixtures": [ + { + "fixture": "staging", + "sandboxId": "bx_b8sawfyp", + "state": "idle", + "hydrated": true, + "health": "ok", + "snapshots": true, + "lastSnapshotAttemptAt": "2026-10-11T03:02:40.556Z", + "lastSnapshotStatus": "in_progress", + "snapshotCompletedAt": "2026-10-11T02:19:59.759Z", + "snapshotVerifiedAt": "2026-10-11T02:19:59.759Z", + "stop": { + "id": "stop_804de979b357c47a", + "status": "failing", + "requestedAt": "2026-10-11T02:34:21.063Z", + "lastAttemptAt": "2026-10-11T03:11:20.544Z", + "endedAt": null, + "error": "Refusing archive/stop: Sandbox machine [redacted] has no successful snapshot in the last 30 minutes (latest completed at 2026-10-11T02:19:59.759Z)." + }, + "completedSnapshots": [ + { + "id": "257de6e4-c150-4c2b-b8e7-026831946617", + "status": "completed", + "kind": "incremental", + "generation": 93, + "createdAt": "2026-10-11T02:19:44.213Z", + "completedAt": "2026-10-11T02:19:59.759Z", + "sizeBytes": 445853, + "fileCount": 81, + "contentSizeBytes": 13367620, + "contentFileCount": 81 + }, + { + "id": "3bcf62e0-cc69-47e9-b493-1e294d74ea51", + "status": "completed", + "kind": "incremental", + "generation": 92, + "createdAt": "2026-10-11T02:18:44.210Z", + "completedAt": "2026-10-11T02:18:59.226Z", + "sizeBytes": 357418, + "fileCount": 71, + "contentSizeBytes": 12994481, + "contentFileCount": 71 + }, + { + "id": "e543237f-89dc-41fc-a1e4-0ad6ad7a7ac4", + "status": "completed", + "kind": "incremental", + "generation": 91, + "createdAt": "2026-10-11T02:17:44.226Z", + "completedAt": "2026-10-11T02:18:02.424Z", + "sizeBytes": 179011, + "fileCount": 54, + "contentSizeBytes": 4214826, + "contentFileCount": 54 + }, + { + "id": "987fbaf1-a3c9-485c-81b1-a3e6c0a687e9", + "status": "completed", + "kind": "incremental", + "generation": 90, + "createdAt": "2026-10-11T02:16:44.225Z", + "completedAt": "2026-10-11T02:17:02.668Z", + "sizeBytes": 272375, + "fileCount": 5, + "contentSizeBytes": 6254470, + "contentFileCount": 5 + }, + { + "id": "6e688e9c-ba76-4c76-9ffd-fa9c457ce3c5", + "status": "completed", + "kind": "incremental", + "generation": 89, + "createdAt": "2026-10-11T02:15:44.223Z", + "completedAt": "2026-10-11T02:16:02.697Z", + "sizeBytes": 207058, + "fileCount": 4, + "contentSizeBytes": 4214662, + "contentFileCount": 4 + } + ], + "hasMoreCompletedSnapshots": true + }, + { + "fixture": "local-main", + "sandboxId": "bx_ndnggjad", + "state": "archiving", + "hydrated": true, + "health": "ok", + "snapshots": true, + "lastSnapshotAttemptAt": "2026-10-11T03:02:42.113Z", + "lastSnapshotStatus": "in_progress", + "snapshotCompletedAt": "2026-10-11T02:41:45.984Z", + "snapshotVerifiedAt": "2026-10-11T02:41:45.984Z", + "stop": { + "id": "stop_e6158ef3a5ab49f1", + "status": "failing", + "requestedAt": "2026-10-11T02:49:01.985Z", + "lastAttemptAt": "2026-10-11T03:02:40.643Z", + "endedAt": null, + "error": "Final snapshot failed; stop aborted to protect unsaved data: archival snapshot 52656953-0970-44b4-b373-f9f29e5f3d50 did not complete (status=failed); machine kept alive" + }, + "completedSnapshots": [ + { + "id": "53e25365-31d5-4949-8913-121ea1d09a2c", + "status": "completed", + "kind": "incremental", + "generation": 279, + "createdAt": "2026-10-11T02:41:40.087Z", + "completedAt": "2026-10-11T02:41:45.984Z", + "sizeBytes": 48739, + "fileCount": 138, + "contentSizeBytes": 174240, + "contentFileCount": 138 + }, + { + "id": "64799060-7ed0-4a52-9b72-52dacb6f20c0", + "status": "completed", + "kind": "incremental", + "generation": 278, + "createdAt": "2026-10-11T02:40:43.045Z", + "completedAt": "2026-10-11T02:40:43.043Z", + "sizeBytes": 179245, + "fileCount": 28, + "contentSizeBytes": null, + "contentFileCount": null + }, + { + "id": "983f8945-6e82-4f06-90d7-76b2ebb00ad4", + "status": "completed", + "kind": "noop", + "generation": null, + "createdAt": "2026-10-11T00:52:13.419Z", + "completedAt": "2026-10-11T00:52:14.041Z", + "sizeBytes": null, + "fileCount": null, + "contentSizeBytes": null, + "contentFileCount": null + }, + { + "id": "35bf2230-1823-4517-99d6-440272916f9c", + "status": "completed", + "kind": "incremental", + "generation": 278, + "createdAt": "2026-10-11T00:50:15.044Z", + "completedAt": "2026-10-11T00:52:12.702Z", + "sizeBytes": 179245, + "fileCount": 28, + "contentSizeBytes": 6443682, + "contentFileCount": 28 + }, + { + "id": "fd62ed3a-723f-4545-8610-3109f6d02396", + "status": "completed", + "kind": "incremental", + "generation": 277, + "createdAt": "2026-10-11T00:46:33.572Z", + "completedAt": "2026-10-11T00:50:14.996Z", + "sizeBytes": 220697, + "fileCount": 496, + "contentSizeBytes": 6252535, + "contentFileCount": 496 + } + ], + "hasMoreCompletedSnapshots": true + } + ] + }, + "deployment17": { + "sourceCommit": "ca8ab88c78556c4711d59f71c6293619543a66a1", + "requestId": "f90ac5bb-503e-4f15-a56e-7ccd2509c187", + "prepareRunId": "38107501313", + "buildRunId": "38107534492", + "result": "Build cancelled before check/deploy after fresh review found admission lifecycle defects. No rollout occurred.", + "reviewFindings": [ + "https://github.com/paperclipai/paperclip/pull/15813#discussion_r4240083053", + "https://github.com/paperclipai/paperclip/pull/15804#discussion_r4240095623" + ], + "correction": "In progress; fresh deployment required after correction." + }, + "providerSelfRecovery0323": { + "recordedAt": "2026-10-11T03:23:54.960011+00:00", + "scope": "Single bounded provider status check; no guest reads or lifecycle actions", + "fixtures": [ + { + "fixture": "staging", + "sandboxId": "bx_b8sawfyp", + "observedAt": "2026-10-11T03:23:54.959691+00:00", + "state": "idle", + "hydrated": true, + "health": "ok", + "snapshots": true, + "lastSnapshotAttemptAt": "2026-10-11T03:16:37.370Z", + "lastSnapshotStatus": "in_progress", + "snapshotCompletedAt": "2026-10-11T02:19:59.759Z", + "snapshotVerifiedAt": "2026-10-11T02:19:59.759Z", + "stop": { + "id": "stop_804de979b357c47a", + "status": "failing", + "requestedAt": "2026-10-11T02:34:21.063Z", + "lastAttemptAt": "2026-10-11T03:23:20.483Z", + "endedAt": null, + "error": "Refusing archive/stop: Sandbox machine [sandbox machine] has no successful snapshot in the last 30 minutes (latest completed at 2026-10-11T02:19:59.759Z)." + }, + "providerErrorPresent": false + }, + { + "fixture": "local-main", + "sandboxId": "bx_ndnggjad", + "observedAt": "2026-10-11T03:23:54.959875+00:00", + "state": "archived", + "hydrated": true, + "health": "ok", + "snapshots": true, + "lastSnapshotAttemptAt": "2026-10-11T03:16:03.007Z", + "lastSnapshotStatus": "completed", + "snapshotCompletedAt": "2026-10-11T03:16:03.591Z", + "snapshotVerifiedAt": "2026-10-11T03:16:03.591Z", + "stop": { + "id": "stop_e6158ef3a5ab49f1", + "status": "completed", + "requestedAt": null, + "lastAttemptAt": "2026-10-11T03:16:03.734Z", + "endedAt": "2026-10-11T03:16:03.734Z", + "error": null + }, + "providerErrorPresent": false + } + ], + "provenance": { + "file": "retry-provider-self-recovery-status.json", + "sha256": "8cfe3e2f31d50468a5d9503aa0ebebb6840b01c50edeed4d7fa109f704291b27" + } + }, + "local37ClaudeResume": { + "checkedAt": "2026-10-11T03:29:05.086Z", + "outcome": "failed during saved-session recovery before independent proof-file verification", + "run": { + "id": "7b6baf95-3e47-456a-99f9-9339183b6508", + "status": "failed", + "created_at": "2026-10-11T03:26:24.365Z", + "started_at": "2026-10-11T03:26:24.393Z", + "finished_at": "2026-10-11T03:28:55.832Z", + "error_code": "native_session_interrupted", + "error": "native_session_recovery_failed: Error: runner_remote_provider_cache_corrupt: [exact agent provider-pack cache directory]; stop tasks using this pack and have an operator remove or quarantine this exact cache directory before retrying", + "native_session_id": "5f2470db-eb0f-4561-95a2-6dddf5646fa6", + "runner_instance_id": "f7c84a3a-b36d-4885-ba30-91441a5f94f8", + "session_id_before": "582fc37b-ddc7-4227-9378-1cbd052a36ea", + "session_id_after": null, + "execution_stage": "preparing", + "controller_lease_expires_at": "2026-10-11T03:28:22.877Z", + "resultKeys": [], + "forceFreshSession": false, + "executionRecovery": null, + "providerPackDigest": null + }, + "providerPackDigest": "sha256:a02b92afc8423b8fdea50a7fd480a5417b17d9c7b31a14dcab4221e57e8c270a", + "sameNormalizedSession": true, + "sameLogicalRunner": true, + "sameProviderThreadBefore": true, + "latestOwner": { + "id": "5ef4911d-a6a9-4d10-9eeb-dce1d02de041", + "phase": "retired", + "generation": 1, + "runId": "7b6baf95-3e47-456a-99f9-9339183b6508", + "sessionKey": "49e146ed7d44b43e9080ba7f0119315bc30625354fb5f70e132d4f1df35d1231", + "deadline": null, + "process": null + }, + "latestLease": { + "id": "0354c41f-d8c2-48bb-889d-b2932082dfc8", + "status": "released", + "heartbeat_run_id": "7b6baf95-3e47-456a-99f9-9339183b6508", + "acquired_at": "2026-10-11T03:26:41.773Z", + "released_at": "2026-10-11T03:28:55.830Z", + "cleanup_status": "success" + }, + "expectedProofHashes": [ + "14fa23911a1a95ede22a93f43fc027934b20a1ba497f8e6120abc8769498e124", + "afaf525f4bf0c38614c5df0cf97140e7d1bf9cad96faa293e2317dc82116af4a" + ], + "proofFilesVerified": false, + "guestReadsPerformed": false, + "cacheMutationsPerformed": false, + "evidence": [ + { + "file": "retry-local37-claude-baseline.json", + "sha256": "8b46b06284ea09fb29567df0296154b8ad18ccc5c26a1a935fd014035a6a48ff" + }, + { + "file": "retry-local37-claude-progress0327.json", + "sha256": "575219156fe2e312c688d9a5f333387ddbe9426c302ef71e525c92491c89c78f" + }, + { + "file": "retry-local37-claude-progress0329.json", + "sha256": "822cbec020ddec5f3d968200b9dbb8f383158c95423ebc95cccdb1260fb94fab" + } + ], + "sourceCommit": "04ce7eb2a8fddbb2a46c8ec99d3a748de53d7ae8", + "ui": "Ordinary follow-up in saved task; UI showed Run failed and Blocked. Screenshot local-37-claude-resume-failed.png." + }, + "deployment18": { + "sourceCommit": "bd86e22e72578fa8a240160b6001802eb2cd40f1", + "requestId": "f8cb13bc-51c4-4639-9323-adc2560dda18", + "prepareRunId": "38108515304", + "status": "in progress; no serving verification yet" } } diff --git a/doc/plans/assets/2026-10-10-boat/local-37-claude-resume-failed.png b/doc/plans/assets/2026-10-10-boat/local-37-claude-resume-failed.png new file mode 100644 index 00000000000..5e1135c5b27 Binary files /dev/null and b/doc/plans/assets/2026-10-10-boat/local-37-claude-resume-failed.png differ