DottaandPaperclip
492d537340
## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution runs in a remote Runner while the webserver owns its controller. > - Encrypted PRP checkpoints and session snapshots already survive a webserver restart in Postgres. > - Ordinary warm and per-turn admission still needs local markers and supervisor metadata to find that state. > - Replacing the webserver with an empty native controller directory can therefore lose a usable retained session. > - This change adds bounded recovery metadata to the existing snapshot transaction and reconstructs local projections from the selected durable source. > - The benefit is native execution continuity across webserver replacement, with the existing ownership and agent lifecycle policies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native remote controller recovery and retained-session continuation during a webserver upgrade. **Subsystem affected** server/ — native runtime, checkpoint persistence, recovery scheduling and internal upgrade reporting. Product E2E qualification also changes. **Current behavior** Without both the local PRP marker and supervisor file, normal continuation can return before discovering the saved Postgres checkpoint. The snapshot already exists in Postgres, but its local reuse compatibility envelope cannot be reconstructed. **Proposed behavior** Select the existing durable source, validate its identity and ownership, restore its PRP state and supervisor projection, then use normal authenticated admission. Missing or inconsistent evidence holds recovery. It does not authorize fresh provider work. **Reason and benefit** A replacement webserver must recover active execution and later continue its retained session without the old server's native files. This is one component of deployment continuity. HTTP traffic switching, workspace storage migration and Railway rollout remain separate. **Breaking changes** No new public endpoint, schema migration or database pool. The private run-profile envelope is additive. Disk-independent rollback also requires a reader for the staged controller format and compact checkpoint part; the first production release containing these readers is the floor. Older file-backed sessions need an owner checkpoint or the bounded internal importer. **Additional context** Related foundations: #17401, #17916, #17254, #14621, #14622. The implementation contract and open acceptance gates are in [the dated plan](doc/plans/2026-10-30-native-controller-checkpoints.md). ## What Changed - Persist a versioned recovery envelope, at most 4 KiB, beside the redacted session snapshot in its existing transaction. Keep it out of public run projections. - Compare frozen admission separately from prepared attachment instructions during staged recovery, preserving exact template/identity/authority checks. - Bound the existing lazy checkpoint pool at three connections after measured two-slot burst backlog; keep request/renewal capacities and SQL operation counts unchanged. The tuning is a separate commit. - Discover and restore the exact selected retained source without local native markers. Preserve the 32-candidate window, progress barriers and current authority checks. - Fence snapshot and result writes. Acquire coordinator, issue and run locks in order and check expiry after lock waits. Combine snapshot ownership/read work into one statement. - Reuse existing recovery deadlines and drain/resume scheduling. Retained-owner waits preserve the original unstarted run and actual lease deadline. Stop, ownership and provider-progress fences precede deferral. Add bounded native evidence reporting, retained-source import and retention protection. - Admit a settled warm source with an authenticated idle goal-read suffix while retaining active-work and mixed-authority holds. - Permit lifecycle `turn.stop` checkpoints after accepted results under existing ownership fences. Claim the existing successor coordinator before retiring the old warm session, and stage shared attachments only after verified retirement. Preserve omitted isolated-home defaults and existing fresh skill-selection order during recovery without hiding real configuration changes. - Carry the exact durable six-field PRP binding during active recovery after a healthy warm handoff. Keep the Runner mismatch guard and preserve opaque attachment IDs. - Preserve an already verified active or idle warm process during remote preparation; revalidate its proof/liveness and retain later authenticated PRP attachment. Ordinary resumed leases retain suspension checks. - Drain retained source events under their original authority, reconcile immutable source accounting and block unknown source tool dispatch before target work. - Persist compact staged handoff intent through authenticated adoption and recover direct/listener command-loss windows without another Runner or duplicate attachment. - Rebuild suspended lifecycle receipts for resumed prepared tasks, and retry missing lifecycle reads after partial PRP materialization without overwriting contradictory evidence. - Add normal warm/per-turn diskless continuation coverage, real Postgres races and SQL-cost checks. Add a reproducible paired load workload. - Give only the explicit outage fixture a 15-minute provider idle window, with a red/green catalog regression preserving all production timing and lifecycle policies. - Publish the goal-less task-session pointer in the existing post-acceptance terminal checkpoint transaction, with current ownership, accepted-result, reset and newer-progress fences. Keep a private bound receipt so late completion cannot duplicate or resurrect publication. - Use a retained warm sandbox in the dedicated continuity fixture, and accept an exact output token followed by an explanation. Keep the strict identity oracle. - Extend the explicit Daytona cell to a five-minute active outage and another replacement after completion. Require retained provider identity, a new tool receipt and a useful follow-up result. Add an owned loopback Postgres E2E fixture for hosts that cannot start embedded Postgres. ## Verification **Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`: ready for the qualified native-controller component.** Full CI, fresh review, local application paths, controlled performance qualification and the strict live Linux/Daytona case pass. Independent resource cleanup is complete. Broader rollout, browser reconnection and the unexecuted transport/platform combinations remain integration gates. Integrated 484 passed full local recursive typecheck/build and ordinary active→completion→cache-loss→C component qualification. Fresh review then found a valid attached-wake failure: preparation appends attachment instructions before the verifier compares execution to the frozen admission. 9f4 fixes that by separately checking the pre-preparation admission and exact prepared execution. Both hashes are captured privately per admission; the full template, identity, configuration and authority checks remain. Its regression fails before the fix; 36 focused cases, 42 staged source/target cases and server typecheck pass afterward. The review thread is resolved. Exact-head full CI passes ([run 41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822): 48 successful jobs, one intended skip); the complete `pnpm test:run` partitions are forced, with the five native suites in the Runner lane. This is not a claim that an unsharded local command passed. The fresh exact-head Greptile check succeeds with 5/5 and all six threads resolved. The original two-connection database comparison closes **steady PASS and burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all operations/ownership remain correct, but candidate arm 3 drains in 1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged 1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2 diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72 ms: three is the smallest tested passing cap, and the repeated two-cap failure survives run-order/cache effects. Separately reviewable `fc68584ce6` changes the existing lazy checkpoint pool default to three connections. Request capacity stays ten and liveness two; idle release, shutdown fencing, transaction counts and pool count are unchanged. Thirteen DB lifecycle/cap tests and DB typecheck pass. Full recursive typecheck/build pass at exactfc68584, with the matching server build stamp. The final paired native feature-cost comparison explicitly uses the **same cap-three policy and common current DB factory in both variants**; its baseline is controlled, not the unmodified historical cap-two configuration. Original gates and workload counts remain unchanged. The one strict Linux/Daytona case started at 04:40:56 UTC only after both new profiles passed; it passed on the first attempt with all 49 workflow checks. Independent cleanup confirms all three owned sandboxes absent, owned Postgres/controllers/observer gone, VM terminated, root volume absent and ephemeral SSH key/agent removed. **Final controlled cap-three qualification: PASS.** Baseline native implementation `59da15066a` and candidate `fc68584ce6` use the same current DB factory and checkpoint policy (request/liveness/checkpoint = 10/2/3). Eight 65-second ABBA arms cover 300 controllers and 10,000 historical tasks. All correctness, absolute and 10% relative gates pass; original cap-two failures remain unchanged. | Mode | DB CPU µs/op, baseline → candidate | WAL B/op, baseline → candidate | DB-request p99 ms, baseline → candidate | Maximum drain | | --- | --- | --- | --- | --- | | Steady | 542.40 → 554.83 (+2.29%) | 1509.01 → 1558.27 (+3.26%) | 107.11 → 111.90 (+4.48%) | 26.35 ms | | Burst | 1036.36 → 1063.11 (+2.58%) | 4153.12 → 4317.64 (+3.96%) | 103.736 → 103.841 (+0.10%) | 930.14 ms | Every matched arm preserves operation/statement/transaction counts, including 3,900 PRP and 3,900 snapshot transactions. All three actual pools are included; there are zero lost owners, missed lease deadlines, dropped operations or errors. Peak Postgres CPU is 18.31%/9.47% of two cores and memory 614.29/684.42 MiB (steady/burst). The benchmark uses one Node process simulating 300 controllers; it does not qualify two-webserver overlap, HTTP latency or hosted capacity. Its observer SQL contributes to measured database CPU. The original unmodified cap-two baseline is separately retained as failed; this cap-three comparison is explicitly controlled. The exact final-source 8/512/8,192-chunk test also passes: zero recovery SQL, transactions or changed rows in each text-only interval, zero idle SQL, seven lifecycle transactions and twelve changed rows per sample. Existing lease traffic remains separate. The final ordinary managed application replay passes active recovery, completion, native-cache removal and C continuation; its UNKNOWN local debug-daemon provenance and partial instruction-observer limits remain recorded. The live case uses the attested Linux producer instead. **Live qualification: PASS, one attempt, no retries.** Campaign `native-controller-retained-fix-fc68584ce6-ec2-20261031T044055Z` started 04:40:56 UTC on the exact final Linux webserver source; suite definition hash `12f9bb97823c1537e43f0398e7cddf39890f77bce7b413ca4dfd944f4a30e5d5`. The existing immutable image and attested Bookworm Runner artifact are unchanged. - A was SIGKILLed and its native cache discarded. B started after 300,149 ms, naturally acquired generation 2, authenticated at 316,624 ms and resumed at 317,682 ms within the original 420,000 ms grace. The same active run, Runner, provider PID/start fingerprint and one submitted/started/accepted turn were preserved; one final result/terminal and workspace finalization completed, with no retry/recovery action. - After active and queued completion, B was SIGKILLed and its native cache discarded again. C received exactly one ordinary task-comment API follow-up, completed a new tool and produced a useful reply while preserving native/provider/Runner identities. This closes the earlier failed post-C gate. A separate C OS-process PID field was not exported; the physical-PID proof is specifically A→B. - Four distinct results were accepted and published; all four finalizations committed at attempt 1, all four accounting-v2 projections settled, and the task-session pointer advanced to C. The coordinator independently checked these facts and all 49 canonical checks against the sanitized receipts. - Canonical cleanup passed and independent checks confirmed three exact sandbox 404s plus complete inventory/zero owned remnants, no owned PG or controller/observer processes, EC2 terminated, root volume absent, and the ephemeral SSH key/agent removed by 04:54:25 UTC. Live scope is classic remote Codex warm continuity with non-native storage dependencies available. Prepared-task completed B→C, full per-turn application disk loss, browser reconnection, two-server load and overall Railway/HTTP continuity were not live-qualified here. Earlier failed campaigns and the original cap-two burst failure remain failed historical evidence. A report-renderer cache setup failure was preserved; the normal sanitizer subsequently succeeded with an explicit cache path and a new output directory, without changing validators. The repair keeps retained events under their original six-field authority until persistence and accounting accept them. Unknown semantic effects stay unresolved. A compact staged target survives source bootstrap, cached-command delivery and authenticated handoff, including an uncertain commit response. Direct and same-guest-listener recovery require the verified original Runner and the exact endpoint. Target snapshot publication marks readiness in memory, with retirement coalesced into an existing command write. Headers contain only the staged descriptor/digest; a prequeue template is written once as a changed checkpoint part and then recovered from the existing command/receipt. No second session snapshot transaction, streaming SQL path, new pool or recurring idle scan is added. | Current repair gate | Evidence and scope | | --- | --- | | Source-tail regression | Real Runner protocol RED reproduced attachment rejection. Frozen core/transport pass five source-tail cases: source usage attribution, held persistence/no premature ACK, lost commit response plus cache loss, unknown semantic hold and accounting rejection hold. | | Handoff loss windows | Direct and listener modes pass queued/prepared/activated/completed boundaries (eight cells), preserving the actual Runner/provider process. Missing adoption proof and changed endpoints reject recovery. | | Compact storage | A 31 KiB attachment grant writes its separate part once, leaves it unchanged across source snapshots and removes it at queueing. Headers do not duplicate the grant. | | Accounting | 21 real-Postgres usage tests and 19 source-binding tests pass. Recovery hydrates immutable source receipts without resetting their producer or billing classification; conflicting/incomplete usage holds. | | Admission and malformed evidence | Driver 16, staged evidence 42, restart classifier 27 and source usage 19 focused cases pass. Four malformed-file regressions pass at d6a7, whose executor is byte-identical at 484, and server typecheck passes. Full executor file: 990 pass; 16 embedded-Postgres `initdb` setup failures before assertions, retained separately. | | Required checks | Exactfc68584recursive typecheck and build pass locally, with matching server stamp. Forced full CI succeeds (48 jobs pass, one intended skip); all PR checks are clean. Maps/token gates pass for their unchanged source surfaces. Fresh exact-head Greptile is 5/5, all six threads resolved. | | Ordinary application path | Exactfc68584managed component passes active recovery after natural expiry, completion, native-cache loss and ordinary C continuation. Production bytes match before/after; temporary fixtures restored and owned DB/services cleaned. UNKNOWN local debug-daemon/partial instruction-observer limits remain; the separate live case uses the attested Linux producer. | | Producer | Locked Bookworm build from exact 5b7 Rust treed03e494217, equal to finalfc68584Rust source. SHA-256 65b8f8cfdd6099e9ad24f1ca154e8a260777269cc82f6e23c6e6a180c531eaaf; 38,515,328 bytes. Actual Debian 12 execution and live admission pass. Controller/admission/DB changes do not relabel the producer source. | | Historical cap-two comparison | Exact 59da baseline vs 5b7 candidate, 300 controllers/10,000 history, ABBA 4×65 s per mode, all three actual pools and real snapshots. Steady CPU/op 541.36→554.64 µs (+2.45%); WAL/op 1512.13→1560.99 B (+3.23%); DB-request p99 102.78→105.99 ms (+3.12%): PASS. Burst CPU/op +4.71%, WAL/op +5.29%, p99 −0.10% pass relative gates, but absolute drains above fail. Identical operation/SQL counts, zero lost leases/drops/errors. One Node process simulates 300 controllers; this does not qualify two-webserver overlap or HTTP capacity. | | Live acceptance | Strict final-source Linux classic Codex case passes 1/1 on attempt 1, all 49 checks. Active recovery within grace and useful retained C continuation succeed; four accepted/published/committed/settled outcomes. Independent provider, PG/process, VM/root and SSH cleanup complete. Prepared/task and broader rollout limits are explicit above. | **Historical qualification snapshot through `9eb9ac6`.** The head at this snapshot was `9eb9ac6f9e312f3ebe8040f5a7565e88ae8824f3`. Server and Runner package sources are byte-identical to `283e05ffaa`; the two later commits change the explicitly scoped qualification and its documentation only. It includes master through the storage-owner fix `45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` (#18357). At that snapshot, CI/review passed but the strict post-C native recovery gate failed; the PR stayed draft and required another native repair. Prior `39a4343` CI and review passed, but its full live workflow failed; that grade remains unchanged. The first repaired `435ccd2` live attempt failed before any injected outage. It exposed a legitimate terminal snapshot before result acceptance; no accepted result ID or publication receipt was observed. Its canonical report remains 0/1 `candidate_failure`, with cleanup passed and independent provider absence confirmed. `0862062` defers publication until result acceptance and fixes the fresh review's coordinator/issue lock-order finding. Both failures have deterministic real-Postgres regressions. The `0862062` live attempt then proved accepted-result publication but failed active recovery: a resumed-lease preparation guard demanded suspension after the exact live Runner was already verified for adoption. `64c0b746` preserves that verified active/idle-warm process, revalidates its proof/liveness, and keeps ordinary suspension checks and later PRP authentication. Its live run passed preparation but exposed a second active-warm boundary: all durable binding fields matched except opaque turn/item attachment IDs, which the server had regenerated. `7a203274` carries the complete verified six-field binding and keeps the strict Runner mismatch guard. Its cloud case then passed active adoption, completion, publication and accounting. The queued continuation exposed accepted-result retirement writes and an implicit-home fingerprint drift; both are fixed with deterministic regressions in `39a4343` before this fresh full qualification. The prior live case exposed a real publication gap. A goal-less native turn could persist its accepted result and terminal snapshot before provider cleanup, while the task-session pointer was still empty. Normal later admission then selected a fresh session. The fix publishes that pointer and a private bound receipt in the **existing post-acceptance terminal checkpoint transaction**, before cleanup. Earlier terminal observations persist without a handled receipt or continuation pointer. No result-acceptance or snapshot transaction was added. The late completion path recognizes the receipt. This also prevents a late callback from restoring a pointer after a user reset. Goal/control publication keeps its existing owner. The dedicated Daytona fixture now uses the existing warm/reusable environment. Its sandbox, Runner and provider storage must survive. The former generic per-turn fixture disposed the sandbox after completion and kept provider-history backup bytes under the discarded directory. Whole per-turn disk-loss acceptance therefore has a separate provider-backup storage dependency. It remains distinct from the passing native per-turn component checks with backup storage available. The 60-second lease, 300-second outage and 420-second grace are unchanged. | Gate | Current repair evidence | | --- | --- | | Assembled regression | Baseline fails with an accepted result and terminal snapshot but a NULL task pointer while real provider cleanup is held. The fix passes normal continuation and reset cases, with no pointer seeding. Exact merged `92b7447` passed 2/2 twice, including after normal package TypeScript output refresh. Production files are byte-identical at frozen `435ccd2`. | | Persistence and concurrency | 19 helper/transaction cases and all 59 existing control-port cases pass on `0862062`. They cover bound result IDs, reset/newer epochs, Stop, lock waits and expiry, receipt replay, missing stubs, foreign bindings and atomic 4 KiB rollback. Existing snapshot transaction count stays one. The new tests reproduce the exact live missing-result-ID error and the inverse replay/status lock order against `435ccd2`, then pass after the correction. Replayed results still reject stale generations. | | Completion | All 32 completion-boundary tests pass on merged `92b7447`; the source is unchanged at `435ccd2`. Native result-shaped session metadata is preserved. | | Qualification fixture | A valid review report reproduced three rejected correct replies. Exact token-boundary matching fixes them. 140 focused E2E/catalog cases and Product E2E typecheck pass. Wrong tokens, marker prefixes/suffixes and wrong authors remain rejected; retained-identity checks are unchanged. | | Required checks | Full recursive typecheck and standard build passed at `435ccd2`. Both exact commands also passed on `0862062`, with its exact server build stamp; both also pass on `7a203274`, with its exact server build stamp. Fresh `0862062` Greptile passed 5/5, with zero unresolved threads. The `64c0b746` normal CI then passed all 49 jobs (48 success, one expected skip) and Greptile passed 5/5 with zero unresolved findings. The `7a203274` normal CI also passed 48 jobs with one expected skip, with fresh Greptile 5/5 and zero unresolved findings. Current `39a4343` recursive typecheck and standard build both pass with persisted exit codes and its exact build stamp; fresh Greptile is 5/5 with zero unresolved findings. Its normal CI completed successfully: 48 jobs passed and one expected job skipped. The fresh live result remains separately failed. The `0862062` normal CI had 46 passing jobs, one expected skip, one worker communication failure and its aggregate failure; GitHub states the self-hosted runner lost communication. No failed assertion or specific OOM/spot cause was established, and no blind retry was used. | | PostgreSQL supplement | One corrected all-pool pair at the preceding repair passed 300-controller/10,000-history correctness and absolute gates. SQL 76,590→75,990; transactions 9,013→8,713; UPDATE rows 11,613 each; 300 correct pointer writes each. PG CPU 31.99→30.46 s, WAL 70.36→67.87 MB, request p99 100.98→103.12 ms. This is one pair, not a repeated improvement claim or an exact `0862062` CPU measurement. Its two later changes defer premature publication and reorder existing locks; they add no transaction, stream/idle writer, pool or timer. The first collapsed-pool attempt remains invalid and retained. This is not an exact final-head CPU/WAL measurement; later fixes change ordering, identity selection and in-memory skill replacement without adding snapshot transactions, pools, stream writes or idle SQL. | | Live publication proof (`0862062`) | Baseline run `06fcc806-f186-4190-9b57-087b61d75533` accepted result `58f912dc-94d9-4b4d-bd3d-d5f689c83fd3`; the publication receipt names the same result, the task pointer has parameters and `lastRunId` matches, and the normalized native identity matches the following active run. Accounting v2 settled. | | Active recovery (`0862062`) | **FAIL.** SIGKILL/cache discard at 23:07:28.666 UTC; hold 300,000.859 ms; B ready at 308,896.486 ms; natural generation-2 ownership at 312,906 ms. At about 319 s, preparation failed `runner_harness_state_mismatch: resumed_not_suspended`. No authenticated after-receipt passed within 420 s. Exact original Runner PID 668/start/fingerprint was independently still alive; provider auto-stop was 15 minutes. No retained C follow-up qualified. Canonical 0/1 `cleanup_failure`/cleanup-failed report remains failed; subsequent independent cleanup deleted all three owned resources with exact 404s and confirmed full intended-account inventory with zero owned resources. | | Verified adoption preparation (`64c0b746`) | 91 focused cases pass, including actual preparation for four active providers and a distinct prior/current warm proof. Unproved resumed state, dead process and stale warm owner remain rejected. The earlier options-only fixture missed this gate. No spawn, backup restore or PRP-authentication bypass was added. Server typecheck passes. | | Active warm identity (`64c0b746`) | **FAIL.** B claimed generation 2 at 312,995 ms. Preparation passed, then strict PRP recovery rejected the generated attachment IDs at about 319 s. The saved and desired Runner, lease, normalized session and run IDs matched; only `turn_attachment_…`/`item_attachment_…` differed from generated `turn-<run>`/`item-<run>`. Canonical 0/1 cleanup-failed report remains failed; independent cleanup confirmed deletion/404s and zero owned provider resources. | | Six-field identity repair (`7a203274`) | Actual server-factory regression RED4→GREEN38. A real Rust Runner test passes healthy first run → random-ID warm handoff → held active second run → restored controller. It verifies a new live authenticated connection, exact identity, same Runner/provider OS identities, no duplicate launch/turn, and no redispatch or invented receipt for an uncertain tool. The synthetic wrong pair still fails the unchanged guard. This protocol case does not claim business completion. Server and Runner typechecks pass. | | Active recovery/completion (`7a203274`) | **PASS for the active component.** Full five-minute SIGKILL/cache-loss hold. B ready at 308,936.651 ms; natural generation 2 observed 312,979 ms; authenticated at 317,720 ms and resumed processing at 318,527 ms. Same run `1e1ccb00-0728-4935-97da-57562d81cb42`, Runner `246f73f7-a689-4177-a9c2-7ce711d63aed`, remote PID 669/start 23:48:27.470, provider lease/session and one submitted turn. Accepted result `49097529-5310-4ac3-af95-5bf9b91d19f4`, publication, finalization and accounting v2 all settled. | | Full workflow (`7a203274`) | **FAIL, preserved.** Queued late-requirement run `5e61f38d-c7d1-4fc1-bcbf-bce9d46ce4eb` failed `runner did not durably suspend before checkpoint` before outage 2. Canonical 0/1 candidate failure, cleanup passed, evidence valid. C retention was not exercised. Independent provider absence, controllers/PG cleanup, EC2 termination and root-volume absence all confirmed; closed receipt SHA-256 `676a3b15eb9937b6bf995c36fb8666c88964d509e5ceb9109fb02da0aa2bc909`. | | Retirement and stable configuration (`39a4343`) | Real Runner + Postgres RED reproduced the exact suspension failure and nine owner rejections. `turn.stop` is now lifecycle checkpoint maintenance under unchanged owner/result/generation fences. GREEN25/25 proves actual persisted suspension, exact identity and completed stop/drain/suspend; unaccepted, foreign, stale and new-work negatives remain rejected. Two default-home fingerprint REDs become GREEN9/9; explicit mode changes still invalidate. No global matching exemption or historical fingerprint rewrite. | | Fresh full Daytona warm cell (`39a4343`) | **FAIL, preserved.** Launched 00:15:29 UTC with automatic retries disabled. Active adoption passed: actual offline hold 300,000.848 ms; B ready 309,922.155 ms; natural generation 2 at 311,821 ms; authenticated 317,989 ms; resumed 330,440 ms. Same run `3e126819-e786-4b3b-b923-c975cf4f5b79`, Runner `4079138a-8d7c-44a4-8558-07a5411d1f55`, PID 668/start and provider/session identities. Accepted result `08e504af-4dec-48a8-bf05-bf0a32280559` and publication passed. Queued run `31d168c9-7ab9-4700-abb6-fafb90912237` failed the same strict suspension error before outage 2. Its row selected a new native identity, but no new third provider process or invocation was observed. Recovery-time configuration still reset only `adapterConfig`; native provider, context and workspace subtrees were unchanged. Canonical 0/1 candidate failure, evidence valid, normal cleanup passed; all campaign provider/PG/process resources independently absent. Closed campaign receipt SHA-256 `5e7d6281c13c812f3a946667f97f3c7d339b5ad86a3a819bd185b2fa73bc4a19`; EC2/SSH remain temporarily held for the concrete follow-up repair. Original 60/300/420-second limits and strict queued/second-replacement checks remain unchanged. Direct-process path; no new live prepared-task claim. | | Native retirement authority (`283e05ffaa`) | Exact `39a4343` facade regression RED3→GREEN9. The successor now uses its existing production coordinator claim and prompt renewal before strict retirement. A real Runner/Postgres case starts in the actual native-observed/unclaimed phase, claims through production code, then closes and durably suspends the predecessor in 1.657 s. All 25 real-Postgres cases pass; unclaimed/foreign writers remain rejected. Shared attachment staging stays after retirement; Stop and denied claims neither close nor stage. No new claim or SQL path was added. | | Configuration continuity (`283e05ffaa`) | Real shared/project-primary environment selection → configuration with four skill versions reproduces `reset: true`, changed category exactly `adapterConfig`, after a Postgres JSONB read. Pinned restoration had reordered the generated preference while actual runtime skills stayed identical. Replacing existing pinned rows in place preserves the original fresh name/key order and fingerprint. The assembled regression passes, as do three existing pinned-version/rename/alias cases and server typecheck. Genuine model/skill-selection changes still invalidate. No fingerprint-format change or broad compatibility exemption. | | Final production repair (`283e05ffaa`) | Full recursive typecheck/build and [normal CI](https://github.com/paperclipai/paperclip/actions/runs/41151145106) PASS: 48 jobs plus one expected skip. Fresh review 5/5, zero unresolved threads. Live baseline, recovered run and queued continuation all succeeded, each accepted/published/committed once with accounting v2 settled. Their native/provider identities and configuration fingerprint match; both recovery and continuation report no reset, no changed categories and task-session reuse. Active outage/authentication remains within original timing. | | Browser-dependent continuation (`283e05ffaa`, definition 6) | **Whole cell FAIL, preserved.** After B was killed at 00:46:38.928 and its cache discarded, C became ready in 9,151.410 ms. `openParent()` then hit a 30-second `page.goto`/DOMContentLoaded timeout before posting the final request. No fourth native run or C continuation was exercised. Cause of navigation timeout is unknown. C task had no execution lock, queued comments, handoff or recovery blocker; all three prior runs were committed/accounted. Normal cleanup passed; independent provider absence, controller/PG cleanup, EC2 termination and exact root-volume absence confirmed. Sanitized report bundle SHA-256 `2a3b841d62efb2ae30ef14f96a431ac4fa7fe3a782f1144831512f78edc65022`. | | Native API continuation qualification (`9eb9ac6f9e`, definition 7) | **FAIL, preserved.** The real post-C API request dispatched run `c71fbb18-f979-4fe4-93ce-9c4be46dbf21`, which failed `native_session_interrupted` / `native_session_recovery_failed` at 01:10:00.640 UTC. It selected the same normalized native identity, Runner and checkpoint-provider identity as the three succeeded turns. No C result was accepted; pending accounting remains explicit. Active recovery passed: actual offline hold 300,001.098 ms, B ready 308,335.019 ms, natural generation 2 observed 311,818 ms, authenticated 315,540 ms, resumed 316,612 ms. Queued continuation succeeded/accepted/published. B was then killed/cache discarded at 01:09:34.500 and C ready in 8,783.803 ms. Original classifier `transient_infrastructure`, whole-case failure and cleanup passed are unchanged; that classifier does not establish the root cause. All owned provider/PG/process resources, EC2, root volume and SSH agent independently absent. Closed receipt SHA-256 `7f21ac4a22efc07b849fa7e17866083b6127b3321af12671b4fc054ce57039a9`; sanitized bundle `71ffef14a30a553e008320a31840a6d6c0de85c1bfdb60f783fdb53a6aae812a`. Exact backend detail is unavailable in approved structured evidence. | | Current required checks (`9eb9ac6f9e`) | Recursive typecheck, standard build and exact build stamp PASS. [Normal CI](https://github.com/paperclipai/paperclip/actions/runs/41152908417) PASS: 48 jobs plus one expected skip. Fresh Greptile 5/5, latest edited summary checked, zero unresolved threads. Exact8/512/8,192 stream probes again report zero recovery SQL/transactions/rows and settled-idle SQL; seven control transactions/twelve changed rows each, with 36 separate legitimate renewals in the longest case. CI success does not supersede the failed live C gate. | The completion supplement observed secondary request/liveness/checkpoint transaction p99 increases of 23.0%/45.7%/15.6% in its single pair. Renewal p99 stayed below 0.865 s, with 900 successful renewals per arm and no lost leases. The named pool caps stayed 10+2+2. An aggregate activity sample peaked at 15 non-observer entries without backend-type classification, so no claim is made that all were client connections. No sampled lock episodes does not establish zero lock waits. Twelve pure reader/selector cases also pass with the new adjacent receipt. This is not another executed process rollback. Development reader `5cf7aead` can parse the recovery state, but its old run-detail redactor can expose receipt IDs/outcome to an authorized run reader. The first production release containing recovery readers, complete attachment-identity handling and both private-field redactions is the rollback floor. The implicit-home fix prevents new recovery-time fingerprint drift; it does not rewrite previously stored fingerprints from failed candidates. The development probe is not a published supported rollback target. Latest actual `0862062` Runner tests independently passed 8/512/8,192 text chunks with zero recovery SQL, transactions and changed rows, plus zero settled-idle SQL. All three observed seven setup/control transactions and twelve changed rows. The 8,192-chunk case separately recorded 31 legitimate lease-renewal statements, with no lost lease. These are component SQL observations, not a production CPU/WAL claim. The new adoption and identity branches affect recovery preparation only. The `64c0b746`, `7a203274` and exact `39a4343` CI independently passed all three stream sizes with the same zero-SQL/idle result; each latest 8,192-chunk case recorded 36 separate renewal statements. Exact `39a4343` Runner 2/2 passed 133 files / 2,317 tests plus two skips; its separate native group passed 150 tests plus eight skips. The first API fixture revision omitted the mandatory final screenshot; this was caught before paid launch and allocated no provider/model work. The correction captures the actual retained Chromium page with an explicit unqualified-browser label, without changing the standard report validator. Automatic approval review rejected exporting raw private API/workflow snapshots because of sensitive payloads and insufficient destination-specific authorization. Sanitized reports and bound/hash-only receipts are retained; raw snapshots were not copied through another channel. This does not change the failed grade. Repair characterization also preserves its failed attempts. An unconditional staging relocation broke three existing same-scope scripted cases; the final implementation defers staging only for an actual superseded warm scope, and all nine order/race cases pass. One baseline transform overlapped an edit and is explicitly invalid; the actual immutable `39a4343` loader then produced the three genuine REDs. The independent task-directory hypothesis produced a RED but was excluded by the live shared-workspace evidence; no directory policy change was landed. A shared-workspace fixture authorization omission was corrected separately from the four-skill semantic RED. Setup failures are retained separately from semantic outcomes. These include an old host port after the owned Postgres fixture restarted and a sandbox-denied TSX IPC pipe. The supported external-Postgres run passed, and the final exact recursive typecheck passes with normal process permissions. Seven stale declaration errors after the master merge cleared after compiling the actual merged SDK/Runner TypeScript sources. No declarations were edited by hand. <details> <summary>Earlier qualification measurements and failed attempts (historical, before this repair)</summary> **Historical blocked gate (before the repair below).** Final native implementation is `abed921e12d543b62f57a56435419c0dafb0bc33`; current branch head is `af9e2d157dcc349e55374ad186c9be2d6cb96bf9`. The two later commits change only qualification fixtures, their matcher and documentation; `server/` and `packages/` are unchanged. It includes master through `fd008979f5794fda3d5f54151d36acd31d0de140`. The live Daytona measurement proves five-minute active recovery, once-only completion and accounting, but **fails retained-session identity after the second replacement**. The saved checkpoint and recovery metadata exist. The task-session pointer remains empty, and ordinary admission creates new native/provider sessions. The exact skipped publication boundary remains unproven. A useful saved-file result alone does not satisfy retained-session continuation. This application-path gap blocks merge independently of CI. A separate new-master CI failure is owned by the storage workstream (fix in #18357). No storage guard, session-identity oracle, lease/grace limit or provider result was relaxed. | Gate | Observed result and exact scope | | --- | --- | | Distinct source recovery and compatible rollback | PASS at A `5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → B `5ea45f8d17e33c9a62bd79982dcf0375b07733d0` → fresh B → A. Four controller processes, natural 60-second lease expiry, absent native projections before replacements, three succeeded/committed unique results, one provider thread, unchanged Runner identity, and strict idle cleanup. 177.782 seconds. | | Immutable installed native component | PASS at installed A `5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → installed B `eaef0d56552c087aaa4fa68a8655282ffbd12667` → fresh B → A. Each controller imports its contained compiled server and DB. Three unique committed results, stable Runner/thread, no extra provider restart on same-contract rollback, strict cleanup. 106.672 seconds. This is compiled component proof, separate from full CLI-native application acceptance. | | Normal public CLI startup | PASS for B `eaef0d56552c087aaa4fa68a8655282ffbd12667`, Node 24.22.1, darwin-arm64: fresh external DB, 415 migrations, health ready, root HTML 200, company POST 201 plus GET readback. The installed tree remained unchanged after execution. | | Five-minute component outage | Actual controller absence 300,168 ms. Natural generation-2 claim at 407.196 seconds after crash, same provider attempt and Runner, original run succeeded/committed. The larger 540-second case timed out before its retained follow-ups, so it is an active-adoption pass, not a complete four-controller pass. The complete short source/installed cases above independently cover follow-up and rollback. | | Drain/resume review regression | Fixed in `226f03e736d1ee99fea1762ab32ea3a8a9241c20`. Five facade-level cases pass: overdue normal/restart waits, future timer cancellation, another facade resuming the original owner, and transient retry. Against exact prior scheduler `5ea45f8d17e33c9a62bd79982dcf0375b07733d0`, four fail and one passes. The later head changes documentation only. | | Final corrected-fixture Daytona | **PARTIAL: active recovery PASS; retained identity FAIL**, exact live source `0b6afc499f`. A was SIGKILLed and native cache discarded; actual offline hold 300,000.375 ms. B was ready at 308,326.961 ms, naturally claimed generation 2 at 312,059 ms, authenticated at 315,996 ms and resumed processing at 316,519 ms, within the original 420-second boundary. The same run, Runner PID/start, session, lease and event prefix survived; one turn submitted. Original execution succeeded and settled accounting. After completion, B was killed and its native cache discarded; C was ready in 9,077.779 ms. C executed the saved project and returned correct `queue_r`, with all four runs succeeded/accounted. However, native and true provider-session identities changed: the unchanged retained oracle fails. No whole-workflow pass is claimed. | | Prior matched-producer Daytona | FAILED at exact `abed921e12`. The matching current-source Runner passed normal admission, completed the first task and settled its accounting. During the active revision, A was SIGKILLed and its native cache discarded. Actual offline hold was 300,000.727 ms; B was ready at 308,304 ms and naturally claimed generation 2 at 309,651 ms. At about 318 seconds, the remote process check reported `process_not_running` for the original admitted Runner. No authenticated after-receipt or retained follow-up passed. The current failure is preserved; no automatic retry, lease/grace relaxation, debt edit or provider replay is used. Bounded forensic read found preserved exact-bound state with no recoverable failure, a transport-reset diagnostic and no terminal delivery; this supports external process loss without proving the exact cause. The sandbox had a five-minute auto-stop policy, conflicting with the required surviving-sandbox assumption. Owned cleanup precedes the separate corrected-fixture case. | **Live retained-session boundary.** Prior completed run `3806…` and C follow-up `8adf1df3-6744-40bf-b35d-100451f29d9f` both have database session snapshots and `nativeControllerRecovery`. Their tool fingerprint, provider configuration, workspace descriptor, runtime-context digest, driver and work mode match. There is no recorded force-fresh/workspace-selection change, and this is an ordinary task rather than a conversation-generation reset. The same issue/agent task-session row has `session_params_json: null` and `last_run_id: null` from baseline time. Both native normalized UUID and actual provider session/thread hashes differ after C. Full application admission therefore did not select the retained source. No fallback to file execution or relaxed identity comparison is accepted as proof of continuity. **Independent cleanup completed.** Full current-account provider inventory confirms no resources owned by either final fixture, with exact-ID absence checks. Owned controllers and campaign Postgres are stopped/removed. EC2 `i-01551c09af098dde5` is terminated, its delete-on-termination root volume returned `InvalidVolume.NotFound` at 22:14:08 UTC, and the temporary SSH key/agent were removed. The canonical failed report and immutable oracle input are retained. Closed qualification receipt SHA-256: `b4b5fefce287ebfdbf2cc2238efaad62d8ad485aed0e9c0628ae95dbc18a8575`. **Original measurement and grader repair remain separate.** The C reply contains the requested marker followed by a newline and the correct output. The old workflow waited for one literal space, so its normal receipt collection stalled; that original attempt was closed and its canonical `cleanup_failure`/interrupted result is preserved. Commit `af9e2d157d` repairs only this reply predicate, with 144 focused cases, E2E typecheck, discovery and diff checks passing. Ordinary string matching stays literal. The corrected predicate accepts the saved reply, but the unchanged retained-identity oracle still fails. Pure active-recovery grading of the captured measurement passes 7/7; this is not a rewritten passing campaign. The sandbox idle-window conflict was corrected only for this explicit fixture (15 minutes). Actual baseline and later sandbox metadata confirm that policy; the original active box was disposed before a post-gate policy read, so its 15-minute setting is established by the normal create/environment configuration rather than an after-read of that box. Production native lifecycle and the 60/300/420-second constants remain unchanged. The development compatibility reader A is a genuine backport commit based on `59ff89a18fe1312b5f0bb908439bbdb1237b0252`, with the required idle-goal reader fixes. It is not a previously published release. Its older executor differs from B. The rollback floor must include this reader capability. The source component cases admitted the Linux ARM64 Runner packaged at `59ff89a18fe1312b5f0bb908439bbdb1237b0252`; the installed component case admitted the `5cf7aead` package. Their Runner bytes are identical: SHA-256 `b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8`. Each admitted artifact remained unchanged throughout its case. [Final public package build](https://github.com/paperclipai/paperclip/actions/runs/41121460635) and [reader-floor build](https://github.com/paperclipai/paperclip/actions/runs/41122170506) both passed all four daemon builds, package assembly and both Darwin clean consumers. Both Linux ARM consumers failed before provider work with the same existing local Codex permission-profile error: bubblewrap attempts `activity.fifo/.git` and reports “Not a directory.” This matched baseline failure remains open; it is not a native remote continuity pass. Full post-run tree and build-stamp revalidation also passed after the compiled native case. The installed component trees are A `c29114c1bf71c5af00f761aeb9e3d84ed3d3075b9f5600dee3ef2543eb2b6ec5` and B `eda575505373a3bdc630fa4919862b359791fe3a7ef637961a25b73b38efe4b0`. **Controlled PostgreSQL qualification passed.** Baseline/candidate/candidate/baseline workloads used 300 controllers, 10,000 historical tasks, real snapshot/encrypted PRP writes and unchanged pools. PostgreSQL and client were each limited to two CPUs and 1536 MiB on an isolated eight-vCPU Linux host. | Workload | PG CPU per operation | WAL per operation | Database request p99 | | --- | ---: | ---: | ---: | | Steady | +1.18% | +1.81% | +1.91% | | Aligned burst | +3.40% | +2.56% | +3.81% | | Two logical servers | +3.11% | +2.56% | +9.68% | All arms passed absolute/correctness gates with unchanged control transaction counts, no dropped operations and no owner loss. Overlap p99 has little margin below the relative threshold. It uses two clients in one Node process, not two HTTP servers. [Closed measurements and limitations](doc/plans/2026-10-30-native-controller-qualification-evidence.json) and exact-source reproduction patches are retained. [Earlier contended-host failures](doc/plans/2026-10-30-native-controller-load-evidence.json) remain unchanged. The production DB-claim probe claimed 300 owners in 2.84 seconds and restored/committed controls and snapshots in 8.74 seconds, with zero owner loss or settled-idle SQL; its provider references were synthetic. The measured checkpoint/snapshot/renewal paths retain the same persistence behavior after integration; later accounting and business writes have separate owners. Actual Runner stream tests at `da8c7f6e1837ddd896dfe83a7524812fee813487` passed 8/512/8,192 chunks: zero recovery SQL, transactions and changed rows during text intervals; zero settled-idle SQL. Each retained seven control transactions and twelve changed records. Existing renewal SQL was measured separately. The guard/scheduler fixes add no stream path, pool or recurring idle scan. The live fixture used current-source producer SHA-256 `f0000c90cb83880191479da639201e659aa6ce0fc7d67c859faa172ec977849e`, a normal locked exact-head build with pinned Rust 1.97.1 in Bookworm. Actual network-isolated, read-only ABI execution passed (GLIBC requirement 2.34 against runtime 2.36). The immutable base image remained `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:36069788f53c51f228764fa36180575443dfb5d1f84a342d7c0daceb28dc61cc`, source `a0ef3ce7`. This was an unpublished development producer, explicitly distinct from the base image and admitted before the fresh fixture began. The earlier host build failed ABI preflight before admission. The preceding old-producer fixture completed its initial task but lacked the usage-basis marker required by current master; it remained pending/unpriced and correctly paused. It was independently cleaned without debt or budget changes. Two preceding local Linux setup attempts OOM-killed before provider allocation, and their cleanup passed. These attempts remain separate from the final recovery failure. **Failures retained and resolved separately.** The earlier `98f05fc` probe rejected a valid settled PRP v2 goal suffix; `eaef0d5` fixed that reader and the fresh follow-up passed. Later component cleanup failures were traced by a scoped exception observer to the fixture adapter rejecting directory downloads. A bounded owned-directory transfer with exact-byte/mode preflight corrected the fixture; source and installed strict-cleanup cases then passed. No production guard was relaxed. The initial local integrated control-port run hit its 30-second embedded Postgres startup hook before executing 59 cases; 18 other focused cases passed. A bounded rerun used a fresh owned external Postgres fixture, applied normal migrations before Vitest, and passed all 59 cases with the original timeouts unchanged. Its container and data were removed; the failed attempt remains retained. The additional opt-in full Product E2E Vitest run is not green: 5,313 passed, seven failed and three skipped. An actual `abed921e12` archive with the same dependencies, Node and IPC permissions reproduces all seven failures with the same reasons. Five are unchanged workflow-policy/profile-pin assertions, including Pi profile version 37 versus the test pin 35. Two database suites failed during embedded Postgres startup before their assertions; semaphore exhaustion is confirmed in the observer log. The observer passes its supported external-Postgres rerun with original timeouts and source unchanged. The controller-database client-kill/postmaster assertions remain unexecuted after `initdb` bootstrap failure; no external override is supported. The unit launcher stopped before its Node-test phase, which was then run independently and passed 181/181. The original failed full run is preserved. These results remain distinct from the passing focused checks and normal repository CI. The first local Product E2E typecheck also encountered stale compiled Runner declarations; a genuine package-local TypeScript compile refreshed them and the unchanged check passed, without a dependency install or producer rebuild. Earlier [Daytona campaign 41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900) failed at sandbox creation before any agent, with the underlying provider error unavailable. Its [sanitized report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-41108162900-2/index.html) remains a `cleanup_failure`. Both old creation names are absent in the current authenticated account, but the old CI account identity was not retained, so account equivalence is unproven. The subsequent exact `eaef0d5` source Product E2E attempt passed image/agent setup and the initial project, then held the controller offline for 300,003.7 ms. Its Mac supervisor failed owned-process inspection before B readiness; no adoption passed. Full current-company lease/label checks, provider deletion, exact-ID 404s and complete inventory confirmed cleanup of that attempt. Its local server and owned Postgres stopped. The prior AWS campaigns and database qualification VM were terminated with root volumes removed. The old admitted test Runner lacks the newer usage-basis marker introduced on master. The integrated component B turns retained unpriced, incomplete accounting obligations with `costAccountingPending: true` and no settled charge. The compatibility-reader A turn used its legacy receipt interpretation but likewise retained pending accounting and no settlement timestamp. This proves obligation preservation, not a whole-run accounting migration. Runner/receiver release ordering remains governed by the usage owner's contract. At native implementation `abed921e12d543b62f57a56435419c0dafb0bc33`, Greptile was 5/5 with zero unresolved threads and CI completed with 54 successful checks and five expected skips. At `0b6afc499f`, Greptile again passed 5/5 with zero unresolved threads, but CI ended with 51 successful checks, five skips and three failures. Server 9/12 fails a deterministic managed-skills maintenance assertion newly inherited from master `198218bf` in merge `c80e00ad`; that test is absent from the PR head, and the storage owner has the explicit-StorageService fix in #18357. Runner 2/2 passed 2,283 unit cases but its runner received a shutdown signal during the long stream probe; its cancelled step and aggregate failure are preserved. No blind retry was used. Current `af9e2d157d` checks are pending; the matcher-only change passes 144 focused cases and Product E2E typecheck. Earlier scheduling-sensitive CI failures and the interrupted unsharded local test run remain recorded in the dated plan. CI runs the repository's normal test partitions. Final verification does not claim Railway rollout, uninterrupted HTTP service or whole-webserver filesystem loss. </details> ## Risks - The task-pointer repair passes its assembled regression. The first live repair exposed pre-acceptance ordering and the fresh review exposed a replay lock-order issue; both are fixed and pass real Postgres tests at `0862062`. The following live attempt exposed the verified-running-versus-suspended preparation gate. Its narrow fix passes 91 focused cases at `64c0b746`. The next live case exposed synthetic attachment IDs; `7a203274` fixes that field selection and passes real Runner authentication. The subsequent queued continuation exposed strict retirement maintenance and a false default-home change; `39a4343` fixes both with real Runner/Postgres and fingerprint regressions. The `39a4343` live continuation failed again despite green CI. `283e05ffaa` repairs the actual native-observed admission ordering and reproduces/fixes the JSONB skill-order fingerprint drift. Both pass targeted real-Postgres regressions and the live active plus queued paths. The definition 6 post-C browser navigation timed out before admission. Explicit definition 7 now verifies native continuation through the ordinary comment API; its strict post-C native gate failed after a real dispatch and remains historical failed evidence. The finalfc68584repair now passes its fresh strict 49-check Linux/Daytona campaign; the prior failed grades remain unchanged. Browser reconnection is handed to its own workstream with the original failed evidence. No prior failed campaign is reclassified as passing. - Incorrect checkpoint/source binding or stale ownership could duplicate external work. Missing evidence remains a hold; existing receipt and finalization owners keep uncertain effects unresolved. - Recovery must retain maintenance writes after accepted results without allowing a stale business turn to write. Real lock-wait and generation tests cover that boundary. - The current two-connection paired burst profile fails its absolute drain gate in both baseline and candidate, despite passing relative load gates and all correctness checks. A causal capacity diagnostic selected three slots; the final controlled cap-three profiles pass unchanged thresholds. Full application replacement and two-server overlap capacity remain separate. There is no new streaming writer, pool or recurring idle scan. - The remote sandbox and Runner must survive the recovery window. Provider auto-stop is a separate lifetime bound from reconnect grace; the explicit qualification fixture uses a longer idle window. - Workspace and provider backup availability remain external dependencies. This PR does not certify whole-webserver filesystem loss, uninterrupted HTTP service or a Railway rollout. ## Model Used Implementation and initial verification used OpenAI GPT-6 through Codex; the initial hosted snapshot was not exposed. The follow-up qualification used three OpenAI `gpt-6.1-sol` workers coordinated through Codex, with repository tools, code execution, CI and scoped test infrastructure. Context-window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused and exact-source application/cost checks pass; full forced CI passes; earlier local embedded-PG setup failures remain recorded) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (exactfc68584full CI and auxiliary checks pass; forced complete test partitions) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (exactfc68584check and edited summary pass; all six threads resolved) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
40 KiB
navigation
| navigation | ||||||
|---|---|---|---|---|---|---|
|
Recovery
People can see why a task stopped, who owns the next action, and whether a safe continuation is available. Recovery retains task context and ownership, honors pause and budget gates, and distinguishes confirmed failure from an uncertain external outcome. A retry acknowledgement alone is not proof that work resumed.
Feature ID: recovery.
Agent access: companion.
Sub-features
diagnosis:recovery/diagnosis— a task/run exposes the recorded cause, evidence, owner, attempt progress, and next action without requiring a raw-log interpretation.bounded-continuation:recovery/bounded-continuation— supported interrupted work continues with preserved conversation context and bounded attempts, without blindly replaying tools. Remote native runners can retain their active turn across main-server replacement using the encrypted Postgres controller journal. See the five-minute contract and qualification recipe in remote continuity. Restart recovery respects live heartbeat preparation before workspace synchronization finishes. PostgreSQL regressions cover unchanged ownership, no duplicate dispatch, and later recovery after preparation leaves the registry.source-action:recovery/source-action— source-scoped recovery actions remain attached to the task; independent repair work may have its own issue and return owner.exhaustion:recovery/exhaustion— an exhausted infrastructure attempt offers ordinary Retry. Internal repair stays system-owned; genuine user decisions retain their specific action.workspace:recovery/workspace— workspace divergence/export failures show the applicable repair, reconcile, or isolated reissue action with evidence and authority checks. Retained-file recovery runs in the harness with observed task progress and a ordinary Retry action after exhaustion or during backoff. It preserves newer files, keeps uncertain sources held, and delivers saved human input through ordinary admission. The task does not expose infrastructure-certification forms or a workspace link when the destination is disabled. Local PostgreSQL and real filesystem regressions cover native and legacy recovery and one-time delivery; hosted provider recovery remains separate acceptance. Daytona directory export can stream its archive when the sandbox cannot store a temporary tar. The transfer remains bounded and validates archive completion and paths before import. The capacity regression exercises a real local PTY and file-size limit. The credential-gated provider test covers stop/start, an ENOSPC archive sink, safe-link filtering, and a second export. Neither test proves the original task's continuation or repairs lost provider session state. Native finalization deadline failures also retain the exact prepared source for the same stop-and-save workflow before Retry admits another executor. A blocked native restart records that preservation obligation atomically with its failure. PostgreSQL regressions cover exact source binding, live ownership, rollback, and no replacement run or wake from file preservation. Retry reconstructs missing preservation for historical failures only from an exact prepared source without a live finalization owner. New messages and orphan cleanup can register missing preservation automatically and wake the existing worker. Saved input still waits for verified stop and file persistence, then enters ordinary admission once. A completed explicit Retry also retires its failed, unowned sandbox predecessor's stale retry and registers preservation atomically. Current task state and live or mismatched owners remain protected; workspace switching still waits for cleanup.unknown-outcome:recovery/unknown-outcome— uncertain external side effects require inspection and an explicit decision, not automatic replay.user-continuation:recovery/user-continuation— saved user messages after a stop are reconsidered once cleanup proves the old execution is stopped; duplicate wakes are avoided. A user comment that also reassigns the task retains this intent, including existing deferred assignment receipts. The comment's identity and author, current assignment, stop proof and normal execution gates remain required.reconnect:recovery/reconnect— browser stream recovery refreshes persisted state without claiming the agent itself restarted or that a failed run succeeded.
How to get to it (user POV)
For recovery/reconnect, also interrupt a write response after the server accepts
it. Keep a title draft bound to its original task while navigating. Restore a
new-task draft with its files selected in another order and require the same
creation key and one task. Verify lost document-create and decision responses,
including an interrupted verification read. Once the bounded retry budget ends,
require a visible Resend now action. Use
durable autosave tests and
task creation tests for the
controlled source cases; repeat the real browser journey for acceptance.
Switch accounts while an inline edit is open or a channel send waits to retry:
the next account must see neither private text nor a retry from the first account.
Returning to the original account restores its retained channel request with the
same key for explicit retry. Unowned legacy receipts stay held for Activity
inspection until a person explicitly discards their browser record; discard does
not replay the text or cancel an accepted update. Typing before the first session
lookup survives that initial lookup and is backed up under the resolved owner. Component evidence lives in the
inline editor tests and
connected-channel tests;
these controlled session changes do not establish hosted authentication acceptance.
Also switch accounts during a restored task create and while its files are
uploading: later uploads stop, the next account sees a fresh form, and the
original request key remains owned by the first account. Reselect colliding
text-document filenames in reverse order and verify their original keys and
contents. Switch between cached document subjects without copying either draft.
A receipt arriving before an in-flight resend must preserve the next message.
The startup browser journey in
disconnect recovery holds the
first session response while a title save waits, simulates an offline event with
API failures while keeping static assets reachable, then verifies the real saved
server title after account resolution and reload. The accepted-comment journey
then completes and assigns a task before repeating its saved comment request;
the original receipt must return once, its activity must appear once, and the task must stay Done. Reusing that
key with different text must return a conflict without changing the task. This
uses a disposable process agent and makes no provider call. A separate healthy-API
startup journey fails the session endpoint with JSON 500 responses, then restores
it while an edit waits; the same edit must save and survive reload without a
network reconnect or another user action.
source-task
Open the affected task from Tasks, Inbox/Blocked, search, or a direct task link. Observe one working, backoff, failed or waiting-for-user state. Use Retry after an infrastructure failure. No process or file certification is required.
A workspace project denial names the blocked access and links to the agent's Permissions / Trust settings. After an authorized person adds the required project, return to the saved task for an explicit Retry task action. Saving permissions alone does not wake the agent. Historical setup denials receive the same notice on reload; unrelated errors and prior assignees do not. Move a denied task to a project the agent can access while retaining a source from another project: the notice must stay blocked until the selected source is also allowed or an authorized workspace selection replaces it. A valid pending selection is checked for the next admission; a stale or unavailable selection must show a non-retryable failure without a project-permission repair action. Project changes alone preserve selected files.
Thinking appears immediately after Retry, before the request completes. Technical source/run details remain in deliberate diagnostics. The task recovery contract describes receipt identity, fresh workspaces, bounded retries, historical adoption and service incidents.
run-detail
Open the agent's run (/agents/:agentId/runs/:runId) from activity or the task's
run link. The run page hosts workspace repair controls only for a failed run
with error code workspace_validation_failed and a linked source task that still
has a live workspace_validation recovery action. Use the source task for other
stopped-run recovery checks. Native workspace export failures also offer the
read-only Inspect recovery control described in the
run diagnostic recipe. It observes one selected
source's recorded identity and does not retry, repair, or authorize execution.
operator-retry
The connected-task regression uses HTTP Retry after an accepted board message fails before provider startup. It checks the canonical intent, same-user identity and actual queue dispatch, then repeats Retry after another contained preparation failure with the same source and a new certified successor; edited source, contradictory execution evidence and mismatched receipts refuse or wait. This local boundary probe deliberately stops before a provider call, so it does not establish hosted recovery of the task.
Use the task's Retry action. It can create an isolated attempt after old execution is contained even when old-file preservation remains unresolved. Approval, credentials, budget and deliberate pause use their existing specific controls.
For a connected conversation, click Retry while an exact retained source still needs containment. Observe a saved wait with no new executor. After verified containment, the same chat receipt must admit one successor with the original source batch. Reload and repeat reconciliation. Check that a paused endpoint, changed source, superseded request or revoked access cannot start that work. The connected-chat database regression in chat channel integration exercises admission and queue continuation. It does not establish hosted provider recovery or successful delivery.
For GitHub, save a follow-up behind a failed run, then reconnect the same App. Click ordinary Retry and verify that one successor receives the saved follow-up instead of replaying the older request. Reload and repeat the same request key; there must be one successor and no duplicate publication. A second reconnect after admission, changed App, revoked sender or changed repository must prevent dispatch. An unadmitted rejection keeps the follow-up available for another deliberate Retry. Historical deliveries must remain unchanged. Repeat with legacy input that has no saved App ID and was refused by a budget before acknowledgement. An earlier processed response from the same endpoint, conversation and agent can prove its App claim. Missing, late, unlinked or conflicting proof must reject Retry. Verify actual execution separately. At dispatch, confirm the admitted successor retains the same source and resolved continuation receipt. If it fails at Retry authorization before provider startup, use the next ordinary Retry against that newest failed run. It must retain the saved follow-up and create one successor; output, an allocated environment or an active controller must prevent treating that failure as never started.
post-stop-message
After a stop, send a new instruction on the same task or inspect one already saved while cleanup was pending. Observe the waiting message, continuation, and eventual result rather than manually changing the task status to look complete.
A continuation denied before startup by an existing execution hold remains a wait. Once the original hold clears, the server verifies the dispatch receipt and absence of execution evidence, ignores any historical false incident for that unused admission, and reconsiders saved input through normal admission. No infrastructure-certification form is needed. The database regressions cover incident prevention, retained audit history, concurrent restart admission, ordered messages, contradictory execution evidence, and unchanged pause and task gates. These tests do not qualify a live provider or prove task completion.
browser-reconnect
While viewing an active or recovering task, interrupt the browser connection and restore it, or suspend and return to the tab. Reload is a separate check. The current task and run state should reconcile with the server.
For a consumed review path, verify that the one automatic repair run receives the instruction to save a one-shot monitor before it promises another check. Repeat with a fresh session, a resumed session, and a connection-pool retry that retains the typed repair marker but changes the wake reason. Confirm a persisted schedule before claiming that follow-up will occur; the repair instruction does not schedule one by itself. The wake regression exercises the real wake builder, serializer and prompt selector with an empty credential-history database boundary. It does not prove a hosted agent scheduled or completed the follow-up. Exhaustion still blocks for a board decision.
Driving it
Preconditions: follow the baseline. Use a disposable instance and reproducible failure. Record source task/run, adapter/runtime mode, cause, attempt count, current owner, and verified stop evidence. Do not kill an unrelated server or induce provider side effects just to get a failure screen. Use the existing deterministic recovery suite first.
Operation checkpoints
Use these checks alongside the entry-point recipes below and the operation records in the access companion. Record the actual actor, surface, saved result and denial separately. Source review and linked tests do not establish a live pass.
| Capability ID | Action and persisted result | Denial, race and retry checks |
|---|---|---|
recovery/diagnosis |
Open the source task and exact run; compare cause, source action, original owner, recovery owner, attempt bounds and next action after reload. | Test an unrelated/private task and a stale run notice. Read-side retirement of a stale action must not create a new attempt, participant, or permission to replay. After workspace recovery, compare task and run instructions; recovered files must not leave instructions to repair a retained sandbox. The stopped run keeps its replay hold until continuation admission. An obsolete exact-session notice retires only after terminal coordinator proof and a separate exact-run workspace hold; no new execution or file cleanup follows. |
recovery/bounded-continuation |
Use a deterministic recoverable interruption and inspect the original run/session, attempt receipts, saved result and eventual useful output. | Repeat restart/receipt delivery, race Stop or a new owner, and retain unknown external outcomes. Accepted completion/finalization recovery must not start a new provider turn or repeat its side effect. |
recovery/source-action |
Inspect active and retained actions, follow the repair owner, and resolve one eligible source-bound action with its exact ID. | Retry the settled resolution and reject a stale/foreign action. Recovery ownership alone cannot mutate the source task; inspect source status, audit and wake independently. |
recovery/exhaustion |
Exhaust or age a bounded fixture; use the currently offered retry once, then inspect its durable acknowledgement and actual continuation. | Reassign, pause, stop, close, add an unresolved dependency, or impose a budget hold before retry. The old notice must not reopen terminal work or run the wrong owner. |
recovery/workspace |
Exercise retained-file recovery, export retry, reconstruction, branch repair/reconciliation and exact-owner finalization as distinct authorized operator paths. | Use exact source/action/run/receipt bindings and preserve existing files. Reject stale source, live/uncertain writers, unsafe archives and insufficient authority. Verify actual files and accepted result separately from queued acknowledgement. |
recovery/unknown-outcome |
Inspect the provider and saved intent before recording a source-specific chat, email or governed-provider outcome. | Use performed, not-performed and unresolved cases. Wrong company/request, stale state and unauthorized resolution must fail. Unknown effects must not replay automatically. An explicit retry must record the operator’s duplicate-risk acceptance and dispatch only the authorized attempt. |
recovery/user-continuation |
Save a distinguishable instruction while Stop cleanup is pending; after verified stop, inspect exactly one eligible continuation and its useful response. | Keep the message waiting under task/ancestor pause, budget or retained-file holds. Do not impersonate a human message or infer delivery from a queued wake. |
recovery/reconnect |
Disconnect the browser, let the server change persisted task/run state, reconnect, then reload and compare. | Inspect duplicate comments/decisions and private/company boundaries after access changes. Browser transport recovery and agent execution recovery require separate results. |
source-task
Automated: recovery actions covers source scoping, bounded attempts, operator stops, quota monitoring, ownership, stale action retirement, and continuation deduplication. Recovery cards cover diagnosis, retry progress, exhaustion, and action availability. The fixture-backed execution-recovery journeys start their own test-drive instances and exercise safe, uncertain, restart, manager-lineage, and legacy-unknown paths:
pnpm exec vitest run server/src/__tests__/issue-recovery-actions.test.ts ui/src/components/IssueRecoveryActionCard.test.tsx
pnpm exec playwright test -c tests/e2e/execution-recovery/playwright.config.ts recovery.spec.ts
Manual: open a stopped task through each affected navigation path. Confirm its explanation, original owner, recovery owner if different, and next action. Follow the supported recovery until the source task produces a useful result or a specific escalation. Reload and verify that resolved notices retire and attempt history remains. A fixture pass does not qualify every live harness/provider.
The native restart integration suite also covers a prepared remote workspace interrupted before the executor's first claim. Real database tests assert one same-run bootstrap claim under concurrent controllers and reject prior claims, launch/provider evidence, and malformed workspace references. These cases prove classification, not a live remote provider restart. For that journey, interrupt a disposable deployment after workspace preparation and before its first executor claim, then confirm one provider turn starts on the original run through the normal admission gates.
The same suite's diskless and diskless-per-turn cases exercise three native
controller processes with both native projections removed between completed
turns. They check history and invocation counts through the real executor and
admission snapshot rebinder. The workspace and sandbox remain available, and
the processes use the same source revision. Separate Postgres tests cover
metadata import, bounded discovery, ownership holds and incomplete-publication
retention. These cases do not qualify full webserver disk loss, distinct releases,
HTTP traffic continuity, or the live Daytona provider path.
The Runner transport regressions exercise a real WebSocket welcome delayed beyond two seconds. Authentication uses the ten-second connection setup budget, clipped by the current lifecycle deadline. The cases also retain failed-welcome ticket fencing, identity checks, and reconnect replay without repeated provider effects. These transport cases do not establish hosted provider acceptance.
The native provider regression sends malformed activity from a real ACPX sidecar process. It checks a structured transport error, one failed terminal, a closed session, and no provider reopening. It proves the error contract and retained failure boundary, not the cause of a historical provider outage or successful hosted task recovery.
Active remote process supervision distinguishes a failed health read from a confirmed exit. The executor tests exercise timeout and repeated gateway-failure recovery with a real listening process, unchanged ownership, and rejected identity evidence. Read-only probes use their shared deadline with bounded backoff; fast failures do not exhaust a three-attempt limit. Signal requests remain single-attempt. The Runner restart tests require positive exit evidence before replacement. The executor regressions also retire a real child during its metadata read, keep unknown adopted-process ownership held, and reject launch before detachment when Node is unavailable. The diagnostic tests cover permission failures and generation-bound zombie receipts. This coverage does not prove recovery of a historical retained sandbox or behavior during a live provider outage. The transport regression also retains a real authenticated Runner and its journal when its close probe rejects, without signalling it, checkpointing it, or starting a replacement. The failed close still retires its local connections, event timers and notification stream. Repeated close cannot report successful retirement.
For a release change during a remote provider session, verify that reattachment uses the controller's admitted pack manifest and runner digest. Check useful output from the same process without replaying a turn. Repeat with a modified sidecar, mismatched receipt scope, replacement lease, and incompatible runner protocol. The authority tests and executor wiring regressions cover selection and fail-closed scope checks. They do not prove a historical retained sandbox was recovered.
For completion recovery, restart after a native completion proposal is appended but before its session checkpoint. Verify that the original result is finalized after provider termination, with no new provider turn or repeated external action. The runtime regressions exercise the real Codex driver with a simulated transport, including a still-active provider and invalid, conflicting, duplicate, or stale proposals. This does not qualify a hosted sandbox restart.
The same database suite pauses an original preparation while recovery claims the remote run. It then verifies that the old preparation yields without opening a provider or changing the recovery owner, for both live and expired leases. Heartbeat boundary tests verify that the detachment signal transfers recovery ownership before cleanup. The suite also rejects an older unmarked preparation and verifies that calling preparation again cannot add the fencing marker from supplied profile data.
For a native workspace owner hold, repeat issue reads and recovery checks after the task becomes done or cancelled. Verify one unchanged action ID, attempt count, and timestamps; task completion must not authorize copyback. Clear only the exact receipt after operator stop verification, then confirm stale action cleanup. The route regressions and heartbeat regression cover this sequence without a live operator recovery claim.
An intentionally Done onboarding parent can still need its validated
child-completion report. Recovery must not mark that callback successful before
execution solely because the parent is Done. The existing
stale-lock sweep suite
covers delayed native/Legacy startup, invalid parent/child/company/assignee
bindings and a dead owner. Process-death and ownership backstops remain active.
Repeat the real onboarding journey and correlate callback turn events with its
reply and downloadable child file; status alone does not prove delivery. The
literal ea6c8eb5e16d61111968f633a66748b397b64a3e live failure is retained,
and the revised behavior still needs live qualification.
run-detail
Automated: run workspace recovery tests this host; it is component coverage, not an end-to-end repair.
Manual: prepare a failed run with error code workspace_validation_failed, a
linked source task, and a live workspace_validation recovery action on that
task. Open that exact run and compare its recovery diagnosis to the source task.
Use an eligible action from the run surface, return to the task,
and verify the same action state, persisted outcome, and restored execution.
Repeat with an actor who lacks authority. Verify that the run-page controls are
absent for other error codes or after the source action retires; continue other
recovery checks on the source task. Full run-detail navigation remains a manual
gap.
operator-retry
The workspace recovery regression uses the real Retry producer and a quiet delivery coordinator. It checks that a committed request starts one physical source preservation, duplicate Retry keeps the active owner's receipt valid, and startup reconstructs a resolved files-only obligation without provider replay. The execution-control lifecycle suite checks independent finalization, stored deadlines, and a bounded pass through completed obligations followed by quiet paused input. The physical export regression holds one export while another completes, then verifies a third Retry can use the free slot and shutdown joins the held work. These local checks do not prove recovery of a deployed allocation. The historical deadline case starts without preservation bookkeeping, holds the provider stop reply, and verifies that HTTP Retry cannot admit a successor until the real temporary files are restored. Duplicate requests then admit one queued turn. GitHub and Telegram admission also retain the exact pending recovery.
Automated: disposition notices tests pending/error/gated states and waiting for the real retry acknowledgement. The recovery-card suite covers workspace divergence, reissue, reconcile, and confirmation. Workspace export covers quiet restoration, legacy/operator assistance and resolved states.
For a retained legacy or native file hold, open the source task after its recovery action has resolved but its files still need recovery. The shared task projection shows actual working/backoff state or ordinary Retry. It does not show a separate infrastructure attention card. Unknown external outcomes remain fenced internally. An authorized board operator can use the resolution API to record verified files and a backup link without changing task status or starting another run. The server independently validates the original stopped-execution receipt. Agent Chat retains the hold rather than suggesting another message. The notice and repair route cover quiet live recovery, read-only access, stale or missing stop proof, source-only repair, and preserved task status. These tests use fixtures; separately verify original files and their attached artifact before recording a real repair.
For a source with a retained-file protection marker, exercise recovery through the assembled environment runtime. Its exact preservation owner may reopen the stopped allocation to read saved files; ordinary resume stays blocked. Reject a changed claim, missing stop intent, wrong lease or wrong environment before a provider call. Repeat accepted-native-result copyback for both reusable lease policies. Require the exact accepted result and completion contract; reject a rejected or unselected result, another allocation owner, and an active host writer. Verify no provider turn or wake is created, then finalize the original result once. Reject malformed or oversized Git metadata through native finalization: the run must fail and retain its file-recovery barrier, rather than successfully omitting the export. Exercise both an oversized sender file and an oversized transfer report. Verify one rejected export, no payload read after rejection, and unchanged host files. Reload a large-result run and its activity: the failure code and task recovery notice remain visible, and metadata repair offers no new provider turn. These database and filesystem controls do not qualify a deployed provider. Verify saved and newer local files, one preservation receipt, and no additional agent turn. The PostgreSQL regression covers this boundary; the opt-in provider probe also uses this runtime path with the protection marker.
For a missing managed allocation, verify a fresh Cloud-signed absence observation under the exact physical recovery owner. The existing Retry may then select an independent workspace through normal admission. Verify that the original run stays failed and missing files are not called recovered. Refuse a foreign stack, source, allocation or owner, expired signatures, changed provider organizations, permission failures and provider outages. Local PostgreSQL and signed-protocol regressions cover these boundaries; deployed recovery requires separate proof.
Repeat source-only repair with a succeeded legacy run that still has a retained
file obligation. Its result must remain successful, its current task owner and
status must stay unchanged, and later saved input must become eligible for normal
admission. A running or queued source and a normal retry of a successful turn
remain denied. The reconciliation regressions
cover these cases and stop-receipt validation without claiming live provider proof.
For an abandoned native copyback owner, the harness verifies the exact stopped
deployment and the reconciler restores saved work automatically. Task users see
no process details or shutdown form. The documented operator API handles older
receipts after independent platform verification, through a separate administrator
gate. The authorization regressions
and route regressions
cover administrator admission, ordinary-member and agent denial, suspended and
foreign membership, missing role data, and stale cloud admin rows.
The native workspace finalizer suite
runs the real reconciler with a valid saved yield from result_accepted,
workspace_finalizing, and retryable_failure. It verifies physical fencing
before verified stop, automatic and operator recovery, successful export and commitment, hold resolution,
unchanged saved results, retained approval/monitor waits, and no new provider wake.
These local regressions do not prove a deployment platform stopped a foreign writer.
Manual: reproduce an exhausted disposable action. Click once, verify pending state prevents repeated clicks, and inspect a successful continuation or an explicit rejection. Reload to confirm the durable outcome. For a workspace repair, record the compared branches/revisions and preserve uncommitted work; confirm the repair's actual result before resuming. Check the denied and stale action paths. A changed card label is insufficient proof of workspace integrity.
For graceful legacy shutdown, interrupt a disposable sandbox run while file export is in flight. Delay settlement past the first 30-second wait. Confirm the lease stays available, no repair hold or successor appears before export, and the finalizer drain waits for the same owner. After export, confirm saved files, successful lease cleanup, and one bounded continuation. Reload before checking the successor. Repeat with controller loss: the expired fence must retain the exact source and require repair, never claim restored files or start a second executor. The database-backed shutdown regression checks finalizer joining, release order, durable retry uniqueness, and reload. The settlement regressions cover other-controller fencing and owner loss. Provider boundaries are controlled in these tests; they do not qualify a live sandbox export.
For an abandoned native controller, follow the stop-verification procedure with a disposable deployment. A new platform-bound owner recovers automatically only after the harness signs proof that the exact deployment fully stopped. Legacy owners require independently verified stop evidence through the operator API; the task has no shutdown form. Confirm the original accepted result commits, the recovery hold resolves, saved files remain intact, and existing approval waits or monitor checks remain. No new agent turn should start. Repeat with a retry-phase owner and an ordinary runtime manager; the first can resume through automatic proof or authorized operator confirmation, while the second cannot use the operator API.
post-stop-message
Automated: legacy continuation, queue routes, and native restart recovery cover their respective recovery paths. Legacy failure continuation adds a browser journey; it is not native runtime proof.
Queued Stop database regressions verify that an operator cancellation before runtime selection releases its own locks, preserves deferred input without launching it, and leaves another active cycle of the same routine unchanged. Provider cancellations and already selected or dispatched runs retain their existing release behavior.
Manual: stop a disposable run, submit a distinguishable follow-up, and observe any cleanup wait. After confirmed stop, verify the follow-up reaches one eligible continuation and produces the requested answer. Repeat with a task/ancestor pause and a budget hold: recovery must not bypass the gate. For unknown external outcomes, inspect the provider before authorizing another operation.
browser-reconnect
Automated: live-update recovery covers client transport recovery with mocks. It does not prove runner restart.
Manual: go offline during a disposable run, let the server reach a new state, then reconnect. Confirm task/run/attention data refresh without duplicate comments or decisions; reload and compare. A healthy websocket proves transport only. Separately inspect the underlying execution if it is still stuck.
Gotchas
- Legacy and native runtimes have different ownership and restart paths. Record which one was exercised instead of inferring coverage from the adapter name.
- A process disappearance, elapsed timeout, or successful API retry does not prove an external tool operation never happened. Unknown outcomes stay unknown until corroborated; automatic replay can duplicate a side effect.
- Active-run watchdogs, task watchdogs, reviewers, and recovery owners have different authority. No recovery card grants board permission by itself.
- Idle Agent Chat is not stranded work. An intentional operator stop must not silently trigger recovery that fights the user's instruction.
- Source inspection and mocked component tests do not qualify real server restart continuity. Use the appropriate runtime suite for that claim.
- See execution semantics for the governing recovery contract and steering for saved follow-ups.
Storybook verification
exhaustion
Capability ID: recovery/exhaustion.
Scope: ui Storybook: Exhausted/overdue attempts, operator retry pending/error, blocked authority/gate and specific escalation. Production: Inspect durable attempt bounds and actual eligible retry/repair; acknowledgment alone does not prove continued execution. Sources: Source: IssueRecoveryActionCard.tsx, DispositionRecoveryNotice.tsx, issue-recovery-actions.ts. Tests: IssueRecoveryActionCard.test.tsx, DispositionRecoveryNotice.test.tsx, issue-recovery-actions.test.ts, recovery.spec.ts
unknown-outcome
Capability ID: recovery/unknown-outcome.
Scope: ui Storybook: Uncertain external outcome, required inspection/evidence and explicit permitted operator decision. Production: Corroborate provider outcome before authorization; enforce no automatic external side-effect replay and reject stale/unauthorized decisions. Sources: Source: IssueRecoveryActionCard.tsx, issue-recovery-actions.ts. Tests: IssueRecoveryActionCard.test.tsx, issue-recovery-actions.test.ts, recovery.spec.ts
user-continuation
Capability ID: recovery/user-continuation.
Scope: ui Storybook: Saved follow-up waiting on stop cleanup, pause/budget gate, one eligible continuation and result/failure. Production: Confirm previous execution stopped before delivery, preserve message, avoid duplicate wakes and honor actual task/ancestor/budget gate in separate legacy/native paths. Sources: Source: TaskChatQueuedMessages.tsx, legacy-continuation.ts, native-restart-recovery.ts. Tests: legacy-continuation.test.ts, native-restart-recovery.test.ts, issue-queued-comments-routes.test.ts, legacy-failure-continuation.spec.ts