From 492d537340532c8a1770f0a6c5285dacd1e430e6 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 31 Oct 2026 07:33:52 -0500 Subject: [PATCH] fix(native): recover controllers across webserver replacement (#18343) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution runs in a remote Runner while the webserver owns its controller. > - Encrypted PRP checkpoints and session snapshots already survive a webserver restart in Postgres. > - Ordinary warm and per-turn admission still needs local markers and supervisor metadata to find that state. > - Replacing the webserver with an empty native controller directory can therefore lose a usable retained session. > - This change adds bounded recovery metadata to the existing snapshot transaction and reconstructs local projections from the selected durable source. > - The benefit is native execution continuity across webserver replacement, with the existing ownership and agent lifecycle policies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native remote controller recovery and retained-session continuation during a webserver upgrade. **Subsystem affected** server/ — native runtime, checkpoint persistence, recovery scheduling and internal upgrade reporting. Product E2E qualification also changes. **Current behavior** Without both the local PRP marker and supervisor file, normal continuation can return before discovering the saved Postgres checkpoint. The snapshot already exists in Postgres, but its local reuse compatibility envelope cannot be reconstructed. **Proposed behavior** Select the existing durable source, validate its identity and ownership, restore its PRP state and supervisor projection, then use normal authenticated admission. Missing or inconsistent evidence holds recovery. It does not authorize fresh provider work. **Reason and benefit** A replacement webserver must recover active execution and later continue its retained session without the old server's native files. This is one component of deployment continuity. HTTP traffic switching, workspace storage migration and Railway rollout remain separate. **Breaking changes** No new public endpoint, schema migration or database pool. The private run-profile envelope is additive. Disk-independent rollback also requires a reader for the staged controller format and compact checkpoint part; the first production release containing these readers is the floor. Older file-backed sessions need an owner checkpoint or the bounded internal importer. **Additional context** Related foundations: #17401, #17916, #17254, #14621, #14622. The implementation contract and open acceptance gates are in [the dated plan](doc/plans/2026-10-30-native-controller-checkpoints.md). ## What Changed - Persist a versioned recovery envelope, at most 4 KiB, beside the redacted session snapshot in its existing transaction. Keep it out of public run projections. - Compare frozen admission separately from prepared attachment instructions during staged recovery, preserving exact template/identity/authority checks. - Bound the existing lazy checkpoint pool at three connections after measured two-slot burst backlog; keep request/renewal capacities and SQL operation counts unchanged. The tuning is a separate commit. - Discover and restore the exact selected retained source without local native markers. Preserve the 32-candidate window, progress barriers and current authority checks. - Fence snapshot and result writes. Acquire coordinator, issue and run locks in order and check expiry after lock waits. Combine snapshot ownership/read work into one statement. - Reuse existing recovery deadlines and drain/resume scheduling. Retained-owner waits preserve the original unstarted run and actual lease deadline. Stop, ownership and provider-progress fences precede deferral. Add bounded native evidence reporting, retained-source import and retention protection. - Admit a settled warm source with an authenticated idle goal-read suffix while retaining active-work and mixed-authority holds. - Permit lifecycle `turn.stop` checkpoints after accepted results under existing ownership fences. Claim the existing successor coordinator before retiring the old warm session, and stage shared attachments only after verified retirement. Preserve omitted isolated-home defaults and existing fresh skill-selection order during recovery without hiding real configuration changes. - Carry the exact durable six-field PRP binding during active recovery after a healthy warm handoff. Keep the Runner mismatch guard and preserve opaque attachment IDs. - Preserve an already verified active or idle warm process during remote preparation; revalidate its proof/liveness and retain later authenticated PRP attachment. Ordinary resumed leases retain suspension checks. - Drain retained source events under their original authority, reconcile immutable source accounting and block unknown source tool dispatch before target work. - Persist compact staged handoff intent through authenticated adoption and recover direct/listener command-loss windows without another Runner or duplicate attachment. - Rebuild suspended lifecycle receipts for resumed prepared tasks, and retry missing lifecycle reads after partial PRP materialization without overwriting contradictory evidence. - Add normal warm/per-turn diskless continuation coverage, real Postgres races and SQL-cost checks. Add a reproducible paired load workload. - Give only the explicit outage fixture a 15-minute provider idle window, with a red/green catalog regression preserving all production timing and lifecycle policies. - Publish the goal-less task-session pointer in the existing post-acceptance terminal checkpoint transaction, with current ownership, accepted-result, reset and newer-progress fences. Keep a private bound receipt so late completion cannot duplicate or resurrect publication. - Use a retained warm sandbox in the dedicated continuity fixture, and accept an exact output token followed by an explanation. Keep the strict identity oracle. - Extend the explicit Daytona cell to a five-minute active outage and another replacement after completion. Require retained provider identity, a new tool receipt and a useful follow-up result. Add an owned loopback Postgres E2E fixture for hosts that cannot start embedded Postgres. ## Verification **Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`: ready for the qualified native-controller component.** Full CI, fresh review, local application paths, controlled performance qualification and the strict live Linux/Daytona case pass. Independent resource cleanup is complete. Broader rollout, browser reconnection and the unexecuted transport/platform combinations remain integration gates. Integrated 484 passed full local recursive typecheck/build and ordinary active→completion→cache-loss→C component qualification. Fresh review then found a valid attached-wake failure: preparation appends attachment instructions before the verifier compares execution to the frozen admission. 9f4 fixes that by separately checking the pre-preparation admission and exact prepared execution. Both hashes are captured privately per admission; the full template, identity, configuration and authority checks remain. Its regression fails before the fix; 36 focused cases, 42 staged source/target cases and server typecheck pass afterward. The review thread is resolved. Exact-head full CI passes ([run 41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822): 48 successful jobs, one intended skip); the complete `pnpm test:run` partitions are forced, with the five native suites in the Runner lane. This is not a claim that an unsharded local command passed. The fresh exact-head Greptile check succeeds with 5/5 and all six threads resolved. The original two-connection database comparison closes **steady PASS and burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all operations/ownership remain correct, but candidate arm 3 drains in 1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged 1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2 diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72 ms: three is the smallest tested passing cap, and the repeated two-cap failure survives run-order/cache effects. Separately reviewable `fc68584ce6` changes the existing lazy checkpoint pool default to three connections. Request capacity stays ten and liveness two; idle release, shutdown fencing, transaction counts and pool count are unchanged. Thirteen DB lifecycle/cap tests and DB typecheck pass. Full recursive typecheck/build pass at exact fc68584, with the matching server build stamp. The final paired native feature-cost comparison explicitly uses the **same cap-three policy and common current DB factory in both variants**; its baseline is controlled, not the unmodified historical cap-two configuration. Original gates and workload counts remain unchanged. The one strict Linux/Daytona case started at 04:40:56 UTC only after both new profiles passed; it passed on the first attempt with all 49 workflow checks. Independent cleanup confirms all three owned sandboxes absent, owned Postgres/controllers/observer gone, VM terminated, root volume absent and ephemeral SSH key/agent removed. **Final controlled cap-three qualification: PASS.** Baseline native implementation `59da15066a` and candidate `fc68584ce6` use the same current DB factory and checkpoint policy (request/liveness/checkpoint = 10/2/3). Eight 65-second ABBA arms cover 300 controllers and 10,000 historical tasks. All correctness, absolute and 10% relative gates pass; original cap-two failures remain unchanged. | Mode | DB CPU µs/op, baseline → candidate | WAL B/op, baseline → candidate | DB-request p99 ms, baseline → candidate | Maximum drain | | --- | --- | --- | --- | --- | | Steady | 542.40 → 554.83 (+2.29%) | 1509.01 → 1558.27 (+3.26%) | 107.11 → 111.90 (+4.48%) | 26.35 ms | | Burst | 1036.36 → 1063.11 (+2.58%) | 4153.12 → 4317.64 (+3.96%) | 103.736 → 103.841 (+0.10%) | 930.14 ms | Every matched arm preserves operation/statement/transaction counts, including 3,900 PRP and 3,900 snapshot transactions. All three actual pools are included; there are zero lost owners, missed lease deadlines, dropped operations or errors. Peak Postgres CPU is 18.31%/9.47% of two cores and memory 614.29/684.42 MiB (steady/burst). The benchmark uses one Node process simulating 300 controllers; it does not qualify two-webserver overlap, HTTP latency or hosted capacity. Its observer SQL contributes to measured database CPU. The original unmodified cap-two baseline is separately retained as failed; this cap-three comparison is explicitly controlled. The exact final-source 8/512/8,192-chunk test also passes: zero recovery SQL, transactions or changed rows in each text-only interval, zero idle SQL, seven lifecycle transactions and twelve changed rows per sample. Existing lease traffic remains separate. The final ordinary managed application replay passes active recovery, completion, native-cache removal and C continuation; its UNKNOWN local debug-daemon provenance and partial instruction-observer limits remain recorded. The live case uses the attested Linux producer instead. **Live qualification: PASS, one attempt, no retries.** Campaign `native-controller-retained-fix-fc68584ce6-ec2-20261031T044055Z` started 04:40:56 UTC on the exact final Linux webserver source; suite definition hash `12f9bb97823c1537e43f0398e7cddf39890f77bce7b413ca4dfd944f4a30e5d5`. The existing immutable image and attested Bookworm Runner artifact are unchanged. - A was SIGKILLed and its native cache discarded. B started after 300,149 ms, naturally acquired generation 2, authenticated at 316,624 ms and resumed at 317,682 ms within the original 420,000 ms grace. The same active run, Runner, provider PID/start fingerprint and one submitted/started/accepted turn were preserved; one final result/terminal and workspace finalization completed, with no retry/recovery action. - After active and queued completion, B was SIGKILLed and its native cache discarded again. C received exactly one ordinary task-comment API follow-up, completed a new tool and produced a useful reply while preserving native/provider/Runner identities. This closes the earlier failed post-C gate. A separate C OS-process PID field was not exported; the physical-PID proof is specifically A→B. - Four distinct results were accepted and published; all four finalizations committed at attempt 1, all four accounting-v2 projections settled, and the task-session pointer advanced to C. The coordinator independently checked these facts and all 49 canonical checks against the sanitized receipts. - Canonical cleanup passed and independent checks confirmed three exact sandbox 404s plus complete inventory/zero owned remnants, no owned PG or controller/observer processes, EC2 terminated, root volume absent, and the ephemeral SSH key/agent removed by 04:54:25 UTC. Live scope is classic remote Codex warm continuity with non-native storage dependencies available. Prepared-task completed B→C, full per-turn application disk loss, browser reconnection, two-server load and overall Railway/HTTP continuity were not live-qualified here. Earlier failed campaigns and the original cap-two burst failure remain failed historical evidence. A report-renderer cache setup failure was preserved; the normal sanitizer subsequently succeeded with an explicit cache path and a new output directory, without changing validators. The repair keeps retained events under their original six-field authority until persistence and accounting accept them. Unknown semantic effects stay unresolved. A compact staged target survives source bootstrap, cached-command delivery and authenticated handoff, including an uncertain commit response. Direct and same-guest-listener recovery require the verified original Runner and the exact endpoint. Target snapshot publication marks readiness in memory, with retirement coalesced into an existing command write. Headers contain only the staged descriptor/digest; a prequeue template is written once as a changed checkpoint part and then recovered from the existing command/receipt. No second session snapshot transaction, streaming SQL path, new pool or recurring idle scan is added. | Current repair gate | Evidence and scope | | --- | --- | | Source-tail regression | Real Runner protocol RED reproduced attachment rejection. Frozen core/transport pass five source-tail cases: source usage attribution, held persistence/no premature ACK, lost commit response plus cache loss, unknown semantic hold and accounting rejection hold. | | Handoff loss windows | Direct and listener modes pass queued/prepared/activated/completed boundaries (eight cells), preserving the actual Runner/provider process. Missing adoption proof and changed endpoints reject recovery. | | Compact storage | A 31 KiB attachment grant writes its separate part once, leaves it unchanged across source snapshots and removes it at queueing. Headers do not duplicate the grant. | | Accounting | 21 real-Postgres usage tests and 19 source-binding tests pass. Recovery hydrates immutable source receipts without resetting their producer or billing classification; conflicting/incomplete usage holds. | | Admission and malformed evidence | Driver 16, staged evidence 42, restart classifier 27 and source usage 19 focused cases pass. Four malformed-file regressions pass at d6a7, whose executor is byte-identical at 484, and server typecheck passes. Full executor file: 990 pass; 16 embedded-Postgres `initdb` setup failures before assertions, retained separately. | | Required checks | Exact fc68584 recursive typecheck and build pass locally, with matching server stamp. Forced full CI succeeds (48 jobs pass, one intended skip); all PR checks are clean. Maps/token gates pass for their unchanged source surfaces. Fresh exact-head Greptile is 5/5, all six threads resolved. | | Ordinary application path | Exact fc68584 managed component passes active recovery after natural expiry, completion, native-cache loss and ordinary C continuation. Production bytes match before/after; temporary fixtures restored and owned DB/services cleaned. UNKNOWN local debug-daemon/partial instruction-observer limits remain; the separate live case uses the attested Linux producer. | | Producer | Locked Bookworm build from exact 5b7 Rust tree d03e49421714f8c7300de451bf3141ff1dd1a74e, equal to final fc68584 Rust source. SHA-256 65b8f8cfdd6099e9ad24f1ca154e8a260777269cc82f6e23c6e6a180c531eaaf; 38,515,328 bytes. Actual Debian 12 execution and live admission pass. Controller/admission/DB changes do not relabel the producer source. | | Historical cap-two comparison | Exact 59da baseline vs 5b7 candidate, 300 controllers/10,000 history, ABBA 4×65 s per mode, all three actual pools and real snapshots. Steady CPU/op 541.36→554.64 µs (+2.45%); WAL/op 1512.13→1560.99 B (+3.23%); DB-request p99 102.78→105.99 ms (+3.12%): PASS. Burst CPU/op +4.71%, WAL/op +5.29%, p99 −0.10% pass relative gates, but absolute drains above fail. Identical operation/SQL counts, zero lost leases/drops/errors. One Node process simulates 300 controllers; this does not qualify two-webserver overlap or HTTP capacity. | | Live acceptance | Strict final-source Linux classic Codex case passes 1/1 on attempt 1, all 49 checks. Active recovery within grace and useful retained C continuation succeed; four accepted/published/committed/settled outcomes. Independent provider, PG/process, VM/root and SSH cleanup complete. Prepared/task and broader rollout limits are explicit above. | **Historical qualification snapshot through `9eb9ac6`.** The head at this snapshot was `9eb9ac6f9e312f3ebe8040f5a7565e88ae8824f3`. Server and Runner package sources are byte-identical to `283e05ffaa`; the two later commits change the explicitly scoped qualification and its documentation only. It includes master through the storage-owner fix `45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` (#18357). At that snapshot, CI/review passed but the strict post-C native recovery gate failed; the PR stayed draft and required another native repair. Prior `39a4343` CI and review passed, but its full live workflow failed; that grade remains unchanged. The first repaired `435ccd2` live attempt failed before any injected outage. It exposed a legitimate terminal snapshot before result acceptance; no accepted result ID or publication receipt was observed. Its canonical report remains 0/1 `candidate_failure`, with cleanup passed and independent provider absence confirmed. `0862062` defers publication until result acceptance and fixes the fresh review's coordinator/issue lock-order finding. Both failures have deterministic real-Postgres regressions. The `0862062` live attempt then proved accepted-result publication but failed active recovery: a resumed-lease preparation guard demanded suspension after the exact live Runner was already verified for adoption. `64c0b746` preserves that verified active/idle-warm process, revalidates its proof/liveness, and keeps ordinary suspension checks and later PRP authentication. Its live run passed preparation but exposed a second active-warm boundary: all durable binding fields matched except opaque turn/item attachment IDs, which the server had regenerated. `7a203274` carries the complete verified six-field binding and keeps the strict Runner mismatch guard. Its cloud case then passed active adoption, completion, publication and accounting. The queued continuation exposed accepted-result retirement writes and an implicit-home fingerprint drift; both are fixed with deterministic regressions in `39a4343` before this fresh full qualification. The prior live case exposed a real publication gap. A goal-less native turn could persist its accepted result and terminal snapshot before provider cleanup, while the task-session pointer was still empty. Normal later admission then selected a fresh session. The fix publishes that pointer and a private bound receipt in the **existing post-acceptance terminal checkpoint transaction**, before cleanup. Earlier terminal observations persist without a handled receipt or continuation pointer. No result-acceptance or snapshot transaction was added. The late completion path recognizes the receipt. This also prevents a late callback from restoring a pointer after a user reset. Goal/control publication keeps its existing owner. The dedicated Daytona fixture now uses the existing warm/reusable environment. Its sandbox, Runner and provider storage must survive. The former generic per-turn fixture disposed the sandbox after completion and kept provider-history backup bytes under the discarded directory. Whole per-turn disk-loss acceptance therefore has a separate provider-backup storage dependency. It remains distinct from the passing native per-turn component checks with backup storage available. The 60-second lease, 300-second outage and 420-second grace are unchanged. | Gate | Current repair evidence | | --- | --- | | Assembled regression | Baseline fails with an accepted result and terminal snapshot but a NULL task pointer while real provider cleanup is held. The fix passes normal continuation and reset cases, with no pointer seeding. Exact merged `92b7447` passed 2/2 twice, including after normal package TypeScript output refresh. Production files are byte-identical at frozen `435ccd2`. | | Persistence and concurrency | 19 helper/transaction cases and all 59 existing control-port cases pass on `0862062`. They cover bound result IDs, reset/newer epochs, Stop, lock waits and expiry, receipt replay, missing stubs, foreign bindings and atomic 4 KiB rollback. Existing snapshot transaction count stays one. The new tests reproduce the exact live missing-result-ID error and the inverse replay/status lock order against `435ccd2`, then pass after the correction. Replayed results still reject stale generations. | | Completion | All 32 completion-boundary tests pass on merged `92b7447`; the source is unchanged at `435ccd2`. Native result-shaped session metadata is preserved. | | Qualification fixture | A valid review report reproduced three rejected correct replies. Exact token-boundary matching fixes them. 140 focused E2E/catalog cases and Product E2E typecheck pass. Wrong tokens, marker prefixes/suffixes and wrong authors remain rejected; retained-identity checks are unchanged. | | Required checks | Full recursive typecheck and standard build passed at `435ccd2`. Both exact commands also passed on `0862062`, with its exact server build stamp; both also pass on `7a203274`, with its exact server build stamp. Fresh `0862062` Greptile passed 5/5, with zero unresolved threads. The `64c0b746` normal CI then passed all 49 jobs (48 success, one expected skip) and Greptile passed 5/5 with zero unresolved findings. The `7a203274` normal CI also passed 48 jobs with one expected skip, with fresh Greptile 5/5 and zero unresolved findings. Current `39a4343` recursive typecheck and standard build both pass with persisted exit codes and its exact build stamp; fresh Greptile is 5/5 with zero unresolved findings. Its normal CI completed successfully: 48 jobs passed and one expected job skipped. The fresh live result remains separately failed. The `0862062` normal CI had 46 passing jobs, one expected skip, one worker communication failure and its aggregate failure; GitHub states the self-hosted runner lost communication. No failed assertion or specific OOM/spot cause was established, and no blind retry was used. | | PostgreSQL supplement | One corrected all-pool pair at the preceding repair passed 300-controller/10,000-history correctness and absolute gates. SQL 76,590→75,990; transactions 9,013→8,713; UPDATE rows 11,613 each; 300 correct pointer writes each. PG CPU 31.99→30.46 s, WAL 70.36→67.87 MB, request p99 100.98→103.12 ms. This is one pair, not a repeated improvement claim or an exact `0862062` CPU measurement. Its two later changes defer premature publication and reorder existing locks; they add no transaction, stream/idle writer, pool or timer. The first collapsed-pool attempt remains invalid and retained. This is not an exact final-head CPU/WAL measurement; later fixes change ordering, identity selection and in-memory skill replacement without adding snapshot transactions, pools, stream writes or idle SQL. | | Live publication proof (`0862062`) | Baseline run `06fcc806-f186-4190-9b57-087b61d75533` accepted result `58f912dc-94d9-4b4d-bd3d-d5f689c83fd3`; the publication receipt names the same result, the task pointer has parameters and `lastRunId` matches, and the normalized native identity matches the following active run. Accounting v2 settled. | | Active recovery (`0862062`) | **FAIL.** SIGKILL/cache discard at 23:07:28.666 UTC; hold 300,000.859 ms; B ready at 308,896.486 ms; natural generation-2 ownership at 312,906 ms. At about 319 s, preparation failed `runner_harness_state_mismatch: resumed_not_suspended`. No authenticated after-receipt passed within 420 s. Exact original Runner PID 668/start/fingerprint was independently still alive; provider auto-stop was 15 minutes. No retained C follow-up qualified. Canonical 0/1 `cleanup_failure`/cleanup-failed report remains failed; subsequent independent cleanup deleted all three owned resources with exact 404s and confirmed full intended-account inventory with zero owned resources. | | Verified adoption preparation (`64c0b746`) | 91 focused cases pass, including actual preparation for four active providers and a distinct prior/current warm proof. Unproved resumed state, dead process and stale warm owner remain rejected. The earlier options-only fixture missed this gate. No spawn, backup restore or PRP-authentication bypass was added. Server typecheck passes. | | Active warm identity (`64c0b746`) | **FAIL.** B claimed generation 2 at 312,995 ms. Preparation passed, then strict PRP recovery rejected the generated attachment IDs at about 319 s. The saved and desired Runner, lease, normalized session and run IDs matched; only `turn_attachment_…`/`item_attachment_…` differed from generated `turn-`/`item-`. Canonical 0/1 cleanup-failed report remains failed; independent cleanup confirmed deletion/404s and zero owned provider resources. | | Six-field identity repair (`7a203274`) | Actual server-factory regression RED4→GREEN38. A real Rust Runner test passes healthy first run → random-ID warm handoff → held active second run → restored controller. It verifies a new live authenticated connection, exact identity, same Runner/provider OS identities, no duplicate launch/turn, and no redispatch or invented receipt for an uncertain tool. The synthetic wrong pair still fails the unchanged guard. This protocol case does not claim business completion. Server and Runner typechecks pass. | | Active recovery/completion (`7a203274`) | **PASS for the active component.** Full five-minute SIGKILL/cache-loss hold. B ready at 308,936.651 ms; natural generation 2 observed 312,979 ms; authenticated at 317,720 ms and resumed processing at 318,527 ms. Same run `1e1ccb00-0728-4935-97da-57562d81cb42`, Runner `246f73f7-a689-4177-a9c2-7ce711d63aed`, remote PID 669/start 23:48:27.470, provider lease/session and one submitted turn. Accepted result `49097529-5310-4ac3-af95-5bf9b91d19f4`, publication, finalization and accounting v2 all settled. | | Full workflow (`7a203274`) | **FAIL, preserved.** Queued late-requirement run `5e61f38d-c7d1-4fc1-bcbf-bce9d46ce4eb` failed `runner did not durably suspend before checkpoint` before outage 2. Canonical 0/1 candidate failure, cleanup passed, evidence valid. C retention was not exercised. Independent provider absence, controllers/PG cleanup, EC2 termination and root-volume absence all confirmed; closed receipt SHA-256 `676a3b15eb9937b6bf995c36fb8666c88964d509e5ceb9109fb02da0aa2bc909`. | | Retirement and stable configuration (`39a4343`) | Real Runner + Postgres RED reproduced the exact suspension failure and nine owner rejections. `turn.stop` is now lifecycle checkpoint maintenance under unchanged owner/result/generation fences. GREEN25/25 proves actual persisted suspension, exact identity and completed stop/drain/suspend; unaccepted, foreign, stale and new-work negatives remain rejected. Two default-home fingerprint REDs become GREEN9/9; explicit mode changes still invalidate. No global matching exemption or historical fingerprint rewrite. | | Fresh full Daytona warm cell (`39a4343`) | **FAIL, preserved.** Launched 00:15:29 UTC with automatic retries disabled. Active adoption passed: actual offline hold 300,000.848 ms; B ready 309,922.155 ms; natural generation 2 at 311,821 ms; authenticated 317,989 ms; resumed 330,440 ms. Same run `3e126819-e786-4b3b-b923-c975cf4f5b79`, Runner `4079138a-8d7c-44a4-8558-07a5411d1f55`, PID 668/start and provider/session identities. Accepted result `08e504af-4dec-48a8-bf05-bf0a32280559` and publication passed. Queued run `31d168c9-7ab9-4700-abb6-fafb90912237` failed the same strict suspension error before outage 2. Its row selected a new native identity, but no new third provider process or invocation was observed. Recovery-time configuration still reset only `adapterConfig`; native provider, context and workspace subtrees were unchanged. Canonical 0/1 candidate failure, evidence valid, normal cleanup passed; all campaign provider/PG/process resources independently absent. Closed campaign receipt SHA-256 `5e7d6281c13c812f3a946667f97f3c7d339b5ad86a3a819bd185b2fa73bc4a19`; EC2/SSH remain temporarily held for the concrete follow-up repair. Original 60/300/420-second limits and strict queued/second-replacement checks remain unchanged. Direct-process path; no new live prepared-task claim. | | Native retirement authority (`283e05ffaa`) | Exact `39a4343` facade regression RED3→GREEN9. The successor now uses its existing production coordinator claim and prompt renewal before strict retirement. A real Runner/Postgres case starts in the actual native-observed/unclaimed phase, claims through production code, then closes and durably suspends the predecessor in 1.657 s. All 25 real-Postgres cases pass; unclaimed/foreign writers remain rejected. Shared attachment staging stays after retirement; Stop and denied claims neither close nor stage. No new claim or SQL path was added. | | Configuration continuity (`283e05ffaa`) | Real shared/project-primary environment selection → configuration with four skill versions reproduces `reset: true`, changed category exactly `adapterConfig`, after a Postgres JSONB read. Pinned restoration had reordered the generated preference while actual runtime skills stayed identical. Replacing existing pinned rows in place preserves the original fresh name/key order and fingerprint. The assembled regression passes, as do three existing pinned-version/rename/alias cases and server typecheck. Genuine model/skill-selection changes still invalidate. No fingerprint-format change or broad compatibility exemption. | | Final production repair (`283e05ffaa`) | Full recursive typecheck/build and [normal CI](https://github.com/paperclipai/paperclip/actions/runs/41151145106) PASS: 48 jobs plus one expected skip. Fresh review 5/5, zero unresolved threads. Live baseline, recovered run and queued continuation all succeeded, each accepted/published/committed once with accounting v2 settled. Their native/provider identities and configuration fingerprint match; both recovery and continuation report no reset, no changed categories and task-session reuse. Active outage/authentication remains within original timing. | | Browser-dependent continuation (`283e05ffaa`, definition 6) | **Whole cell FAIL, preserved.** After B was killed at 00:46:38.928 and its cache discarded, C became ready in 9,151.410 ms. `openParent()` then hit a 30-second `page.goto`/DOMContentLoaded timeout before posting the final request. No fourth native run or C continuation was exercised. Cause of navigation timeout is unknown. C task had no execution lock, queued comments, handoff or recovery blocker; all three prior runs were committed/accounted. Normal cleanup passed; independent provider absence, controller/PG cleanup, EC2 termination and exact root-volume absence confirmed. Sanitized report bundle SHA-256 `2a3b841d62efb2ae30ef14f96a431ac4fa7fe3a782f1144831512f78edc65022`. | | Native API continuation qualification (`9eb9ac6f9e`, definition 7) | **FAIL, preserved.** The real post-C API request dispatched run `c71fbb18-f979-4fe4-93ce-9c4be46dbf21`, which failed `native_session_interrupted` / `native_session_recovery_failed` at 01:10:00.640 UTC. It selected the same normalized native identity, Runner and checkpoint-provider identity as the three succeeded turns. No C result was accepted; pending accounting remains explicit. Active recovery passed: actual offline hold 300,001.098 ms, B ready 308,335.019 ms, natural generation 2 observed 311,818 ms, authenticated 315,540 ms, resumed 316,612 ms. Queued continuation succeeded/accepted/published. B was then killed/cache discarded at 01:09:34.500 and C ready in 8,783.803 ms. Original classifier `transient_infrastructure`, whole-case failure and cleanup passed are unchanged; that classifier does not establish the root cause. All owned provider/PG/process resources, EC2, root volume and SSH agent independently absent. Closed receipt SHA-256 `7f21ac4a22efc07b849fa7e17866083b6127b3321af12671b4fc054ce57039a9`; sanitized bundle `71ffef14a30a553e008320a31840a6d6c0de85c1bfdb60f783fdb53a6aae812a`. Exact backend detail is unavailable in approved structured evidence. | | Current required checks (`9eb9ac6f9e`) | Recursive typecheck, standard build and exact build stamp PASS. [Normal CI](https://github.com/paperclipai/paperclip/actions/runs/41152908417) PASS: 48 jobs plus one expected skip. Fresh Greptile 5/5, latest edited summary checked, zero unresolved threads. Exact8/512/8,192 stream probes again report zero recovery SQL/transactions/rows and settled-idle SQL; seven control transactions/twelve changed rows each, with 36 separate legitimate renewals in the longest case. CI success does not supersede the failed live C gate. | The completion supplement observed secondary request/liveness/checkpoint transaction p99 increases of 23.0%/45.7%/15.6% in its single pair. Renewal p99 stayed below 0.865 s, with 900 successful renewals per arm and no lost leases. The named pool caps stayed 10+2+2. An aggregate activity sample peaked at 15 non-observer entries without backend-type classification, so no claim is made that all were client connections. No sampled lock episodes does not establish zero lock waits. Twelve pure reader/selector cases also pass with the new adjacent receipt. This is not another executed process rollback. Development reader `5cf7aead` can parse the recovery state, but its old run-detail redactor can expose receipt IDs/outcome to an authorized run reader. The first production release containing recovery readers, complete attachment-identity handling and both private-field redactions is the rollback floor. The implicit-home fix prevents new recovery-time fingerprint drift; it does not rewrite previously stored fingerprints from failed candidates. The development probe is not a published supported rollback target. Latest actual `0862062` Runner tests independently passed 8/512/8,192 text chunks with zero recovery SQL, transactions and changed rows, plus zero settled-idle SQL. All three observed seven setup/control transactions and twelve changed rows. The 8,192-chunk case separately recorded 31 legitimate lease-renewal statements, with no lost lease. These are component SQL observations, not a production CPU/WAL claim. The new adoption and identity branches affect recovery preparation only. The `64c0b746`, `7a203274` and exact `39a4343` CI independently passed all three stream sizes with the same zero-SQL/idle result; each latest 8,192-chunk case recorded 36 separate renewal statements. Exact `39a4343` Runner 2/2 passed 133 files / 2,317 tests plus two skips; its separate native group passed 150 tests plus eight skips. The first API fixture revision omitted the mandatory final screenshot; this was caught before paid launch and allocated no provider/model work. The correction captures the actual retained Chromium page with an explicit unqualified-browser label, without changing the standard report validator. Automatic approval review rejected exporting raw private API/workflow snapshots because of sensitive payloads and insufficient destination-specific authorization. Sanitized reports and bound/hash-only receipts are retained; raw snapshots were not copied through another channel. This does not change the failed grade. Repair characterization also preserves its failed attempts. An unconditional staging relocation broke three existing same-scope scripted cases; the final implementation defers staging only for an actual superseded warm scope, and all nine order/race cases pass. One baseline transform overlapped an edit and is explicitly invalid; the actual immutable `39a4343` loader then produced the three genuine REDs. The independent task-directory hypothesis produced a RED but was excluded by the live shared-workspace evidence; no directory policy change was landed. A shared-workspace fixture authorization omission was corrected separately from the four-skill semantic RED. Setup failures are retained separately from semantic outcomes. These include an old host port after the owned Postgres fixture restarted and a sandbox-denied TSX IPC pipe. The supported external-Postgres run passed, and the final exact recursive typecheck passes with normal process permissions. Seven stale declaration errors after the master merge cleared after compiling the actual merged SDK/Runner TypeScript sources. No declarations were edited by hand.
Earlier qualification measurements and failed attempts (historical, before this repair) **Historical blocked gate (before the repair below).** Final native implementation is `abed921e12d543b62f57a56435419c0dafb0bc33`; current branch head is `af9e2d157dcc349e55374ad186c9be2d6cb96bf9`. The two later commits change only qualification fixtures, their matcher and documentation; `server/` and `packages/` are unchanged. It includes master through `fd008979f5794fda3d5f54151d36acd31d0de140`. The live Daytona measurement proves five-minute active recovery, once-only completion and accounting, but **fails retained-session identity after the second replacement**. The saved checkpoint and recovery metadata exist. The task-session pointer remains empty, and ordinary admission creates new native/provider sessions. The exact skipped publication boundary remains unproven. A useful saved-file result alone does not satisfy retained-session continuation. This application-path gap blocks merge independently of CI. A separate new-master CI failure is owned by the storage workstream (fix in #18357). No storage guard, session-identity oracle, lease/grace limit or provider result was relaxed. | Gate | Observed result and exact scope | | --- | --- | | Distinct source recovery and compatible rollback | PASS at A `5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → B `5ea45f8d17e33c9a62bd79982dcf0375b07733d0` → fresh B → A. Four controller processes, natural 60-second lease expiry, absent native projections before replacements, three succeeded/committed unique results, one provider thread, unchanged Runner identity, and strict idle cleanup. 177.782 seconds. | | Immutable installed native component | PASS at installed A `5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → installed B `eaef0d56552c087aaa4fa68a8655282ffbd12667` → fresh B → A. Each controller imports its contained compiled server and DB. Three unique committed results, stable Runner/thread, no extra provider restart on same-contract rollback, strict cleanup. 106.672 seconds. This is compiled component proof, separate from full CLI-native application acceptance. | | Normal public CLI startup | PASS for B `eaef0d56552c087aaa4fa68a8655282ffbd12667`, Node 24.22.1, darwin-arm64: fresh external DB, 415 migrations, health ready, root HTML 200, company POST 201 plus GET readback. The installed tree remained unchanged after execution. | | Five-minute component outage | Actual controller absence 300,168 ms. Natural generation-2 claim at 407.196 seconds after crash, same provider attempt and Runner, original run succeeded/committed. The larger 540-second case timed out before its retained follow-ups, so it is an active-adoption pass, not a complete four-controller pass. The complete short source/installed cases above independently cover follow-up and rollback. | | Drain/resume review regression | Fixed in `226f03e736d1ee99fea1762ab32ea3a8a9241c20`. Five facade-level cases pass: overdue normal/restart waits, future timer cancellation, another facade resuming the original owner, and transient retry. Against exact prior scheduler `5ea45f8d17e33c9a62bd79982dcf0375b07733d0`, four fail and one passes. The later head changes documentation only. | | Final corrected-fixture Daytona | **PARTIAL: active recovery PASS; retained identity FAIL**, exact live source `0b6afc499f`. A was SIGKILLed and native cache discarded; actual offline hold 300,000.375 ms. B was ready at 308,326.961 ms, naturally claimed generation 2 at 312,059 ms, authenticated at 315,996 ms and resumed processing at 316,519 ms, within the original 420-second boundary. The same run, Runner PID/start, session, lease and event prefix survived; one turn submitted. Original execution succeeded and settled accounting. After completion, B was killed and its native cache discarded; C was ready in 9,077.779 ms. C executed the saved project and returned correct `queue_r`, with all four runs succeeded/accounted. However, native and true provider-session identities changed: the unchanged retained oracle fails. No whole-workflow pass is claimed. | | Prior matched-producer Daytona | FAILED at exact `abed921e12`. The matching current-source Runner passed normal admission, completed the first task and settled its accounting. During the active revision, A was SIGKILLed and its native cache discarded. Actual offline hold was 300,000.727 ms; B was ready at 308,304 ms and naturally claimed generation 2 at 309,651 ms. At about 318 seconds, the remote process check reported `process_not_running` for the original admitted Runner. No authenticated after-receipt or retained follow-up passed. The current failure is preserved; no automatic retry, lease/grace relaxation, debt edit or provider replay is used. Bounded forensic read found preserved exact-bound state with no recoverable failure, a transport-reset diagnostic and no terminal delivery; this supports external process loss without proving the exact cause. The sandbox had a five-minute auto-stop policy, conflicting with the required surviving-sandbox assumption. Owned cleanup precedes the separate corrected-fixture case. | **Live retained-session boundary.** Prior completed run `3806…` and C follow-up `8adf1df3-6744-40bf-b35d-100451f29d9f` both have database session snapshots and `nativeControllerRecovery`. Their tool fingerprint, provider configuration, workspace descriptor, runtime-context digest, driver and work mode match. There is no recorded force-fresh/workspace-selection change, and this is an ordinary task rather than a conversation-generation reset. The same issue/agent task-session row has `session_params_json: null` and `last_run_id: null` from baseline time. Both native normalized UUID and actual provider session/thread hashes differ after C. Full application admission therefore did not select the retained source. No fallback to file execution or relaxed identity comparison is accepted as proof of continuity. **Independent cleanup completed.** Full current-account provider inventory confirms no resources owned by either final fixture, with exact-ID absence checks. Owned controllers and campaign Postgres are stopped/removed. EC2 `i-01551c09af098dde5` is terminated, its delete-on-termination root volume returned `InvalidVolume.NotFound` at 22:14:08 UTC, and the temporary SSH key/agent were removed. The canonical failed report and immutable oracle input are retained. Closed qualification receipt SHA-256: `b4b5fefce287ebfdbf2cc2238efaad62d8ad485aed0e9c0628ae95dbc18a8575`. **Original measurement and grader repair remain separate.** The C reply contains the requested marker followed by a newline and the correct output. The old workflow waited for one literal space, so its normal receipt collection stalled; that original attempt was closed and its canonical `cleanup_failure`/interrupted result is preserved. Commit `af9e2d157d` repairs only this reply predicate, with 144 focused cases, E2E typecheck, discovery and diff checks passing. Ordinary string matching stays literal. The corrected predicate accepts the saved reply, but the unchanged retained-identity oracle still fails. Pure active-recovery grading of the captured measurement passes 7/7; this is not a rewritten passing campaign. The sandbox idle-window conflict was corrected only for this explicit fixture (15 minutes). Actual baseline and later sandbox metadata confirm that policy; the original active box was disposed before a post-gate policy read, so its 15-minute setting is established by the normal create/environment configuration rather than an after-read of that box. Production native lifecycle and the 60/300/420-second constants remain unchanged. The development compatibility reader A is a genuine backport commit based on `59ff89a18fe1312b5f0bb908439bbdb1237b0252`, with the required idle-goal reader fixes. It is not a previously published release. Its older executor differs from B. The rollback floor must include this reader capability. The source component cases admitted the Linux ARM64 Runner packaged at `59ff89a18fe1312b5f0bb908439bbdb1237b0252`; the installed component case admitted the `5cf7aead` package. Their Runner bytes are identical: SHA-256 `b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8`. Each admitted artifact remained unchanged throughout its case. [Final public package build](https://github.com/paperclipai/paperclip/actions/runs/41121460635) and [reader-floor build](https://github.com/paperclipai/paperclip/actions/runs/41122170506) both passed all four daemon builds, package assembly and both Darwin clean consumers. Both Linux ARM consumers failed before provider work with the same existing local Codex permission-profile error: bubblewrap attempts `activity.fifo/.git` and reports “Not a directory.” This matched baseline failure remains open; it is not a native remote continuity pass. Full post-run tree and build-stamp revalidation also passed after the compiled native case. The installed component trees are A `c29114c1bf71c5af00f761aeb9e3d84ed3d3075b9f5600dee3ef2543eb2b6ec5` and B `eda575505373a3bdc630fa4919862b359791fe3a7ef637961a25b73b38efe4b0`. **Controlled PostgreSQL qualification passed.** Baseline/candidate/candidate/baseline workloads used 300 controllers, 10,000 historical tasks, real snapshot/encrypted PRP writes and unchanged pools. PostgreSQL and client were each limited to two CPUs and 1536 MiB on an isolated eight-vCPU Linux host. | Workload | PG CPU per operation | WAL per operation | Database request p99 | | --- | ---: | ---: | ---: | | Steady | +1.18% | +1.81% | +1.91% | | Aligned burst | +3.40% | +2.56% | +3.81% | | Two logical servers | +3.11% | +2.56% | +9.68% | All arms passed absolute/correctness gates with unchanged control transaction counts, no dropped operations and no owner loss. Overlap p99 has little margin below the relative threshold. It uses two clients in one Node process, not two HTTP servers. [Closed measurements and limitations](doc/plans/2026-10-30-native-controller-qualification-evidence.json) and exact-source reproduction patches are retained. [Earlier contended-host failures](doc/plans/2026-10-30-native-controller-load-evidence.json) remain unchanged. The production DB-claim probe claimed 300 owners in 2.84 seconds and restored/committed controls and snapshots in 8.74 seconds, with zero owner loss or settled-idle SQL; its provider references were synthetic. The measured checkpoint/snapshot/renewal paths retain the same persistence behavior after integration; later accounting and business writes have separate owners. Actual Runner stream tests at `da8c7f6e1837ddd896dfe83a7524812fee813487` passed 8/512/8,192 chunks: zero recovery SQL, transactions and changed rows during text intervals; zero settled-idle SQL. Each retained seven control transactions and twelve changed records. Existing renewal SQL was measured separately. The guard/scheduler fixes add no stream path, pool or recurring idle scan. The live fixture used current-source producer SHA-256 `f0000c90cb83880191479da639201e659aa6ce0fc7d67c859faa172ec977849e`, a normal locked exact-head build with pinned Rust 1.97.1 in Bookworm. Actual network-isolated, read-only ABI execution passed (GLIBC requirement 2.34 against runtime 2.36). The immutable base image remained `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:36069788f53c51f228764fa36180575443dfb5d1f84a342d7c0daceb28dc61cc`, source `a0ef3ce7`. This was an unpublished development producer, explicitly distinct from the base image and admitted before the fresh fixture began. The earlier host build failed ABI preflight before admission. The preceding old-producer fixture completed its initial task but lacked the usage-basis marker required by current master; it remained pending/unpriced and correctly paused. It was independently cleaned without debt or budget changes. Two preceding local Linux setup attempts OOM-killed before provider allocation, and their cleanup passed. These attempts remain separate from the final recovery failure. **Failures retained and resolved separately.** The earlier `98f05fc` probe rejected a valid settled PRP v2 goal suffix; `eaef0d5` fixed that reader and the fresh follow-up passed. Later component cleanup failures were traced by a scoped exception observer to the fixture adapter rejecting directory downloads. A bounded owned-directory transfer with exact-byte/mode preflight corrected the fixture; source and installed strict-cleanup cases then passed. No production guard was relaxed. The initial local integrated control-port run hit its 30-second embedded Postgres startup hook before executing 59 cases; 18 other focused cases passed. A bounded rerun used a fresh owned external Postgres fixture, applied normal migrations before Vitest, and passed all 59 cases with the original timeouts unchanged. Its container and data were removed; the failed attempt remains retained. The additional opt-in full Product E2E Vitest run is not green: 5,313 passed, seven failed and three skipped. An actual `abed921e12` archive with the same dependencies, Node and IPC permissions reproduces all seven failures with the same reasons. Five are unchanged workflow-policy/profile-pin assertions, including Pi profile version 37 versus the test pin 35. Two database suites failed during embedded Postgres startup before their assertions; semaphore exhaustion is confirmed in the observer log. The observer passes its supported external-Postgres rerun with original timeouts and source unchanged. The controller-database client-kill/postmaster assertions remain unexecuted after `initdb` bootstrap failure; no external override is supported. The unit launcher stopped before its Node-test phase, which was then run independently and passed 181/181. The original failed full run is preserved. These results remain distinct from the passing focused checks and normal repository CI. The first local Product E2E typecheck also encountered stale compiled Runner declarations; a genuine package-local TypeScript compile refreshed them and the unchanged check passed, without a dependency install or producer rebuild. Earlier [Daytona campaign 41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900) failed at sandbox creation before any agent, with the underlying provider error unavailable. Its [sanitized report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-41108162900-2/index.html) remains a `cleanup_failure`. Both old creation names are absent in the current authenticated account, but the old CI account identity was not retained, so account equivalence is unproven. The subsequent exact `eaef0d5` source Product E2E attempt passed image/agent setup and the initial project, then held the controller offline for 300,003.7 ms. Its Mac supervisor failed owned-process inspection before B readiness; no adoption passed. Full current-company lease/label checks, provider deletion, exact-ID 404s and complete inventory confirmed cleanup of that attempt. Its local server and owned Postgres stopped. The prior AWS campaigns and database qualification VM were terminated with root volumes removed. The old admitted test Runner lacks the newer usage-basis marker introduced on master. The integrated component B turns retained unpriced, incomplete accounting obligations with `costAccountingPending: true` and no settled charge. The compatibility-reader A turn used its legacy receipt interpretation but likewise retained pending accounting and no settlement timestamp. This proves obligation preservation, not a whole-run accounting migration. Runner/receiver release ordering remains governed by the usage owner's contract. At native implementation `abed921e12d543b62f57a56435419c0dafb0bc33`, Greptile was 5/5 with zero unresolved threads and CI completed with 54 successful checks and five expected skips. At `0b6afc499f`, Greptile again passed 5/5 with zero unresolved threads, but CI ended with 51 successful checks, five skips and three failures. Server 9/12 fails a deterministic managed-skills maintenance assertion newly inherited from master `198218bf` in merge `c80e00ad`; that test is absent from the PR head, and the storage owner has the explicit-StorageService fix in #18357. Runner 2/2 passed 2,283 unit cases but its runner received a shutdown signal during the long stream probe; its cancelled step and aggregate failure are preserved. No blind retry was used. Current `af9e2d157d` checks are pending; the matcher-only change passes 144 focused cases and Product E2E typecheck. Earlier scheduling-sensitive CI failures and the interrupted unsharded local test run remain recorded in the dated plan. CI runs the repository's normal test partitions. Final verification does not claim Railway rollout, uninterrupted HTTP service or whole-webserver filesystem loss.
## Risks - The task-pointer repair passes its assembled regression. The first live repair exposed pre-acceptance ordering and the fresh review exposed a replay lock-order issue; both are fixed and pass real Postgres tests at `0862062`. The following live attempt exposed the verified-running-versus-suspended preparation gate. Its narrow fix passes 91 focused cases at `64c0b746`. The next live case exposed synthetic attachment IDs; `7a203274` fixes that field selection and passes real Runner authentication. The subsequent queued continuation exposed strict retirement maintenance and a false default-home change; `39a4343` fixes both with real Runner/Postgres and fingerprint regressions. The `39a4343` live continuation failed again despite green CI. `283e05ffaa` repairs the actual native-observed admission ordering and reproduces/fixes the JSONB skill-order fingerprint drift. Both pass targeted real-Postgres regressions and the live active plus queued paths. The definition 6 post-C browser navigation timed out before admission. Explicit definition 7 now verifies native continuation through the ordinary comment API; its strict post-C native gate failed after a real dispatch and remains historical failed evidence. The final fc68584 repair now passes its fresh strict 49-check Linux/Daytona campaign; the prior failed grades remain unchanged. Browser reconnection is handed to its own workstream with the original failed evidence. No prior failed campaign is reclassified as passing. - Incorrect checkpoint/source binding or stale ownership could duplicate external work. Missing evidence remains a hold; existing receipt and finalization owners keep uncertain effects unresolved. - Recovery must retain maintenance writes after accepted results without allowing a stale business turn to write. Real lock-wait and generation tests cover that boundary. - The current two-connection paired burst profile fails its absolute drain gate in both baseline and candidate, despite passing relative load gates and all correctness checks. A causal capacity diagnostic selected three slots; the final controlled cap-three profiles pass unchanged thresholds. Full application replacement and two-server overlap capacity remain separate. There is no new streaming writer, pool or recurring idle scan. - The remote sandbox and Runner must survive the recovery window. Provider auto-stop is a separate lifetime bound from reconnect grace; the explicit qualification fixture uses a longer idle window. - Workspace and provider backup availability remain external dependencies. This PR does not certify whole-webserver filesystem loss, uninterrupted HTTP service or a Railway rollout. ## Model Used Implementation and initial verification used OpenAI GPT-6 through Codex; the initial hosted snapshot was not exposed. The follow-up qualification used three OpenAI `gpt-6.1-sol` workers coordinated through Codex, with repository tools, code execution, CI and scoped test infrastructure. Context-window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused and exact-source application/cost checks pass; full forced CI passes; earlier local embedded-PG setup failures remain recorded) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (exact fc68584 full CI and auxiliary checks pass; forced complete test partitions) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (exact fc68584 check and edited summary pass; all six threads resolved) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 49 + doc/evals.md | 5 +- doc/execution-semantics.md | 163 +- ...026-10-30-native-controller-checkpoints.md | 583 +++ .../2026-10-30-native-controller-claim.patch | 170 + ...10-30-native-controller-load-evidence.json | 390 ++ ...2026-10-30-native-controller-overlap.patch | 76 + ...ive-controller-qualification-evidence.json | 3628 +++++++++++++++++ ...ative-controller-source-probe-receipt.json | 70 + doc/testing-native-remote-continuity.md | 131 +- maps/feature-map/recovery.md | 9 + packages/db/src/checkpoint-pool.test.ts | 16 +- packages/db/src/client.ts | 5 +- .../src/bin/fake-codex-app-server.rs | 4 +- .../durable-prp-control-plane.test.ts | 63 + .../durable-prp-control-plane.ts | 233 +- .../control-plane/runner-execution-driver.ts | 4 + packages/paperclip-runner/src/index.ts | 3 + .../src/live/runnerd-codex-transport.test.ts | 256 +- .../src/live/runnerd-codex-transport.ts | 317 +- .../src/live/runnerd-staged-handoff.test.ts | 241 ++ .../benchmark-native-controller-recovery.ts | 996 +++++ .../heartbeat-execution-configuration.test.ts | 161 +- .../remote-warm-controller.test-worker.ts | 3 + .../native-harness-backup-expiration.test.ts | 9 + .../native-session-resumption.test.ts | 166 + server/src/services/company-skills.ts | 9 +- server/src/services/heartbeat.ts | 27 +- .../heartbeat/execution-configuration.ts | 6 +- .../native-recovery-deadlines.test.ts | 102 + .../src/services/heartbeat/recovery.test.ts | 17 +- server/src/services/heartbeat/recovery.ts | 8 +- .../services/heartbeat/run-completion.test.ts | 11 +- .../src/services/heartbeat/run-completion.ts | 8 + .../heartbeat/runtime-execution.test.ts | 21 +- .../services/heartbeat/runtime-execution.ts | 21 + .../services/heartbeat/upgrade-readiness.ts | 11 +- .../native-controller-recovery-inventory.ts | 61 + ...ve-controller-recovery.integration.test.ts | 165 + .../native-controller-recovery.test.ts | 43 + .../native-controller-recovery.ts | 270 ++ .../native-harness-backup-expiration.ts | 3 + .../native-recovery-usage-delivery.test.ts | 92 + .../native-recovery-usage-delivery.ts | 47 + .../native-restart-recovery.test.ts | 12 + .../native-runtime/native-restart-recovery.ts | 33 +- .../native-runner-checkpoint.ts | 118 +- .../native-runner-execution-driver.test.ts | 35 + .../native-runner-execution-driver.ts | 14 + ...unner-restart-recovery.integration.test.ts | 296 +- .../native-session-executor.test.ts | 185 +- .../native-runtime/native-session-executor.ts | 470 ++- .../native-session-resume-projection.test.ts | 13 +- .../native-runtime/native-session-resume.ts | 16 +- .../native-staged-attachment-recovery.test.ts | 187 + .../native-staged-attachment-recovery.ts | 124 + .../native-task-session-publication.test.ts | 319 ++ .../native-task-session-publication.ts | 145 + .../native-usage-delivery.integration.test.ts | 87 +- .../native-runtime/native-usage-delivery.ts | 44 +- .../paperclip-control-plane-port.test.ts | 58 + .../paperclip-control-plane-port.ts | 130 +- .../remote-warm-session-recovery.test.ts | 43 + .../remote-warm-session-recovery.ts | 46 +- server/src/services/run-secret-redaction.ts | 2 +- server/src/vendor/paperclip-runner/index.ts | 3 + tests/runner-e2e/EVERYDAY-WORKFLOWS.md | 7 + tests/runner-e2e/README.md | 57 +- tests/runner-e2e/catalog.test.ts | 41 + tests/runner-e2e/catalog.ts | 18 +- tests/runner-e2e/controller-outage.test.ts | 2 +- tests/runner-e2e/everyday-flow.ts | 121 +- tests/runner-e2e/everyday-observations.ts | 25 +- tests/runner-e2e/everyday.test.ts | 45 + .../external-controller-database.test.ts | 52 + .../external-controller-database.ts | 47 + tests/runner-e2e/harness-env.ts | 2 + tests/runner-e2e/installed-cli.test.ts | 2 +- tests/runner-e2e/installed-cli.ts | 1 + tests/runner-e2e/instance-isolation.ts | 20 +- tests/runner-e2e/launch.ts | 13 +- .../retained-controller-continuation.test.ts | 65 + .../retained-controller-continuation.ts | 59 + tests/runner-e2e/runner.spec.ts | 7 +- 84 files changed, 11213 insertions(+), 424 deletions(-) create mode 100644 doc/plans/2026-10-30-native-controller-checkpoints.md create mode 100644 doc/plans/2026-10-30-native-controller-claim.patch create mode 100644 doc/plans/2026-10-30-native-controller-load-evidence.json create mode 100644 doc/plans/2026-10-30-native-controller-overlap.patch create mode 100644 doc/plans/2026-10-30-native-controller-qualification-evidence.json create mode 100644 doc/plans/2026-10-30-native-controller-source-probe-receipt.json create mode 100644 packages/paperclip-runner/src/live/runnerd-staged-handoff.test.ts create mode 100644 scripts/benchmark-native-controller-recovery.ts create mode 100644 server/src/services/heartbeat/native-recovery-deadlines.test.ts create mode 100644 server/src/services/native-runtime/native-controller-recovery-inventory.ts create mode 100644 server/src/services/native-runtime/native-controller-recovery.integration.test.ts create mode 100644 server/src/services/native-runtime/native-controller-recovery.test.ts create mode 100644 server/src/services/native-runtime/native-controller-recovery.ts create mode 100644 server/src/services/native-runtime/native-recovery-usage-delivery.test.ts create mode 100644 server/src/services/native-runtime/native-recovery-usage-delivery.ts create mode 100644 server/src/services/native-runtime/native-staged-attachment-recovery.test.ts create mode 100644 server/src/services/native-runtime/native-staged-attachment-recovery.ts create mode 100644 server/src/services/native-runtime/native-task-session-publication.test.ts create mode 100644 server/src/services/native-runtime/native-task-session-publication.ts create mode 100644 server/src/services/native-runtime/remote-warm-session-recovery.test.ts create mode 100644 tests/runner-e2e/external-controller-database.test.ts create mode 100644 tests/runner-e2e/external-controller-database.ts create mode 100644 tests/runner-e2e/retained-controller-continuation.test.ts create mode 100644 tests/runner-e2e/retained-controller-continuation.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index b4bec17b31c..13f3868869a 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -2649,6 +2649,26 @@ and `packages/adapter-utils/src/sandbox-callback-bridge.test.ts` for these contr ### Native runner restart recovery +For remote controller disk-loss work, use the existing control-plane snapshot +transaction and `native-controller-recovery.ts`; do not add a second snapshot +writer. The private versioned metadata is limited to 4 KiB. Its importer handles +one verified retained source while the source disk is available. Workspace and +provider-backup migrations remain with their storage owners. See +[the continuity verification guide](testing-native-remote-continuity.md) for +the direct/prepared transport and three-controller continuation fixtures. +An upgrade readiness report contains preliminary native evidence, not permission +to remove a deployment volume. Keep the checkpoint pool bounded and measure all +pools when qualifying database cost; the current size of two is a default. + +For retained Codex event tails, keep source drain/accounting and target admission +separate. `native-staged-attachment-recovery.ts` validates private source/target +header evidence; `native-recovery-usage-delivery.ts` binds the source's accounting +receiver. The transport persists its staged target before source bootstrap, then +uses the existing authenticated attachment protocol after durable drain. Do not +rotate the identity first, acknowledge before receipt persistence, or weaken the +Runner's settled-provider check. A staged target does not yet certify a resumable +session. Its eventual snapshot still uses the existing checkpoint transaction. + Custom source-mode HTTP launchers should create their listener with `createPaperclipHttpServer` from `server/src/http/server.ts`, passing the selected API origin as `apiUrl`. This registers the native Runner WebSocket transport @@ -2791,6 +2811,35 @@ and hard restarts with live and dead runners, the result-finalization race, incomplete bootstrap, repeated crashes with steering, and fail-closed process identity mismatches. +Native task-session publication is part of the terminal checkpoint transaction, +not resource cleanup. `native-task-session-publication.ts` owns its accepted-result, +owner, task-epoch and reset checks; heartbeat supplies prepared session metadata. +The goal-less regression in `native-session-resumption.test.ts` holds provider +cleanup open and then exercises a normal follow-up and an explicit session reset. +Control-port tests also cover terminal snapshots before accepted results and +result-replay contention with the status writer's coordinator-first lock order. +Remote adoption tests must invoke actual external-state preparation. Inspecting +transport options alone does not exercise the resumed-lease harness checks. +Include a healthy warm handoff before the crash so recovery exercises the +handoff's turn and item attachment IDs as well as first-run identities. +Retirement tests must exercise the preceding `turn.stop` as well as drain and +suspend after result acceptance. Use the actual native-observed, unclaimed +admission phase and the production claim before retiring the predecessor; +a synthetic legacy preparation does not cover that boundary. Verify failed +claims and Stop leave the predecessor and shared attachment staging untouched. +Configuration tests compare fresh and recovered +home-mode fingerprints while preserving rejection of real mode changes. +Include multiple skill versions read back from Postgres JSONB: version-map key +order must not change the generated preference or session fingerprint. Exercise +the real environment-selection and configuration services with a retained shared +workspace, and keep actual model or skill-version changes incompatible. +Run it against an isolated database with `PAPERCLIP_TEST_DATABASE_URL` when the +host cannot start embedded Postgres: + +```sh +pnpm exec vitest run server/src/__tests__/native-session-resumption.test.ts -t 'publishes the completed task session before provider cleanup' +``` + ## Version-based skill authoring and delivery Managed content uses `managed-skill-content.ts`: prepare and verify outside the diff --git a/doc/evals.md b/doc/evals.md index 353fe60e268..e396d676907 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -55,8 +55,9 @@ original bounds. Its source registration is not an AWS or TestOps pass. The broader YAML catalog remains planning inventory. The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/README.md) -extends the existing native restart oracle to a 125-second controller outage, -using the recorded Claude and Codex models in disposable development fixtures. +extends the existing native restart oracle to a 300-second controller outage +and another replacement after completion, using the recorded Claude and Codex +models in disposable development fixtures. It requires same-process adoption, saved files and finalization without replay; local regression checks and catalog discovery are separate from live acceptance. diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index c59bab4b3f6..9409d0d608e 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -2167,8 +2167,11 @@ digest. Earlier live qualification results keep their original profile identity. Healthy native warm turns retain their runner and provider processes. After a controller crash, continuation can recover a completed remote session from its -saved checkpoint and exact sandbox process identity. The prior controller must -be stopped, its result committed, and the successor must hold current task and +saved checkpoint and exact sandbox process identity. For an externally persisted +controller with validated recovery metadata, the prior lease must be released +and its result committed. A PID in the replacement container does not establish +whether the prior remote controller is alive. Legacy file-backed recovery retains +its stopped-process check. The successor must hold current task and coordinator authority. Recovery reconnects through the prior run's route before the authenticated run attachment changes authority. Unavailable identity probes have a shared 30-second retry budget; an identity conflict never permits replay. @@ -2647,3 +2650,159 @@ The existing prior-owner, suspended-state and backup checks still decide whether the next turn may start. Missing checkpoints and ambiguous identity remain blocked; restoration does not authorize a new provider session or change the configured lifecycle policy. + +Remote admission also handles the absence of both native controller projections. +`native-controller-recovery.ts` selects the existing bounded task/session source +and restores its encrypted PRP records before admission reads local identity. +It reconstructs the supervisor envelope from `runnerProfileJson.sessionCheckpoint` +and the private `nativeControllerRecovery` metadata. Missing, incompatible, or +mixed evidence holds the continuation; it does not authorize fresh execution. +The 32-candidate boundary and newer-progress barriers remain in force. Exhausted +history and current-run progress cannot establish fresh-bootstrap proof. + +A settled warm journal may contain an authenticated idle goal-read snapshot +after `run.terminal`. Recovery accepts only the supported PRP event v1/v2 +envelopes with the same complete attachment identity, a null goal and +`workingNow: false`. All commands must already be complete. Active goals, new +work, unknown versions and mixed bindings remain holds. This check adds no +streaming persistence or database work. + +If admission finds a live retained controller owner, it preserves the unstarted +business run and records the owner's actual deadline through the existing native +resumption path. This does not consume a provider attempt or stop the retained +sandbox. A locked run/coordinator check rejects Stop, changed ownership or any +provider progress. Startup restores even an overdue wait to the same scheduler; +ordinary resumption rechecks current permissions and task authority. No idle +scan is added. Per-turn continuation reconstructs a missing suspended-state +receipt from the admitted sandbox or prepared-task owner, including after a +previous read failed with only a PRP projection written. + +The version-one metadata contains a digest of the exact redacted stored snapshot, +configuration digests and the PRP checkpoint binding. It is bounded to 4 KiB, +excluded from public run projections, and saved by `checkpointSession` in its +existing snapshot transaction. No second snapshot writer, per-text-chunk query, +pool, or recurring scan is added. A null checkpoint binding records an incomplete +publication and prevents expiration. Snapshot and result writes recheck the +coordinator generation and lease after lock waits, including the existing limited +post-result maintenance exception. + +Goal-less native turns publish their task-session pointer with the first accepted, +successful terminal checkpoint, before provider or instruction cleanup can delay +the heartbeat completion callback. The pointer uses the native normalized UUID +and the checkpoint's provider display identity. Its prepared configuration metadata +comes from normal admission; neither streaming events nor ordinary checkpoints +add task-session writes. Goal checkpoints retain their existing publication path. + +The snapshot transaction validates the exact accepted result and current owner, +then updates only the authenticated run's existing task-session projection. Its +private `nativeTaskSessionPublication` receipt shares the 4 KiB recovery-metadata +budget. Coordinator, issue and run locks precede the ownership clock check; the +issue's `NO KEY UPDATE` lock remains compatible with result insertion's parent +`KEY SHARE`. Reset/deletion, a newer task-session epoch or a competing execution +prevents publication. Later maintenance and heartbeat completion honor the receipt, +so they cannot restore a cleared pointer or overwrite its successor. Unsupported +legacy bindings keep the existing completion path. The receipt is excluded from +public run projections and contains no session parameters or credentials. + +A terminal snapshot can arrive before result acceptance. That snapshot publishes +neither a continuation pointer nor a handled publication receipt. The existing +post-acceptance checkpoint completes publication in its normal transaction. +The first production release containing recovery readers, complete attachment +identity handling and redaction for both private recovery fields is the +disk-independent rollback floor. Earlier +development-reader probes do not establish a published rollback target. + +Active remote adoption and compatible idle warm reuse carry an existing verified +Runner proof. Preparation revalidates that proof and the exact process's liveness, +then preserves its in-place harness state. The later protocol attachment still +authenticates the durable PRP authority. A provider lease acquisition described as +`resumed` does not establish that this already-running harness was suspended. +Ordinary resumed leases retain their verified suspension and backup checks. + +Reattachment keeps the complete durable PRP binding, including its turn and item +attachment IDs. A healthy warm handoff can create new attachment IDs within the +retained Runner. Replacing that run's controller must preserve those IDs; deriving +them again from the business run ID would request an invalid authority rotation. + +Cold Codex continuation can restore a completed provider with a retained event +suffix. Native remote controllers first persist a private staged attachment under +the target run's existing checkpoint owner, then restore and drain the exact +source authority before attaching the target. The source's run, turn and item +identity stays on every old event. Receipt persistence precedes acknowledgement; +unknown semantic work holds the handoff and never reaches the target's tool +authority. The provider's settled-session attachment guard remains mandatory. + +The staged controller format is +`paperclip.runner.durable.control-plane-state.staged-attachment.v1`. It contains +the source and target attachment tuples and one bound target attachment template +inside the existing encrypted PRP authority, subject to its existing size limits. +The header stores a bounded descriptor and digest. Before command delivery, the +template is a separate changed-record part; after queueing, recovery uses the +existing command or receipt. Control checkpoints do not rewrite another copy of +instructions or tool context in the header. +It is separate from the 4 KiB run-profile recovery envelope. Startup discovers +this target obligation through bounded encrypted-header reads before a target +session snapshot exists. It verifies both frozen runs, the accepted and released +source, its original Runner artifact and the exact source header. Remote process +verification then chooses authenticated source adoption or verified suspended +source restoration. An in-memory rebound snapshot supplies normal admission; +an existing target snapshot always takes precedence over that source baseline. +The intent remains through command delivery and authenticated handoff, including +an uncertain commit response. Source and target receipt phases require their +exact physical authority; receipt replay cannot start a second Runner or attach +again under a different endpoint. After the ordinary target snapshot commits, +an in-memory notification permits intent retirement in the next existing command +write. There is no separate metadata/snapshot transaction for that retirement. + +Source accounting uses the source's immutable producer receipts and billing +classification. Opening this receiver does not reset accounting ownership or +reopen a finalized run. Equal finalized usage retains the existing audit policy; +changed, incomplete or conflicting usage remains unacknowledged for +reconciliation. The target's usage receiver never receives source events. Staging +adds recovery control work only: text streaming and settled idle operation add no +queries, snapshot transactions, pools or recurring scans. The first rollback +release supporting this continuation must also read the staged controller format; +older readers fail closed. Prepared-launch and other provider paths retain their +existing admission and recovery contracts. + +Accepted-result retirement still stops the old provider turn before draining and +suspending its Runner. The checkpoint codec classifies `turn.stop` as lifecycle +maintenance. Existing exact ownership, generation and accepted-result fences +govern this cleanup; starting provider work still requires an active owner. +A successor claims its existing native coordinator and starts renewal before +retiring a superseded warm session. An unclaimed preparation cannot authorize +the predecessor's checkpoint writes. Shared attachment staging follows verified +retirement; a failed claim or Stop cannot close that predecessor or stage files. +Recovery also preserves the implicit isolated-home default when binding admitted +configuration, so an unchanged home does not invalidate the retained session. +Explicit home-mode changes retain their normal configuration-change checks. +Restoring pinned skill versions also preserves the fresh selection's existing +name/key order. JSONB map ordering must not change the generated skill preference +and invalidate the session when the selected keys, versions and bytes are unchanged. + +For per-turn direct sandbox recovery, the executor reads the retained Runner's +suspended lifecycle through the recorded sandbox lease and validates its complete +identity and lack of pending delivery/cleanup. Only a lifecycle projection is +cached locally; provider files remain on the execution host. Prepared tasks use +their existing `read_state` contract. The normal snapshot rebinder, tool-contract +checks and current admission authority still govern the next turn. + +An active remote owner causes startup recovery to retain the obligation at the +actual lease deadline. The existing recovery dispatch timer is rearmed on drain +resume; successful reconciliation leaves no recurring recovery work. +Follow-up waits retain their original dispatch mode and owner across a drain, +including when another heartbeat service facade resumes the work. A failed +dispatch retains a bounded retry obligation rather than losing its deadline. +Upgrade diagnostics retain their existing `ready` meaning and add bounded retained-pointer +header evidence. That inventory is preliminary, reports partial coverage, and +does not certify workspace durability or volume removal. + +`importNativeControllerRecovery` is an internal one-source operation for the +migration owner while old controller files remain available. It requires the +admitted configuration digest and exact supervisor snapshot, locks the issue, +coordinator and run in order, rejects live or unsettled ownership, and publishes +only missing metadata. Repeating the same import is a no-op. Normal lifecycle +checkpoints publish the metadata without a separate migration write. Disk-free +rollback requires a server that reads this envelope and the supported settled +warm journal, including its PRP v2 idle goal suffix. Earlier local-only readers +and readers lacking that suffix handling remain outside that rollback contract. diff --git a/doc/plans/2026-10-30-native-controller-checkpoints.md b/doc/plans/2026-10-30-native-controller-checkpoints.md new file mode 100644 index 00000000000..8c4780f08aa --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-checkpoints.md @@ -0,0 +1,583 @@ +# Native-controller recovery during webserver upgrades + +Date: 2026-10-30. Status: implementation delivered; final qualification is tracked +in [PR #18343](https://github.com/paperclipai/paperclip/pull/18343). +Source inspected: `0c70cedce9a803bdc201324528abb35e423e7cd0` in the current task worktree. + +## Goal and finish line + +Make native controller state independent of the main Paperclip webserver's disk, +as one prerequisite for zero-downtime webserver deployments. A person starts a +task, the webserver is replaced, and the same remote execution continues. After +completion, another replacement can continue the retained session without the +previous webserver's files. Recovery must preserve Stop, current authorization, +accepted results, pending tool decisions, and single-owner execution. + +The implementation endpoint is a tested, reviewable PR for the native-controller +portion only. General HTTP readiness, traffic switching, request draining, +browser delivery, storage migration and Railway rollout belong to other chats. +Native shutdown integrates with the existing server lifecycle coordinator. + +Initial supported environment: native remote Runner execution, external Postgres, +a stable externally supplied instance encryption key, surviving remote sandbox +storage, and mutually compatible server/Runner releases. Qualify the existing +Daytona path, including its prepared-task transport. Preserve local/self-hosted +behavior. No new database, queue service, or shared filesystem is needed. + +Keep the existing outage contract: a 300-second controller outage, with recovery +completed within the Runner's existing 420-second reconnect grace. The remote +controller lease is currently 60 seconds. The 300-second fault-tolerance test is +not the permitted downtime of the webserver deployment. Do not shorten it or renew the Runner's +disconnected lifetime on each retry to make a test pass. This contract permits +server-dependent tools to wait; it does not promise uninterrupted HTTP service. + +## What already exists + +The earlier [Explore zero-downtime Railway deploy](codex://threads/01a1836e-8c8e-7b83-b09c-5db49a5e89de) +plan included remotely persisted native controller checkpoints. Much of that work +has since landed. Build on it instead of introducing another checkpoint store. + +| Existing capability | Owner / evidence inspected | +| --- | --- | +| Encrypted, company/task/run-bound PRP checkpoints in Postgres | [`native-runner-checkpoint.ts`](../../server/src/services/native-runtime/native-runner-checkpoint.ts), [`native_runner_checkpoints`](../../packages/db/src/schema/native_runner_checkpoints.ts); landed in `09ad9adce2` / PR #17439 | +| Opaque checkpoint codec, restore, authenticated replay cursor and persistence-before-effect barriers | [`durable-prp-control-plane.ts`](../../packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts) | +| Expiring ownership, controller generations, stale-writer fencing and isolated renewal/checkpoint pools | [`native-restart-recovery.ts`](../../server/src/services/native-runtime/native-restart-recovery.ts), [`native-execution-lease.ts`](../../server/src/services/native-runtime/native-execution-lease.ts) | +| Active remote reattachment from DB state and the admitted Runner artifact | [`native-session-executor.ts`](../../server/src/services/native-runtime/native-session-executor.ts), `executePaperclipNativeSessionWithinScope` | +| Bounded database discovery of prior task/session checkpoints | [`native-session-resume.ts`](../../server/src/services/native-runtime/native-session-resume.ts) | +| Empty-controller-directory, receipt-recovery, concurrency, retention and SQL-cost tests | [`native-runner-restart-recovery.integration.test.ts`](../../server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts), [continuity runbook](../testing-native-remote-continuity.md) | +| Read-only active-run upgrade eligibility | [`heartbeat/upgrade-readiness.ts`](../../server/src/services/heartbeat/upgrade-readiness.ts); landed in `567c043b17` / PR #18037 | + +The related [Enable uninterrupted agent upgrades](codex://threads/01a15e84-545f-7023-a854-9a642cbc76e1) +chat reports actual PAP staging replacements preserving active executions. It is +also working on broader default enablement. Those are reported results from that +chat, not a new qualification of this checkout or proof that the entire webserver +filesystem can be discarded. Reconcile that work before implementing overlapping +changes; do not duplicate its rollout or configuration work. + +## Remaining dependencies and evidence gaps + +1. **Ordinary continuation still discovers PRP state through local identity.** + `restoreWarmRunnerdController` and `restorePerTurnRunnerdController` first read + the local full state or external marker. They return when neither supplies a + matching identity. Active restart recovery instead starts from a DB run + binding. Extend that approach to ordinary continuation without skipping the + existing prior-owner, session, configuration, and suspended-state checks. +2. **Warm supervisor state has its own local persistence path.** + `persistWarmNativeCheckpoint` writes `paperclip.native-session-supervisor.v1` + with the snapshot, configuration digest, and process-configuration digest; + `loadWarmNativeCheckpoint` and warm process reuse read that file. The separate + Postgres PRP restore writes only `control-plane-state.json`. Existing saved + `runnerProfileJson.sessionCheckpoint` and admitted execution input provide + reusable data, but the complete warm reuse decision must become reconstructible. +3. **Current focused tests do not establish the whole application contract.** + The empty-directory transport fixture restores explicitly and sets the old + lease expired. It is valuable protocol evidence, but does not prove normal + admission with no markers, natural ownership expiry, HTTP routing during + overlap, or a second fresh-server continuation. Existing supervisor/Product + E2E coverage may prove some of these; reconcile it before adding new cells. +4. **Upgrade eligibility is narrower than disk independence.** + The current report checks active runs and bounded PRP headers. It does not + certify completed retained sessions or dereference workspace recovery files. + An empty active-run list must not imply that all controller history is safe + to discard. + +These are inspected dependencies and missing proof, not newly reproduced +production failures. Begin with characterization tests to determine which paths +already reconstruct sufficient state through their callers. + +## Design + +### One durable source for each kind of state + +| State | Durable owner | Local treatment | +| --- | --- | --- | +| PRP authentication, command receipts, control events and replay state | Existing encrypted `native_runner_checkpoints` | Reconstructed cache/marker | +| Run authority and finalization | Existing `native_run_finalizations`, run and issue locks | No local substitute | +| Provider/session snapshot and native reuse compatibility | Existing run profile plus a versioned native recovery envelope where fields are missing | Reconstructed supervisor projection | +| Session lookup | Existing task-session pointer and bounded prior-run selection | Never depends on directory enumeration or a marker | +| Provider process, provider history and unacknowledged outbox | Surviving remote Runner/sandbox | Never copied from an older controller cache over live state | +| Workspace descriptors, manifests, archives and provider backup blobs | Their existing workspace/backup owners, with remote durability supplied by that workstream | Logical references; independent eligibility requirement | + +Keep Runner's codec and opaque record contract in the Runner package. Native +runtime owns session selection, compatibility, and the server-side recovery +envelope. The database adapter owns encryption and transactional fencing. +Heartbeat only coordinates these contracts. Extract the affected persistence and +lookup responsibility from the large executor into a cohesive native-runtime +module rather than expanding the executor or creating a generic recovery system. + +### Reconstruct from durable identity + +Use the exact active run for restart adoption. For a subsequent turn, use the +existing task/session pointer and `native-session-resume.ts` selection policy to +locate the eligible prior run. Validate company, task, agent, normalized session, +workspace scope, admitted execution, provider configuration and current authority. +Retain the existing progress barriers: a newer incompatible or incomplete record +must not make recovery select an older convenient checkpoint. + +Load and validate the PRP checkpoint and native snapshot before materializing +local projections. The destination is computed from the replacement's local +root; absolute paths recorded by the old controller never identify durable data. +An existing local projection must agree with the selected committed state or be +replaced under the same authority. Missing, corrupt, undecodable or mismatched +state remains a specific recovery hold, without a fresh provider turn. + +Default persistence design: reuse `runnerProfileJson.sessionCheckpoint` and add +only a private, versioned native recovery envelope for missing compatibility and +checkpoint-binding metadata. Persist the snapshot and its metadata together +under the current run's fenced ownership. Do not duplicate the PRP journal or +store server-specific fields as arbitrary Runner checkpoint parts. If existing +profile metadata already supplies a field, reference it instead of copying it. +Use a schema migration only if characterization demonstrates that the existing +run/session records cannot express a required durable lookup or concurrency check. + +The private `nativeControllerRecovery` envelope is limited to 4 KiB serialized. +It contains a format version, digest of the exact redacted stored snapshot, +native configuration digest, process-configuration digest or explicit absence, +and PRP attachment/checkpoint binding. Run/company/task/agent/session identity is +validated against existing records and the snapshot, not duplicated execution +inputs. Keep credentials in the +existing encrypted authority; persist logical resource identities, not secret +values or host paths. Any newly sensitive recovery material uses the existing +encryption provider and remains excluded from public run projections. + +`PaperclipControlPlanePort.checkpointSession` already persists the snapshot before +the local supervisor callback. Add the envelope to that same transaction and +update; do not introduce a second snapshot writer. Compute stable compatibility +inputs once per admission/attachment, preserve unrelated profile fields, and +recheck durable ownership after lock waits with the established lock ordering. +Keep legitimate post-result suspension/cleanup checkpoints permitted. The generic +Runner persistence port must not learn Paperclip database fields. Remote local +files become projections; local-only consumers retain authoritative files. + +The recovery owner resolves, validates, materializes and attaches through existing +owners. Outcomes distinguish recoverable, waiting for an owner with its actual +deadline, missing/inconsistent evidence, unsupported format/capability, dependency +unavailable and proven no-prior-execution. Missing evidence is not fresh bootstrap. +Keep the 32-candidate selection window and newer-progress barriers; fetch the full +payload only for the selected source. PRP attachment identity and current business +run write authority differ during warm handoff and must remain explicit. + +Publish a resumable snapshot only after its referenced PRP boundary is durable. +Across separate commits, interruption may leave a pending obligation, never a +false ready state. Recovery reconciles that exact obligation without rerunning +the provider. Completion acceptance, workspace settlement, session reuse, and +cleanup remain separate facts. + +### Preserve ownership during replacement + +The new server may load recovery data while the old server serves, but it cannot +send commands, acknowledge events, or settle results until it owns the execution. +Reuse the existing boot identity, generation and lease-owner fences. Cross-host +takeover uses DB lease authority, never a local check of another container's PID. + +Graceful shutdown fences new local controller work, completes outstanding durable +barriers, detaches supported sessions, and releases only its own authority. A +crash uses natural expiry. The successor takes the existing fenced claim, +restores state, revalidates permissions and attaches to the exact remote process. +Lost authority detaches the stale controller without killing the surviving Runner. + +Retain wake/recovery obligations while the previous lease is live. Reconcile at +startup and the actual lease/retry deadline through the shared scheduler. Cover +a lost local release notification with durable state and the deadline. Add no +permanent checkpoint scans or independent polling timers. A future shared +cross-instance notification mechanism can accelerate this path without becoming +its source of truth. + +Exercise Stop and tool decisions arriving at either instance. Use existing durable +intents and receipts; fix a demonstrated missing native delivery path in its +owner. Never retransmit an uncertain external mutation just because its HTTP +response was lost. Current access, pause, cancellation and budget controls still +apply; restoring an admitted run is not a new grant of authority. + +### Compatibility, adoption and retention + +Use additive readers/writers. Before an ephemeral deployment is eligible, its +rollback target must understand the recovery envelope, sparse checkpoint policy +and authenticated cursor. A v2 decoder alone is insufficient. Preserve the +original Runner artifact identity; an old running binary does not gain new +capabilities merely because the webserver upgraded. + +For existing file-backed sessions, allow the current legitimate owner to publish +a validated durable snapshot at a normal checkpoint boundary. A bounded explicit +backfill may import retained supervisor state while the disk is still available, +after rechecking the same bindings and ownership. Never fabricate missing +credentials, checkpoints or remote process evidence. Unprovable sessions remain +ineligible until normal completion/retirement or explicit recovery. + +Keep local mode intact. Require the same externally supplied encryption key on +replacement; a recovery read must not generate a new key. Broad external-mode +startup validation belongs to the parent persistence workstream. + +Extend existing retention reference checks for any new envelope references. +Retain active owners, unresolved handoffs/finalization and selected continuation +sources. Cleanup must not race a new continuation claim or allow a late writer to +resurrect deleted authority. Preserve bounded startup cleanup and zero idle SQL. + +Expose an additive, versioned **native controller durability** diagnostic through +the existing internal upgrade report, with explicit unknown/ineligible reasons. +It must distinguish active adoption from retained-session continuation, and it +must never claim whole-instance volume removal readiness. Keep the existing +`ready` meaning compatible; the broader deploy coordinator composes this evidence +with workspace and other persistence checks. Derive eligibility from committed +records and supported readers, not a persisted boolean that can become stale. +Report active adoption, selected retained continuation, outstanding persistence or +handoff obligations and unsupported evidence separately. Include observation time, +bounded reason counts and coverage/truncation. Use bounded batched metadata/header +reads and pagination, and report incomplete inventory as unknown coverage. Add no +public endpoint. Establish the first envelope-capable release as the rollback +floor; an older local-only reader is not a valid empty-disk rollback target. + +## PostgreSQL load contract + +- Text streaming adds zero recovery SQL, transactions or checkpoint rows per chunk. +- Ordinary usage remains coalesced into control checkpoints. Preserve changed-part + PRP writes and exact removals; do not add a journal, transcript or archive copy. +- Snapshot metadata rides the existing snapshot transaction, including unchanged + metadata; no parallel writer, per-agent connection or new pool. +- Idle servers add no recurring SQL/network work once obligations settle. +- Scoped candidate selection fetches only the selected payload. Diagnostics use + bounded headers/metadata, never an automatic scan of historical payloads. + +Keep the existing dedicated checkpoint pool initially with its two-connection +default, lazy creation, idle release and shutdown fencing. Two is not an +architectural invariant. The invariant is bounded checkpoint traffic that meets +recovery deadlines without starving requests or renewals. Include both servers' +pools during overlap. Measure queue wait separately from SQL and lock wait. If +capacity fails acceptance, compare bounded sizes under the same workload and +choose the smallest passing size in a separately reviewable tuning change. Do not +increase the cap to conceal slow queries or lock contention. + +Run paired baseline/candidate workloads with identical fixtures and counts and +all database clients included. Record statement/transaction classes, changed rows, +persisted bytes, WAL, CPU, memory, connections, request latency, checkpoint queue/ +lock/execution latency, renewal latency/deadlines, throughput, retries and failures. +Use the existing 8/512-chunk test plus a stream beyond the retained-event window; +require exactly zero extra recovery SQL during text-only intervals. Repeat the +documented 300-controller/10,000-historical-task steady and aligned-burst workload, +alternating order and including actual session snapshot persistence. + +Gates: unchanged lifecycle sequence has no additional normal checkpoint +transactions; no new idle work, missed lease deadlines, lost owners, dropped +operations or premature ACKs; existing absolute latency/throughput gates pass. +Investigate any repeatable increase above 10% in steady CPU per completed operation, +WAL per completed operation or request p99 before handoff; report absolute values +too. Measure replacement bursts separately within the 420-second grace. Never +relax persistence-before-effect guarantees to meet a performance target. + +## Implementation slices + +| Slice | Owned surfaces and useful result | Dependency / review evidence | +| --- | --- | --- | +| 1. Complete native recovery state | Native checkpoint adapter, executor persistence/lookup, session-resume owner and focused tests. Normal active, warm and per-turn paths reconstruct with no local marker or supervisor file. | Reuse landed checkpoint work; first reproduce the exact missing path. Test current local mode and incompatible-session barriers. Add only demonstrated missing envelope fields. | +| 2. Replacement ownership and compatibility | Native lifecycle, shared recovery scheduling, shutdown wiring, retention and internal upgrade diagnostics. Competing servers hand off safely, and historical sessions have an honest eligibility path. | Slice 1; real Postgres ownership races, natural expiry, stale callbacks, Stop, migration/read compatibility and rollback tests. Keep heartbeat wiring small. | +| 3. Application and hosted qualification | Existing restart supervisor/Product E2E suites, continuity docs and acceptance evidence. Two replacements preserve execution and later continuation. | Slices 1–2; workspace durability dependency for full-home cases. Real immutable A/B releases and compatible rollback; no synthetic claims of Daytona qualification. | + +Tests and documentation travel with the behavior they verify. These are review +boundaries, not a fixed PR count. Combine tightly coupled implementation where +splitting would create an unusable intermediate state. Rebase onto the current +default branch and reconcile the active upgrades chat before writing source. + +## Acceptance matrix + +| Scenario | Required result and independently inspectable evidence | +| --- | --- | +| Fresh active remote run; graceful replacement | Same run, Runner process/start identity, provider session and active turn; one provider start, one final result | +| SIGKILL and absent controller storage | Normal startup claims after natural expiry; no manually edited lease, injected recovery marker or manual restore call | +| Output-only gap and pending tool/decision | Authenticated cursor recovers; pending input settles once; independent business receipt/effect count confirms no duplicate mutation | +| Completion while disconnected or during shutdown | Accepted result, accounting and finalization survive; incomplete persistence resumes without another provider turn | +| Completed warm and per-turn sessions | B can discover the correct source from DB; a later C continues it after B's controller storage is removed; no new provider session solely because a marker was missing | +| Warm config changes | Verified close/suspend and permitted saved-history resume; never incorrectly reuse changed credentials, tool contract or workspace | +| Old and new server overlap | B respects A's live lease; only one successor wins; delayed A cannot checkpoint, ACK, dispatch or finalize after losing ownership | +| Stop, revoked access, pause and cross-company attempts | Saved intent and current authorization prevail; no recovery bypass or delayed mutation | +| Corrupt/missing checkpoint, key or unsupported version | Precise ineligibility/hold; no silent fallback, replay, key generation or destruction of the last recoverable copy | +| Existing records and rollback | Valid local state can be imported under authority; unsupported state stays explicit; compatible prior release reads newly written records | +| Cleanup and idle | Selected checkpoints survive cleanup races; settled/idle states issue no new recurring SQL or network work; next legitimate producer reactivates recovery | +| Full application with empty homes | A → B → C uses the normal API/admission path and independently verifies the saved result; blocked until required workspace/backup dependencies are remotely durable | + +## Feedback path and verification + +First experiment: extend the existing real-Postgres remote continuation fixture +with a completed warm turn, remove both PRP marker and supervisor directories, +then submit the next turn through normal admission. Keep the remote Runner alive. +Assert the prior run/session chosen, process identity, one new turn and a useful +result. Also run the per-turn control. Do not prepopulate the desired recovery +state. This distinguishes a lookup/envelope gap from missing workspace material. + +Use the existing local Linux/container harness and scripted provider initially; +it avoids paid model calls while preserving database locks and process lifetime. +Measure setup separately from edit → run → inspected evidence. Start with a short +outage; run the real 300-second outage after the narrow behavior passes. Record +elapsed time and the exact revision in this plan when implementation begins. + +Then exercise separate compatible application revisions with distinct native roots +and shared Postgres, through normal startup/admission. Remove access to A's native +controller storage. Identify surviving non-native dependencies explicitly: this +is not evidence of whole-webserver filesystem loss. Test both SIGTERM and SIGKILL, B starting +while A still owns the run, decisions/Stop during handoff, and a second B → C +replacement. Retain an independent effects ledger and process identity evidence. +If this reaches a workspace/backup dependency, keep that failure as explicit +parent-workstream evidence; do not hide it with a copied controller directory. + +Qualify one disposable Daytona environment using the existing Product E2E path, +including completion and follow-up, plus a compatible rollback. Record server +revisions, Runner artifacts, run/session/sandbox identities, lease timestamps, +receipt counts, recovery latency and cleanup. Reuse an existing healthy campaign +only after checking its owner and exact revision. No fleet rollout is part of +this checkpoint feature. + +Required handoff checks for implementation: focused affected suites, then +`pnpm -r typecheck`, `pnpm test:run`, and `pnpm build`. If schema changes prove +necessary, use `pnpm db:generate` and the migration/snapshot checks. Update +`doc/execution-semantics.md`, `doc/testing-native-remote-continuity.md`, relevant +developer/database documentation and affected capability maps. Run +`node scripts/check-maps.mjs` after map edits. PRs use the complete repo template. +Report skipped or unavailable checks without promoting older evidence to a pass. + +## Parent-workstream dependencies and non-goals + +Native checkpoint success alone does not permit removing the webserver volume. +Workspace sync still resolves local descriptors/manifests in +[`native-workspace-sync.ts`](../../server/src/services/native-runtime/native-workspace-sync.ts). +That owner must provide durable logical recovery references and saved generations; +this plan consumes that contract rather than moving workspace archives into PRP. +Provider backup blobs and prepared resources needed after a process has suspended +must likewise resolve without the old server disk. + +Agent files/instructions, logs and accounting journals, imports, mutable packages, +global background-work claims, browser live-event delivery, HTTP readiness/drain, +and Cloud/Railway volume migration retain their own slices in the broader plan. +Do not turn native checkpoints into a second transcript: the existing sparse +checkpoint excludes display-only output, and this work does not strengthen the +run log's durability guarantee. Destroyed sandbox recovery, legacy local stdio +continuity and permanent horizontal scaling are outside this release. + +Railway still prevents overlapping deployments on a service with an attached +volume. Its separate overlap/drain controls become useful after all required +volume dependencies and concurrent-server behavior are qualified. Sources checked +2026-10-30: [volume caveats](https://docs.railway.com/volumes/reference#caveats), +[deployment teardown](https://docs.railway.com/deployments/deployment-teardown). + +## Current state and next action + +The observations below retain earlier qualification attempts and their limits. +The PR's Verification section is the current acceptance record. The final native +runtime candidate is `eaef0d56552c087aaa4fa68a8655282ffbd12667`; documentation-only +updates do not change those runtime bytes. Its CI and fresh 5/5 review passed. +The corrected source-process test has completed active adoption and a fresh +retained follow-up. Distinct compatible rollback, installed-application and live +Daytona gates remain under verification as of 2026-10-30 20:28 UTC. Preserve failed +attempts when adding later passes. The rollback floor must include the idle-goal +reader fix; an envelope-only reader is insufficient. + +Implementation is underway on `codex/native-controller-disk-independent-recovery`. +The related Cloud policy work merged as paperclip-cloud PR #1048; it retains +compatibility checks and drain fallback and does not implement this component. +Verified locally: metadata/source-selection/lock-wait checks (21 tests), readiness +and recovery scheduling (55 tests), durable discovery/import/inventory (7 tests), +and the complete native executor suite. The combined executor/selection/retention +run passed 1,063 tests and exposed two overbroad retention guards; those guards +were corrected and all 75 retention cases then passed. The snapshot fence test +also checks stale result acceptance and lease expiry during a run-row lock wait. + +Both warm and per-turn fixtures passed three separate controller processes, +deleting both native controller directories before B and again before C. They +check provider history, Runner identity, invocation counts, and the existing +suspension policy. Per-turn uses the production admission selector and snapshot +rebinder. These processes use the same candidate source, not distinct release +artifacts. Workspace and remote sandbox storage remain available. This is not +full-webserver filesystem-loss evidence. + +Both direct and prepared-task transport cases passed the 300-second outage with +natural lease expiry. The 8/512/8,192-chunk SQL test passed: each sample used seven +control transactions and twelve changed checkpoint records, with zero recovery +SQL/transactions/rows during the text interval and zero idle SQL. The long sample +included normal lease renewals. Its cluster-wide WAL values include simultaneous +test migrations and are diagnostic only, not a load acceptance measurement. + +Workspace typecheck, build, and map structural checks passed on the earlier +candidate; the later check dispositions are recorded below. The paired +300-controller/10,000-task workload now includes actual snapshot persistence. +Its initial host pilot missed throughput gates; the subsequent isolated Linux +measurements are recorded below. Database cost acceptance, distinct-release application startup, +compatible rollback, and native Daytona qualification remain open. The published Daytona qualification image was independently resolved from successful +workflow 40709784515 to `sha256:b38ed051d00fed409092d354eea9ae1b01fb232bcb7d5a187ed763382dda2d0b` +(source `8fe5d5defcdfb55a1b9049a801cfa5498629cebc`). The exact Linux Runner was +extracted from that image. The local attempt failed during the environment probe before agent creation. A read-only lookup independently returned `DaytonaAuthenticationError: Invalid credentials`. The private database and cleanup obligation remain preserved. The protected single-cell workflow [41093316372](https://github.com/paperclipai/paperclip/actions/runs/41093316372), targeting `512cc6d6dc`, was cancelled during image/artifact builds before provider allocation so acceptance can use the rebased candidate. Its protected secret is separate from the invalid local fixture. +No production deployment or complete zero-downtime acceptance is claimed. + +Failures retained in the implementation record: sandbox listener denial and host +embedded-Postgres startup limits; a corrected missing `phase` in a new fixture; +a corrected unexecuted clock query in result persistence; Linux fixture builds +first failed on shared build storage and then missing protocol source files, +resolved by a container-local build with the complete inputs. The 8,192-chunk +case exposed a test-provider cap of 4,096 (expanded in the test binary), then +exceeded the old short-stream wait limit. Its longer measurement now uses the +normal 60-second lease and renewal rather than an artificially long owner lease. +The long-stream renewal fixture initially omitted the controller PID/start identity; +renewal correctly rejected it. Complete fixture identity fixed the test. Per-turn +characterization then exposed the missing remote suspended-lifecycle projection; +that is now verified against the bound surviving sandbox. The fixture also now +uses the existing admission snapshot rebinder, which its warm-only predecessor +omitted. The first benchmark attempts lacked ES-module setup and the observer's +pg_stat_statements extension; the isolated Linux pilot needed its native Sharp +dependency. No database-load acceptance claim has been made from failed attempts. +There is no Paperclip issue bound to this local chat; this versioned source plan +is the handoff, rather than an upload to an unrelated company or issue. + +The remaining native qualification and compatibility cells require reproducible +measurements and a reviewed PR. Full-home application +acceptance remains dependent on workspace/backup durability; no controller-only +test can waive that gate. + +Qualification update: the legacy and changed-process-configuration warm cases +exposed a marker left by suspension before authority rotation. Restoring the +committed PRP checkpoint before constructing the replacement backend fixes both; +both real-process cases pass. The initial rerun used the wrong fixture executable +path and failed before provider startup; its log is retained. All 59 control-plane +port tests pass after narrowing the snapshot read to the binding and profile +columns it actually needs. The initial paired 300-controller ABBA comparison +completed every offered operation with no ownership losses, but candidate latency +and CPU increased above the investigation threshold. It is not performance +acceptance. A new paired run will measure the narrower snapshot projection. + + +Qualification update at `512cc6d6dc`: the snapshot port now combines coordinator +and run locks into one statement, with a database-clock check after both waits. +All 59 real-Postgres port tests pass. Snapshot and PRP transaction counts remain +unchanged; the first completed 300-controller baseline/candidate pair each issued +74,790 total statements and 7,800 control transactions across all pools. The +candidate adds the metadata bytes but no SQL per streamed chunk. + +The dedicated Daytona suite now fixes the active outage at 300 seconds and adds +normal UI follow-up after a second replacement following completion. Its oracle +requires a new successful native run, retained native/provider session identity, +a completed tool receipt and correct saved-project output. Focused E2E support +checks passed (56 tests, one optional external-database case skipped), plus all +137 Node support tests and E2E typecheck. The explicit external-database fixture +was separately exercised against owned Postgres earlier. The broad E2E unit +attempt lacked local listener permissions and was stopped; that failure remains +in `/tmp/native-controller-e2e-unit.log`, and broad qualification is still open. + +Database load remains **unqualified**. The narrowed-read and combined-query ABBA +runs became heavily contended as other local checks ran, including our E2E support +suite during the final pair. Both baseline and candidate lost absolute gates; +dropped operations invalidate relative comparisons. The combined-query first +pair completed all work, but its candidate p99 was 303 ms versus 131 ms and PG CPU +35.69 s versus 25.89 s. Unchanged business queries also slowed. These observations +require a quiet-host paired rerun; they are neither a performance pass nor proof +that the implementation caused all observed regression. The connection limit +remains unchanged. Summarized counters are retained in the adjacent +`2026-10-30-native-controller-load-evidence.json`; raw measurements remain under +`/tmp/native-controller-load/`. The reproducible script records failed absolute +and relative gates and marks unequal-operation comparisons ineligible. + +Remaining critical path: finish the protected Daytona attempt and cleanup; +complete full repository checks on the final rebased revision; finish the quiet +steady/burst/replacement load gates and explicit lock-wait latency measurement; +qualify distinct installed A/B/A artifacts; then resolve review on the draft +PR. Current server startup smoke and three-controller component tests are useful +layer-specific evidence, not substitutes for those remaining gates. + + +The full local `pnpm test:run` attempt was stopped after roughly 45 minutes while +still progressing through the general chat suite. Its output is retained at +`/tmp/native-controller-full-tests.log`; it is incomplete, not a pass. Broad +checks will run on hosted Linux for the rebased candidate. The broad E2E support +rerun with two workers was also stopped before rebasing; focused support checks +remain the completed evidence. A separate `server-config.test.ts` diagnostic +confirmed the original restricted run's `listen EPERM` cause. + + +### Rebased review and qualification follow-up + +Draft implementation: https://github.com/paperclipai/paperclip/pull/18343. +At rebased head `3aa90fbf0b5553ffa66a9cdc0fb153f348b56d75`, all 66 focused +persistence/discovery cases and full `pnpm -r typecheck` passed. The PR's general +server shard found 190 executor unit failures where synthetic IDs and a partial +Db double reached the new real discovery boundary. That fixture boundary is +being corrected; no production validation is relaxed. Greptile scored 3/5 and +identified missing prepared-task lifecycle reconstruction and retry after a +failed lifecycle read. Corrections and regression cases are under verification. +A separately identified retained-owner admission wait now uses the existing +same-run retry record/deadline, with unstarted-run and current-authority fences. + +Protected Daytona workflow `41094539946` failed before any test cell ran: the +image publisher exhausted disk (`No space left on device`, exit 102), including +its cleanup/log writes. It is not live recovery evidence, and repeating the +same build without a capacity fix is not justified. The default-branch workflow +has no per-run disk-capacity or upgrade-manifest override. CI infrastructure owns +that dependency. The invalid local credential attempt remains unresolved evidence. + +Local review-check attempts also encountered exhausted macOS Postgres semaphores; +only these owned test processes were stopped and moved to dedicated Docker +Postgres fixtures. One additional direct-read retry scenario timed out during +its source-controller fixture startup before a Runner appeared. Fixture import +readiness is now measured separately from the unchanged 60-second admission +limit. These failed/incomplete attempts remain distinct from subsequent passes. + +Review-fix verification: all 1,066 cases across native executor, durable +discovery, startup recovery and heartbeat runtime execution passed against owned +Docker Postgres. After the final database-clock and operator-Stop fences, all +nine durable discovery/import/wait tests passed and server typecheck passed. +The direct per-turn failed-read regression passed through normal admission with +three real controller processes: the failed read leaves the reconstructed PRP +projection, the next admission reconstructs the missing lifecycle receipt, and +the next replacement still continues the same provider history. Prepared-task +resumed and failed-read/retry cases also pass for Codex and Claude fixtures. +These are scripted-provider component results with remote workspace storage +available, not live Daytona or distinct-release acceptance. + + +### Bounded qualification campaign on October 30 + +The follow-up campaign began at 18:33 UTC with an approximately one-hour budget. +Three GPT-6.1 Sol workers own live Daytona, database load, and distinct-reader +replacement/rollback; the coordinator owns integration, artifact builds and review. +The exact native implementation under test is `a0ef3ce7493c8bb76c4766ff007e8a390118d145`. +Qualification commit `da8c7f6e1837ddd896dfe83a7524812fee813487` adds measurement and +installed-journey admission only; its native runtime, Runner, shared and database +production files are byte-identical to that implementation. + +The independent Linux database campaign uses an eight-vCPU, 32 GiB disposable +host, with PostgreSQL and its Node workload each limited to two CPUs and 1536 MiB. +It preserves the existing 10/2/2 request/renewal/checkpoint pool caps per logical +server. Steady and aligned-burst ABBA comparisons passed the existing absolute, +correctness and 10% relative regression gates with 300 controllers and 10,000 +historical tasks. Neither changes snapshot/PRP transaction counts. The separate +claim-capacity probe waits for natural lease expiry, invokes the production +claim path for 300 synthetic executions, restores real encrypted checkpoints +under empty controller directories, and measures current-owner checkpoint and +renewal work. Synthetic provider references do not establish sandbox existence +or authenticated provider reattachment. + +The closed, sanitized measurements and limits are recorded in +`2026-10-30-native-controller-qualification-evidence.json`. The earlier failed, +contended-host measurements remain in `2026-10-30-native-controller-load-evidence.json`. +Lock measurements use calibrated PostgreSQL wait-start timestamps and sampled +duration bounds. Observer traffic contributes to the measured database cost; +short unsampled waits and full HTTP latency remain outside that measurement. + +Live qualification uses protected campaign +[41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900). +Its initial catalog timeout occurred before provider allocation after Azure +reported a dependency-cache egress limit. A scoped exact-entry cache invalidation +and targeted retry forced a successful cold install. Only this campaign's +ephemeral image publisher received a larger, delete-on-termination disk; image +identity, source admission and provider authorization gates remain in force. + +Provider-free package builds pin first reader `59ff89a18fe1312b5f0bb908439bbdb1237b0252` +and candidate `a0ef3ce7493c8bb76c4766ff007e8a390118d145` in runs +[41108342354](https://github.com/paperclipai/paperclip/actions/runs/41108342354) and +[41108352138](https://github.com/paperclipai/paperclip/actions/runs/41108352138). +The source-process crash probe independently observed B waiting for A's live +lease and claiming the same execution after natural expiry, with no lease edits. +Its initial all-turns/single-provider-process assertion also crossed completion +contract revision 1 to revision 2. The existing Runner attachment policy can +retire a provider at a changed semantic contract while preserving its conversation; +phase-specific observations distinguish that boundary from same-contract rollback. +Earlier strict failures remain retained until that distinction is demonstrated. + +The final live, reader, installed-artifact, cleanup and CI outcomes are maintained +in [PR #18343](https://github.com/paperclipai/paperclip/pull/18343), with their exact +revisions and evidence limits. Campaign registration, successful artifact builds +and this plan entry do not themselves certify that a live journey passed. diff --git a/doc/plans/2026-10-30-native-controller-claim.patch b/doc/plans/2026-10-30-native-controller-claim.patch new file mode 100644 index 00000000000..0a3fe495bba --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-claim.patch @@ -0,0 +1,170 @@ +--- scripts/benchmark-native-controller-recovery.ts ++++ scripts/benchmark-native-controller-recovery.ts +@@ -14 +14 @@ +-const repoRoot = fileURLToPath(new URL("../", import.meta.url)); ++const repoRoot = path.resolve(required("BENCH_CANDIDATE_ROOT")); +@@ -241,0 +242,112 @@ ++async function claimCapacityPhase(db: any, u: string, items: any[], monitor: any) { ++ const { claimNativeRestartRecoveries } = await load(repoRoot, "server/src/services/native-runtime/native-restart-recovery.ts"); ++ const { loadNativeRunnerCheckpoint } = await load(repoRoot, "server/src/services/native-runtime/native-runner-checkpoint.ts"); ++ const replacement = createDb(u, { maxConnections: 10, applicationName: "benchmark-claim-replacement" }); ++ const successor = { bootId: randomUUID(), pid: process.pid, processStartedAt: new Date() }; ++ const result: any = { schema: "paperclip.native-controller-claim-capacity/v1", ...provenance, ++ scope: "Production DB claim function against synthetic prepared-workspace references and real persisted snapshots/PRP records. Natural lease expiry; two server database clients. No provider attachment or external sandbox existence claim.", ++ errors: [], claims: [], reads: [], samples: [], sqlByClient: {}, renewedOwnerLosses: 0, restoredCheckpoints: 0, ++ retainedOldControllerLeaseMs: 60000, remoteReconnectGraceMs: 420000, checkpointPoolPerServer: 2 }; ++ const clients = [ ++ ["old_request", db.$client], ["old_liveness", livenessDb(db).$client], ["old_checkpoint", checkpointDb(db).$client], ++ ["new_request", replacement.$client], ["new_liveness", livenessDb(replacement).$client], ["new_checkpoint", checkpointDb(replacement).$client], ++ ]; ++ for (const [name, client] of clients) { result.sqlByClient[name] = 0; client.options.debug = () => { result.sqlByClient[name]++; }; } ++ // Data is persisted by the ordinary snapshot port during the preconditioning phase. ++ const validity = await db.$client.unsafe("select count(*)::int as valid from heartbeat_runs r join native_run_finalizations f on f.run_id=r.id where r.status='running' and r.runner_profile_json->'nativeWorkspaceSync'->>'state'='prepared' and r.runner_profile_json->'sessionCheckpoint'->'identity'->>'runId'=r.id::text and exists(select 1 from native_runner_checkpoints c where c.run_id=r.id)"); ++ assert.equal(validity[0].valid, 300, "All claim inputs need real snapshots/checkpoints and prepared references"); ++ const before = pgResources(), started = performance.now(), requests = new Set>(), renewals: any[] = []; ++ const observed = observeLockWaits(monitor); ++ let tick = 0, offered = 0, completed = 0; ++ const traffic = setInterval(() => { ++ for (let j = 0; j < 20; j++) { ++ const index = tick++, item = items[index % items.length], client = index % 2 ? replacement : db; ++ const at = performance.now(); offered++; ++ const task = loadAgentRunAuthentication(client, item).then(() => { completed++; result.reads.push(performance.now() - at); }) ++ .catch((e: any) => result.errors.push({ source: "request", message: e.message })).finally(() => requests.delete(task)); ++ requests.add(task); ++ } ++ }, 100); // 200 database authentication requests/second, split evenly between servers. ++ let sampling: Promise | null = null; ++ const sample = setInterval(() => { ++ if (sampling) return; ++ sampling = monitor`select application_name,count(*)::int as connections,count(*) filter(where state='active')::int as active from pg_stat_activity where datname=current_database() and application_name!='benchmark-observer' group by application_name` ++ .then((rows: any) => result.samples.push({ elapsedMs: performance.now() - started, rows, postgresMemoryBytes: pgResources().memoryBytes })) ++ .catch((e: any) => result.errors.push({ source: "observer", message: e.message })).finally(() => { sampling = null; }); ++ }, 1000); ++ try { ++ const [lease] = await db.$client.unsafe("select greatest(0,extract(epoch from(max(lease_expires_at)-clock_timestamp()))*1000)::float8 as remaining from native_run_finalizations where lease_owner is not null"); ++ result.naturalExpiryWaitMs = Number(lease.remaining); ++ await sleep(result.naturalExpiryWaitMs + 100); ++ const [expired] = await db.$client.unsafe("select count(*)::int as expired from native_run_finalizations where lease_owner is not null and lease_expires_at<=clock_timestamp()"); ++ assert.equal(expired.expired, 300, "Never rewrite or shorten a controller lease for the probe"); ++ const claimStarted = performance.now(); ++ const dispositions = await claimNativeRestartRecoveries({ db: replacement, controller: successor, restartKind: "hard", ++ runIds: items.map(item => item.runId), limit: 300, ++ onClaim: (claim: any, scope: any) => { ++ result.claims.push({ runId: claim.runId, kind: claim.kind, elapsedMs: performance.now() - claimStarted, generation: claim.controllerGeneration }); ++ renewals.push(startNativeSessionExecutionLeaseRenewal({ db: replacement, ++ companyId: scope.companyId, issueId: scope.issueId, runId: claim.runId, leaseOwner: claim.leaseOwner, ++ attempt: claim.providerAttempt, controller: successor, remote: true, ++ onLost: async () => { result.renewedOwnerLosses++; } })); ++ }, ++ }); ++ result.claimDrainMs = performance.now() - claimStarted; ++ assert.equal(dispositions.length, 300); ++ assert(dispositions.every((claim: any) => claim.kind === "reattach_remote_runner"), "Every DB claim must independently validate its evidence"); ++ // Restore persisted records into empty replacement directories, then persist a real new control and snapshot. ++ const work = [...dispositions]; ++ await Promise.all(Array.from({ length: 4 }, async () => { ++ while (work.length) { ++ const claim: any = work.shift(), item = items.find(i => i.runId === claim.runId); ++ const checkpoint = await loadNativeRunnerCheckpoint(replacement, { companyId: item.companyId, issueId: item.issueId, runId: item.runId }); ++ assert(checkpoint); ++ const directory = root + "/replacement-states/" + item.runId; ++ checkpoint.restoreSessionDirectory(directory); ++ const header = JSON.parse(checkpoint.records.get("header")!); ++ const core = new CandidateCore({ stateDirectory: directory, identity: header.state.identity, ++ expectedRunnerVersion: checkpoint.runnerVersion, expectedRunnerDigest: checkpoint.runnerDigest }); ++ const binding = { companyId: item.companyId, issueId: item.issueId, runId: item.runId, ++ authorityRunId: item.runId, bootId: successor.bootId, generation: claim.controllerGeneration, leaseOwner: claim.leaseOwner }; ++ core.setCheckpointPersistence(CandidateWriter(replacement, binding)); ++ core.queueCommand("semantic_tool.result", { callId: "replacement-capacity", operationId: "fixture_read", result: { body: "x".repeat(4096) }, isError: false }, "replacement-control"); ++ await core.flushCheckpoint(); ++ const identity = item.snapshot.identity; ++ const port = new CandidatePort(replacement, { ...identity, sourceInstanceId: checkpoint.recoveryEvidence.runnerInstanceId, ++ controlPlaneSourceInstanceId: "replacement-controller", completionContractId: item.completionContractId, completionContractSha256: item.completionContractSha256 }, { nativeRecovery: { ++ compatibility: { configDigest: "sha256:" + "a".repeat(64), processConfigDigest: null }, owner: binding } }); ++ await port.openRun({ identity, backendKind: "codex", sourceInstanceId: checkpoint.recoveryEvidence.runnerInstanceId }); ++ await port.checkpointSession(item.snapshot); ++ result.restoredCheckpoints++; ++ await core.stop(); ++ } ++ })); ++ result.claimAndRestoreDrainMs = performance.now() - claimStarted; ++ // Observe one natural 20-second renewal from the last replacement claim. ++ await sleep(Math.max(0, 21000 - (performance.now() - claimStarted - result.claimDrainMs))); ++ const [saved] = await replacement.$client.unsafe("select count(*)::int as owners,min(controller_generation)::int as min_generation,max(controller_generation)::int as max_generation,min(extract(epoch from(lease_expires_at-clock_timestamp()))*1000)::float8 as minimum_remaining_ms from native_run_finalizations where controller_boot_id=$1 and lease_owner is not null", [successor.bootId]); ++ result.postRenewalOwners = saved; ++ assert.equal(saved.owners, 300); assert.equal(saved.min_generation, 2); assert.equal(saved.max_generation, 2); ++ assert(saved.minimum_remaining_ms > 40000, "Replacement leases should have naturally renewed from the database clock"); ++ } catch (e: any) { result.errors.push({ source: "claim-or-restore", message: e.message, stack: e.stack }); } ++ finally { ++ clearInterval(traffic); clearInterval(sample); ++ await Promise.allSettled([...requests]); await sampling; ++ await Promise.allSettled(renewals.map(r => r.stop())); ++ result.lockWait = await observed.stop(); ++ } ++ result.elapsedMs = performance.now() - started; ++ result.requestsOffered = offered; result.requestsCompleted = completed; result.requestLatency = summary(result.reads); delete result.reads; ++ result.postgresCpuUsec = pgResources().cpuUsec - before.cpuUsec; ++ const totalSql = () => Object.values(result.sqlByClient).reduce((a: number, b: any) => a + Number(b), 0); ++ const idleStart = totalSql(); await sleep(1000); result.settledIdleSql = totalSql() - idleStart; ++ result.passed = result.errors.length === 0 && result.lockWait.errors.length === 0 && result.claims.length === 300 ++ && result.restoredCheckpoints === 300 && completed === offered && result.renewedOwnerLosses === 0 ++ && result.claimAndRestoreDrainMs < 420000 && result.requestLatency.p99 < 250 && result.settledIdleSql === 0; ++ await writeFile(root + "/claim-capacity.json", JSON.stringify(result, null, 2) + "\n"); ++ console.log(JSON.stringify({ phase: "claim-capacity-complete", passed: result.passed, claims: result.claims.length, ++ claimDrainMs: result.claimDrainMs, claimAndRestoreDrainMs: result.claimAndRestoreDrainMs, ++ requests: result.requestLatency, errors: result.errors.slice(0, 3) })); ++ if (!result.passed) process.exitCode = 1; ++} ++ +@@ -429,0 +542 @@ ++ runnerProfileJson: { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: randomUUID(), remoteCwd: "/workspace", resourceDisposition: "destroy" } }, +@@ -544,0 +658,2 @@ ++ completionContractId: contractId, ++ completionContractSha256: "fixture", +@@ -905,0 +1021 @@ ++ await claimCapacityPhase(db, u, items, monitor); +@@ -917,6 +1033 @@ +- const profile = [ +- ["baseline", 300], +- ["candidate", 300], +- ["candidate", 300], +- ["baseline", 300], +- ] as const; ++ const profile = [["candidate", 300]] as const; +@@ -953,35 +1064 @@ +- const perOperation = (r: any, key: string) => +- r[key] / (r.normalCompleted + r.normalWritesCompleted + r.controls + r.admissionsCompleted); +- const average = (values: number[]) => values.reduce((a, b) => a + b, 0) / values.length; +- const comparable = +- result.every((r) => r.correctnessPassed) && +- result.every( +- (r) => +- r.controls === result[0].controls && +- r.normalCompleted === result[0].normalCompleted && +- r.normalWritesCompleted === result[0].normalWritesCompleted, +- ); +- const relative = {}; +- for (const key of ["postgresCpuUsec", "walBytes", "requestP99"]) { +- const value = (r: any) => (key === "requestP99" ? r.normalReadSummary.p99 : perOperation(r, key)); +- const baseline = average(result.filter((r) => r.variant === "baseline").map(value)); +- const candidate = average(result.filter((r) => r.variant === "candidate").map(value)); +- relative[key] = { +- baseline, +- candidate, +- increasePercent: comparable ? 100 * (candidate / baseline - 1) : null, +- passed: comparable ? candidate <= baseline * 1.1 : null, +- }; +- } +- Object.assign(report, { +- relative, +- comparable, +- passed: result.every((r) => r.passed) && Object.values(relative).every((r: any) => r.passed), +- measurementLimits: [ +- "Database request latency is not HTTP latency", +- "Pool queue includes client encoding; BEGIN response includes network time", +- "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed", +- "Peak memory is sampled once a second", +- "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide", +- ], +- }); ++ Object.assign(report, { passed: result.every(r => r.passed), scope: "Preconditioning only for the separate claim-capacity.json report" }); diff --git a/doc/plans/2026-10-30-native-controller-load-evidence.json b/doc/plans/2026-10-30-native-controller-load-evidence.json new file mode 100644 index 00000000000..1fadc46a050 --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-load-evidence.json @@ -0,0 +1,390 @@ +{ + "schema": "paperclip.native-controller-load-summary/v1", + "baselineRevision": "59e191c368", + "candidateRevision": "4f710fd7e7", + "qualification": "failed; host contention and unequal completed work invalidate the ABBA comparison", + "sourceReport": "/tmp/native-controller-load/combined-steady/report.json", + "fixture": { + "mode": "steady", + "normalReadsPerSecond": 500, + "normalWritesPerSecond": 100, + "databaseAdmissionEverySeconds": 5, + "checkpointControlEverySeconds": 5, + "controlPayloadBytes": 4096, + "phaseSeconds": 65, + "snapshotTransactionsPerControl": 1, + "baselineCheckpointPool": 2, + "baselineRenewalIntervalSeconds": 20, + "candidateRenewalIntervalSeconds": 20, + "historicalIssues": 10000 + }, + "limits": [ + "Database request latency is not HTTP latency", + "Pool queue includes client encoding; BEGIN response includes network time", + "pg_stat_statements execution includes lock waits; samples report concurrent lock waiters, not a lock latency distribution", + "Peak memory is sampled once a second", + "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide" + ], + "results": [ + { + "label": "01_baseline_300", + "controls": 3900, + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesCompleted": 6500, + "dropped": 0, + "renewalLosses": 0, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "changedRows": 7800, + "changedBytes": 17371500, + "walBytes": 63819704, + "postgresCpuUsec": 25887716, + "postgresPeakSampledMemoryBytes": 697917440, + "normalReadSummary": { + "n": 32500, + "p50": 53.82264099999884, + "p95": 83.97040299999935, + "p99": 131.4543599999888, + "max": 249.75419500000135 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 3.5057919999962905, + "p95": 14.36212499999965, + "p99": 79.32949999999255, + "max": 220.70087599998806 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.04241599999659229, + "p95": 4.89183399999456, + "p99": 25.90625, + "max": 210.7756250000093 + }, + "liveness": { + "n": 900, + "p50": 401.07175099999586, + "p95": 1987.666584000006, + "p99": 2269.4225429999933, + "max": 2296.889626000004 + }, + "checkpoint": { + "n": 3900, + "p50": 0.05258299999695737, + "p95": 0.14449999999487773, + "p99": 41.99183400000038, + "max": 188.36633400000574 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 3.6501250000001164, + "p95": 14.756584000002476, + "p99": 62.41062499998952, + "max": 240.88450000000012 + }, + "liveness": { + "n": 900, + "p50": 406.2205419999955, + "p95": 2044.2198760000028, + "p99": 2277.324250999998, + "max": 2308.3716669999994 + }, + "checkpoint": { + "n": 3900, + "p50": 3.297791999997571, + "p95": 14.160208000001148, + "p99": 79.12295900000026, + "max": 220.50066700000025 + } + }, + "drainMs": 26.716459000002942, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "startedAt": "2026-10-30T16:40:56.503Z", + "endedAt": "2026-10-30T16:42:01.534Z" + }, + { + "label": "02_candidate_300", + "controls": 3900, + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesCompleted": 6500, + "dropped": 0, + "renewalLosses": 0, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "changedRows": 7800, + "changedBytes": 17371500, + "walBytes": 66746424, + "postgresCpuUsec": 35685605, + "postgresPeakSampledMemoryBytes": 666578944, + "normalReadSummary": { + "n": 32500, + "p50": 68.02364699999453, + "p95": 127.37661100001424, + "p99": 303.326441000012, + "max": 474.2829830000119 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 5.670542000007117, + "p95": 241.97500100001344, + "p99": 479.3440830000036, + "max": 553.2187080000003 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.05162500002188608, + "p95": 37.2935839999991, + "p99": 136.02920900000026, + "max": 420.89487499999814 + }, + "liveness": { + "n": 900, + "p50": 536.7470420000027, + "p95": 3620.454543, + "p99": 4146.325211000018, + "max": 4238.34758500001 + }, + "checkpoint": { + "n": 3900, + "p50": 0.057792000006884336, + "p95": 209.87512499999139, + "p99": 443.83324999999604, + "max": 528.2805419999931 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 6.640833000012208, + "p95": 66.06445900001563, + "p99": 184.52537499999744, + "max": 449.3999579999945 + }, + "liveness": { + "n": 900, + "p50": 545.1022919999959, + "p95": 3665.246376999974, + "p99": 4172.158336000022, + "max": 4248.078752000001 + }, + "checkpoint": { + "n": 3900, + "p50": 5.484875000009197, + "p95": 241.83212600002298, + "p99": 478.8787080000038, + "max": 553.1449999999895 + } + }, + "drainMs": 76.8760000000184, + "absoluteGates": { + "requestP99": false, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": false, + "startedAt": "2026-10-30T16:42:16.915Z", + "endedAt": "2026-10-30T16:43:21.995Z" + }, + { + "label": "03_candidate_300", + "controls": 3887, + "normalOffered": 32500, + "normalCompleted": 32379, + "normalWritesCompleted": 6470, + "dropped": 164, + "renewalLosses": 0, + "checkpointTransactions": 3887, + "snapshotTransactions": 3887, + "sqlCount": 54484, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15549, + "changedRows": 7774, + "changedBytes": 17313559, + "walBytes": 66470080, + "postgresCpuUsec": 53715752, + "postgresPeakSampledMemoryBytes": 670437376, + "normalReadSummary": { + "n": 32379, + "p50": 87.29533699998865, + "p95": 407.7868620000081, + "p99": 715.8863589999964, + "max": 1413.3376139999891 + }, + "checkpointLatencySummary": { + "n": 3887, + "p50": 35.722666999994544, + "p95": 2254.7568759999995, + "p99": 4195.890377000003, + "max": 4784.590544000006 + }, + "poolQueueByPool": { + "request": { + "n": 3900, + "p50": 0.12583400000585243, + "p95": 218.51912499999162, + "p99": 498.72595799999544, + "max": 1368.8183760000102 + }, + "liveness": { + "n": 900, + "p50": 2733.2611259999976, + "p95": 7990.602337999997, + "p99": 8707.207421, + "max": 8915.786796 + }, + "checkpoint": { + "n": 3887, + "p50": 3.155832999997074, + "p95": 2207.667333999998, + "p99": 4154.626294000016, + "max": 4726.676418999996 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3900, + "p50": 27.709541999996873, + "p95": 276.61000000001513, + "p99": 625.9165419999918, + "max": 1627.7409179999959 + }, + "liveness": { + "n": 900, + "p50": 2782.7440009999846, + "p95": 8019.347628999996, + "p99": 8734.34838000001, + "max": 8964.291171000019 + }, + "checkpoint": { + "n": 3887, + "p50": 35.49954200000502, + "p95": 2254.554459999985, + "p99": 4195.5352520000015, + "max": 4784.371294000011 + } + }, + "drainMs": 4928.940960000007, + "absoluteGates": { + "requestP99": false, + "drain": false, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": false, + "startedAt": "2026-10-30T16:43:41.479Z", + "endedAt": "2026-10-30T16:44:51.414Z" + }, + { + "label": "04_baseline_300", + "controls": 2390, + "normalOffered": 32500, + "normalCompleted": 10713, + "normalWritesCompleted": 1503, + "dropped": 28305, + "renewalLosses": 0, + "checkpointTransactions": 2390, + "snapshotTransactions": 2390, + "sqlCount": 21797, + "livenessSqlCount": 4502, + "checkpointSqlCount": 9561, + "changedRows": 4780, + "changedBytes": 10643952, + "walBytes": 31871472, + "postgresCpuUsec": 55047197, + "postgresPeakSampledMemoryBytes": 643657728, + "normalReadSummary": { + "n": 10713, + "p50": 291.001007999992, + "p95": 841.957837999973, + "p99": 2601.1680469999555, + "max": 3106.3534709999803 + }, + "checkpointLatencySummary": { + "n": 2390, + "p50": 6933.93029400002, + "p95": 16553.939049999986, + "p99": 17176.082591000013, + "max": 17434.499467000016 + }, + "poolQueueByPool": { + "request": { + "n": 2392, + "p50": 0.10450000001583248, + "p95": 520.4013340000529, + "p99": 798.97454200004, + "max": 2614.81958499999 + }, + "liveness": { + "n": 900, + "p50": 5497.6995860000025, + "p95": 16836.80413299997, + "p99": 17748.473174999992, + "max": 18895.275049999997 + }, + "checkpoint": { + "n": 2390, + "p50": 6882.121711000043, + "p95": 16492.48646600003, + "p99": 17097.382091999985, + "max": 17352.715134 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 2392, + "p50": 55.64137500000652, + "p95": 638.8517920000013, + "p99": 2574.2484169999952, + "max": 2879.7436260000104 + }, + "liveness": { + "n": 900, + "p50": 5623.322503000032, + "p95": 16872.938590999984, + "p99": 18552.84471799998, + "max": 18921.25067499996 + }, + "checkpoint": { + "n": 2390, + "p50": 6933.636502000038, + "p95": 16553.763133, + "p99": 17175.909424000012, + "max": 17433.771509000042 + } + }, + "drainMs": 12508.142965000006, + "absoluteGates": { + "requestP99": false, + "drain": false, + "throughput": false, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": false, + "startedAt": "2026-10-30T16:45:39.710Z", + "endedAt": "2026-10-30T16:46:57.219Z" + } + ] +} diff --git a/doc/plans/2026-10-30-native-controller-overlap.patch b/doc/plans/2026-10-30-native-controller-overlap.patch new file mode 100644 index 00000000000..4963f5045e2 --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-overlap.patch @@ -0,0 +1,76 @@ +--- scripts/benchmark-native-controller-recovery.ts ++++ scripts/benchmark-native-controller-recovery.ts +@@ -0,0 +1 @@ ++// Temporary qualification probe derived from frozen benchmark SHA256 abaa29d50eb95013836b54b8dacc94c72e080edf24744cc5d291cc5eb1ff2818. +@@ -14 +15 @@ +-const repoRoot = fileURLToPath(new URL("../", import.meta.url)); ++const repoRoot = path.resolve(required("BENCH_CANDIDATE_ROOT")); +@@ -187 +188 @@ +- const byPool = Object.fromEntries(["benchmark-application", "paperclip-controller-lease", "paperclip-runner-checkpoint"].map(pool => { ++ const byPool = Object.fromEntries(["benchmark-application", "benchmark-replacement-application", "paperclip-controller-lease", "paperclip-runner-checkpoint"].map(pool => { +@@ -250,0 +252 @@ ++ const replacementDb = createDb(u, { maxConnections: 10, applicationName: "benchmark-replacement-application" }); +@@ -256,0 +259 @@ ++ const replacementOwner = { ...owner, bootId: randomUUID() }; +@@ -299,0 +303,3 @@ ++ ["replacement_request", replacementDb.$client], ++ ["replacement_liveness", livenessDb(replacementDb).$client], ++ ["replacement_checkpoint", checkpointDb(replacementDb).$client], +@@ -369,0 +376 @@ ++ replacementDb.$client.options.debug = db.$client.options.debug; +@@ -386,0 +394,2 @@ ++ livenessDb(replacementDb).$client.options.debug = livenessDb(db).$client.options.debug; ++ checkpointDb(replacementDb).$client.options.debug = checkpointDb(db).$client.options.debug; +@@ -389,0 +399,2 @@ ++ const controllerDb = i % 2 ? replacementDb : db; ++ const controllerOwner = i % 2 ? replacementOwner : owner; +@@ -443,3 +454,3 @@ +- controllerBootId: owner.bootId, +- controllerPid: owner.pid, +- controllerProcessStartedAt: owner.processStartedAt, ++ controllerBootId: controllerOwner.bootId, ++ controllerPid: controllerOwner.pid, ++ controllerProcessStartedAt: controllerOwner.processStartedAt, +@@ -467 +478 @@ +- const writer = (variant === "candidate" ? CandidateWriter : BaselineWriter)(db, { ++ const writer = (variant === "candidate" ? CandidateWriter : BaselineWriter)(controllerDb, { +@@ -472 +483 @@ +- bootId: owner.bootId, ++ bootId: controllerOwner.bootId, +@@ -511 +522 @@ +- db, ++ controllerDb, +@@ -531 +542 @@ +- bootId: owner.bootId, ++ bootId: controllerOwner.bootId, +@@ -544,0 +556,2 @@ ++ db: controllerDb, ++ owner: controllerOwner, +@@ -592 +605 @@ +- for (let i = 0; i < 100; i++) await loadAgentRunAuthentication(db, items[i % n]); ++ for (let i = 0; i < 100; i++) await loadAgentRunAuthentication(items[i % n].db, items[i % n]); +@@ -639 +652 @@ +- const input = { db, ...item, attempt: 1, controller: owner }; ++ const input = { ...item, attempt: 1, controller: item.owner }; +@@ -664 +677 @@ +- const rows = await loadAgentRunAuthentication(db, item); ++ const rows = await loadAgentRunAuthentication(item.db, item); +@@ -700 +713 @@ +- await db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, item.issueId)); ++ await item.db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, item.issueId)); +@@ -713 +726 @@ +- await db.transaction(async (tx: any) => { ++ await (metrics.admissionsOffered % 2 ? replacementDb : db).transaction(async (tx: any) => { +@@ -917,6 +930 @@ +- const profile = [ +- ["baseline", 300], +- ["candidate", 300], +- ["candidate", 300], +- ["baseline", 300], +- ] as const; ++ const profile = [["baseline", 300], ["candidate", 300], ["candidate", 300], ["baseline", 300]] as const; +@@ -929 +937 @@ +- "Paired baseline/candidate, including real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", ++ "Two logical servers in one client process, 150 controllers each, separate request/liveness/checkpoint pools at unchanged caps; same aggregate aligned-burst workload. Matched baseline/candidate ABBA pool-overlap component qualification; no takeover latency or provider reattachment claim. Includes real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", +@@ -930,0 +939 @@ ++ logicalServers: 2, diff --git a/doc/plans/2026-10-30-native-controller-qualification-evidence.json b/doc/plans/2026-10-30-native-controller-qualification-evidence.json new file mode 100644 index 00000000000..1b8e6f50444 --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-qualification-evidence.json @@ -0,0 +1,3628 @@ +{ + "schema": "paperclip.native-controller-qualification-evidence/v1", + "recordedAtUtc": "2026-10-30T19:08:26.343965+00:00", + "qualification": "Passed isolated PostgreSQL steady, aligned-burst and matched two-logical-server ABBA correctness, absolute and relative gates, plus production database recovery-claim/restore capacity. These are component measurements.", + "baselineRevision": "59e191c368cb3119ff3206621fefa9c5d4ad3110", + "candidateRevision": "a0ef3ce7493c8bb76c4766ff007e8a390118d145", + "measurementHarnessRevision": "da8c7f6e1837ddd896dfe83a7524812fee813487", + "measurementHarnessSha256": "abaa29d50eb95013836b54b8dacc94c72e080edf24744cc5d291cc5eb1ff2818", + "overlapProbeSha256": "fae5f15053eab0eeac15f4689035c7225488e2b8fd9cf1920e3543b71187f25e", + "sourceInputs": [ + { + "path": "server/src/services/native-runtime/native-runner-checkpoint.ts", + "candidateSha256": "d79f0e3ab44c2cea158571d9f0235aba1f9985e1e3c30ae4b347eb3e15772acf", + "baselineSha256": "d798cd0859ca363ae3be33c607f10b4755420d77d796824d1ee9570d0f7a253c" + }, + { + "path": "server/src/services/native-runtime/paperclip-control-plane-port.ts", + "candidateSha256": "b9143eea07733b67eab80e7e000cf1018202b047c755c47533a6679b67d57c51", + "baselineSha256": "a1f807c5d8fbdcb30e966ebf8a4789974f6a8a7086a0056d9351da6e9817e5f2" + }, + { + "path": "packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts", + "candidateSha256": "a8a8bed373cdd18397a762b9f67b04c789d9e5d63208f06c5ebe000faff090ac", + "baselineSha256": "a8a8bed373cdd18397a762b9f67b04c789d9e5d63208f06c5ebe000faff090ac" + }, + { + "path": "server/src/services/native-runtime/native-execution-lease.ts", + "candidateSha256": "03e70c33342ce505011ff883f8f67892e3ac5f53c58eb758316b67b4b0b2b386", + "baselineSha256": "03e70c33342ce505011ff883f8f67892e3ac5f53c58eb758316b67b4b0b2b386" + } + ], + "environment": { + "host": "One isolated disposable AWS m7i.2xlarge Linux x86_64 VM", + "hostVcpus": 8, + "hostMemoryGiB": 32, + "hostSwapMiB": 0, + "nodeVersion": "v24.22.2", + "postgresMajor": 17, + "pnpmVersion": "9.15.4", + "postgresCpuLimit": 2, + "postgresMemoryLimitMiB": 1536, + "nodeCpuLimit": 2, + "nodeMemoryLimitMiB": 1536, + "perServerPoolLimits": { + "request": 10, + "liveness": 2, + "checkpoint": 2 + }, + "observerConnectionLimit": 1, + "leaseDurationMs": 60000, + "remoteReconnectGraceMs": 420000 + }, + "fixture": { + "historicalTasks": 10000, + "controllers": 300, + "phaseDurationSeconds": 65, + "controlIntervalSeconds": 5, + "controlPayloadBytes": 4096, + "snapshotsPerControl": 1, + "mainProfileOrder": [ + "baseline", + "candidate", + "candidate", + "baseline" + ], + "baselineAndCandidateUseIdenticalDependencyBytes": true, + "normalRequestAndLeaseRenewalImplementationsAreCommonControls": true + }, + "profiles": { + "steady": { + "schema": "paperclip.normal-recovery-load/v1", + "scope": "Paired baseline/candidate, including real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", + "hardware": { + "postgresCpuLimit": 2, + "postgresMemoryLimitMiB": 1536, + "mainPool": 10, + "livenessPool": 2, + "candidateCheckpointPool": 2 + }, + "workload": { + "mode": "steady", + "normalReadsPerSecond": 500, + "normalWritesPerSecond": 100, + "databaseAdmissionEverySeconds": 5, + "checkpointControlEverySeconds": 5, + "controlPayloadBytes": 4096, + "phaseSeconds": 65, + "snapshotTransactionsPerControl": 1, + "baselineCheckpointPool": 2, + "baselineRenewalIntervalSeconds": 20, + "candidateRenewalIntervalSeconds": 20, + "historicalIssues": 10000 + }, + "passed": true, + "comparable": true, + "measurementLimits": [ + "Database request latency is not HTTP latency", + "Pool queue includes client encoding; BEGIN response includes network time", + "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed", + "Peak memory is sampled once a second", + "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide" + ], + "sourceReport": "/tmp/native-controller-load-qualification-20261030/remote/evidence/steady/report.json", + "relative": { + "postgresCpuUsec": { + "baseline": 656.5912310954723, + "candidate": 664.3645165800573, + "increasePercent": 1.183885059143397, + "passed": true + }, + "walBytes": { + "baseline": 1532.2154125789389, + "candidate": 1559.9179735744415, + "increasePercent": 1.8080069400212562, + "passed": true + }, + "requestP99": { + "baseline": 91.71616899999935, + "candidate": 93.46717100001115, + "increasePercent": 1.9091530087915087, + "passed": true + } + }, + "results": [ + { + "label": "01_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:43:31.121Z", + "endedAt": "2026-10-30T18:44:36.144Z", + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesOffered": 6500, + "normalWritesCompleted": 6500, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 67048632, + "postgresCpuUsec": 28159533, + "postgresPeakSampledMemoryBytes": 917352448, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 19.461555999994744, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 32500, + "p50": 23.630726999996114, + "p95": 104.30216500000097, + "p99": 107.72847300001013, + "max": 111.46659400001226 + }, + "normalWriteSummary": { + "n": 6500, + "p50": 28.412395999999717, + "p95": 109.15322800001013, + "p99": 112.54311900000903, + "max": 117.06165799999872 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 14.350610333334771, + "p95": 21.62447166666243, + "p99": 42.645023666656925, + "max": 92.58285566666018 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 4.567198000004282, + "p95": 6.6246809999938705, + "p99": 11.499447000001965, + "max": 34.80159999999887 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 4.406659000000218, + "p95": 8.00715200000559, + "p99": 12.106628999998065, + "max": 81.07146300000022 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.031578000001900364, + "p95": 3.2165359999999055, + "p99": 6.9602370000066, + "max": 65.06650499999887 + }, + "liveness": { + "n": 900, + "p50": 222.9600139999966, + "p95": 408.67597899999964, + "p99": 436.3592290000015, + "max": 446.41857600000367 + }, + "checkpoint": { + "n": 3900, + "p50": 0.03176600000006147, + "p95": 0.044882999995024875, + "p99": 0.18860399999903166, + "max": 17.319057000000612 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 4.37169300000096, + "p95": 8.198765000001004, + "p99": 12.514188999994076, + "max": 81.02461400000175 + }, + "liveness": { + "n": 900, + "p50": 226.25227900000027, + "p95": 412.00253099999827, + "p99": 440.3486919999996, + "max": 448.6205390000032 + }, + "checkpoint": { + "n": 3900, + "p50": 4.4467920000024606, + "p95": 6.499621000002662, + "p99": 11.388421999999991, + "max": 34.62909299999956 + } + }, + "sqlAllApplicationPools": 74790, + "sampledPeakDatabaseActivityRows": 15, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2563, + "gapsMs": { + "n": 2563, + "p50": 25.121846000001824, + "p95": 26.47124900000199, + "p99": 28.41609900000185, + "max": 86.58039899999858 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2563, + "total_exec_time": 222.99437100000054, + "rows": 0 + } + ], + "nodeCpu": { + "user": 20498304, + "system": 1761522 + }, + "nodeCpuAverageOneCorePercent": 34.23372797056988, + "eventLoopDelay": { + "p95Ms": 11.337727, + "p99Ms": 12.173311, + "maxMs": 70.975487 + }, + "peakSampledNodeRssBytes": 829227008, + "peakSampledNodeHeapBytes": 579761248, + "postgresStatementExecutionMs": 7781.774002, + "lockObserverExecutionMs": 222.99437100000054 + }, + { + "label": "02_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:44:42.986Z", + "endedAt": "2026-10-30T18:45:48.008Z", + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesOffered": 6500, + "normalWritesCompleted": 6500, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 67046184, + "postgresCpuUsec": 28544913, + "postgresPeakSampledMemoryBytes": 901398528, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 18.60484000001452, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 32500, + "p50": 52.137395999990986, + "p95": 73.59120199998142, + "p99": 80.18543700000737, + "max": 160.65012499998556 + }, + "normalWriteSummary": { + "n": 6500, + "p50": 56.79107799999474, + "p95": 79.06722500000615, + "p99": 124.89146799998707, + "max": 164.84574299998349 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 14.875081999998656, + "p95": 22.424880666658282, + "p99": 59.884479666667175, + "max": 174.85218533332227 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 4.532260000007227, + "p95": 6.7100529999879654, + "p99": 14.624643999995897, + "max": 70.33155800000532 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 4.792860999994446, + "p95": 8.380057000002125, + "p99": 14.55555699999968, + "max": 96.5399219999963 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.03197399998316541, + "p95": 3.4958060000208206, + "p99": 8.42655800000648, + "max": 86.68744899999001 + }, + "liveness": { + "n": 900, + "p50": 226.07839499998954, + "p95": 631.9207260000112, + "p99": 687.6313799999916, + "max": 700.4564780000073 + }, + "checkpoint": { + "n": 3900, + "p50": 0.03252500000235159, + "p95": 0.04554400000779424, + "p99": 1.5021660000056727, + "max": 30.859597000002395 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 4.757719000001089, + "p95": 8.515767000004416, + "p99": 14.811911999990116, + "max": 96.50402399999439 + }, + "liveness": { + "n": 900, + "p50": 228.28790900000604, + "p95": 634.5756129999936, + "p99": 689.5229259999905, + "max": 703.2544249999919 + }, + "checkpoint": { + "n": 3900, + "p50": 4.425221999990754, + "p95": 6.599863999988884, + "p99": 14.496323999992455, + "max": 70.18957200000295 + } + }, + "sqlAllApplicationPools": 74790, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2564, + "gapsMs": { + "n": 2564, + "p50": 25.082806999998866, + "p95": 26.16317899999558, + "p99": 28.29036499999347, + "max": 96.75551200000336 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2564, + "total_exec_time": 222.22588199999979, + "rows": 0 + } + ], + "nodeCpu": { + "user": 21447274, + "system": 1827953 + }, + "nodeCpuAverageOneCorePercent": 35.79597645613477, + "eventLoopDelay": { + "p95Ms": 10.903551, + "p99Ms": 12.877823, + "maxMs": 70.778879 + }, + "peakSampledNodeRssBytes": 861179904, + "peakSampledNodeHeapBytes": 612108352, + "postgresStatementExecutionMs": 7869.057230000005, + "lockObserverExecutionMs": 222.22588199999979 + }, + { + "label": "03_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:45:55.253Z", + "endedAt": "2026-10-30T18:47:00.271Z", + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesOffered": 6500, + "normalWritesCompleted": 6500, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 66835336, + "postgresCpuUsec": 28474836, + "postgresPeakSampledMemoryBytes": 890142720, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 14.864391999988584, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 32500, + "p50": 70.75364599999739, + "p95": 101.21604299999308, + "p99": 106.74890500001493, + "max": 184.49763100000564 + }, + "normalWriteSummary": { + "n": 6500, + "p50": 75.31148000000394, + "p95": 106.2007309999899, + "p99": 112.02943600001163, + "max": 189.8124850000022 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 14.700123333343072, + "p95": 22.93642300000647, + "p99": 74.96976966666989, + "max": 167.43050899999798 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 4.546646000002511, + "p95": 6.599335000006249, + "p99": 16.56861200000276, + "max": 66.67637499998091 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 4.787251000001561, + "p95": 9.346571000001859, + "p99": 17.659020999999484, + "max": 89.22574900000473 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.032805000024382025, + "p95": 3.5023479999799747, + "p99": 7.330482000019401, + "max": 22.141313000000082 + }, + "liveness": { + "n": 900, + "p50": 229.3470420000085, + "p95": 658.2537230000016, + "p99": 692.6467640000046, + "max": 699.1289490000054 + }, + "checkpoint": { + "n": 3900, + "p50": 0.034085999999661, + "p95": 0.048119000013684854, + "p99": 4.801466999982949, + "max": 22.820584999979474 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 4.7497089999960735, + "p95": 9.576310999982525, + "p99": 17.61802799999714, + "max": 89.19793199998094 + }, + "liveness": { + "n": 900, + "p50": 232.34173700000974, + "p95": 665.4881870000099, + "p99": 694.4492489999975, + "max": 701.2261050000088 + }, + "checkpoint": { + "n": 3900, + "p50": 4.438666999980342, + "p95": 6.487386000022525, + "p99": 16.5198870000022, + "max": 66.55283199998667 + } + }, + "sqlAllApplicationPools": 74790, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2560, + "gapsMs": { + "n": 2560, + "p50": 25.087980000011157, + "p95": 26.18886699998984, + "p99": 29.31080700000166, + "max": 87.73278699998627 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2560, + "total_exec_time": 228.186316, + "rows": 0 + } + ], + "nodeCpu": { + "user": 21036535, + "system": 1821778 + }, + "nodeCpuAverageOneCorePercent": 35.15690455164375, + "eventLoopDelay": { + "p95Ms": 11.026431, + "p99Ms": 13.180927, + "maxMs": 74.579967 + }, + "peakSampledNodeRssBytes": 867966976, + "peakSampledNodeHeapBytes": 609559232, + "postgresStatementExecutionMs": 7841.367742000012, + "lockObserverExecutionMs": 228.186316 + }, + { + "label": "04_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:47:06.695Z", + "endedAt": "2026-10-30T18:48:11.712Z", + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesOffered": 6500, + "normalWritesCompleted": 6500, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 64455288, + "postgresCpuUsec": 28193066, + "postgresPeakSampledMemoryBytes": 876048384, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 14.899754000012763, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 32500, + "p50": 44.329154999984894, + "p95": 69.53714800003218, + "p99": 75.70386499998858, + "max": 144.0518920000177 + }, + "normalWriteSummary": { + "n": 6500, + "p50": 49.25532100000419, + "p95": 75.08631300000707, + "p99": 122.02264599999762, + "max": 148.6692589999875 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 14.625869666662766, + "p95": 21.669017333333613, + "p99": 61.56695066665998, + "max": 185.71976100001484 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 4.66447999997763, + "p95": 6.80966199998511, + "p99": 16.252097999997204, + "max": 80.11221200000728 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 4.480548000021372, + "p95": 8.228377999999793, + "p99": 14.50026900001103, + "max": 103.96219200000633 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.032313000003341585, + "p95": 3.0398300000233576, + "p99": 7.722563000017544, + "max": 90.98564000002807 + }, + "liveness": { + "n": 900, + "p50": 225.77165700000478, + "p95": 577.020279999997, + "p99": 636.1102529999916, + "max": 644.8716770000174 + }, + "checkpoint": { + "n": 3900, + "p50": 0.033404999994672835, + "p95": 0.04764499998418614, + "p99": 3.443251999997301, + "max": 24.120947999996133 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 4.446676999999909, + "p95": 8.55906799994409, + "p99": 14.462981000018772, + "max": 103.91490999999223 + }, + "liveness": { + "n": 900, + "p50": 228.31017700000666, + "p95": 580.3901179999812, + "p99": 639.0160339999711, + "max": 646.8926890000002 + }, + "checkpoint": { + "n": 3900, + "p50": 4.548077000014018, + "p95": 6.678080999990925, + "p99": 15.536747000005562, + "max": 79.98216100002173 + } + }, + "sqlAllApplicationPools": 74790, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2555, + "gapsMs": { + "n": 2555, + "p50": 25.13867300003767, + "p95": 26.488039000018034, + "p99": 28.65206400002353, + "max": 90.49584500002675 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2555, + "total_exec_time": 226.8500740000005, + "rows": 0 + } + ], + "nodeCpu": { + "user": 21150924, + "system": 1781032 + }, + "nodeCpuAverageOneCorePercent": 35.27067846892734, + "eventLoopDelay": { + "p95Ms": 11.354111, + "p99Ms": 12.230655, + "maxMs": 70.909951 + }, + "peakSampledNodeRssBytes": 869142528, + "peakSampledNodeHeapBytes": 618486624, + "postgresStatementExecutionMs": 7824.61821599999, + "lockObserverExecutionMs": 226.8500740000005 + } + ] + }, + "burst": { + "schema": "paperclip.normal-recovery-load/v1", + "scope": "Paired baseline/candidate, including real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", + "hardware": { + "postgresCpuLimit": 2, + "postgresMemoryLimitMiB": 1536, + "mainPool": 10, + "livenessPool": 2, + "candidateCheckpointPool": 2 + }, + "workload": { + "mode": "burst", + "normalReadsPerSecond": 100, + "normalWritesPerSecond": 20, + "databaseAdmissionEverySeconds": 5, + "checkpointControlEverySeconds": 5, + "controlPayloadBytes": 4096, + "phaseSeconds": 65, + "snapshotTransactionsPerControl": 1, + "baselineCheckpointPool": 2, + "baselineRenewalIntervalSeconds": 20, + "candidateRenewalIntervalSeconds": 20, + "historicalIssues": 10000 + }, + "passed": true, + "comparable": true, + "measurementLimits": [ + "Database request latency is not HTTP latency", + "Pool queue includes client encoding; BEGIN response includes network time", + "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed", + "Peak memory is sampled once a second", + "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide" + ], + "sourceReport": "/tmp/native-controller-load-qualification-20261030/remote/evidence/burst/report.json", + "relative": { + "postgresCpuUsec": { + "baseline": 1212.0604029710578, + "candidate": 1253.290446512422, + "increasePercent": 3.4016492445672952, + "passed": true + }, + "walBytes": { + "baseline": 4214.911636643046, + "candidate": 4322.979253820542, + "increasePercent": 2.563935533973982, + "passed": true + }, + "requestP99": { + "baseline": 99.50419749999128, + "candidate": 103.2992319999903, + "increasePercent": 3.813944130346214, + "passed": true + } + }, + "results": [ + { + "label": "01_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:48:31.444Z", + "endedAt": "2026-10-30T18:49:37.203Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23487, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 50551328, + "postgresCpuUsec": 14274208, + "postgresPeakSampledMemoryBytes": 987574272, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 754.7449020000058, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 74.4481569999989, + "p95": 98.20989800000098, + "p99": 101.2616299999936, + "max": 169.48861500000203 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 76.8531340000045, + "p95": 100.53923999999824, + "p99": 103.82765900000231, + "max": 171.57443800000328 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 504.43135499999335, + "p95": 904.9732890000014, + "p99": 987.5739109999995, + "max": 1058.7055260000052 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 464.0462219999972, + "p95": 860.9761490000092, + "p99": 942.788827999997, + "max": 998.9880149999954 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 5.402625000002445, + "p95": 9.567354999999225, + "p99": 32.887894999999844, + "max": 80.82659199999762 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.01833500000066124, + "p95": 0.03541399999812711, + "p99": 1.1884140000038315, + "max": 35.858127999992575 + }, + "liveness": { + "n": 900, + "p50": 577.4616719999976, + "p95": 935.9824230000013, + "p99": 967.3116440000013, + "max": 986.1326500000141 + }, + "checkpoint": { + "n": 3900, + "p50": 458.1738780000014, + "p95": 856.6399959999981, + "p99": 936.7588849999993, + "max": 995.3760920000059 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 5.375102999998489, + "p95": 9.639744000000064, + "p99": 39.09893100000045, + "max": 90.62846400000126 + }, + "liveness": { + "n": 900, + "p50": 582.0743040000016, + "p95": 940.3565129999988, + "p99": 971.0303759999952, + "max": 991.3499540000048 + }, + "checkpoint": { + "n": 3900, + "p50": 463.99203199999465, + "p95": 860.9391810000088, + "p99": 942.7536680000012, + "max": 998.9510200000077 + } + }, + "sqlAllApplicationPools": 43590, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2555, + "gapsMs": { + "n": 2555, + "p50": 25.1262850000021, + "p95": 26.189194999999017, + "p99": 62.625012000000424, + "max": 86.79922200000146 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2555, + "total_exec_time": 216.60417100000035, + "rows": 0 + } + ], + "nodeCpu": { + "user": 18625390, + "system": 871929 + }, + "nodeCpuAverageOneCorePercent": 29.649551657605677, + "eventLoopDelay": { + "p95Ms": 10.977279, + "p99Ms": 12.943359, + "maxMs": 70.647807 + }, + "peakSampledNodeRssBytes": 809156608, + "peakSampledNodeHeapBytes": 509942048, + "postgresStatementExecutionMs": 3523.919157, + "lockObserverExecutionMs": 216.60417100000035 + }, + { + "label": "02_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:49:43.659Z", + "endedAt": "2026-10-30T18:50:49.402Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23487, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 50596552, + "postgresCpuUsec": 14685071, + "postgresPeakSampledMemoryBytes": 990670848, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 739.15390200002, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 66.31109799999103, + "p95": 99.20572600000014, + "p99": 102.6184629999916, + "max": 152.73279199999524 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 69.19254099999671, + "p95": 101.30469299998367, + "p99": 151.40168099998846, + "max": 156.07804199999373 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 545.065419999999, + "p95": 943.5618349999859, + "p99": 1053.6890769999882, + "max": 1114.0595559999929 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 503.06268100001034, + "p95": 899.0765079999983, + "p99": 1003.1476519999997, + "max": 1050.1976159999904 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 5.609040000010282, + "p95": 10.175828000006732, + "p99": 66.40549299999839, + "max": 77.22545900000841 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.018979999993462116, + "p95": 0.03818100001080893, + "p99": 1.3919429999950808, + "max": 60.06078600001638 + }, + "liveness": { + "n": 900, + "p50": 618.575686000011, + "p95": 1052.3832659999898, + "p99": 1074.6600569999719, + "max": 1082.593469000014 + }, + "checkpoint": { + "n": 3900, + "p50": 498.0827250000002, + "p95": 893.5146509999904, + "p99": 996.6987400000071, + "max": 1045.0579469999939 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 5.584780999997747, + "p95": 10.301430999999866, + "p99": 66.71016699999745, + "max": 79.74655799999891 + }, + "liveness": { + "n": 900, + "p50": 625.6686590000027, + "p95": 1054.6769740000018, + "p99": 1076.872750999988, + "max": 1084.9572240000125 + }, + "checkpoint": { + "n": 3900, + "p50": 503.0248439999996, + "p95": 899.0387380000029, + "p99": 1003.1150250000064, + "max": 1050.135020000016 + } + }, + "sqlAllApplicationPools": 43590, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2542, + "gapsMs": { + "n": 2542, + "p50": 25.161112000001594, + "p95": 26.290963999999803, + "p99": 69.67558000001009, + "max": 85.49304400000256 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2542, + "total_exec_time": 216.69468099999958, + "rows": 0 + } + ], + "nodeCpu": { + "user": 17900804, + "system": 947711 + }, + "nodeCpuAverageOneCorePercent": 28.670043958722392, + "eventLoopDelay": { + "p95Ms": 11.100159, + "p99Ms": 12.918783, + "maxMs": 71.368703 + }, + "peakSampledNodeRssBytes": 870387712, + "peakSampledNodeHeapBytes": 576073168, + "postgresStatementExecutionMs": 3486.1393980000007, + "lockObserverExecutionMs": 216.69468099999958 + }, + { + "label": "03_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:50:56.635Z", + "endedAt": "2026-10-30T18:52:02.382Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23487, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 50673560, + "postgresCpuUsec": 14674511, + "postgresPeakSampledMemoryBytes": 973410304, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 744.6864390000119, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 65.08340400000452, + "p95": 101.02233700000215, + "p99": 103.98000099998899, + "max": 108.83414200000698 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 75.7316179999907, + "p95": 103.83602499999688, + "p99": 107.78453299999819, + "max": 111.01445499999681 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 496.9523610000033, + "p95": 920.7745010000071, + "p99": 1049.187115000008, + "max": 1091.9349159999983 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 457.1240420000104, + "p95": 873.8922820000153, + "p99": 995.4707909999997, + "max": 1029.6669490000058 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 5.59725800002343, + "p95": 9.960343999991892, + "p99": 63.98731100000441, + "max": 81.85869700001786 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.019010999996680766, + "p95": 0.034908000001451, + "p99": 1.545111999992514, + "max": 38.025777999981074 + }, + "liveness": { + "n": 900, + "p50": 624.3187389999803, + "p95": 1033.8756909999938, + "p99": 1092.2892289999872, + "max": 1099.7822960000194 + }, + "checkpoint": { + "n": 3900, + "p50": 451.83978900001966, + "p95": 868.8492180000176, + "p99": 990.5199409999768, + "max": 1025.1646750000073 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 5.570027000008849, + "p95": 10.061665000015637, + "p99": 64.13720699999249, + "max": 81.82990699997754 + }, + "liveness": { + "n": 900, + "p50": 630.1176869999908, + "p95": 1036.091023999994, + "p99": 1094.1707950000127, + "max": 1101.9002290000208 + }, + "checkpoint": { + "n": 3900, + "p50": 457.09078199998476, + "p95": 873.8558170000033, + "p99": 995.4385109999857, + "max": 1029.6382080000185 + } + }, + "sqlAllApplicationPools": 43590, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2544, + "gapsMs": { + "n": 2544, + "p50": 25.161667999986093, + "p95": 26.483623999985866, + "p99": 65.51908900000853, + "max": 86.48127200000454 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 1, + "observedLowerMs": { + "n": 1, + "p50": 0.694, + "p95": 0.694, + "p99": 0.694, + "max": 0.694 + }, + "estimatedUpperMs": { + "n": 1, + "p50": 26.656120999978157, + "p95": 26.656120999978157, + "p99": 26.656120999978157, + "max": 26.656120999978157 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 2, + "observedLowerMs": { + "n": 2, + "p50": 3.427, + "p95": 3.427, + "p99": 3.427, + "max": 3.427 + }, + "estimatedUpperMs": { + "n": 2, + "p50": 81.2373620000144, + "p95": 81.2373620000144, + "p99": 81.2373620000144, + "max": 81.2373620000144 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2544, + "total_exec_time": 218.81579299999981, + "rows": 3 + } + ], + "nodeCpu": { + "user": 18483524, + "system": 895970 + }, + "nodeCpuAverageOneCorePercent": 29.475632426669986, + "eventLoopDelay": { + "p95Ms": 11.198463, + "p99Ms": 12.804095, + "maxMs": 74.317823 + }, + "peakSampledNodeRssBytes": 871841792, + "peakSampledNodeHeapBytes": 566295128, + "postgresStatementExecutionMs": 3610.8802799999994, + "lockObserverExecutionMs": 218.81579299999981 + }, + { + "label": "04_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:52:08.252Z", + "endedAt": "2026-10-30T18:53:14.009Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23487, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 48187192, + "postgresCpuUsec": 14119519, + "postgresPeakSampledMemoryBytes": 961527808, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 753.365232000011, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 70.38637200000812, + "p95": 94.09619499999098, + "p99": 97.74676499998895, + "max": 185.41928999999072 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 72.84908000001451, + "p95": 96.762837999966, + "p99": 105.79805099999066, + "max": 190.5021940000006 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 472.87740499997744, + "p95": 851.2413069999893, + "p99": 927.7114349999465, + "max": 1017.7665159999742 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 425.16228799999226, + "p95": 785.2056810000213, + "p99": 846.040737000003, + "max": 879.5945269999793 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 5.294340000022203, + "p95": 9.09644999995362, + "p99": 13.077560000005178, + "max": 79.4648959999904 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.01867399999173358, + "p95": 0.03246000001672655, + "p99": 1.0189679999893997, + "max": 38.69638399995165 + }, + "liveness": { + "n": 900, + "p50": 450.48504699999467, + "p95": 818.5160719999694, + "p99": 846.6014129999967, + "max": 852.2024820000515 + }, + "checkpoint": { + "n": 3900, + "p50": 420.15820399997756, + "p95": 780.1903570000432, + "p99": 842.1725940000033, + "max": 876.1292810000014 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 5.267995000001974, + "p95": 9.177220999990823, + "p99": 13.902025000017602, + "max": 80.02995799999917 + }, + "liveness": { + "n": 900, + "p50": 455.3887499999837, + "p95": 824.6480000000156, + "p99": 848.961249999993, + "max": 854.1895970000187 + }, + "checkpoint": { + "n": 3900, + "p50": 425.0832320000045, + "p95": 785.1541730000172, + "p99": 845.993524999998, + "max": 879.5596040000091 + } + }, + "sqlAllApplicationPools": 43590, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2567, + "gapsMs": { + "n": 2567, + "p50": 25.124033000000054, + "p95": 26.08253999997396, + "p99": 50.32741100003477, + "max": 183.68320700002369 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 1, + "observedLowerMs": { + "n": 1, + "p50": 0.264, + "p95": 0.264, + "p99": 0.264, + "max": 0.264 + }, + "estimatedUpperMs": { + "n": 1, + "p50": 25.689116000023205, + "p95": 25.689116000023205, + "p99": 25.689116000023205, + "max": 25.689116000023205 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2567, + "total_exec_time": 212.36981699999987, + "rows": 1 + } + ], + "nodeCpu": { + "user": 17769478, + "system": 879767 + }, + "nodeCpuAverageOneCorePercent": 28.36105586844359, + "eventLoopDelay": { + "p95Ms": 10.936319, + "p99Ms": 12.083199, + "maxMs": 177.602559 + }, + "peakSampledNodeRssBytes": 872235008, + "peakSampledNodeHeapBytes": 588085240, + "postgresStatementExecutionMs": 3513.288256999997, + "lockObserverExecutionMs": 212.36981699999987 + } + ] + }, + "overlap-burst": { + "schema": "paperclip.normal-recovery-load/v1", + "scope": "Two logical servers in one client process, 150 controllers each, separate request/liveness/checkpoint pools at unchanged caps; same aggregate aligned-burst workload. Component overlap capacity only; no takeover latency or provider reattachment claim. Includes real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", + "hardware": { + "logicalServers": 2, + "postgresCpuLimit": 2, + "postgresMemoryLimitMiB": 1536, + "mainPool": 10, + "livenessPool": 2, + "candidateCheckpointPool": 2 + }, + "workload": { + "mode": "burst", + "normalReadsPerSecond": 100, + "normalWritesPerSecond": 20, + "databaseAdmissionEverySeconds": 5, + "checkpointControlEverySeconds": 5, + "controlPayloadBytes": 4096, + "phaseSeconds": 65, + "snapshotTransactionsPerControl": 1, + "baselineCheckpointPool": 2, + "baselineRenewalIntervalSeconds": 20, + "candidateRenewalIntervalSeconds": 20, + "historicalIssues": 10000 + }, + "passed": true, + "comparable": false, + "measurementLimits": [ + "Two logical server clients share one two-CPU Node process; this is a bounded component pool-overlap probe", + "Each logical server retains its own 10 request, 2 liveness, and 2 checkpoint connection limits", + "Same aggregate workload as the one-server aligned burst; no actual takeover or provider reattachment latency", + "Lock wait uses sampled PostgreSQL waitstart ages; waits shorter than polling gaps may be missed", + "Observer nominally adds 40 queries/second, included in database CPU", + "Database request latency is not HTTP latency; PostgreSQL CPU/WAL are cluster-wide" + ], + "sourceReport": "/tmp/native-controller-load-qualification-20261030/remote/evidence/overlap-burst/report.json", + "relative": null, + "results": [ + { + "label": "01_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:54:12.653Z", + "endedAt": "2026-10-30T18:55:18.280Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23494, + "livenessSqlCount": 4504, + "checkpointSqlCount": 15602, + "walBytes": 58953856, + "postgresCpuUsec": 16207490, + "postgresPeakSampledMemoryBytes": 1184854016, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 622.7349329999997, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 71.72571099999186, + "p95": 99.64835199999652, + "p99": 103.41825999999128, + "max": 167.0082839999959 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 73.87743999999657, + "p95": 101.96136399998795, + "p99": 106.03463099998771, + "max": 167.02244699999574 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 510.7785539999895, + "p95": 847.7751439999993, + "p99": 955.7120819999982, + "max": 1019.8632959999959 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 464.80313800000295, + "p95": 799.8753710000019, + "p99": 903.230091999998, + "max": 973.0980429999981 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 8.885602000002109, + "p95": 18.187949999999546, + "p99": 75.0605080000023, + "max": 103.59090300000025 + }, + "poolQueueByPool": { + "request": { + "n": 1956, + "p50": 0.01652999999350868, + "p95": 0.032654000009642914, + "p99": 0.5533529999956954, + "max": 32.65080599999783 + }, + "liveness": { + "n": 450, + "p50": 679.932338999999, + "p95": 983.9729049999951, + "p99": 1005.0673800000004, + "max": 1008.1818359999961 + }, + "checkpoint": { + "n": 1950, + "p50": 455.7207139999955, + "p95": 789.6575360000061, + "p99": 879.0804459999927, + "max": 927.088631999999 + }, + "replacement_request": { + "n": 1957, + "p50": 0.016338000001269393, + "p95": 0.030588000001444016, + "p99": 0.04346399999485584, + "max": 34.07667899999069 + }, + "replacement_liveness": { + "n": 450, + "p50": 680.664604000005, + "p95": 985.0139530000015, + "p99": 1007.0503020000033, + "max": 1013.1946360000002 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 451.09980499999074, + "p95": 795.8284449999992, + "p99": 905.3483259999994, + "max": 968.1059260000038 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 1956, + "p50": 8.849019000001135, + "p95": 18.71429999999964, + "p99": 79.22411199999988, + "max": 103.53132499999992 + }, + "liveness": { + "n": 450, + "p50": 690.4811160000027, + "p95": 989.9459690000003, + "p99": 1008.2045910000015, + "max": 1011.9318149999963 + }, + "checkpoint": { + "n": 1950, + "p50": 465.5724979999941, + "p95": 798.1944409999996, + "p99": 891.0923289999992, + "max": 935.9049690000029 + }, + "replacement_request": { + "n": 1957, + "p50": 8.899836000011419, + "p95": 20.206529000002774, + "p99": 72.87767399999575, + "max": 96.72780800001055 + }, + "replacement_liveness": { + "n": 450, + "p50": 690.242895000003, + "p95": 991.1967210000003, + "p99": 1010.9223639999982, + "max": 1015.3685380000024 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 461.74945899999875, + "p95": 805.4827579999983, + "p99": 914.226066999996, + "max": 973.0625819999987 + } + }, + "sqlAllApplicationPools": 43600, + "sampledPeakDatabaseActivityRows": 26, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2532, + "gapsMs": { + "n": 2532, + "p50": 25.151506999998674, + "p95": 26.541017999999895, + "p99": 66.61091300000408, + "max": 87.0732219999918 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "benchmark-replacement-application": { + "observedEpisodes": 2, + "observedLowerMs": { + "n": 2, + "p50": 1.992, + "p95": 1.992, + "p99": 1.992, + "max": 1.992 + }, + "estimatedUpperMs": { + "n": 2, + "p50": 77.9257440000074, + "p95": 77.9257440000074, + "p99": 77.9257440000074, + "max": 77.9257440000074 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 4, + "observedLowerMs": { + "n": 4, + "p50": 2.681, + "p95": 3.012, + "p99": 3.012, + "max": 3.012 + }, + "estimatedUpperMs": { + "n": 4, + "p50": 89.75422199998866, + "p95": 90.08522199999425, + "p99": 90.08522199999425, + "max": 90.08522199999425 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 6, + "observedLowerMs": { + "n": 6, + "p50": 1.226, + "p95": 3.516, + "p99": 3.516, + "max": 3.516 + }, + "estimatedUpperMs": { + "n": 6, + "p50": 28.208660000003874, + "p95": 64.30090599999676, + "p99": 64.30090599999676, + "max": 64.30090599999676 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2532, + "total_exec_time": 228.71010199999986, + "rows": 12 + } + ], + "nodeCpu": { + "user": 18403801, + "system": 860810 + }, + "nodeCpuAverageOneCorePercent": 29.35486475249997, + "eventLoopDelay": { + "p95Ms": 11.182079, + "p99Ms": 13.328383, + "maxMs": 72.482815 + }, + "peakSampledNodeRssBytes": 835289088, + "peakSampledNodeHeapBytes": 527752512, + "postgresStatementExecutionMs": 4084.128306999997, + "lockObserverExecutionMs": 228.71010199999986 + } + ] + }, + "overlap-abba": { + "schema": "paperclip.normal-recovery-load/v1", + "scope": "Two logical servers in one client process, 150 controllers each, separate request/liveness/checkpoint pools at unchanged caps; same aggregate aligned-burst workload. Matched baseline/candidate ABBA pool-overlap component qualification; no takeover latency or provider reattachment claim. Includes real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", + "hardware": { + "logicalServers": 2, + "postgresCpuLimit": 2, + "postgresMemoryLimitMiB": 1536, + "mainPool": 10, + "livenessPool": 2, + "candidateCheckpointPool": 2 + }, + "workload": { + "mode": "burst", + "normalReadsPerSecond": 100, + "normalWritesPerSecond": 20, + "databaseAdmissionEverySeconds": 5, + "checkpointControlEverySeconds": 5, + "controlPayloadBytes": 4096, + "phaseSeconds": 65, + "snapshotTransactionsPerControl": 1, + "baselineCheckpointPool": 2, + "baselineRenewalIntervalSeconds": 20, + "candidateRenewalIntervalSeconds": 20, + "historicalIssues": 10000 + }, + "passed": true, + "comparable": true, + "relative": { + "postgresCpuUsec": { + "baseline": 1334.8748826090668, + "candidate": 1376.3816699393838, + "increasePercent": 3.109414063525584, + "passed": true + }, + "walBytes": { + "baseline": 4209.21437718774, + "candidate": 4317.105779902672, + "increasePercent": 2.5632194762913674, + "passed": true + }, + "requestP99": { + "baseline": 106.66748450000887, + "candidate": 116.98804850000306, + "increasePercent": 9.675454566469433, + "passed": true + } + }, + "measurementLimits": [ + "Database request latency is not HTTP latency", + "Pool queue includes client encoding; BEGIN response includes network time", + "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed", + "Peak memory is sampled once a second", + "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide" + ], + "results": [ + { + "label": "01_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:59:52.122Z", + "endedAt": "2026-10-30T19:00:57.905Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23496, + "livenessSqlCount": 4504, + "checkpointSqlCount": 15602, + "walBytes": 50462184, + "postgresCpuUsec": 15645449, + "postgresPeakSampledMemoryBytes": 1343967232, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 721.3357259999902, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 18.673452000002726, + "p95": 101.14544200000091, + "p99": 105.69386800000211, + "max": 219.4722890000121 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 22.73070000000007, + "p95": 103.43886099999872, + "p99": 116.90596599999844, + "max": 224.30324400000973 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 487.1431490000032, + "p95": 804.2522070000123, + "p99": 901.6204700000017, + "max": 941.495895 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 436.6414150000055, + "p95": 746.4600769999961, + "p99": 826.2692409999945, + "max": 891.541333000001 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 8.546736999996938, + "p95": 15.356733000000531, + "p99": 75.99347599998873, + "max": 98.56840399999783 + }, + "poolQueueByPool": { + "request": { + "n": 1956, + "p50": 0.016741999999794643, + "p95": 0.03154400000494206, + "p99": 0.6992969999992056, + "max": 5.09585099999822 + }, + "liveness": { + "n": 450, + "p50": 501.6249909999897, + "p95": 876.1304410000012, + "p99": 913.067286999998, + "max": 919.2936289999998 + }, + "checkpoint": { + "n": 1950, + "p50": 427.4311180000004, + "p95": 738.933901999997, + "p99": 813.4696679999979, + "max": 886.4332880000002 + }, + "replacement_request": { + "n": 1957, + "p50": 0.0168159999884665, + "p95": 0.02985600000829436, + "p99": 0.04265299999678973, + "max": 10.915440000000672 + }, + "replacement_liveness": { + "n": 450, + "p50": 503.20186999999714, + "p95": 886.2206920000026, + "p99": 913.0074840000016, + "max": 918.3099900000016 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 426.9944880000112, + "p95": 739.3993829999963, + "p99": 821.3317409999981, + "max": 878.8103770000016 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 1956, + "p50": 8.524627999999211, + "p95": 15.395046999998158, + "p99": 76.11285200000202, + "max": 97.21485499999835 + }, + "liveness": { + "n": 450, + "p50": 510.662817000004, + "p95": 886.2594030000037, + "p99": 916.8632429999998, + "max": 921.3872209999972 + }, + "checkpoint": { + "n": 1950, + "p50": 436.56327500000043, + "p95": 745.776893000002, + "p99": 822.4703049999989, + "max": 891.5069800000056 + }, + "replacement_request": { + "n": 1957, + "p50": 8.534450999999535, + "p95": 15.9994200000001, + "p99": 75.55317300000024, + "max": 98.5201530000013 + }, + "replacement_liveness": { + "n": 450, + "p50": 513.2629840000009, + "p95": 890.6694930000012, + "p99": 915.7612549999976, + "max": 920.3067900000024 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 436.6130089999933, + "p95": 746.4048050000129, + "p99": 830.9584389999982, + "max": 887.7181599999967 + } + }, + "nodeCpu": { + "user": 17806507, + "system": 879516 + }, + "nodeCpuAverageOneCorePercent": 28.40549223176189, + "eventLoopDelay": { + "p95Ms": 11.010047, + "p99Ms": 12.697599, + "maxMs": 183.107583 + }, + "sqlAllApplicationPools": 43602, + "sampledPeakDatabaseActivityRows": 29, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2551, + "gapsMs": { + "n": 2551, + "p50": 25.145184999997582, + "p95": 26.409488999997848, + "p99": 59.03993599999376, + "max": 229.8140570000105 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "benchmark-replacement-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 10, + "observedLowerMs": { + "n": 10, + "p50": 2.481, + "p95": 11.224, + "p99": 11.224, + "max": 11.224 + }, + "estimatedUpperMs": { + "n": 10, + "p50": 27.13945600000443, + "p95": 36.74406199999794, + "p99": 36.74406199999794, + "max": 36.74406199999794 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2551, + "total_exec_time": 230.25003400000065, + "rows": 10 + } + ], + "peakSampledNodeRssBytes": 815190016, + "peakSampledNodeHeapBytes": 503697472, + "postgresStatementExecutionMs": 4126.207232999998, + "lockObserverExecutionMs": 230.25003400000065 + }, + { + "label": "02_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T19:01:04.413Z", + "endedAt": "2026-10-30T19:02:10.241Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23496, + "livenessSqlCount": 4504, + "checkpointSqlCount": 15602, + "walBytes": 50562024, + "postgresCpuUsec": 16161232, + "postgresPeakSampledMemoryBytes": 1324052480, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 825.5109480000101, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 14.214422999997623, + "p95": 103.00589499998023, + "p99": 109.18438099999912, + "max": 124.26887600000191 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 16.232166000001598, + "p95": 104.91053900000406, + "p99": 111.67462099998374, + "max": 124.31452200000058 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 497.3481239999819, + "p95": 844.0066550000047, + "p99": 960.1377219999995, + "max": 998.2879429999884 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 452.44012500000827, + "p95": 791.6675080000132, + "p99": 904.2310109999962, + "max": 948.914775000012 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 8.975842999992892, + "p95": 16.999474000011105, + "p99": 80.54993900000409, + "max": 92.77781900000991 + }, + "poolQueueByPool": { + "request": { + "n": 1956, + "p50": 0.01796499999181833, + "p95": 0.03485300000465941, + "p99": 0.5488950000144541, + "max": 16.1574099999998 + }, + "liveness": { + "n": 450, + "p50": 634.4087819999986, + "p95": 971.200656999994, + "p99": 989.703067999988, + "max": 996.0718149999884 + }, + "checkpoint": { + "n": 1950, + "p50": 443.02296800000477, + "p95": 778.6681580000004, + "p99": 884.4173180000071, + "max": 918.9185089999955 + }, + "replacement_request": { + "n": 1957, + "p50": 0.017980999997234903, + "p95": 0.03247500001452863, + "p99": 0.04606999998213723, + "max": 39.24437300002319 + }, + "replacement_liveness": { + "n": 450, + "p50": 634.364161000005, + "p95": 969.9561950000061, + "p99": 989.6641479999962, + "max": 995.097293999992 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 440.39501599999494, + "p95": 786.2747190000082, + "p99": 904.2403229999909, + "max": 943.5718220000126 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 1956, + "p50": 8.969991000019945, + "p95": 17.73811299999943, + "p99": 80.50918799999636, + "max": 92.72936200001277 + }, + "liveness": { + "n": 450, + "p50": 645.302377, + "p95": 976.2641570000123, + "p99": 993.3791540000093, + "max": 1001.4772759999905 + }, + "checkpoint": { + "n": 1950, + "p50": 453.5094059999974, + "p95": 786.6211449999828, + "p99": 893.5732319999952, + "max": 926.4708950000058 + }, + "replacement_request": { + "n": 1957, + "p50": 8.951632000011159, + "p95": 17.27179700000852, + "p99": 79.54755400000431, + "max": 92.341597000006 + }, + "replacement_liveness": { + "n": 450, + "p50": 645.1912089999969, + "p95": 975.069325000004, + "p99": 992.4281219999975, + "max": 997.0403169999918 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 451.6021180000098, + "p95": 793.450920000003, + "p99": 914.5407790000027, + "max": 948.8798110000062 + } + }, + "nodeCpu": { + "user": 18799633, + "system": 889825 + }, + "nodeCpuAverageOneCorePercent": 29.910693792544862, + "eventLoopDelay": { + "p95Ms": 11.124735, + "p99Ms": 12.779519, + "maxMs": 80.478207 + }, + "sqlAllApplicationPools": 43602, + "sampledPeakDatabaseActivityRows": 28, + "postgresThrottledUsec": 117766, + "lockWait": { + "intervalMs": 25, + "samples": 2548, + "gapsMs": { + "n": 2548, + "p50": 25.149598000003607, + "p95": 26.418762000001152, + "p99": 73.46682900001178, + "max": 90.0133790000109 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 2, + "observedLowerMs": { + "n": 2, + "p50": 57.415, + "p95": 57.415, + "p99": 57.415, + "max": 57.415 + }, + "estimatedUpperMs": { + "n": 2, + "p50": 78.82680399998208, + "p95": 78.82680399998208, + "p99": 78.82680399998208, + "max": 78.82680399998208 + } + }, + "benchmark-replacement-application": { + "observedEpisodes": 1, + "observedLowerMs": { + "n": 1, + "p50": 0.817, + "p95": 0.817, + "p99": 0.817, + "max": 0.817 + }, + "estimatedUpperMs": { + "n": 1, + "p50": 25.092009999993024, + "p95": 25.092009999993024, + "p99": 25.092009999993024, + "max": 25.092009999993024 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 11, + "observedLowerMs": { + "n": 11, + "p50": 3.97, + "p95": 58.82, + "p99": 58.82, + "max": 58.82 + }, + "estimatedUpperMs": { + "n": 11, + "p50": 30.69183900000644, + "p95": 83.27365099999588, + "p99": 83.27365099999588, + "max": 83.27365099999588 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2548, + "total_exec_time": 234.671226, + "rows": 14 + } + ], + "peakSampledNodeRssBytes": 870621184, + "peakSampledNodeHeapBytes": 593590232, + "postgresStatementExecutionMs": 4403.789213000001, + "lockObserverExecutionMs": 234.671226 + }, + { + "label": "03_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T19:02:16.491Z", + "endedAt": "2026-10-30T19:03:22.313Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23496, + "livenessSqlCount": 4504, + "checkpointSqlCount": 15602, + "walBytes": 50570496, + "postgresCpuUsec": 16081885, + "postgresPeakSampledMemoryBytes": 1332752384, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 818.6222430000198, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 39.5862119999947, + "p95": 74.14522999999463, + "p99": 124.791716000007, + "max": 129.85977700000512 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 41.993144000007305, + "p95": 77.44222500000615, + "p99": 125.49473299999954, + "max": 132.81708999999682 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 487.02540600000066, + "p95": 836.5916340000113, + "p99": 955.1296190000139, + "max": 1000.8564950000145 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 444.6453029999975, + "p95": 790.974338, + "p99": 903.6869510000106, + "max": 944.6195900000166 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 8.99197100001038, + "p95": 17.247062000009464, + "p99": 77.24802900000941, + "max": 89.03845799999544 + }, + "poolQueueByPool": { + "request": { + "n": 1956, + "p50": 0.01789499999722466, + "p95": 0.03461500001139939, + "p99": 1.2706449999823235, + "max": 7.25288700000965 + }, + "liveness": { + "n": 450, + "p50": 668.1329050000058, + "p95": 943.7231139999931, + "p99": 964.8324849999917, + "max": 971.0895420000015 + }, + "checkpoint": { + "n": 1950, + "p50": 436.23111700001755, + "p95": 776.24320300002, + "p99": 884.2146380000049, + "max": 935.5753669999831 + }, + "replacement_request": { + "n": 1957, + "p50": 0.017906000022776425, + "p95": 0.03174499998567626, + "p99": 0.04958200000692159, + "max": 37.47016100000474 + }, + "replacement_liveness": { + "n": 450, + "p50": 660.5174439999973, + "p95": 939.0108659999969, + "p99": 963.677406999981, + "max": 968.687052000023 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 436.32736499997554, + "p95": 788.8090800000064, + "p99": 902.0589020000189, + "max": 938.6039999999921 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 1956, + "p50": 8.946074000006774, + "p95": 17.429038000002038, + "p99": 76.37251200000173, + "max": 89.003899000003 + }, + "liveness": { + "n": 450, + "p50": 678.8731129999796, + "p95": 945.945057999983, + "p99": 967.5097349999996, + "max": 973.1065839999937 + }, + "checkpoint": { + "n": 1950, + "p50": 444.8119090000109, + "p95": 784.2202169999946, + "p99": 893.1187290000089, + "max": 943.4892260000051 + }, + "replacement_request": { + "n": 1957, + "p50": 9.007467999996152, + "p95": 18.345773000008194, + "p99": 78.94358199997805, + "max": 89.00447099999292 + }, + "replacement_liveness": { + "n": 450, + "p50": 668.120407999988, + "p95": 941.9229499999783, + "p99": 966.5471529999922, + "max": 971.1461609999824 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 444.2886780000117, + "p95": 798.4288989999914, + "p99": 912.1802920000046, + "max": 944.5919719999947 + } + }, + "nodeCpu": { + "user": 18563401, + "system": 892958 + }, + "nodeCpuAverageOneCorePercent": 29.559203827490894, + "eventLoopDelay": { + "p95Ms": 11.214847, + "p99Ms": 13.647871, + "maxMs": 80.478207 + }, + "sqlAllApplicationPools": 43602, + "sampledPeakDatabaseActivityRows": 28, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2541, + "gapsMs": { + "n": 2541, + "p50": 25.158021000010194, + "p95": 26.61356500000693, + "p99": 70.55964399999357, + "max": 90.20055800001137 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 3, + "observedLowerMs": { + "n": 3, + "p50": 2.624, + "p95": 42.948, + "p99": 42.948, + "max": 42.948 + }, + "estimatedUpperMs": { + "n": 3, + "p50": 68.97033699997701, + "p95": 72.40636000002269, + "p99": 72.40636000002269, + "max": 72.40636000002269 + } + }, + "benchmark-replacement-application": { + "observedEpisodes": 2, + "observedLowerMs": { + "n": 2, + "p50": 2.623, + "p95": 2.623, + "p99": 2.623, + "max": 2.623 + }, + "estimatedUpperMs": { + "n": 2, + "p50": 59.07080300000962, + "p95": 59.07080300000962, + "p99": 59.07080300000962, + "max": 59.07080300000962 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 4, + "observedLowerMs": { + "n": 4, + "p50": 1.948, + "p95": 4.972, + "p99": 4.972, + "max": 4.972 + }, + "estimatedUpperMs": { + "n": 4, + "p50": 56.89580300002126, + "p95": 76.26736000002711, + "p99": 76.26736000002711, + "max": 76.26736000002711 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 8, + "observedLowerMs": { + "n": 8, + "p50": 2.616, + "p95": 62.06, + "p99": 62.06, + "max": 62.06 + }, + "estimatedUpperMs": { + "n": 8, + "p50": 28.899388999998337, + "p95": 87.05610200000228, + "p99": 87.05610200000228, + "max": 87.05610200000228 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2541, + "total_exec_time": 230.91749600000003, + "rows": 17 + } + ], + "peakSampledNodeRssBytes": 887791616, + "peakSampledNodeHeapBytes": 633808520, + "postgresStatementExecutionMs": 4428.595989999998, + "lockObserverExecutionMs": 230.91749600000003 + }, + { + "label": "04_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T19:03:29.319Z", + "endedAt": "2026-10-30T19:04:35.119Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23496, + "livenessSqlCount": 4504, + "checkpointSqlCount": 15602, + "walBytes": 48142872, + "postgresCpuUsec": 15625330, + "postgresPeakSampledMemoryBytes": 1307955200, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 796.7394680000143, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 14.284052000002703, + "p95": 102.27245000004768, + "p99": 107.64110100001562, + "max": 141.18229500000598 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 16.601036000007298, + "p95": 104.65098200005013, + "p99": 110.24340700003086, + "max": 141.02557800000068 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 511.11750200000824, + "p95": 835.1623750000144, + "p99": 932.1108870000462, + "max": 966.9618110000156 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 466.6611930000363, + "p95": 785.0629789999803, + "p99": 882.4489049999975, + "max": 911.663005999988 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 8.741485000005923, + "p95": 21.27701399999205, + "p99": 79.73378300003242, + "max": 98.61222199999611 + }, + "poolQueueByPool": { + "request": { + "n": 1956, + "p50": 0.017884999979287386, + "p95": 0.035715999954845756, + "p99": 0.7316219999920577, + "max": 6.9568530000397 + }, + "liveness": { + "n": 450, + "p50": 612.479364000028, + "p95": 931.2895360000257, + "p99": 968.4246090000088, + "max": 974.2110239999893 + }, + "checkpoint": { + "n": 1950, + "p50": 457.2952529999893, + "p95": 777.7845369999995, + "p99": 872.6880790000141, + "max": 904.4663130000117 + }, + "replacement_request": { + "n": 1957, + "p50": 0.017825999995693564, + "p95": 0.032011000032071024, + "p99": 0.05147999996552244, + "max": 58.91463199999998 + }, + "replacement_liveness": { + "n": 450, + "p50": 611.0748399999866, + "p95": 924.6075030000065, + "p99": 966.4776800000109, + "max": 972.2507760000008 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 458.15926700001, + "p95": 773.1846900000237, + "p99": 871.5540899999905, + "max": 906.4964309999486 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 1956, + "p50": 8.748108999978285, + "p95": 21.241075000027195, + "p99": 79.69776200002525, + "max": 96.44556800002465 + }, + "liveness": { + "n": 450, + "p50": 623.5618810000014, + "p95": 938.7270220000064, + "p99": 971.3052120000066, + "max": 976.3884749999852 + }, + "checkpoint": { + "n": 1950, + "p50": 466.64239300001645, + "p95": 786.580037000007, + "p99": 882.414477000013, + "max": 910.3289480000385 + }, + "replacement_request": { + "n": 1957, + "p50": 8.698626999976113, + "p95": 22.4105860000127, + "p99": 80.60295999998925, + "max": 98.5777620000008 + }, + "replacement_liveness": { + "n": 450, + "p50": 620.2892739999807, + "p95": 935.5929790000082, + "p99": 969.3374020000047, + "max": 975.1367830000236 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 466.62874600000214, + "p95": 782.9779160000035, + "p99": 880.8073159999913, + "max": 911.626213999989 + } + }, + "nodeCpu": { + "user": 19222778, + "system": 829326 + }, + "nodeCpuAverageOneCorePercent": 30.474186242549273, + "eventLoopDelay": { + "p95Ms": 10.977279, + "p99Ms": 14.262271, + "maxMs": 82.444287 + }, + "sqlAllApplicationPools": 43602, + "sampledPeakDatabaseActivityRows": 28, + "postgresThrottledUsec": 86175, + "lockWait": { + "intervalMs": 25, + "samples": 2544, + "gapsMs": { + "n": 2544, + "p50": 25.112813999992795, + "p95": 26.291275000025053, + "p99": 71.21967900000163, + "max": 94.01448299997719 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "benchmark-replacement-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 7, + "observedLowerMs": { + "n": 7, + "p50": 3.134, + "p95": 62.973, + "p99": 62.973, + "max": 62.973 + }, + "estimatedUpperMs": { + "n": 7, + "p50": 29.69778799993219, + "p95": 91.62701000005472, + "p99": 91.62701000005472, + "max": 91.62701000005472 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2544, + "total_exec_time": 225.15488799999974, + "rows": 7 + } + ], + "peakSampledNodeRssBytes": 900505600, + "peakSampledNodeHeapBytes": 621405448, + "postgresStatementExecutionMs": 4218.09364, + "lockObserverExecutionMs": 225.15488799999974 + } + ], + "sourceReport": "/tmp/native-controller-load-qualification-20261030/remote/evidence/overlap-abba/report.json" + } + }, + "lockObserverCalibration": { + "intentionalHoldMs": 250, + "releasedAtMs": 249.4750299999996, + "transactionElapsedMs": 304.0121600000002, + "observed": { + "intervalMs": 25, + "samples": 12, + "gapsMs": { + "n": 12, + "p50": 25.226912000000084, + "p95": 35.78078900000037, + "p99": 35.78078900000037, + "max": 35.78078900000037 + }, + "errors": [], + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 1, + "observedLowerMs": { + "n": 1, + "p50": 229.153, + "p95": 229.153, + "p99": 229.153, + "max": 229.153 + }, + "estimatedUpperMs": { + "n": 1, + "p50": 254.2638969999989, + "p95": 254.2638969999989, + "p99": 254.2638969999989, + "max": 254.2638969999989 + } + } + }, + "episodes": [ + { + "pid": 88, + "application_name": "paperclip-runner-checkpoint", + "query_start": "2026-10-30 18:42:31.6856+00", + "wait_event": "transactionid", + "wait_start": "2026-10-30 18:42:31.685612+00", + "wait_age_ms": 229.153, + "firstMs": 12704.136085, + "lastMs": 12899.644581, + "precedingGapMs": 35.78078900000037, + "observedLowerMs": 229.153, + "estimatedUpperMs": 254.2638969999989, + "rightCensored": false + } + ], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + } + }, + "overlapInterpretation": { + "scope": "Two logical server clients share one two-CPU Node process; each has its own bounded request/liveness/checkpoint pools. Same aggregate workload as one-server aligned burst.", + "postgresCpuSeconds": 16.20749, + "oneServerCandidateBurstMeanPostgresCpuSeconds": 14.679791, + "increasePercent": 10.406817099780241, + "interpretation": "Capacity passed. This additional CPU cost comes from a different topology and one sample; it is not an improvement claim or a repeatable baseline/candidate regression finding." + }, + "retainedFailures": { + "earlierReport": "doc/plans/2026-10-30-native-controller-load-evidence.json", + "earlierReportUnchanged": true, + "freshAttemptLedger": "/tmp/native-controller-load-qualification-20261030/failure-ledger.json", + "localFailedPilot": { + "requestP99Ms": 1211.193594999997, + "droppedOperations": 33, + "postgresThrottledUsec": 12294711, + "outcome": "Ineligible for acceptance; local scheduling pressure and unmatched work. A subsequent identical local pilot passed, demonstrating temporal host variability." + }, + "remoteSetupFailures": [ + "Abbreviated Git revision fetch rejected; exact full revision fetched.", + "Missing built plugin SDK export; compiled from locked installed dependencies.", + "Observer database missing pg_stat_statements; harness now creates the extension in the owned observer database." + ], + "allFailedAttemptsAccountedFor": true + }, + "limits": [ + "Database authentication/read latency is not HTTP latency or full application latency.", + "The baseline supplies writer, snapshot port and PRP controller; normal request and lease-renewal paths are common controls.", + "PostgreSQL CPU and WAL are cluster-wide and include observer overhead and PostgreSQL background work.", + "Lock observer nominally adds 40 queries/second; actual calls and SQL execution time are included in each result.", + "Lock episodes use PostgreSQL waitstart identities and server-reported ages. Short waits between polls can be missed, and client scheduling/network delay widens estimated upper bounds.", + "Pool admission includes client encoding; SQL execution statistics include lock waits.", + "Memory and database activity are sampled once per second. Activity-row counts include non-client database backends, so they are not exact client-connection counts.", + "The two-server overlap probe uses two logical clients in one Node process and does not measure real process replacement or authenticated provider reattachment.", + "The separate claim-capacity probe uses synthetic prepared-workspace references. Provider authentication and external sandbox existence are outside its scope." + ], + "rawEvidence": { + "root": "/tmp/native-controller-load-qualification-20261030", + "initialArchive": "/tmp/native-controller-load-qualification-20261030/remote-evidence-initial.tar.gz", + "initialArchiveSha256": "d1f2da9429e2ae1944df35d40da77c0333c506336fed4f78dcfd1d13447403f8", + "cloudInstanceCleanup": "Owned benchmark containers stopped and owned disposable VM confirmed terminated with no attached root volumes.", + "finalDeltaArchive": "/tmp/native-controller-load-qualification-20261030/remote-evidence-final-delta.tar.gz", + "finalDeltaArchiveSha256": "1e77edbfd3ee9e05bd5de6a9ecefb82fdb12c89e52e575f9e8b48ea0c94d26ed", + "cleanupReceipt": "/tmp/native-controller-load-qualification-20261030/ec2-final-state.json" + }, + "claimCapacity": { + "schema": "paperclip.native-controller-claim-capacity/v1", + "scope": "Production DB claim function against synthetic prepared-workspace references and real persisted snapshots/PRP records. Natural lease expiry; two server database clients. No provider attachment or external sandbox existence claim.", + "passed": true, + "harnessSha256": "b083e2e34eccf21a1921e3b32a434ac7d5b1dec24c903b0b3b5052f9f48adf2e", + "candidateRevision": "a0ef3ce7493c8bb76c4766ff007e8a390118d145", + "baselineRevision": "59e191c368cb3119ff3206621fefa9c5d4ad3110", + "errors": [], + "naturalExpiryWaitMs": 54026.14, + "claimDrainMs": 2838.7143720000167, + "claimAndRestoreDrainMs": 8742.97213000001, + "restoredCheckpoints": 300, + "renewedOwnerLosses": 0, + "postRenewalOwners": { + "owners": 300, + "min_generation": 2, + "max_generation": 2, + "minimum_remaining_ms": 56198.788 + }, + "elapsedMs": 77971.21296299998, + "requestsOffered": 15400, + "requestsCompleted": 15400, + "requestLatency": { + "n": 15400, + "p50": 2.684827999997651, + "p95": 5.376435999991372, + "p99": 9.759766000002855, + "max": 64.86076799999864 + }, + "postgresCpuUsec": 11280308, + "sqlByClient": { + "old_request": 7703, + "old_liveness": 0, + "old_checkpoint": 0, + "new_request": 12512, + "new_liveness": 1502, + "new_checkpoint": 1502 + }, + "settledIdleSql": 0, + "retainedOldControllerLeaseMs": 60000, + "remoteReconnectGraceMs": 420000, + "checkpointPoolPerServer": 2, + "claimsByKind": { + "reattach_remote_runner": 300 + }, + "distinctClaimedRuns": 300, + "claimGenerations": [ + 2 + ], + "fixtureDatabaseRequestsPerSecond": 200, + "sourceCode": "/tmp/native-controller-load-qualification-20261030/benchmark-native-controller-claim-capacity.ts", + "sourceReport": "/tmp/native-controller-load-qualification-20261030/remote/evidence/claim-capacity/claim-capacity.json", + "sourceReportSha256": "cd9fc360f1cc1df0b6e82ca50694f6b2ad4c3ff3f638381637dcc99dd3385bc2", + "peakSampledDatabaseActivityRows": 25, + "lockWait": { + "samples": 2987, + "gapsMs": { + "n": 2987, + "p50": 25.607273999994504, + "p95": 29.01517500000773, + "p99": 43.6128069999977, + "max": 85.42577500001062 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + } + }, + "matchedOverlapAbba": { + "status": "passed", + "sourceCode": "/tmp/native-controller-load-qualification-20261030/benchmark-native-controller-overlap-abba.ts", + "sourceSha256": "2cc52afee21c34efab7f65af88bbd911e6efdd424094f7020bfccd4a16b4d441", + "requestP99IncreasePercent": 9.675454566469433, + "interpretation": "All relative gates passed; request p99 increase is close to the 10 percent threshold." + }, + "reproduction": { + "baseHarnessSha256": "abaa29d50eb95013836b54b8dacc94c72e080edf24744cc5d291cc5eb1ff2818", + "patches": { + "overlap": { + "path": "doc/plans/2026-10-30-native-controller-overlap.patch", + "patchSha256": "e8e14ddbdbed289f47e57fa0fe8ae2be337e77f2919c755d7003afd73150faf4", + "reconstructedSourceSha256": "2cc52afee21c34efab7f65af88bbd911e6efdd424094f7020bfccd4a16b4d441", + "verifiedPatchReconstruction": true + }, + "claim": { + "path": "doc/plans/2026-10-30-native-controller-claim.patch", + "patchSha256": "1fcbf028654499562c5f6b789f36e7c4681ec5d28c3dec6d13e2a51650496e2e", + "reconstructedSourceSha256": "b083e2e34eccf21a1921e3b32a434ac7d5b1dec24c903b0b3b5052f9f48adf2e", + "verifiedPatchReconstruction": true + } + }, + "instructions": [ + "Copy scripts/benchmark-native-controller-recovery.ts to a new probe source file.", + "Apply the corresponding dated patch with: patch /absolute/path/to/probe.ts /absolute/path/to/patch.", + "Set existing benchmark inputs plus BENCH_CANDIDATE_ROOT to the repository root; use the pinned baseline and identical installed dependencies.", + "Run Node 24 with the repository tsx loader and the patched probe file followed by burst. Each report directory must be fresh." + ], + "requiredInputs": [ + "BENCH_DISPOSABLE_POSTGRES=1", + "BENCH_DATABASE_URL", + "BENCH_BASELINE_ROOT", + "BENCH_REPORT_ROOT", + "BENCH_POSTGRES_CGROUP", + "BENCH_CANDIDATE_ROOT" + ] + }, + "streamingSqlIndependence": { + "status": "passed", + "sourceRevision": "da8c7f6e1837ddd896dfe83a7524812fee813487", + "sourceReport": "/tmp/native-controller-release-qualification-20261030/stream-cost-current.json", + "sourceLog": "/tmp/native-controller-release-qualification-20261030/stream-cost-current.log", + "samples": [ + { + "chunks": 8, + "streamSql": 0, + "streamTransactions": 0, + "streamChangedRows": 0, + "idleSql": 0, + "checkpointWrites": 8, + "checkpointTransactions": 7, + "changedRows": 12 + }, + { + "chunks": 512, + "streamSql": 0, + "streamTransactions": 0, + "streamChangedRows": 0, + "idleSql": 0, + "checkpointWrites": 8, + "checkpointTransactions": 7, + "changedRows": 12 + }, + { + "chunks": 8192, + "streamSql": 0, + "streamTransactions": 0, + "streamChangedRows": 0, + "idleSql": 0, + "checkpointWrites": 8, + "checkpointTransactions": 7, + "changedRows": 12 + } + ], + "positiveCalibrationPassed": true, + "existingRenewalSqlFor8192ChunkCase": 66, + "limits": [ + "Separate real Runner fixture executed by the release qualification agent.", + "Concurrent owned fixtures may affect its cluster-wide WAL and elapsed diagnostics; those values are excluded from performance claims." + ] + } +} diff --git a/doc/plans/2026-10-30-native-controller-source-probe-receipt.json b/doc/plans/2026-10-30-native-controller-source-probe-receipt.json new file mode 100644 index 00000000000..73d05dc06a6 --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-source-probe-receipt.json @@ -0,0 +1,70 @@ +{ + "schema": "paperclip.native-controller.release-qualification-receipt/v1", + "finishedAt": "2026-10-30T19:33:20.587023+00:00", + "sourceIntegrity": { + "A": { + "revision": "59ff89a18fe1312b5f0bb908439bbdb1237b0252", + "nativeRuntimeSourceBytesUnchanged": true, + "mismatches": [] + }, + "B": { + "revision": "98f05fcd5b452704131a35ff09486f5c04c6959a", + "nativeRuntimeSourceBytesUnchanged": true, + "mismatches": [] + } + }, + "sourceAndArtifact": { + "A": "59ff89a18fe1312b5f0bb908439bbdb1237b0252", + "B": "98f05fcd5b452704131a35ff09486f5c04c6959a", + "admittedRunnerSha256": "b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8", + "runnerArtifact": { + "role": "admitted Runner artifact A, unchanged through every controller", + "sourceRevision": "59ff89a18fe1312b5f0bb908439bbdb1237b0252", + "binary": "/private/tmp/native-controller-qualification-20261030-1833/installed-A/consumer/node_modules/@paperclipai/server/dist/vendor/paperclip-runner/bin/linux-arm64/paperclip-runnerd", + "sha256": "b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8", + "bytes": 35345144, + "installationReceipt": "/private/tmp/native-controller-qualification-20261030-1833/installed-A/receipt.json" + } + }, + "passed": [ + "Distinct source-controller active SIGKILL followed by natural 60-second lease takeover; exact same run/provider attempt/Runner retained", + "Current real-PostgreSQL stream oracle at 8,512,8192 chunks: zero incremental checkpoint SQL/transactions/changed rows; idle SQL zero; positive control passed", + "B1 recovered run completed and committed its result with ready native snapshot and valid v1 recovery envelope" + ], + "failed": [ + "Fresh B2 retained continuation rejected after B1 completed because durable control journal has a null-goal snapshot after run.terminal", + "Final 98f05fc live probe still rejects actual paperclip.prp.event.v2 because proposed guard accepts only v1" + ], + "notQualified": [ + "Complete A/B/A native installed-application startup/rollback gate", + "Uninterrupted warm provider process through changed completion-contract boundary", + "First compatible published empty-disk rollback reader" + ], + "limits": [ + "Source controller processes and scripted Codex-compatible provider in Docker; no live-provider or live-Daytona claim", + "Qualified immutable A release Runner used; source controller native modules hash-verified, not full installed server execution", + "Only native controller projections deleted; non-native configuration, encryption key, workspace/sandbox dependencies preserved", + "Local stream WAL/elapsed diagnostics excluded from quantitative performance due concurrent isolated fixture" + ], + "evidence": { + "finalProbe": "/tmp/native-controller-release-qualification-qualified-A-20261030/evidence.json", + "finalProbeLog": "/tmp/native-controller-release-qualification-qualified-A-20261030/run-final-fix.log", + "controlLog": "/tmp/native-controller-release-qualification-qualified-A-20261030/controller.log", + "streamOracle": "/tmp/native-controller-release-qualification-20261030/stream-cost-current.json", + "streamLog": "/tmp/native-controller-release-qualification-20261030/stream-cost-current.log", + "earlierThreeTurnBusinessContinuation": "/tmp/native-controller-release-qualification-20261030/evidence-run-06.json" + }, + "finalProbeFailure": { + "message": "Error: runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate\n at deny (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/remote-warm-session-recovery.ts:56:37)\n at inspectRemoteWarmSessionRecovery (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/remote-warm-session-recovery.ts:85:67)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async executePaperclipNativeSessionWithinScope (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/native-session-executor.ts:8237:28)\n at async executePaperclipNativeSession (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/native-session-executor.ts:8057:12)\n at async process. (/private/tmp/native-controller-release-qualification-qualified-A-20261030/controller-worker.mjs:36:16)", + "cause": null + }, + "candidateCorrection": { + "change": "Accept supported PRP event v1 and v2 idle goal suffixes; reject unknown versions", + "fullSourceReplay": "not rerun after the final v2 correction within the one-hour qualification window", + "focusedTests": { + "result": "pass", + "cases": 14, + "test": "server/src/services/native-runtime/remote-warm-session-recovery.test.ts" + } + } +} diff --git a/doc/testing-native-remote-continuity.md b/doc/testing-native-remote-continuity.md index 91887a1b71b..65671f91edf 100644 --- a/doc/testing-native-remote-continuity.md +++ b/doc/testing-native-remote-continuity.md @@ -7,7 +7,7 @@ and backup validation against a deterministic fake Codex process in Linux. It checks the same runner process identity and one provider process across three warm turns, provider history reuse, and verified suspension at idle close. Separate cases kill the actual controller process and recover the next turn. They reject -a still-live controller and a replacement sandbox before reconnecting. Legacy +a live execution lease and a replacement sandbox before reconnecting. Legacy checkpoints and changed process configuration require verified suspension and replacement, while preserving the conversation and starting each turn once. Recovery can restart a provider to refresh its tool contract; process reuse is @@ -24,6 +24,39 @@ subscription. The controller must register the connection before subscribing so completed message processing can drain and the suspended session can retire. It does not qualify a live model or a sandbox provider API. +The `diskless` and `diskless-per-turn` cases delete both native controller +directories after A completes, continue in a separate B process, delete those +directories after B completes, and continue in C. They use the production source +selector and snapshot rebinder for per-turn admission, followed by the real +native executor. They do not inject PRP files or edit leases to produce recovery. +Warm mode verifies compatible process reuse; per-turn mode verifies the existing +suspension and history-resume policy. These component tests preserve workspace +and remote sandbox storage, and use one source revision in three processes. +They do not qualify distinct application releases or complete webserver disk loss. + +`native-controller-recovery.integration.test.ts` separately verifies encrypted +database discovery, one-source import, idempotency, live-owner waiting, corrupt +bindings and paginated inventory. `native-session-resume-projection.test.ts` +retains the 32-candidate and newer-progress barrier checks. Snapshot transaction +counts and stale-generation rejection are exercised by the control-plane port +tests; incomplete-publication retention is exercised by the expiration tests. + +The cold Codex transport regressions additionally retain a real provider event +after completion, hold its persistence callback and require the original source +tuple and Runner outbox to survive until acceptance. A separate case deletes the +controller cache after its staged target is durably recorded and resumes through +the same source before target attachment. Unknown semantic tails must stay held +without target tool dispatch or a fabricated response. The staged restart helper +and classifier tests cover target discovery before its first session snapshot, +exact encrypted source/header/artifact bindings and the existing compatibility +rebinder. Source accounting tests use real PostgreSQL to distinguish immutable +receipt replay, equal finalized usage and late usage requiring reconciliation. +Handoff crash tests must restore the exact queued command or authenticated +receipt, reuse the verified physical Runner and preserve endpoint binding. Test +both direct callbacks and the same guest listener when that ingress is enabled; +changing an endpoint or losing physical proof must hold recovery. +These component cases do not replace the normal three-controller Daytona gate. + Per-turn plugin continuation uses a stopped checkpoint on a replacement lease. The executor tests cover Claude and Codex, including reopening blocked work. The lease-authority integration test uses PostgreSQL to check that checkpoint @@ -151,14 +184,21 @@ local file becomes a marker that requires restoration from the external checkpoint before restart. Ordinary control changes do not rewrite a second local journal. The sandbox runner keeps its own journal. -Checkpoint transactions share a separate pool capped at two connections per +Checkpoint transactions share a separate pool capped at three connections per server database client. They queue without occupying the application pool or the existing lease-renewal pool. This can delay a control acknowledgement during a burst; it does not release that acknowledgement before its commit. The cost is -up to two additional active database connections per instance, not per agent. +up to three additional active database connections per instance, not per agent. There is no new SQL per text chunk or while idle, and the number of control transactions is unchanged. SQL cost collection must include the checkpoint pool. +The cap follows a bounded 2 → 3 → 4 → 2 diagnostic: both cap-2 trials failed +the existing burst-drain gate; 3 was the smallest tested passing cap. Retain +those failures and the diagnostic separately. The subsequent native feature-cost +comparison uses the common current DB factory and equal cap-3 policy for both +variants, so its baseline is a controlled baseline rather than the unmodified +historical cap-2 configuration. All existing benchmark gates remain unchanged. + A sparse checkpoint is marked `checkpointEventPolicy: "control-v1"`. Its peer must advertise `recoveryCursorVersion: 1`. During reconnection the controller echoes the runner's durable `resume.ackedSourceSeq` as `recoveryAckedSourceSeq` in the signed @@ -270,8 +310,10 @@ adapters, database loss and destroyed sandboxes are outside this guarantee. The empty-controller-cache test uses a real Rust runner and scripted provider in separate directories, removes the controller directory, then restores solely -from Postgres. It checks unchanged runner/provider process identities and one -provider turn start. Build the host runner and fake provider first, then run: +from Postgres. It waits for natural lease expiry and checks unchanged +runner/provider process identities and one provider turn start. It explicitly +restores the transport checkpoint, so it does not prove application startup +discovery by itself. Build the host runner and fake provider first, then run: ```sh PAPERCLIP_TEST_CONTROLLER_OUTAGE_MS=300000 pnpm exec vitest run \ @@ -284,6 +326,85 @@ This is protocol and persistence evidence. The separate qualifies the actual Daytona/provider path, including completion and workspace finalization. Neither test substitutes for the other's failure boundary. +The explicit `daytona-restart-continuity.runner-codex.daytona.recover-controller` +cell fixes the outage at 300 seconds and adds a second replacement after +completion. Its new turn executes the saved project through normal admission. +The oracle compares hashed native/provider session identities and requires a new +completed tool receipt and correct answer. An installed A/B/A artifact manifest +uses the last replacement for this retained-session continuation/rollback check. +The general everyday suite retains its 125-second default. These intervals are +fault tests, not permitted HTTP deployment downtime. Both fixtures retain the +workspace and remote sandbox; they do not prove whole-webserver disk loss. + +The dedicated suite's version 6 uses the existing reusable warm environment, +with a 15-minute provider idle window and unchanged 60/300/420-second ownership, +outage and reconnect limits. Its prior per-turn/no-reuse measurement failed +retained native/provider identity and remains failed. The test retires the whole +`runtime/paperclip-runner` directory, including provider failover-backup bytes. +Per-turn application qualification after sandbox disposal and this disk loss +therefore awaits the provider-backup owner's durable storage contract; logical +native checkpoints do not replace provider history or weaken the identity oracle. + +The rollback reader must support both the recovery envelope and the settled PRP +v2 idle goal suffix. Record its actual source commit and installed tree hash. +An older reader rejecting that journal is an unsupported rollback, not a pass. +A backported compatibility build is a development fixture until it is released; +do not describe it as a previously published version. Distinct source-controller +tests, compiled native-module tests, installed CLI startup and the full Product +E2E journey provide different evidence and must retain those scope labels. + +## Database cost qualification + +`scripts/benchmark-native-controller-recovery.ts` compares a pinned source +baseline with the candidate's real PRP writer, control-plane snapshot port and +lease renewal. It uses identical fixtures with 300 controllers and 10,000 prior +tasks, in baseline/candidate/candidate/baseline order. `steady` distributes control +checkpoints; `burst` aligns them every five seconds. `pilot` checks instrumentation +with ten controllers. No provider is called. + +Run on an otherwise idle disposable Linux Postgres cluster with +`pg_stat_statements` preloaded and its extension available. Give Postgres and the +Node client two CPUs and 1536 MiB each. Set these explicit inputs: + +```sh +BENCH_DISPOSABLE_POSTGRES=1 \ +BENCH_DATABASE_URL=postgres://postgres:fixture-password@127.0.0.1:5432/postgres \ +BENCH_BASELINE_ROOT=/absolute/path/to/pinned-baseline \ +BENCH_REPORT_ROOT=/absolute/path/to/new-report-directory \ +BENCH_POSTGRES_CGROUP=/absolute/path/to/postgres-cgroup \ +node --import ./cli/node_modules/tsx/dist/loader.mjs \ + scripts/benchmark-native-controller-recovery.ts steady +``` + +The baseline needs the same installed dependency versions and built workspace +packages. It supplies the baseline writer, snapshot port and PRP controller; +the request fixtures, renewal path and pool policy are common controls. When +running an exported tree without Git metadata, supply its exact +`BENCH_BASELINE_REVISION`, `BENCH_CANDIDATE_REVISION` and +`BENCH_CANDIDATE_DIFF`. Record the export source and verify its bytes separately. +Use a fresh report directory for every attempt. Only new random databases are +created; successful trial databases are dropped, and the owned template remains +for inspection until the disposable cluster is removed. + +The report counts SQL across all three pools, actual snapshot/PRP transactions, +changed records/bytes, WAL, database CPU, sampled memory, pool admission, +transaction latency and database request latency. Pool admission ends at +connection assignment/BEGIN encoding; BEGIN response time is separate. +Postgres execution time includes lock waits. A 25 ms observer uses PostgreSQL's +`pg_locks.waitstart` to report duration bounds for observed lock episodes by pool. +The pilot calibrates it against an intentional 250 ms row-lock wait. Short waits +can be missed, and delayed samples widen the estimated upper bound. The nominal +40 observer queries per second contribute to measured database CPU and statement +statistics. Reports include actual sample counts, sampling gaps, CPU pressure, +quota throttling and the measurement harness hash. They do not provide exact +lock-exclusive SQL execution time or HTTP latency; this is component load +qualification. + +Correctness and absolute gates must pass for acceptance. +Dropped operations invalidate paired comparisons. The report retains failures +and refuses an overall pass if p99, drain, throughput, memory, CPU or the 10% +relative regression gate fails. Do not change pool limits to conceal a failure. + ## Required native usage acceptance Built-in native remote Runner execution accepts structured accounting before diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index 071404c8b83..2a2041d2e1a 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -272,6 +272,15 @@ provider restart. For that journey, interrupt a disposable deployment after workspace preparation and before its first executor claim, then confirm one provider turn starts on the original run through the normal admission gates. +The same suite's `diskless` and `diskless-per-turn` cases exercise three native +controller processes with both native projections removed between completed +turns. They check history and invocation counts through the real executor and +admission snapshot rebinder. The workspace and sandbox remain available, and +the processes use the same source revision. Separate Postgres tests cover +metadata import, bounded discovery, ownership holds and incomplete-publication +retention. These cases do not qualify full webserver disk loss, distinct releases, +HTTP traffic continuity, or the live Daytona provider path. + The [Runner transport regressions](../../packages/paperclip-runner/runner/crates/runner-core/src/durable/transport.rs) exercise a real WebSocket welcome delayed beyond two seconds. Authentication uses the ten-second connection setup budget, clipped by the current lifecycle diff --git a/packages/db/src/checkpoint-pool.test.ts b/packages/db/src/checkpoint-pool.test.ts index 05dd02a8208..e75842053b6 100644 --- a/packages/db/src/checkpoint-pool.test.ts +++ b/packages/db/src/checkpoint-pool.test.ts @@ -1,5 +1,5 @@ import { afterEach, expect, it } from "vitest"; -import { checkpointDb, closeCheckpointDb, closeRegisteredClients, createDb, type Db } from "./client.js"; +import { checkpointDb, closeCheckpointDb, closeRegisteredClients, createDb, livenessDb, type Db } from "./client.js"; // postgres.js pools are lazy: these tests exercise actual client lifetimes but // never connect to a database or depend on embedded Postgres availability. @@ -17,6 +17,7 @@ it("fences checkpoints when shutdown precedes the first pool request", async () it("keeps an already-created checkpoint pool ended after shutdown", async () => { const db = createDb(url); const pool = checkpointDb(db); + expect(pool.$client.options.max).toBe(3); await closeCheckpointDb(db, 1); expect(checkpointDb(db)).toBe(pool); await expect(pool.$client`select 1`).rejects.toMatchObject({ code: "CONNECTION_ENDED" }); @@ -30,3 +31,16 @@ it("preserves other owner lifetimes and non-owning test-double behavior", async await closeCheckpointDb(nonOwner, 1); expect(checkpointDb(nonOwner)).toBe(nonOwner); }); + +it("bounds one cached checkpoint pool per owner without changing request or renewal capacity", () => { + const first = createDb(url, { maxConnections: 10 }); + const second = createDb(url, { maxConnections: 10 }); + const pool = checkpointDb(first); + expect(pool).not.toBe(first); + expect(checkpointDb(first)).toBe(pool); + expect(checkpointDb(second)).not.toBe(pool); + expect(pool.$client.options.max).toBe(3); + expect(checkpointDb(second).$client.options.max).toBe(3); + expect(first.$client.options.max).toBe(10); + expect(livenessDb(first).$client.options.max).toBe(2); +}); diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index b43d472d19e..66ff29b3aa6 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -295,9 +295,10 @@ export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise Db>(); const checkpointDbs = new WeakMap(); +const CHECKPOINT_POOL_MAX_CONNECTIONS = 3; const closedCheckpointOwners = new WeakSet(); /** The bounded controller checkpoint pool; transactions and test doubles keep @@ -342,7 +343,7 @@ function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: ob ...resolved, maxConnections: LIVENESS_POOL_MAX_CONNECTIONS, applicationName: "paperclip-controller-lease", }, db)); checkpointDbFactories.set(db, () => createDbClient(url, { - ...resolved, maxConnections: 2, applicationName: "paperclip-runner-checkpoint", + ...resolved, maxConnections: CHECKPOINT_POOL_MAX_CONNECTIONS, applicationName: "paperclip-runner-checkpoint", }, db)); return db; } diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs index 88f6c18c27e..cf710278233 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs @@ -769,8 +769,8 @@ fn run() -> Result<(), Box> { .map(|value| value.parse::()) .transpose()? .unwrap_or(64); - if !(1..=4096).contains(&held_output_count) { - return Err("--held-output-count must be between 1 and 4096".into()); + if !(1..=8192).contains(&held_output_count) { + return Err("--held-output-count must be between 1 and 8192".into()); } let exit_after_turn_start = args.iter().any(|value| value == "--exit-after-turn-start"); let exit_after_turn_completion = args diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts index b9f982401d2..c4f29bdbb6a 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts @@ -32,6 +32,8 @@ import { DurablePrpControlPlane, decodeDurablePrpCheckpoint, inspectDurablePrpCheckpointHeader, + inspectStagedRunAttachment, + inspectStagedRunAttachmentHeader, type DurablePrpCheckpoint, durableRecoveryInternals, SemanticToolNotDispatchedError, @@ -53,6 +55,66 @@ const identity: DurableRecoveryIdentity = { const expectedRunnerVersion = "0.3.0"; const expectedRunnerDigest = `sha256:${"a".repeat(64)}`; +it("persists an undispatched source-to-target attachment before bootstrap and preserves it after cache loss", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-staged-attachment-")); + const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" }; + const template = { provider: { provider: "codex", command: "fixture", instructions: "assigned instructions ".repeat(1_500) }, authorizedTools: {} }; + const core = new DurablePrpControlPlane({ stateDirectory: resolve(root, "first"), identity, expectedRunnerVersion, expectedRunnerDigest }); + let checkpoint!: DurablePrpCheckpoint, templateWrites = 0; + core.setCheckpointPersistence(async value => { checkpoint = value; if (value.changedRecords.has("staged-attachment-template")) templateWrites++; }); + let restored: DurablePrpControlPlane | undefined; + let client: AuthenticatedClient | null = null; + try { + core.stageRunAttachment(target, template); + await core.flushCheckpoint(); + expect(checkpoint.runId).toBe(target.runId); + expect(checkpoint.predecessor).toEqual({ runId: identity.runId, normalizedSessionId: identity.normalizedSessionId }); + expect(checkpoint.workCommandIds).toEqual([]); + const proof = inspectStagedRunAttachmentHeader(checkpoint.records.get("header")!); + expect(proof).toMatchObject({ attachment: { sourceIdentity: identity, targetIdentity: target, commandId: null } }); + expect(proof?.attachment).not.toHaveProperty("runAttachTemplate"); + expect(checkpoint.records.get("header")!.length).toBeLessThan(JSON.stringify(template).length / 2); + expect(JSON.parse(checkpoint.records.get("staged-attachment-template")!)).toEqual(template); + expect(proof?.runnerVersion).toBe(expectedRunnerVersion); + expect(proof?.runnerDigest).toBe(expectedRunnerDigest); + checkpoint.restoreSessionDirectory(resolve(root, "replacement")); + restored = new DurablePrpControlPlane({ stateDirectory: resolve(root, "replacement", "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest }); + expect(inspectStagedRunAttachment(restored.store.state, target)).toMatchObject({ ...proof?.attachment, runAttachTemplate: template }); + expect(restored.store.state.commands).toEqual([]); + expect(() => restored!.stageRunAttachment({ ...target, itemId: "foreign" }, template)).toThrow("conflicts"); + expect(() => restored!.queueCommand("turn.start", {})).toThrow("maintenance only"); + expect(() => restored!.queueCommand("run.attach", { ...template, paperclipNextAuthority: { identity: { ...target, itemId: "foreign" } } })).toThrow("conflicts"); + const corruptedRecords = new Map(checkpoint.records); + corruptedRecords.set("staged-attachment-template", JSON.stringify({ provider: { command: "changed" } })); + expect(() => decodeDurablePrpCheckpoint(corruptedRecords)).toThrow("native_runner_checkpoint_invalid"); + expect(inspectStagedRunAttachment(restored.store.state, { ...target, turnId: "foreign" })).toBeNull(); + for (const field of ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId"] as const) { + const changed = JSON.parse(checkpoint.records.get("header")!); + changed.state.stagedRunAttachment.targetIdentity[field] = "foreign"; + expect(inspectStagedRunAttachmentHeader(JSON.stringify(changed))).toBeNull(); + } + const downgraded = JSON.parse(checkpoint.records.get("header")!); + downgraded.state.schema = "paperclip.runner.durable.control-plane-state.v1"; + expect(inspectStagedRunAttachmentHeader(JSON.stringify(downgraded))).toBeNull(); + const beforeRetry = checkpoint; + core.stageRunAttachment(target, template); await core.flushCheckpoint(); expect(checkpoint).toBe(beforeRetry); + core.queueCommand("session.snapshot", {}); await core.flushCheckpoint(); + expect(checkpoint.changedRecords.has("staged-attachment-template")).toBe(false); + await core.start(); client = await authenticate(core, core.issueBootstrapTicket()); + expect(client).not.toBeNull(); + core.queueCommand("run.attach", { ...template, paperclipNextAuthority: { identity: target, + connection: { mode: "connect", connectUrl: core.connectUrl } } }, "staged-next-attach"); + await core.flushCheckpoint(); + expect(templateWrites).toBe(1); + expect(checkpoint.records.has("staged-attachment-template")).toBe(false); + expect(checkpoint.removedRecords).toContain("staged-attachment-template"); + expect(() => decodeDurablePrpCheckpoint(checkpoint.records)).not.toThrow(); + } finally { + client?.socket.destroy(); + await restored?.stop(); await core.stop(); rmSync(root, { recursive: true, force: true }); + } +}); + it("continues externally persisted commands after losing the local controller directory", async () => { const root = mkdtempSync(resolve(tmpdir(), "runner-external-store-")); const options = { stateDirectory: resolve(root, "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest }; @@ -106,6 +168,7 @@ it("restores authentication and commands from opaque checkpoint records", async try { const ticket = first.issueBootstrapTicket(); first.queueCommand("session.snapshot", {}, "maintenance"); + first.queueCommand("turn.stop", {}, "retirement-maintenance"); first.queueCommand("turn.start", {}, "execution"); first.persistPreparedLaunchConfiguration(identity, "retained-configuration"); await first.flushCheckpoint(); diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index eb67e6d9849..ea82cd674e0 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -54,6 +54,7 @@ const protocolVersion = RUNNER_PRP_PROTOCOL_MAX; const secureFrameSchema = "paperclip.runner.secure-frame.v1"; const websocketGuid = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; const coreStateSchema = "paperclip.runner.durable.control-plane-state.v1"; +const stagedCoreStateSchema = "paperclip.runner.durable.control-plane-state.staged-attachment.v1"; const externalCoreStateSchema = "paperclip.runner.external-control-plane-state.v1"; const transitionCoreStateSchema = "paperclip.runner.durable.control-plane-state.warm-transition.v1"; @@ -140,8 +141,17 @@ interface ConnectionLeaseRecord { revokedAt: string | null; } +export interface StagedRunAttachment { + readonly sourceIdentity: DurableRecoveryIdentity; + readonly targetIdentity: DurableRecoveryIdentity; + readonly runAttachTemplate: Record; + readonly templateDigest: string; + readonly commandId: string | null; +} + interface StoredCoreState { - schema: typeof coreStateSchema | typeof transitionCoreStateSchema; + schema: typeof coreStateSchema | typeof transitionCoreStateSchema | typeof stagedCoreStateSchema; + stagedRunAttachment?: Omit & { runAttachTemplate?: Record }; identity: DurableRecoveryIdentity; /** External checkpoints contain control state, not a duplicate output log. */ checkpointEventPolicy?: "control-v1"; @@ -311,10 +321,12 @@ type CheckpointHeader = { schema: "paperclip.runner.controller-checkpoint.v2"; runnerVersion: string; runnerDigest: string; - state: Omit; + state: Omit + & { stagedRunAttachment?: Omit }; }; function checkpointRunId(state: StoredCoreState): string { + if (state.stagedRunAttachment) return state.stagedRunAttachment.targetIdentity.runId; if (state.warmTransition) return state.warmTransition.receipt.newIdentity.runId; const attachment = state.commands.findLast(command => command.type === "run.attach"); const next = attachment?.payload.paperclipNextAuthority as { identity?: { runId?: string } } | undefined; @@ -347,17 +359,19 @@ function checkpointChanges(records: ReadonlyMap, runId: string, function captureCheckpoint(state: StoredCoreState, runnerVersion: string, runnerDigest: string): DurablePrpCheckpoint { const runId = checkpointRunId(state); const records = new Map(); - const { commands, committedEvents, ackedSourceSeq: _cursor, replayDeliveries: _replays, ...metadata } = state; + const { commands, committedEvents, ackedSourceSeq: _cursor, replayDeliveries: _replays, stagedRunAttachment, ...metadata } = state; + const stagedDescriptor = stagedRunAttachment ? (({ runAttachTemplate: _template, ...descriptor }) => descriptor)(stagedRunAttachment) : undefined; + if (stagedRunAttachment?.commandId === null) records.set("staged-attachment-template", JSON.stringify(stagedRunAttachment.runAttachTemplate)); const latestUsage = committedEvents.findLast(event => event.eventType === "usage.reported"); const recoveryEvents = committedEvents.filter(event => !isStreamingEvent(event) || event === latestUsage); for (const command of commands) records.set(`command:${command.controllerSeq}`, JSON.stringify(command)); for (const event of recoveryEvents) records.set(`event:${event.sourceSeq}`, JSON.stringify(event)); const header: CheckpointHeader = { schema: "paperclip.runner.controller-checkpoint.v2", - runnerVersion, runnerDigest, state: { ...metadata, checkpointEventPolicy: "control-v1" } }; + runnerVersion, runnerDigest, state: { ...metadata, ...(stagedDescriptor ? { stagedRunAttachment: stagedDescriptor } : {}), checkpointEventPolicy: "control-v1" } }; records.set("header", JSON.stringify(header)); records.set("cursor", JSON.stringify({ ackedSourceSeq: state.ackedSourceSeq, replayDeliveries: state.replayDeliveries, commandCount: commands.length, eventCount: recoveryEvents.length })); - const maintenance = new Set(["runner.drain", "runner.suspend", "session.snapshot", "session.close"]); + const maintenance = new Set(["turn.stop", "runner.drain", "runner.suspend", "session.snapshot", "session.close"]); return { runnerVersion, runnerDigest, runId, records, ...checkpointChanges(records, runId), recoveryEvidence: { runnerInstanceId: state.identity.runnerInstanceId, normalizedSessionId: state.identity.normalizedSessionId, @@ -397,11 +411,43 @@ export function inspectDurablePrpCheckpointHeader(contents: string): DurableReco const header = parseCheckpointHeader(contents); if (header.schema !== "paperclip.runner.controller-checkpoint.v2" || header.state.checkpointEventPolicy !== "control-v1" - || ![coreStateSchema, transitionCoreStateSchema].includes(String(header.state.schema))) return null; + || ![coreStateSchema, transitionCoreStateSchema, stagedCoreStateSchema].includes(String(header.state.schema)) + || (header.state.schema === stagedCoreStateSchema && !validStagedRunAttachment(header.state as unknown as StoredCoreState, true))) return null; return { ...header.state.identity }; } catch { return null; } } +/** Bounded checkpoint metadata inspection; never itself grants recovery authority. */ +export function inspectStagedRunAttachmentHeader(contents: string): { + attachment: Omit; runnerVersion: string; runnerDigest: string; + protocolIdentity: DurableRecoveryIdentity; + phase: "source_bootstrap" | "attachment_queued" | "awaiting_result" | "prepared" | "activated" | "completed"; + receipt: DurableWarmRunTransition | null; +} | null { + try { + if (Buffer.byteLength(contents) > maxStateBytes) return null; + const header = parseCheckpointHeader(contents), state = header.state as unknown as StoredCoreState; + if (header.schema !== "paperclip.runner.controller-checkpoint.v2" + || state.checkpointEventPolicy !== "control-v1" || !validStagedRunAttachment(state, true) + || !state.stagedRunAttachment) return null; + const transition = state.warmTransition; + const completed = stagedTargetActivated(state) ? state.completedWarmTransition : undefined; + const bound = transition ?? completed; + if (bound) { + const receipt = bound.receipt; + const result = transition?.phase === "awaiting_result" ? transition.expectedResult : bound.command.result; + const expected = warmTransitionReceipt(receipt.oldIdentity, bound.command, result!, receipt.oldAckedSourceSeq, + { leaseId: receipt.leaseId, expiresAtUnixMs: receipt.leaseExpiresAtUnixMs, revocationEpoch: receipt.leaseRevocationEpoch }, + header.runnerVersion, header.runnerDigest); + if (canonicalJson(expected) !== canonicalJson(receipt)) return null; + } + return { attachment: structuredClone(state.stagedRunAttachment), protocolIdentity: structuredClone(state.identity), + phase: transition?.phase ?? (completed ? "completed" : state.stagedRunAttachment.commandId ? "attachment_queued" : "source_bootstrap"), + receipt: bound ? structuredClone(bound.receipt) : null, + runnerVersion: header.runnerVersion, runnerDigest: header.runnerDigest }; + } catch { return null; } +} + function checkpointState(records: ReadonlyMap) { const header = parseCheckpointHeader(records.get("header") ?? "null"); const read = (key: string): unknown => { @@ -423,9 +469,14 @@ function checkpointState(records: ReadonlyMap) { } const cursor = v2 ? read("cursor") : header.state; if (!isRecord(cursor) || (v2 && (cursor.commandCount !== commandKeys.length || cursor.eventCount !== eventKeys.length - || records.size !== commandKeys.length + eventKeys.length + 2))) throw new Error("native_runner_checkpoint_incomplete"); + || records.size !== commandKeys.length + eventKeys.length + 2 + (isRecord(header.state.stagedRunAttachment) && header.state.stagedRunAttachment.commandId === null ? 1 : 0)))) throw new Error("native_runner_checkpoint_incomplete"); const state = { ...header.state, ackedSourceSeq: cursor.ackedSourceSeq, replayDeliveries: cursor.replayDeliveries, - commands: commandKeys.map(read), committedEvents: eventKeys.map(read) }; + commands: commandKeys.map(read), committedEvents: eventKeys.map(read) } as unknown as StoredCoreState; + if (state.stagedRunAttachment?.commandId === null) { + const template = read("staged-attachment-template"); + if (!isRecord(template)) throw new Error("native_runner_checkpoint_invalid"); + state.stagedRunAttachment = { ...state.stagedRunAttachment, runAttachTemplate: template }; + } if (!isStoredCoreState(state, header.state.identity)) throw new Error("native_runner_checkpoint_invalid"); if (state.commands.some((command, index) => commandKeys[index] !== `command:${command.controllerSeq}`) || state.committedEvents.some((event, index) => eventKeys[index] !== `event:${event.sourceSeq}`)) { @@ -683,9 +734,9 @@ function warmTransitionReceipt( function validStoredWarmTransition(state: StoredCoreState): boolean { const transition = state.warmTransition; - if (!transition) return state.schema === coreStateSchema; + if (!transition) return state.schema === coreStateSchema || (state.schema === stagedCoreStateSchema && validStagedRunAttachment(state)); if ( - state.schema !== transitionCoreStateSchema || + (state.schema !== transitionCoreStateSchema && state.schema !== stagedCoreStateSchema) || !["awaiting_result", "prepared", "activated"].includes(transition.phase) || !isRecord(transition.receipt) || !isRecord(transition.command) || @@ -802,6 +853,76 @@ function unsettledSemanticInput( } } +function stagedCommandMatches(staged: Pick, command: DurableRecoveryCoreCommand): boolean { + if (command.type !== "run.attach" || command.commandId !== staged.commandId) return false; + const { paperclipNextAuthority, ...template } = command.payload; + return isRecord(paperclipNextAuthority) + && canonicalJson(paperclipNextAuthority.identity) === canonicalJson(staged.targetIdentity) + && `sha256:${canonicalDigest(template)}` === staged.templateDigest; +} + +function stagedAttachmentTemplate(state: StoredCoreState): Record | null { + const staged = state.stagedRunAttachment; + if (!staged) return null; + if (staged.commandId === null) return staged.runAttachTemplate ?? null; + const command = state.commands?.find(command => command.commandId === staged.commandId) + ?? state.warmTransition?.command ?? state.completedWarmTransition?.command; + if (!command || !stagedCommandMatches(staged, command)) return null; + const { paperclipNextAuthority: _authority, ...template } = command.payload; + return template; +} + +function stagedTargetActivated(state: StoredCoreState): boolean { + const staged = state.stagedRunAttachment, completed = state.completedWarmTransition; + return !!staged && !state.warmTransition && !!completed + && canonicalJson(state.identity) === canonicalJson(staged.targetIdentity) + && completed.receipt.commandId === staged.commandId + && canonicalJson(completed.receipt.oldIdentity) === canonicalJson(staged.sourceIdentity) + && canonicalJson(completed.receipt.newIdentity) === canonicalJson(staged.targetIdentity) + && stagedCommandMatches(staged, completed.command); +} + +function validStagedRunAttachment(state: StoredCoreState, headerOnly = false): boolean { + const staged = state.stagedRunAttachment; + if (staged === undefined) return state.schema !== stagedCoreStateSchema; + if (!(state.schema === stagedCoreStateSchema && isRecord(staged) + && Object.keys(staged).sort().join(",") === (headerOnly || staged.commandId !== null ? "commandId,sourceIdentity,targetIdentity,templateDigest" : "commandId,runAttachTemplate,sourceIdentity,targetIdentity,templateDigest") + && exactIdentity(staged.sourceIdentity) && exactIdentity(staged.targetIdentity) + && staged.sourceIdentity.runId !== staged.targetIdentity.runId + && ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId"].every(key => + staged.sourceIdentity[key as keyof DurableRecoveryIdentity] === staged.targetIdentity[key as keyof DurableRecoveryIdentity]) + && /^sha256:[0-9a-f]{64}$/.test(staged.templateDigest) + && (headerOnly || staged.commandId !== null || (isRecord(staged.runAttachTemplate) && isRecord(staged.runAttachTemplate.provider) + && staged.runAttachTemplate.paperclipNextAuthority === undefined + && `sha256:${canonicalDigest(staged.runAttachTemplate)}` === staged.templateDigest + && Buffer.byteLength(JSON.stringify(staged.runAttachTemplate)) <= maxCommandBytes)) + && (staged.commandId === null || (typeof staged.commandId === "string" && stableIdPattern.test(staged.commandId))))) return false; + const sourceCurrent = canonicalJson(state.identity) === canonicalJson(staged.sourceIdentity); + if (!sourceCurrent && canonicalJson(state.identity) !== canonicalJson(staged.targetIdentity)) return false; + if (state.warmTransition) { + const transition = state.warmTransition; + return staged.commandId !== null && stagedCommandMatches(staged, transition.command) + && canonicalJson(transition.receipt.oldIdentity) === canonicalJson(staged.sourceIdentity) + && canonicalJson(transition.receipt.newIdentity) === canonicalJson(staged.targetIdentity) + && (sourceCurrent ? transition.phase !== "activated" : transition.phase === "activated"); + } + if (!sourceCurrent) return stagedTargetActivated(state); + if (staged.commandId === null) return true; + // Header-only inspection cannot certify the separate command record. Full + // restoration below requires its exact tuple, payload and command ID. + return !Array.isArray(state.commands) || state.commands.some(command => stagedCommandMatches(staged, command)); +} + +/** Inspection is evidence only; the caller still owns recovery authorization. */ +export function inspectStagedRunAttachment(state: unknown, expectedTarget?: DurableRecoveryIdentity): StagedRunAttachment | null { + try { + if (!isRecord(state) || !exactIdentity(state.identity) || !isStoredCoreState(state, state.identity) + || !state.stagedRunAttachment || (expectedTarget && canonicalJson(state.stagedRunAttachment.targetIdentity) !== canonicalJson(expectedTarget))) return null; + const template = stagedAttachmentTemplate(state); + return template ? { ...structuredClone(state.stagedRunAttachment), runAttachTemplate: structuredClone(template) } : null; + } catch { return null; } +} + function isStoredCoreState( value: unknown, identity: DurableRecoveryIdentity, @@ -811,7 +932,7 @@ function isStoredCoreState( const events = value.committedEvents; if ( (value.schema !== coreStateSchema && - value.schema !== transitionCoreStateSchema) || + value.schema !== transitionCoreStateSchema && value.schema !== stagedCoreStateSchema) || canonicalJson(value.identity) !== canonicalJson(identity) || (value.checkpointEventPolicy !== undefined && value.checkpointEventPolicy !== "control-v1") || !isRecord(value.tickets) || @@ -828,6 +949,7 @@ function isStoredCoreState( ) { return false; } + if (!validStagedRunAttachment(value as unknown as StoredCoreState)) return false; if (!validStoredWarmTransition(value as unknown as StoredCoreState)) return false; if (value.completedWarmTransition !== undefined) { @@ -1758,6 +1880,7 @@ class AuthorityConnection { /** Authenticated, replay-safe PRP transport authority. Business operations are caller supplied. */ export class DurablePrpControlPlane { #identity: DurableRecoveryIdentity; + #stagedTargetCheckpointPublished: string | null = null; readonly #store: DurableCoreStore; #expectedRunnerVersion: string; #expectedRunnerDigest: string; @@ -1854,7 +1977,12 @@ export class DurablePrpControlPlane { cached.set(part, prior?.record === record ? prior : { record, command: JSON.parse(record) }); } this.#publicationCommands = cached; - return { ...header.state, ...cursor, commands: [...cached.values()].map(value => value.command), + const descriptor = header.state.stagedRunAttachment; + const workingStage = this.#store.state.stagedRunAttachment; + const staged = descriptor?.commandId === null ? { ...descriptor, + runAttachTemplate: workingStage?.templateDigest === descriptor.templateDigest && workingStage.runAttachTemplate + ? workingStage.runAttachTemplate : JSON.parse(snapshot.records.get("staged-attachment-template")!) as Record } : descriptor; + return { ...header.state, ...(staged ? { stagedRunAttachment: staged } : {}), ...cursor, commands: [...cached.values()].map(value => value.command), // Event records are immutable after insertion. Retain their array, not // another deep copy of the streaming payload window. committedEvents: this.#store.state.committedEvents }; @@ -2323,6 +2451,23 @@ export class DurablePrpControlPlane { return ticket; } + /** Import exact durable handoff evidence for a caller-verified surviving participant; issues no credential. */ + prepareWarmTransitionAdoption(input: { transitionId: string; runnerState: Record }): void { + this.#store.assertWritable(); + const pending = input.runnerState.warmTransition; + const proof = warmTransitionRecoveryProof({ controlPlaneState: this.#store.state, runnerState: input.runnerState, + expectedNewIdentity: (isRecord(pending) && isRecord(pending.receipt) ? pending.receipt.newIdentity : null) as DurableRecoveryIdentity, + expectedRunnerVersion: this.#expectedRunnerVersion, expectedRunnerDigest: this.#expectedRunnerDigest }); + if (!proof || proof.transition.receipt.transitionId !== input.transitionId || !this.#store.state.stagedRunAttachment) { + throw new Error("Warm transition adoption snapshot proof is not authorized."); + } + if (this.#store.state.warmTransition && canonicalJson(this.#store.state.warmTransition) === canonicalJson(proof.transition)) return; + const candidate = structuredClone(this.#store.state); + candidate.schema = stagedCoreStateSchema; + candidate.warmTransition = structuredClone(proof.transition); + this.#store.commit(candidate); + } + /** Caller-owned recovery admission is required; a receipt is not a credential. */ issueWarmTransitionBootstrapTicket( input: { @@ -2361,7 +2506,7 @@ export class DurablePrpControlPlane { original.expiresAtUnixMs, ); const candidate = structuredClone(this.#store.state); - candidate.schema = transitionCoreStateSchema; + candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : transitionCoreStateSchema; candidate.warmTransition = structuredClone(transition); candidate.tickets[material.credentialId] = { recordId: `bootstrap_ticket_${randomUUID()}`, @@ -2380,6 +2525,36 @@ export class DurablePrpControlPlane { return ticket; } + /** Caller confirms its existing session-checkpoint transaction committed. No journal write. */ + markStagedTargetCheckpointPublished(identity: DurableRecoveryIdentity): boolean { + if (!stagedTargetActivated(this.#store.state) || canonicalJson(identity) !== canonicalJson(this.#store.state.identity)) return false; + this.#stagedTargetCheckpointPublished = canonicalJson(identity); + return true; + } + + /** Durable intent precedes source restoration; it is never a dispatched command. */ + stageRunAttachment(targetIdentity: DurableRecoveryIdentity, runAttachTemplate: Record): void { + this.#store.assertWritable(); + const existing = this.#store.state.stagedRunAttachment; + if (existing && !(stagedTargetActivated(this.#store.state) + && this.#stagedTargetCheckpointPublished === canonicalJson(this.#store.state.identity) + && targetIdentity.runId !== existing.targetIdentity.runId)) { + if (!validStagedRunAttachment(this.#store.state) + || canonicalJson(existing.targetIdentity) !== canonicalJson(targetIdentity) + || canonicalJson(stagedAttachmentTemplate(this.#store.state)) !== canonicalJson(runAttachTemplate)) { + throw new Error("Staged run attachment conflicts with its durable intent."); + } + return; + } + const candidate = structuredClone(this.#store.state); + candidate.schema = stagedCoreStateSchema; + candidate.stagedRunAttachment = { sourceIdentity: structuredClone(candidate.identity), + targetIdentity: structuredClone(targetIdentity), runAttachTemplate: structuredClone(runAttachTemplate), + templateDigest: `sha256:${canonicalDigest(runAttachTemplate)}`, commandId: null }; + if (!validStagedRunAttachment(candidate)) throw new Error("Staged run attachment binding is invalid."); + this.#store.commit(candidate); + } + queueCommand( type: string, payload: Record = {}, @@ -2387,6 +2562,21 @@ export class DurablePrpControlPlane { deliverImmediately = false, ): DurableRecoveryCoreCommand { this.#store.assertWritable(); + const staged = this.#store.state.stagedRunAttachment; + const retireStaged = stagedTargetActivated(this.#store.state) && (type === "turn.start" + || (this.#stagedTargetCheckpointPublished === canonicalJson(this.#store.state.identity) + && !["runner.drain", "session.snapshot", "turn.stop", "runner.suspend", "session.close"].includes(type))); + if (staged && !retireStaged) { + if (type === "run.attach") { + const { paperclipNextAuthority, ...template } = payload; + if (!isRecord(paperclipNextAuthority) + || canonicalJson(paperclipNextAuthority.identity) !== canonicalJson(staged.targetIdentity) + || `sha256:${canonicalDigest(template)}` !== staged.templateDigest + || (staged.commandId !== null && staged.commandId !== commandId)) throw new Error("Staged run attachment command conflicts."); + } else if (!stagedTargetActivated(this.#store.state) && !["runner.drain", "session.open", "session.snapshot", "turn.stop", "runner.suspend", "session.close", "semantic_tool.result"].includes(type)) { + throw new Error("Staged run attachment permits recovery maintenance only."); + } + } const transition = this.#store.state.warmTransition; if (transition && transition.phase !== "activated") { if ( @@ -2461,6 +2651,16 @@ export class DurablePrpControlPlane { if (Buffer.byteLength(JSON.stringify(command)) > maxCommandBytes) throw new CommandJournalLimitError("command_payload_too_large"); this.#store.state.commands.push(command); + if (staged && !retireStaged && type === "run.attach") { + const { runAttachTemplate: _template, ...descriptor } = staged; + this.#store.state.stagedRunAttachment = { ...descriptor, commandId: command.commandId }; + } else if (retireStaged) { + // Native runtime persists the recovered session before its first turn. + // Read/goal controls during opening must not retire startup evidence. + delete this.#store.state.stagedRunAttachment; + this.#store.state.schema = coreStateSchema; + this.#stagedTargetCheckpointPublished = null; + } this.#store.save(); void this.flushCheckpoint().catch(() => undefined); if (deliverImmediately) { @@ -2677,7 +2877,7 @@ export class DurablePrpControlPlane { } if (transition) { const candidate = structuredClone(this.#store.state); - candidate.schema = coreStateSchema; + candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : coreStateSchema; candidate.leases[transition.credentialId]!.identity = structuredClone( receipt.newIdentity, ); @@ -3205,8 +3405,9 @@ export class DurablePrpControlPlane { } if (transition.phase === "prepared") { const candidate = initialCoreState(transition.receipt.newIdentity); + candidate.stagedRunAttachment = structuredClone(this.#store.state.stagedRunAttachment); candidate.preparedLaunchConfigurations = structuredClone(this.#store.state.preparedLaunchConfigurations); - candidate.schema = transitionCoreStateSchema; + candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : transitionCoreStateSchema; candidate.warmTransition = { ...structuredClone(transition), phase: "activated", @@ -3507,7 +3708,7 @@ export class DurablePrpControlPlane { )!; completed.status = "completed"; completed.result = structuredClone(result); - candidate.schema = transitionCoreStateSchema; + candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : transitionCoreStateSchema; candidate.warmTransition = { receipt, phase: "prepared", diff --git a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts index f94fa464082..526d628d9f7 100644 --- a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts +++ b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts @@ -36,6 +36,10 @@ export interface RunnerExecutionDriver { checkpoint?: (settlement: "settled" | "unsettled") => Promise | void; release: (reason?: "detach") => Promise | void; }>; + /** Register source and target checkpoint ownership before staging changes the stored run. */ + onStagedRunAttachment?: (authority: DurablePrpControlPlane, proof: { + sourceIdentity: DurableRecoveryIdentity; targetIdentity: DurableRecoveryIdentity; templateDigest: string; + }) => Promise; /** Override process ownership; omission uses the SDK's local process owner. */ runnerProcessLauncher?: ( spec: RunnerProcessLaunchSpec, diff --git a/packages/paperclip-runner/src/index.ts b/packages/paperclip-runner/src/index.ts index 264fb7ed4ed..9f628c39f0a 100644 --- a/packages/paperclip-runner/src/index.ts +++ b/packages/paperclip-runner/src/index.ts @@ -38,6 +38,9 @@ export { RUNNER_PRP_PROTOCOL_MIN, RUNNER_PRP_PROTOCOL_MAX, inspectWarmRunTransition, + inspectStagedRunAttachment, + inspectStagedRunAttachmentHeader, + type StagedRunAttachment, type DurablePrpControlPlaneOptions, type DurablePrpCheckpoint, decodeDurablePrpCheckpoint, diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index 68fc16f53f7..b1749f4653a 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -7646,6 +7646,122 @@ it("rotates PRP authority in place for a warm cross-run attachment", async () => } }, 30_000); +it("recovers an active second warm business run with its opaque attachment identity", async () => { + const root = await mkdtemp(join(tmpdir(), "runnerd-warm-second-active-")); + const firstRoot = join(root, "first-controller"), nextRoot = join(root, "next-controller"); + const server = createServer(); + let authority: DurablePrpControlPlane | null = null, generation = 0; + let checkpoint: durableControlPlane.DurablePrpCheckpoint | undefined; + const registered = new WeakSet(); + server.on("upgrade", (request, socket, head) => authority + ? authority.handleUpgrade(request, socket, "/runner", head) : socket.destroy()); + await new Promise(done => server.listen(0, "127.0.0.1", done)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Missing owned Runner route"); + const registration = async (core: DurablePrpControlPlane) => { + if (!registered.has(core)) { + core.setCheckpointPersistence(async value => { checkpoint = value; }); + registered.add(core); + } + authority = core; const current = ++generation; + return { connectUrl: `ws://127.0.0.1:${address.port}/runner`, release: async () => { if (generation === current) authority = null; } }; + }; + const initialIdentity = { runnerInstanceId: "runner-warm-second-active", environmentLeaseId: "lease-warm-second-active", + runId: "run-warm-first-healthy", normalizedSessionId: "session-warm-second-active", turnId: "turn-first", itemId: "item-first" }; + const options = { runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex, + codexArgs: fakeCodexArgs(root, "--emit-tool-call", "--durable-tool-ids"), stateDirectory: firstRoot, + prpIdentity: initialIdentity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 }, controlPlaneRegistration: registration }; + const first = createCapabilityRunnerdCodexTransport(options); + const within = async (label: string, promise: Promise) => { + let timer: ReturnType | undefined; + try { return await Promise.race([promise, new Promise((_, reject) => { timer = setTimeout(() => reject(new Error(`warm active recovery stage timed out: ${label}`)), 5_000); })]); } + finally { clearTimeout(timer); } + }; + const waitComplete = async (transport: typeof first.transport) => { + for await (const notification of transport.notifications()) if (notification.method === "turn/completed") return; + }; + let calls = 0, runnerPid: number | null = null, providerPid: number | null = null; + let recovered: ReturnType | undefined; + const result = { ok: true, task: { id: "task-1" } }; + first.transport.setServerRequestHandler(async () => ++calls === 1 + ? { success: true, contentItems: [{ type: "inputText", text: JSON.stringify(result) }] } + : new Promise>(() => undefined)); + try { + const opened = await within("first thread/start", first.transport.request("thread/start", { cwd: tmpdir(), dynamicTools: [{ name: "get_task_context", description: "Read current task", + inputSchema: { type: "object", properties: {}, additionalProperties: false } }] })); + runnerPid = first.evidence().runnerPid; + providerPid = first.evidence().codexPid; + expect(providerPid).toEqual(expect.any(Number)); + const providerStart = execFileSync("ps", ["-p", String(providerPid), "-o", "lstart="], { encoding: "utf8" }).trim(); + const startedAt = first.transport.processInfo().startedAt; + const firstCompleted = waitComplete(first.transport); + await within("first turn/start", first.transport.request("turn/start", { input: [{ type: "text", text: "first healthy business run" }] })); + await within("healthy first turn completion", firstCompleted); + const attachment = { runId: "run-warm-second-active", turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` }; + await within("warm second business attachment", first.transport.attachRun!(attachment)); + const active = await within("second active turn/start", first.transport.request("turn/start", { input: [{ type: "text", text: "second business run held at tool" }] })); + await vi.waitFor(() => expect(calls).toBe(2), { timeout: 5_000 }); + const activeCore = authority!; + const savedIdentity = { ...activeCore.store.state.identity }; + expect(savedIdentity).toEqual({ ...initialIdentity, ...attachment }); + await activeCore.flushCheckpoint(); + await within("controller detach", first.detachControllerForRestart()); + expect(() => process.kill(runnerPid!, 0)).not.toThrow(); + checkpoint!.restoreSessionDirectory(nextRoot); + const noSpawn = vi.fn(() => { throw new Error("Duplicate Runner launch"); }); + const openedThread = opened.thread as Record; + const activeTurn = active.turn as Record; + expect(openedThread.id).toEqual(expect.any(String)); + expect(activeTurn.id).toEqual(expect.any(String)); + const shared = { ...options, stateDirectory: nextRoot, + resumeProviderSession: { driverSessionId: String(openedThread.id), providerSessionId: String(openedThread.sessionId) }, + resumeActiveTurnId: String(activeTurn.id), runnerArtifact: readRunnerdArtifactBinding(options.runnerBinary), + runnerStateDirectory: join(firstRoot, "runner"), readRunnerState: async () => JSON.parse(await readFile(join(firstRoot, "runner/runner-state.json"), "utf8")), + prepareExternalRunnerState: async () => {}, archiveExternalRunnerState: async () => { throw new Error("Active authority must not rotate"); }, + runnerProcessLauncher: noSpawn, adoptExistingRunner: { pid: runnerPid!, processGroupId: runnerPid, startedAt, + isAlive: () => { try { process.kill(runnerPid!, 0); return true; } catch { return false; } }, + signal: (signal: NodeJS.Signals) => { try { process.kill(runnerPid!, signal); return true; } catch { return false; } } } }; + // The old server's synthetic pair must continue failing the strict Runner guard. + const wrong = createCapabilityRunnerdCodexTransport({ ...shared, prpIdentity: { ...savedIdentity, + turnId: `turn-${attachment.runId}`, itemId: `item-${attachment.runId}` } }); + await expect(wrong.transport.request("thread/read", {})).rejects.toThrow("PRP recovery identity does not match the durable session binding"); + await wrong.detachControllerForRestart(); + // Leave the pending tool outcome unknown. Reattachment must authenticate + // and preserve it without fabricating a receipt or redispatching its handler. + recovered = createCapabilityRunnerdCodexTransport({ ...shared, prpIdentity: savedIdentity }); + await within("replacement thread/read", recovered.transport.request("thread/read", {})); + expect(recovered.evidence()).toMatchObject({ runnerPid, runnerExited: false }); + expect(authority!.activeRunnerConnectionCount()).toBe(1); + expect(() => process.kill(providerPid!, 0)).not.toThrow(); + expect(execFileSync("ps", ["-p", String(providerPid), "-o", "lstart="], { encoding: "utf8" }).trim()).toBe(providerStart); + expect(calls).toBe(2); + expect(authority!.store.state.commands.filter(command => command.type === "semantic_tool.result")).toHaveLength(0); + expect(recovered.transport.processInfo().startedAt).toBe(startedAt); + expect(authority!.store.state.identity).toEqual(savedIdentity); + expect(authority!.store.state.commands.filter(command => command.type === "turn.start")).toHaveLength(1); + expect(noSpawn).not.toHaveBeenCalled(); + } catch (error) { + const saved = JSON.parse(await readFile(join(firstRoot, "runner/runner-state.json"), "utf8").catch(() => "{}")); + const stderr = await readFile(join(firstRoot, "diagnostics/runnerd.stderr.log"), "utf8").catch(() => ""); + throw new Error(`${String(error)}; owned protocol evidence ${JSON.stringify({ + transport: recovered?.evidence(), connections: authority?.activeRunnerConnectionCount(), + identity: authority?.store.state.identity, commands: authority?.store.state.commands.map(c => ({type:c.type,status:c.status})), + events: authority?.store.state.committedEvents.map(e => e.eventType).slice(-8), + runner: { lifecycle:saved.lifecycle, runId:saved.runId, turnId:saved.turnId, itemId:saved.itemId, + outbox:saved.outbox?.length }, stderr:stderr.slice(-1500), + })}`); + } finally { + await (recovered ? within("recovered cleanup", recovered.transport.close()).catch(() => undefined) : undefined); + await within("first cleanup", first.transport.close()).catch(() => undefined); + if (providerPid) { try { process.kill(providerPid, "SIGKILL"); } catch {} } + await (authority as DurablePrpControlPlane | null)?.stop(); + if (runnerPid) { try { process.kill(-runnerPid, "SIGKILL"); } catch {} } + server.closeAllConnections(); + if (server.listening) await new Promise(done => server.close(() => done())); + await rm(root, { recursive: true, force: true }); + } +}, 30_000); + it("waits for a warm runner to re-authenticate before probing attachment readiness", async () => { const stateDirectory = await mkdtemp( join(tmpdir(), "runnerd-warm-reattach-before-probe-"), @@ -8123,7 +8239,7 @@ it("still fails closed when a real close grace period cannot fit a durable suspe } }, 30_000); -it("cold-restores a suspended provider session under its durable run binding", async () => { +it.each([false, true, "restart-before-bootstrap", "unknown-semantic", "accounting-rejected"] as const)("cold-restores a suspended provider session under its durable run binding (pending tail: %s)", async (pendingTail) => { const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-cold-attach-")); const tracePath = join(stateDirectory, "provider-trace.ndjson"); const skillRoot = join(stateDirectory, "runtime-skill"); @@ -8216,14 +8332,57 @@ it("cold-restores a suspended provider session under its durable run binding", a throw new Error("cold attach fixture did not return a provider thread"); } + // Model a retained provider-owned suffix: rotation must deliver and durably + // accept it before attempting run.attach, rather than dropping or ACKing it. + const retainedDelta = "cold-recovery-retained-tail"; + if (pendingTail) { + const providerFile = join(stateDirectory, "runner", "codex-provider-state.json"); + const provider = JSON.parse(await readFile(providerFile, "utf8")); + expect(provider.pendingEvents).toEqual([]); + provider.pendingEvents.push({ + executorEventId: `codex_provider_${String(provider.nextProviderEventSeq++).padStart(16, "0")}`, + eventType: pendingTail === "unknown-semantic" ? "semantic_tool.input" : "usage.reported", + priority: "p0", + payload: pendingTail === "unknown-semantic" ? { semantic_tool: { + schema: "paperclip.prp.semantic_tool.v1", schemaVersion: 1, phase: "input", + callId: "unknown-source-call", operationId: "get_task_context", input: {}, idempotencyKey: null, + correlation: { runId: baseIdentity.runId, normalizedSessionId: baseIdentity.normalizedSessionId, + turnId: baseIdentity.turnId, itemId: baseIdentity.itemId }, + content: { digest: `sha256:${createHash("sha256").update("{}").digest("hex")}`, redactionDisposition: "digest_only", references: [] }, + } } : { provider: "codex", kind: "usage", delta: retainedDelta, + providerTurnId: provider.completed_provider_turn_id, runDelta: { inputTokens: 20, outputTokens: 2 }, + runDeltaAvailable: true, runDeltaBasis: "run" }, + }); + await writeFile(providerFile, JSON.stringify(provider)); + } + const retainedEvents: PrpEvent[] = []; + let releaseRetained!: () => void, retainedEntered!: () => void; + const retainedAccepted = new Promise(done => { releaseRetained = done; }); + const retainedAdmission = new Promise(done => { retainedEntered = done; }); const secondIdentity = { ...baseIdentity, runId: "run-cold-second", turnId: "turn-cold-second", itemId: "item-cold-second", }; - const rotated = createCapabilityRunnerdCodexTransport({ + let stagedCheckpoint: durableControlPlane.DurablePrpCheckpoint | undefined; + let rejectStagedCommit = pendingTail === "restart-before-bootstrap"; + const rotatedOptions = { ...options, + onStagedRunAttachment: pendingTail === "restart-before-bootstrap" ? async (authority: DurablePrpControlPlane) => { + authority.setCheckpointPersistence(async checkpoint => { + stagedCheckpoint = checkpoint; + if (rejectStagedCommit) throw new Error("fixture lost controller after durable intent"); + }); + } : undefined, + onRecoveryAccountingEvent: pendingTail ? async (event: PrpEvent) => { + if (event.payload.delta === retainedDelta) { + retainedEvents.push(structuredClone(event)); + retainedEntered(); + if (pendingTail === "accounting-rejected") throw new Error("source accounting receipt unavailable"); + await retainedAccepted; + } + } : undefined, environment: { ...options.environment, PAPERCLIP_GITHUB_BROKER_TOKEN: "test-run-B-capability" }, codexArgs: options.codexArgs.map((arg) => arg.replace('/run/A', '/run/B')), resumeDynamicTools: dynamicTools, @@ -8232,14 +8391,81 @@ it("cold-restores a suspended provider session under its durable run binding", a criterionIds: ["criterion-second"], }, prpIdentity: secondIdentity, - }); - rotated.transport.setServerRequestHandler(async () => ({ - success: true, - contentItems: [], - })); + }; + let rotated = createCapabilityRunnerdCodexTransport(rotatedOptions); + if (pendingTail === "restart-before-bootstrap") { + await expect(rotated.transport.request("thread/read", {})).rejects.toThrow("fixture lost controller after durable intent"); + expect(rotated.evidence().runnerPid).toBeNull(); + expect(stagedCheckpoint?.runId).toBe(secondIdentity.runId); + expect([...stagedCheckpoint!.records].filter(([key]) => key.startsWith("command:")).map(([, value]) => JSON.parse(value).type)).not.toContain("run.attach"); + await rotated.detachControllerForRestart().catch(() => undefined); + rejectStagedCommit = false; + await rm(join(stateDirectory, "control-plane"), { recursive: true, force: true }); + stagedCheckpoint!.restoreSessionDirectory(stateDirectory); + rotated = createCapabilityRunnerdCodexTransport(rotatedOptions); + } + const currentHandler = vi.fn(async () => ({ success: true, contentItems: [] })); + rotated.transport.setServerRequestHandler(currentHandler); + if (pendingTail === "accounting-rejected") { + const reading = rotated.transport.request("thread/read", {}); + void reading.catch(() => undefined); + try { + await Promise.race([retainedAdmission, reading]); + const source = JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8")); + expect(source.identity).toEqual(baseIdentity); + expect(source.stagedRunAttachment.targetIdentity).toEqual(secondIdentity); + expect(source.commands.some((command: { type: string }) => command.type === "run.attach")).toBe(false); + const retainedRunner = JSON.parse(await readFile(join(stateDirectory, "runner", "runner-state.json"), "utf8")); + expect(retainedRunner.outbox.some((entry: { envelope: { payload: PrpEvent } }) => + entry.envelope.payload.sourceEventId === retainedEvents[0]!.sourceEventId)).toBe(true); + expect(currentHandler).not.toHaveBeenCalled(); + } finally { + await expect(rotated.transport.close()).rejects.toMatchObject({ code: "native_session_close_unrecoverable" }); + await expect(reading).rejects.toThrow(); + await releaseMaterializedNativeRuntimeSkills(join(stateDirectory, "codex-home", "skills")); + await rm(stateDirectory, { recursive: true, force: true }); + } + return; + } + if (pendingTail === "unknown-semantic") { + try { + await expect(rotated.transport.request("thread/read", {})).rejects.toThrow("native_runner_source_semantic_reconciliation_required"); + expect(currentHandler).not.toHaveBeenCalled(); + const retained = JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8")); + expect(retained.identity).toEqual(baseIdentity); + expect(retained.stagedRunAttachment.targetIdentity).toEqual(secondIdentity); + expect(retained.commands.some((command: { type: string }) => ["run.attach", "semantic_tool.result"].includes(command.type))).toBe(false); + expect(retained.committedEvents.some((event: PrpEvent) => event.eventType === "semantic_tool.input")).toBe(true); + } finally { + await expect(rotated.transport.close()).rejects.toMatchObject({ code: "native_session_close_unrecoverable" }); + await releaseMaterializedNativeRuntimeSkills(join(stateDirectory, "codex-home", "skills")); + await rm(stateDirectory, { recursive: true, force: true }); + } + return; + } try { - const read = await rotated.transport.request("thread/read", {}); + const reading = rotated.transport.request("thread/read", {}); + void reading.catch(() => undefined); + if (pendingTail) { + await Promise.race([retainedAdmission, reading]); + const retainedRunner = JSON.parse(await readFile(join(stateDirectory, "runner", "runner-state.json"), "utf8")); + expect(retainedRunner.outbox.some((entry: { envelope: { payload: PrpEvent } }) => + entry.envelope.payload.sourceEventId === retainedEvents[0]!.sourceEventId)).toBe(true); + const retainedCore = stagedCheckpoint ? { + identity: JSON.parse(stagedCheckpoint.records.get("header")!).state.identity, + commands: [...stagedCheckpoint.records].filter(([key]) => key.startsWith("command:")).map(([, value]) => JSON.parse(value)), + } : JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8")); + expect(retainedCore.identity).toEqual(baseIdentity); + expect(retainedCore.commands.some((command: { type: string }) => command.type === "run.attach")).toBe(false); + } + releaseRetained(); + const read = await reading; + expect(retainedEvents).toHaveLength(pendingTail ? 1 : 0); + if (pendingTail) expect(retainedEvents[0]).toMatchObject({ + runId: baseIdentity.runId, turnId: baseIdentity.turnId, itemId: baseIdentity.itemId, + }); const persistedProvider = JSON.parse(await readFile(join(stateDirectory, "runner", "codex-provider-state.json"), "utf8")); + expect(persistedProvider.pendingEvents).toEqual([]); expect(persistedProvider.config.args.join("\n")).toContain('/run/B'); expect(JSON.stringify(persistedProvider)).not.toContain("test-run-B-capability"); expect(read.thread).toMatchObject({ @@ -8256,8 +8482,14 @@ it("cold-restores a suspended provider session under its durable run binding", a expect(rotated.evidence().diagnostics).toContain( "runnerd attached the durable provider session to a fresh PRP run authority", ); - expect(await stat(join(stateDirectory, "authority-epochs"))).toBeDefined(); + if (pendingTail) { + const target = stagedCheckpoint ? JSON.parse(stagedCheckpoint.records.get("header")!).state + : JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8")); + expect(target.stagedRunAttachment).toBeUndefined(); + expect(target.completedWarmTransition.receipt).toMatchObject({ oldIdentity: baseIdentity, newIdentity: secondIdentity }); + } else expect(await stat(join(stateDirectory, "authority-epochs"))).toBeDefined(); } finally { + releaseRetained(); await rotated.transport.close(); } const requestFrames = (await readFile(tracePath, "utf8")) @@ -8294,6 +8526,12 @@ it("cold-restores a suspended provider session under its durable run binding", a ); } + if (pendingTail) { + await releaseMaterializedNativeRuntimeSkills(join(stateDirectory, "codex-home", "skills")); + await rm(stateDirectory, { recursive: true, force: true }); + return; + } + // A remote process owner keeps runner-state outside the controller's local // session root. Resume must defer to its explicit state reader instead of // rejecting recovery before the remote checkpoint can be made available. diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index c7df8fe8bc9..811133f5b0a 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -69,6 +69,7 @@ import type { } from "../contracts/harness-driver.js"; import { DurablePrpControlPlane, + inspectStagedRunAttachment, SemanticToolNotDispatchedError, durableRecoveryInternals, inspectWarmRunTransition, @@ -1241,6 +1242,7 @@ export interface CapabilityRunnerdCodexTransportOptions extends RunnerExecutionD observeCredentials?: boolean; onCredentialsUpdated?: (event: PrpEvent) => Promise; onAccountingEvent?: (event: PrpEvent) => Promise; + onRecoveryAccountingEvent?: (event: PrpEvent, sourceIdentity: DurableRecoveryIdentity) => Promise; recoverSemanticToolInput?: DurablePrpControlPlaneOptions["recoverSemanticToolInput"]; opencodePermissionMode?: NativeOpenCodePermissionMode; acpxAgent?: QualifiedAcpxAgent; @@ -1379,7 +1381,7 @@ export interface CapabilityRunnerdCodexTransportOptions extends RunnerExecutionD itemId: string; }; /** Existing server-owned routes only; never provision two ingress owners. */ - warmTransitionRegistrationMode?: "routed_connect"; + warmTransitionRegistrationMode?: "routed_connect" | "routed_same_endpoint"; /** Rechecks the server-owned recovery claim after each asynchronous boundary. */ authorizeWarmTransitionRecovery?: ( stage: "before_bootstrap" | "before_spawn" | "before_authentication", @@ -3563,6 +3565,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { #pump: NodeJS.Timeout | null = null; #eventSourceSeq = 0; #eventIdentity: DurableRecoveryIdentity | null = null; + #sourceRecoveryIdentity: DurableRecoveryIdentity | null = null; + #sourceAttachmentDeadline: number | null = null; #pendingWarmRecoveryCompletion: { transitionId: string; identity: DurableRecoveryIdentity; @@ -3794,6 +3798,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ): Promise> { if (this.#closed) throw new Error("PRP Codex transport is closed"); this.#throwIfFailed(); + if (this.#sourceRecoveryIdentity && !["initialize", "collaborationMode/list"].includes(method)) { + throw new Error("native_runner_source_attachment_pending"); + } if ( this.#pendingWarmRecoveryCompletion !== null && !["thread/read", "initialize", "collaborationMode/list"].includes(method) @@ -4103,7 +4110,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } #recoverPendingSemanticInputsWhenReady(): void { - if (this.#startupComplete && this.#handlerInstalled) { + if (this.#startupComplete && this.#handlerInstalled && this.#sourceRecoveryIdentity === null && this.#pendingWarmRecoveryCompletion === null) { void Promise.resolve().then(() => this.#core?.recoverPendingSemanticInputs()) .catch(error => this.#failTransport(error instanceof Error ? error : new Error(String(error)))); } @@ -4113,7 +4120,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { // Remote runner ingress already has a bounded reconnect budget. Reuse the // same budget here so a transient tunnel reconnect cannot trip the shorter // generic command timeout and replace an otherwise healthy warm runner. - const reconnectGraceMs = this.options.runnerReconnectGraceMs ?? 5_000; + const reconnectGraceMs = Math.min(this.options.runnerReconnectGraceMs ?? 5_000, + this.#sourceAttachmentDeadline === null ? Infinity : Math.max(0, this.#sourceAttachmentDeadline - Date.now())); await waitForWarmAttachmentReadiness({ graceMs: reconnectGraceMs, waitForConnection: (deadline) => this.#awaitWarmRunnerConnection(deadline), @@ -4243,7 +4251,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { .update(`${prior.runId}:${desired.runId}:${desired.turnId}`) .digest("hex") .slice(0, 32)}`; - const runAttachTemplate = this.#runAttachTemplate + const stagedCandidate = inspectStagedRunAttachment(core.store.state); + const staged = stagedCandidate && recoveryIdentityMatches(stagedCandidate.targetIdentity, desired) ? stagedCandidate : undefined; + const runAttachTemplate = staged ? structuredClone(staged.runAttachTemplate) : this.#runAttachTemplate ? retargetRunAttachPayload( this.#runAttachTemplate, desired, @@ -4260,7 +4270,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { currentRunGrant?.runtimeContext, currentRunGrant ? { text: currentRunGrant.instructions, context: currentRunGrant.runtimeContext } : undefined, ); - if (this.options.prepareLaunch) { + if (this.options.prepareLaunch && !staged) { const priorProvider = record(runAttachTemplate.provider); const priorRequirements = new RunnerLaunchPreparationStore(this.#root).load(prior)?.requirements; await this.#prepareOwnedProvider(runAttachTemplate, desired, @@ -4270,6 +4280,11 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } this.#assertStartupOpen(); this.#runAttachTemplate = structuredClone(runAttachTemplate); + if (!staged && (this.options.provider ?? "codex") === "codex" && !this.options.prepareLaunch && this.options.onStagedRunAttachment) { + // Registration already installed the target writer. Stage and queue + // share the existing attachment checkpoint; no duplicate owner reads. + core.stageRunAttachment(desired, runAttachTemplate); + } const payload = { ...runAttachTemplate, paperclipNextAuthority: { identity: desired, connection, @@ -4285,7 +4300,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await this.#waitCommand( "run.attach", commandId, - this.#coldAdmissionDeadline(), + this.#sourceAttachmentDeadline === null ? this.#coldAdmissionDeadline() + : Math.min(Date.now() + 30_000, this.#sourceAttachmentDeadline), true, ); const attached = core.getCommand(commandId); @@ -4295,6 +4311,21 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { activationStarted = true; core.rotateRunIdentity(desired, runAttachTemplate); + if (core.store.state.stagedRunAttachment && registration?.connection?.mode === "listen" && this.options.readRunnerState) { + const deadline = Math.min(Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000), this.#sourceAttachmentDeadline ?? Infinity); + const artifact = this.options.runnerArtifact ?? approvedRunnerArtifact(this.options.runnerBinary ?? defaultCapabilityRunnerdBinary()); + for (;;) { + this.#assertStartupOpen(); + const physical = await this.options.readRunnerState(); + const proof = inspectWarmRunTransition({ controlPlaneState: core.store.state, runnerState: physical, + expectedNewIdentity: desired, expectedRunnerVersion: artifact.version, expectedRunnerDigest: artifact.digest }); + if (proof && recoveryIdentityMatches(proof.runnerIdentity, desired)) { + await previousRelease?.(); previousReleased = true; break; + } + if (Date.now() >= deadline || await this.#runnerHasExited()) throw new Error("native_runner_warm_transition_activation_pending"); + await new Promise(done => setTimeout(done, 100)); + } + } await registration?.activate?.(); if (registration?.failure) { void registration.failure.catch((error: unknown) => { @@ -4304,13 +4335,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { }); } await this.#awaitRegistrationReady(registration?.ready); - const activationDeadline = - Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000); + const activationDeadline = Math.min( + Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000), this.#sourceAttachmentDeadline ?? Infinity); while ( !recoveryIdentityMatches(core.store.state.identity, desired) || core.store.state.warmTransition !== undefined || core.activeRunnerConnectionCount() === 0 ) { + this.#assertStartupOpen(); if ( Date.now() >= activationDeadline || (await this.#runnerHasExited()) @@ -4319,11 +4351,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } await new Promise((resolveWait) => setTimeout(resolveWait, 10)); } + this.#assertStartupOpen(); this.#eventIdentity = structuredClone(desired); this.#eventSourceSeq = 0; this.#deferredTurnStartEvents = []; this.#durableTurnId = desired.turnId; - await previousRelease?.(); + if (!previousReleased) await previousRelease?.(); previousReleased = true; this.#controlPlaneRelease = registration?.release ?? null; } catch (error) { @@ -5251,7 +5284,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { onSemanticToolInput: (call) => this.#handleSemanticToolInput(call), recoverSemanticToolInput: call => this.#recoverSemanticToolInput(call), onCommittedEvent: async (event) => { - await this.options.onAccountingEvent?.(event); + if (this.#sourceRecoveryIdentity && event.runId === this.#sourceRecoveryIdentity.runId) { + if (!this.options.onRecoveryAccountingEvent && this.options.onAccountingEvent) throw new Error("native_runner_source_accounting_unavailable"); + await this.options.onRecoveryAccountingEvent?.(event, this.#sourceRecoveryIdentity); + } else await this.options.onAccountingEvent?.(event); if (event.eventType === "auth.credentials.updated") { await this.options.onCredentialsUpdated?.(event); } @@ -5393,7 +5429,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { async #resumeWork(): Promise { if (this.options.acpxCandidateAdmission?.agent === "antigravity") throw new Error("ACPX candidate durable runner recovery is not admitted"); - const desiredIdentity = this.options.prpIdentity; + let desiredIdentity = this.options.prpIdentity; if (desiredIdentity === undefined) { throw new Error("PRP provider resume state is unavailable"); } @@ -5457,6 +5493,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ) { candidateRunnerState = await this.options.readRunnerState(); } + const staged = inspectStagedRunAttachment(controlPlaneState, desiredIdentity); + if (controlPlaneState?.stagedRunAttachment !== undefined && staged === null) { + throw new Error("native_runner_staged_attachment_binding_mismatch"); + } let warmRecovery: { runnerState: Record; runnerIdentity: DurableRecoveryIdentity; @@ -5482,8 +5522,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { !controlPlaneState || !candidateRunnerState || (this.options.controlPlaneRegistration !== undefined && - this.options.warmTransitionRegistrationMode !== "routed_connect") || - this.options.adoptExistingRunner !== undefined + this.options.warmTransitionRegistrationMode !== "routed_connect" + && !(staged && this.options.adoptExistingRunner && this.options.warmTransitionRegistrationMode === "routed_same_endpoint")) || + (this.options.adoptExistingRunner !== undefined && !staged) ) { throw new Error( "native_runner_warm_transition_requires_exact_owned_endpoint", @@ -5508,14 +5549,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ? new URL(connection.connectUrl) : null; if ( - connection.mode !== "connect" || - endpoint === null || - endpoint.search || - endpoint.hash || - endpoint.username || - endpoint.password || + (connection.mode !== "connect" && !(connection.mode === "listen" && staged + && this.options.adoptExistingRunner && this.options.controlPlaneRegistration + && this.options.warmTransitionRegistrationMode === "routed_same_endpoint")) || + (connection.mode === "connect" && (endpoint === null || endpoint.search || endpoint.hash || endpoint.username || endpoint.password)) || (!this.options.controlPlaneRegistration && - (endpoint.protocol !== "ws:" || + (endpoint === null || endpoint.protocol !== "ws:" || endpoint.hostname !== "127.0.0.1" || endpoint.pathname !== "/durableRecovery/connect" || !endpoint.port)) @@ -5529,7 +5568,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { transitionId: receipt.transitionId, commandId: receipt.commandId, ...(!this.options.controlPlaneRegistration - ? { port: Number(endpoint.port) } + ? { port: Number(endpoint!.port) } : {}), }; // Fence concurrent public work before exposing the core or awaiting @@ -5539,6 +5578,40 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { identity: structuredClone(desiredIdentity), }; } + const completedStaged = staged && recoveryIdentityMatches(identity, staged.targetIdentity) + && record(record(controlPlaneState?.completedWarmTransition).receipt).commandId === staged.commandId; + const queuedSourceAttachment = staged && staged.commandId !== null && warmRecovery === null + && recoveryIdentityMatches(identity, staged.sourceIdentity) ? staged : null; + if (queuedSourceAttachment && !this.options.adoptExistingRunner) { + throw new Error("native_runner_queued_attachment_requires_verified_live_source"); + } + let sourceAttachmentTarget: DurableRecoveryIdentity | null = null; + if (controlPlaneState && warmRecovery === null && (this.options.provider ?? "codex") === "codex" + && !this.options.prepareLaunch && (staged || this.options.onStagedRunAttachment || this.options.onRecoveryAccountingEvent) + && identity.runId !== desiredIdentity.runId) { + // Restore and acknowledge the source under its own authority. The target + // remains a durable, undispatched intent until every old obligation settles. + if (!this.options.adoptExistingRunner) { + if (localStateOwner) { + if (!candidateRunnerState) throw new Error("native_runner_authority_rotation_state_unavailable"); + assertSuspendedRunnerState(candidateRunnerState, identity); + } else { + if (!this.options.readRunnerState || !this.options.prepareExternalRunnerState) throw new Error("native_runner_prp_run_rotation_unavailable"); + await this.options.prepareExternalRunnerState(); + assertSuspendedRunnerState(await this.options.readRunnerState(), identity); + } + } + sourceAttachmentTarget = queuedSourceAttachment ? null : desiredIdentity; + desiredIdentity = identity; + this.#sourceRecoveryIdentity = structuredClone(identity); + this.#sourceAttachmentDeadline = Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000); + } else if (staged && !completedStaged && !warmRecovery) { + throw new Error("native_runner_staged_attachment_profile_unsupported"); + } + if (warmRecovery && staged) { + this.#sourceRecoveryIdentity = structuredClone(staged.sourceIdentity); + this.#sourceAttachmentDeadline = Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000); + } const exactAuthority = controlPlaneState !== null && identity.runnerInstanceId === desiredIdentity.runnerInstanceId && @@ -5704,7 +5777,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { onSemanticToolInput: (call) => this.#handleSemanticToolInput(call), recoverSemanticToolInput: call => this.#recoverSemanticToolInput(call), onCommittedEvent: async (event) => { - await this.options.onAccountingEvent?.(event); + if (this.#sourceRecoveryIdentity && event.runId === this.#sourceRecoveryIdentity.runId) { + if (!this.options.onRecoveryAccountingEvent && this.options.onAccountingEvent) throw new Error("native_runner_source_accounting_unavailable"); + await this.options.onRecoveryAccountingEvent?.(event, this.#sourceRecoveryIdentity); + } else await this.options.onAccountingEvent?.(event); if (event.eventType === "auth.credentials.updated") { await this.options.onCredentialsUpdated?.(event); } @@ -5775,14 +5851,15 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { preparedContext, preparedInstructions, saved?.prepared, saved?.requirements); this.#runAttachTemplate = structuredClone(template); } - if (rotatedAuthority) { + if (rotatedAuthority || sourceAttachmentTarget) { if (provider === "acpx" && this.options.acpxAgent === "copilot" && typeof this.options.baseInstructions !== "string") { throw new Error("native_runner_current_instruction_grant_unavailable"); } + const attachmentIdentity = sourceAttachmentTarget ?? desiredIdentity; const runAttachTemplate = rotatedRunAttachPayload( controlPlaneState, - desiredIdentity, + attachmentIdentity, this.#authorizedTools, this.options.resumeCompletionContract, runtimeContext, @@ -5823,8 +5900,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { runAttachTemplate.provider = environmentPrepared.provider; } this.#runAttachTemplate = structuredClone(runAttachTemplate); - core.persistRunAttachTemplate(runAttachTemplate); - core.queueCommand("run.attach", runAttachTemplate); + if (sourceAttachmentTarget) { + await this.options.onStagedRunAttachment?.(core, { sourceIdentity: desiredIdentity, + targetIdentity: sourceAttachmentTarget, templateDigest: commandDigest(runAttachTemplate) }); + core.stageRunAttachment(sourceAttachmentTarget, runAttachTemplate); + } else { + core.persistRunAttachTemplate(runAttachTemplate); + core.queueCommand("run.attach", runAttachTemplate); + } } const committedEvents = core.store.state.committedEvents; const runAttachment = recoveredRunAttachment(core.store.state); @@ -5836,7 +5919,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ? `command_resume_probe_${randomUUID().replaceAll("-", "")}` : null; if (recoveryProbeCommandId !== null) { - core.queueCommand("runner.drain", {}, recoveryProbeCommandId); + core.queueCommand(sourceAttachmentTarget ? "session.open" : "runner.drain", {}, recoveryProbeCommandId); } // A controller retry can open the exact authority after run.attach has // already reached a durable outcome. Re-observe that command instead of @@ -5885,6 +5968,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { : "restored provider identity from the exact durable checkpoint; awaiting live confirmation", ); } + if (staged && !sourceAttachmentTarget) { + await this.options.onStagedRunAttachment?.(core, { sourceIdentity: staged.sourceIdentity, + targetIdentity: staged.targetIdentity, templateDigest: staged.templateDigest }); + if (warmRecovery && this.options.adoptExistingRunner) core.prepareWarmTransitionAdoption({ + transitionId: warmRecovery.transitionId, runnerState: warmRecovery.runnerState }); + } type RecoveryRegistration = Awaited< ReturnType< NonNullable< @@ -5895,67 +5984,48 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { let oldTransitionRegistration: RecoveryRegistration | null = null; let newTransitionRegistration: RecoveryRegistration | null = null; this.#assertStartupOpen(); - if (warmRecovery && this.options.controlPlaneRegistration) { + const recoveringAttachment = warmRecovery || queuedSourceAttachment; + const queuedCommand = queuedSourceAttachment ? core.getCommand(queuedSourceAttachment.commandId!) : undefined; + const expectedRecoveryConnection = (warmRecovery?.proof.receipt.connection + ?? record(queuedCommand?.payload.paperclipNextAuthority).connection) as RunnerProcessConnection | undefined; + const recoverySource = warmRecovery?.proof.receipt.oldIdentity ?? queuedSourceAttachment?.sourceIdentity; + const recoveryTarget = warmRecovery?.proof.receipt.newIdentity ?? queuedSourceAttachment?.targetIdentity; + if (recoveringAttachment && this.options.controlPlaneRegistration) { try { - oldTransitionRegistration = await this.options.controlPlaneRegistration( - core, - warmRecovery.proof.receipt.oldIdentity, - ); - const oldConnection = oldTransitionRegistration.connection ?? { - mode: "connect", - connectUrl: oldTransitionRegistration.connectUrl, - }; - if ( - oldConnection.mode !== "connect" || - typeof oldConnection.connectUrl !== "string" - ) { - throw new Error( - "native_runner_warm_transition_registered_endpoint_mismatch", - ); + if (!recoverySource || !recoveryTarget || !expectedRecoveryConnection + || (expectedRecoveryConnection.mode !== "connect" && !(staged && this.options.adoptExistingRunner + && this.options.warmTransitionRegistrationMode === "routed_same_endpoint"))) { + throw new Error("native_runner_warm_transition_requires_exact_owned_endpoint"); } + oldTransitionRegistration = await this.options.controlPlaneRegistration(core, recoverySource); this.#assertStartupOpen(); - newTransitionRegistration = await this.options.controlPlaneRegistration( - core, - warmRecovery.proof.receipt.newIdentity, - ); - const newConnection = newTransitionRegistration.connection ?? { - mode: "connect", - connectUrl: newTransitionRegistration.connectUrl, - }; - if ( - newConnection.mode !== "connect" || - durableRecoveryInternals.canonicalJson(newConnection) !== - durableRecoveryInternals.canonicalJson( - warmRecovery.proof.receipt.connection, - ) - ) { - throw new Error( - "native_runner_warm_transition_registered_endpoint_mismatch", - ); + newTransitionRegistration = await this.options.controlPlaneRegistration(core, recoveryTarget); + const connection = newTransitionRegistration.connection ?? { mode: "connect", connectUrl: newTransitionRegistration.connectUrl }; + if (durableRecoveryInternals.canonicalJson(connection) !== durableRecoveryInternals.canonicalJson(expectedRecoveryConnection)) { + throw new Error("native_runner_warm_transition_registered_endpoint_mismatch"); } } catch (error) { - await Promise.allSettled( - [oldTransitionRegistration, newTransitionRegistration].map((entry) => - Promise.resolve().then(() => entry?.release()), - ), - ); + await Promise.allSettled([oldTransitionRegistration, newTransitionRegistration].map(entry => Promise.resolve().then(() => entry?.release()))); throw error; } } - const registration = - warmRecovery && oldTransitionRegistration && newTransitionRegistration - ? recoveryIdentityMatches( - warmRecovery.runnerIdentity, - warmRecovery.proof.receipt.oldIdentity, - ) - ? oldTransitionRegistration - : newTransitionRegistration - : this.options.controlPlaneRegistration - ? await this.options.controlPlaneRegistration(core) - : null; + const registration = oldTransitionRegistration && newTransitionRegistration + ? recoveryIdentityMatches(warmRecovery?.runnerIdentity ?? queuedSourceAttachment!.sourceIdentity, recoverySource!) + ? oldTransitionRegistration : newTransitionRegistration + : this.options.controlPlaneRegistration + ? await this.options.controlPlaneRegistration(core, this.#sourceRecoveryIdentity ?? undefined) : null; this.#startupFailureCode = registration?.startupFailureCode ?? "runner_local_connect_failed"; - if (registration === null) await core.start(warmRecovery?.port); + if (registration === null) { + let queuedPort: number | undefined; + if (queuedSourceAttachment) { + const endpoint = expectedRecoveryConnection?.mode === "connect" ? new URL(expectedRecoveryConnection.connectUrl) : null; + if (!endpoint || endpoint.protocol !== "ws:" || endpoint.hostname !== "127.0.0.1" || endpoint.pathname !== "/durableRecovery/connect" + || !endpoint.port || endpoint.search || endpoint.hash || endpoint.username || endpoint.password) throw new Error("native_runner_warm_transition_snapshot_mismatch"); + queuedPort = Number(endpoint.port); + } + await core.start(warmRecovery?.port ?? queuedPort); + } else { this.#controlPlaneCheckpoint = registration.checkpoint ?? null; this.#controlPlaneRelease = @@ -5976,7 +6046,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#assertStartupOpen(); // Route and immutable endpoint admission precedes every launch-material // write. A refused transition leaves its retained home/context untouched. - if (runtimeContext !== null && !this.options.prepareLaunch) { + if (runtimeContext !== null && !this.options.prepareLaunch && this.#sourceRecoveryIdentity === null) { writeFileSync( localRuntimeContextPath, `${JSON.stringify(runtimeContext)}\n`, @@ -5985,7 +6055,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } if (provider === "aws_agentcore" && !this.options.prepareLaunch) mkdirSync(codexHome, { recursive: true, mode: 0o700 }); - if (!this.options.prepareLaunch && provider === "codex" && this.options.homeMode !== "preconfigured") { + if (!this.options.prepareLaunch && provider === "codex" && this.options.homeMode !== "preconfigured" && this.#sourceRecoveryIdentity === null) { await releaseMaterializedNativeRuntimeSkills( resolve(localCodexHome, "skills"), ); @@ -6025,7 +6095,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } // A replacement executor restores its cold provider before acknowledging recovery. this.#assertStartupOpen(); - const admissionDeadline = this.#coldAdmissionDeadline(adoptedRunner === undefined); + const admissionDeadline = this.#sourceAttachmentDeadline === null + ? this.#coldAdmissionDeadline(adoptedRunner === undefined) + : Math.min(Date.now() + 30_000, this.#sourceAttachmentDeadline); const handle = adoptedRunner ? null : spawnRunner({ @@ -6071,18 +6143,30 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await this.#publishSpawnedProcess(handle); } this.#assertStartupOpen(); + let targetRecoveryActivated = registration === newTransitionRegistration; if (oldTransitionRegistration && newTransitionRegistration) { - await oldTransitionRegistration.activate?.(); + // A guest listener still serves only its actual physical run. Open that + // route first; probing the future/retired path cannot prove readiness. + await registration?.activate?.(); this.#assertStartupOpen(); - await newTransitionRegistration.activate?.(); + if (registration === oldTransitionRegistration) { + // Source authentication can consume its receipt and disappear before + // a sampled count sees it. The durable/physical proof loop below owns + // that progress; only target authentication completes recovery. + if (expectedRecoveryConnection?.mode !== "listen") { + await newTransitionRegistration.activate?.(); targetRecoveryActivated = true; + } + } } else await registration?.activate?.(); this.#assertStartupOpen(); - if (registration?.failure) { - void registration.failure.catch((error: unknown) => { - this.#failTransport( - error instanceof Error ? error : new Error(String(error)), - ); - }); + if (registration?.failure && !recoveringAttachment) { + void registration.failure.catch((error: unknown) => this.#failTransport(error instanceof Error ? error : new Error(String(error)))); + } else if (recoveringAttachment) { + // The old listener can retire normally during recovery. Authentication + // and the shared deadline, rather than its stale readiness promise, own + // the transition. Only the target remains live after activation. + void oldTransitionRegistration?.failure?.catch(() => undefined); + void newTransitionRegistration?.failure?.catch(() => undefined); } if (!adoptedRunner) await this.#awaitRegistrationReady(registration?.ready); this.#assertStartupOpen(); @@ -6095,22 +6179,40 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#evidence.runnerProcessGroupId = handle?.processGroupId ?? null; } this.#publish(); - if (warmRecovery) { - const deadline = - Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000); + if (recoveringAttachment) { + const deadline = Math.min(Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000), this.#sourceAttachmentDeadline ?? Infinity); + let sourceRouteRetired = false, nextPhysicalInspection = 0, previousConnections = core.activeRunnerConnectionCount(); while ( - !recoveryIdentityMatches(core.store.state.identity, desiredIdentity) || - core.store.state.warmTransition !== undefined + !recoveryIdentityMatches(core.store.state.identity, recoveryTarget!) || + core.store.state.warmTransition !== undefined || core.activeRunnerConnectionCount() !== 1 ) { this.#assertStartupOpen(); + const connections = core.activeRunnerConnectionCount(); + if (connections === 0 && previousConnections > 0) nextPhysicalInspection = 0; + previousConnections = connections; + if (!sourceRouteRetired && expectedRecoveryConnection?.mode === "listen" && oldTransitionRegistration + && newTransitionRegistration && this.options.readRunnerState && Date.now() >= nextPhysicalInspection) { + nextPhysicalInspection = Date.now() + 250; + const physical = await this.options.readRunnerState(); + const proof = inspectWarmRunTransition({ controlPlaneState: core.store.state, runnerState: physical, + expectedNewIdentity: recoveryTarget!, expectedRunnerVersion: runnerArtifact.version, expectedRunnerDigest: runnerArtifact.digest }); + if (proof && recoveryIdentityMatches(proof.runnerIdentity, recoveryTarget!)) { + // The same participant has consumed the old ACK and now serves + // the target path. Stop obsolete ingress retries before they can + // compete with the target's authenticated connection. + await oldTransitionRegistration.release(); sourceRouteRetired = true; + if (!targetRecoveryActivated) { await newTransitionRegistration.activate?.(); targetRecoveryActivated = true; } + } + } if (Date.now() >= deadline || (await this.#runnerHasExited())) { throw new Error("native_runner_warm_transition_recovery_pending"); } await new Promise((resolveWait) => setTimeout(resolveWait, 10)); } - if (core.getCommand(warmRecovery.commandId)?.status !== "completed") { + if (core.getCommand(warmRecovery?.commandId ?? queuedSourceAttachment!.commandId!)?.status !== "completed") { throw new Error("native_runner_warm_transition_result_unproven"); } + desiredIdentity = recoveryTarget!; this.#durableTurnId = desiredIdentity.turnId; this.#eventIdentity = structuredClone(desiredIdentity); this.#eventSourceSeq = 0; @@ -6119,8 +6221,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await oldTransitionRegistration.release(); this.#assertStartupOpen(); this.#controlPlaneRelease = newTransitionRegistration.release; - this.#controlPlaneCheckpoint = - newTransitionRegistration.checkpoint ?? null; + this.#controlPlaneCheckpoint = newTransitionRegistration.checkpoint ?? null; + void newTransitionRegistration.failure?.catch(error => this.#failTransport(error instanceof Error ? error : new Error(String(error)))); } } this.#assertStartupOpen(); @@ -6128,7 +6230,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { if (adoptedRunner) { await this.#awaitAdoptedRunnerConnection( adoptedRunner, - registration?.ready, + newTransitionRegistration?.ready ?? registration?.ready, ); } // Only a replacement executor can be restoring a cold Pi process. Share @@ -6137,7 +6239,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await this.#waitCommand("run.attach", runAttachment.commandId, admissionDeadline, true); } if (recoveryProbeCommandId !== null) { - await this.#waitCommand("runner.drain", recoveryProbeCommandId, admissionDeadline, true); + await this.#waitCommand(sourceAttachmentTarget ? "session.open" : "runner.drain", recoveryProbeCommandId, admissionDeadline, true); if (this.#checkpointProviderIdentityExpectation !== null) { // A replacement runner can restore the exact provider while its fresh // session.resumed event is compacted or delayed behind the completed @@ -6166,8 +6268,20 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { admissionDeadline, ); this.#startupComplete = true; + if (recoveringAttachment && staged) { + this.#sourceRecoveryIdentity = null; + this.#sourceAttachmentDeadline = null; + } + if (sourceAttachmentTarget) { + await this.#awaitWarmRunAttachmentReady(); + if (!core.semanticToolResultsSettled()) throw new Error("native_runner_source_semantic_reconciliation_required"); + if (runtimeContext !== null) writeFileSync(localRuntimeContextPath, `${JSON.stringify(runtimeContext)}\n`, { mode: 0o600 }); + await this.#attachRunWork(sourceAttachmentTarget); + this.#sourceRecoveryIdentity = null; + this.#sourceAttachmentDeadline = null; + } this.#diagnostic( - rotatedAuthority + rotatedAuthority || sourceAttachmentTarget ? "runnerd attached the durable provider session to a fresh PRP run authority" : "runnerd restored its durable PRP session and provider thread", ); @@ -6176,6 +6290,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } async #recoverSemanticToolInput(call: Parameters>[0]) { + if (this.#sourceRecoveryIdentity) throw new Error("native_runner_source_semantic_reconciliation_required"); if (call.operationId === "paperclip_finish" || call.operationId === "paperclip_block") { return this.#handleSemanticToolInput(call); } @@ -6188,6 +6303,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { NonNullable >[0], ) { + if (this.#sourceRecoveryIdentity) throw new Error("native_runner_source_semantic_reconciliation_required"); try { this.#throwIfFailed(); } catch { @@ -6529,6 +6645,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await completing; if (this.#pendingWarmRecoveryCompletion === completion) { this.#pendingWarmRecoveryCompletion = null; + this.#recoverPendingSemanticInputsWhenReady(); } } finally { if (this.#warmRecoveryCompletionInFlight === completing) { diff --git a/packages/paperclip-runner/src/live/runnerd-staged-handoff.test.ts b/packages/paperclip-runner/src/live/runnerd-staged-handoff.test.ts new file mode 100644 index 00000000000..23a465d4b56 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-staged-handoff.test.ts @@ -0,0 +1,241 @@ +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { createServer, request } from "node:http"; +import { createHash, randomBytes } from "node:crypto"; +import { execFileSync } from "node:child_process"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { expect, it, vi } from "vitest"; +import { + DurablePrpControlPlane, inspectStagedRunAttachmentHeader, + type DurablePrpCheckpoint, type RunnerProcessConnection, type PrpWireConnection, +} from "../control-plane/durable-prp-control-plane.js"; +import { + createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary, + readRunnerdArtifactBinding, +} from "./runnerd-codex-transport.js"; + +// The guest rejects WebSocket extensions. Use the same uncompressed wire +// contract as production, without importing the server connector into Runner. +async function openOwnedWire(url: string): Promise<{ wire: PrpWireConnection; closed: Promise; close(): void }> { + return new Promise((resolvePeer, reject) => { + const key = randomBytes(16).toString("base64"); + const req = request(url.replace("ws:", "http:"), { headers: { + Connection: "Upgrade", Upgrade: "websocket", "Sec-WebSocket-Version": "13", "Sec-WebSocket-Key": key, + } }); + req.on("error", reject); req.on("response", response => { response.resume(); reject(new Error(`Guest upgrade refused: ${response.statusCode}`)); }); + req.setTimeout(1_000, () => req.destroy(new Error("Guest upgrade timeout"))); + req.on("upgrade", (response, socket, head) => { + socket.setTimeout(0); + const expected = createHash("sha1").update(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").digest("base64"); + if (response.headers["sec-websocket-accept"] !== expected) { socket.destroy(); reject(new Error("Invalid guest upgrade")); return; } + const listeners: Array<(value: unknown) => void> = [], closes: Array<() => void> = []; + let buffer: Buffer = head, ended = false; + let closeResolve!: () => void; + const closed = new Promise(done => { closeResolve = done; }); + const finish = () => { if (ended) return; ended = true; for (const close of closes) close(); closeResolve(); }; + const send = (payload: Buffer, opcode: number) => { + const mask = randomBytes(4), prefix = payload.length < 126 ? Buffer.from([0x80 | opcode, 0x80 | payload.length]) : Buffer.alloc(payload.length <= 65535 ? 4 : 10); + if (payload.length >= 126) { prefix[0] = 0x80 | opcode; prefix[1] = 0x80 | (payload.length <= 65535 ? 126 : 127); + if (payload.length <= 65535) prefix.writeUInt16BE(payload.length, 2); else prefix.writeBigUInt64BE(BigInt(payload.length), 2); } + const masked = Buffer.from(payload); for (let index = 0; index < masked.length; index++) masked[index] ^= mask[index % 4]!; + socket.write(Buffer.concat([prefix, mask, masked])); + }; + const parse = () => { + while (buffer.length >= 2) { + const opcode = buffer[0]! & 15, initial = buffer[1]! & 127; + if (buffer.length < (initial === 126 ? 4 : initial === 127 ? 10 : 2)) return; + const size = initial === 126 ? buffer.readUInt16BE(2) : initial === 127 ? Number(buffer.readBigUInt64BE(2)) : initial; + const offset = initial === 126 ? 4 : initial === 127 ? 10 : 2; + if (buffer.length < offset + size) return; + const payload = buffer.subarray(offset, offset + size); buffer = buffer.subarray(offset + size); + if (opcode === 8) { socket.destroy(); finish(); return; } + if (opcode === 9) send(payload, 10); + if (opcode === 1 || opcode === 2) for (const listener of listeners) listener(JSON.parse(payload.toString("utf8"))); + } + }; + socket.on("data", bytes => { buffer = Buffer.concat([buffer, bytes]); parse(); }); socket.on("close", finish); socket.on("error", finish); + const close = () => { socket.destroy(); finish(); }; + resolvePeer({ wire: { sendJson: value => send(Buffer.from(JSON.stringify(value)), 1), close, + onJson: listener => { listeners.push(listener); }, onClose: listener => { closes.push(() => listener({})); } }, closed, close }); + queueMicrotask(parse); + }); req.end(); + }); +} + +// Real encrypted PRP and real Runner/provider processes; no model or server imports. +it.each(["connect", "listen"] as const)("recovers a staged %s handoff at every durable boundary without another process", async mode => { + const root = await mkdtemp(join(tmpdir(), "runner-staged-handoff-")); + const shared = join(root, "shared"), calls = join(root, "calls.log"); + const source = { runnerInstanceId: "runner-staged", environmentLeaseId: "lease-staged", runId: "source-run", + normalizedSessionId: "session-staged", turnId: "source-turn", itemId: "source-item" }; + const target = { ...source, runId: "target-run", turnId: "target-turn", itemId: "target-item" }; + const http = createServer(); + const routes = new Map(); + http.on("upgrade", (request, socket, head) => { + const route = routes.get(request.url ?? ""); + if (route) route.core.handleUpgrade(request, socket, request.url!, head); else socket.destroy(); + }); + await new Promise(done => http.listen(0, "127.0.0.1", done)); + const address = http.address(); + if (!address || typeof address === "string") throw new Error("Missing owned endpoint"); + const guest = createServer(); + await new Promise(done => guest.listen(0, "127.0.0.1", done)); + const guestAddress = guest.address(); + if (!guestAddress || typeof guestAddress === "string") throw new Error("Missing owned guest endpoint"); + const guestPort = guestAddress.port; + await new Promise(done => guest.close(() => done())); + const within = async (promise: Promise, label: string, ms = 8_000): Promise => { + let timer: ReturnType | undefined; + try { return await Promise.race([promise, new Promise((_, reject) => { timer = setTimeout(() => reject(new Error(`staged fixture timeout: ${label}`)), ms); })]); } + finally { clearTimeout(timer); } + }; + const dead = (pid: number) => { try { process.kill(pid, 0); return false; } catch (error) { return (error as NodeJS.ErrnoException).code === "ESRCH"; } }; + // Each boundary uses a fresh company-free fixture so no previous command or + // activation can satisfy a later assertion. + try { + for (const boundary of ["queued", "prepared", "activated", "completed"] as const) { + const directory = join(shared, boundary), callLog = `${calls}-${boundary}`; + let checkpoint!: DurablePrpCheckpoint, currentCore!: DurablePrpControlPlane; + let lost = false, release!: () => void, reached!: () => void; + const held = new Promise(done => { release = done; }); + const atBoundary = new Promise(done => { reached = done; }); + const cores = new Set(); + const registered = new WeakSet(); + let generation = 0; + const registration = async (core: DurablePrpControlPlane, identity = core.store.state.identity) => { + currentCore = core; + if (!registered.has(core)) { + registered.add(core); cores.add(core); + const epoch = generation; + core.setCheckpointPersistence(async saved => { + if (epoch !== generation) throw new Error("retired fixture controller"); + checkpoint = saved; + const proof = inspectStagedRunAttachmentHeader(saved.records.get("header")!); + if (!lost && proof?.phase === (boundary === "queued" ? "attachment_queued" : boundary)) { + lost = true; reached(); await held; + if (epoch !== generation) throw new Error("retired fixture controller"); + } + }); + } + const path = `/api/runner/v1/connect/${identity.runId}`, token = Symbol(path); + const connection: RunnerProcessConnection = mode === "connect" + ? { mode: "connect", connectUrl: `ws://127.0.0.1:${address.port}${path}` } + : { mode: "listen", listenAddress: "0.0.0.0", listenPort: guestPort, listenPath: path }; + if (mode === "connect") { + routes.set(path, { core, token }); + return { connection, release: () => { if (routes.get(path)?.token === token) routes.delete(path); } }; + } + let stopped = false, socket: { close(): void } | undefined; + let readyResolve!: () => void, readyReject!: (error: Error) => void; + const ready = new Promise((done, reject) => { readyResolve = done; readyReject = reject; }); + void ready.catch(() => undefined); + const connect = async () => { + const deadline = Date.now() + 8_000; + while (!stopped && Date.now() < deadline) { + try { + const peer = await openOwnedWire(`ws://127.0.0.1:${guestPort}${path}`); socket = peer; + if (stopped) { peer.close(); return; } + core.attachWireConnection(peer.wire); readyResolve(); await peer.closed; + } catch { /* The target listener appears only after the old receipt ACK. */ } + // Match the production ingress connector's first retry delay (250ms). + if (!stopped) await new Promise(done => setTimeout(done, 250)); + } + if (!stopped) readyReject(new Error("Owned listener did not become available")); + }; + return { connection, activate: () => { void connect(); }, ready: () => ready, + release: () => { stopped = true; socket?.close(); } }; + }; + const options = { runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"), + codexArgs: ["--call-log", callLog, "--record-process-start", "--state-file", join(directory, "fake.json")], + stateDirectory: directory, readRunnerState: async () => JSON.parse(await readFile(join(directory, "runner/runner-state.json"), "utf8")), lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 }, + runnerReconnectGraceMs: 5_000, prpIdentity: source, controlPlaneRegistration: registration, + onStagedRunAttachment: async () => undefined, onRecoveryAccountingEvent: async () => undefined }; + const first = createCapabilityRunnerdCodexTransport(options); + let replacement: ReturnType | undefined; + let runnerPid: number | null = null, providerPid: number | null = null; + try { + const opened = await within(first.transport.request("thread/start", { cwd: root, dynamicTools: [] }), "source open"); + runnerPid = first.evidence().runnerPid; providerPid = first.evidence().providerPid; + const startedAt = first.transport.processInfo().startedAt; + const providerStart = execFileSync("ps", ["-p", String(providerPid), "-o", "lstart="], { encoding: "utf8" }).trim(); + expect(runnerPid).toBeGreaterThan(0); expect(providerPid).toBeGreaterThan(0); + const attach = first.transport.attachRun!({ runId: target.runId, turnId: target.turnId, itemId: target.itemId }); + void attach.catch(() => undefined); + await within(atBoundary, `reach ${boundary}`); + expect(inspectStagedRunAttachmentHeader(checkpoint.records.get("header")!)?.attachment.targetIdentity).toEqual(target); + generation += 1; + // Disconnect the controller only. The bounded failed commit is released + // after its route is retired; a stale callback cannot publish generation B. + const detached = first.detachControllerForRestart(); void detached.catch(() => undefined); + release(); await within(detached, "detach").catch(() => undefined); + const next = join(root, `replacement-${boundary}`); + checkpoint.restoreSessionDirectory(next); + const noSpawn = vi.fn(() => { throw new Error("Duplicate Runner launch"); }); + const recoveryOptions: Parameters[0] = { ...options, stateDirectory: next, + prpIdentity: target, runnerArtifact: readRunnerdArtifactBinding(options.runnerBinary), + runnerStateDirectory: join(directory, "runner"), readRunnerState: async () => JSON.parse(await readFile(join(directory, "runner/runner-state.json"), "utf8")), + prepareExternalRunnerState: async () => { throw new Error("Verified live handoff must not restore a backup"); }, + runnerProcessLauncher: noSpawn, warmTransitionRegistrationMode: mode === "listen" ? "routed_same_endpoint" : "routed_connect", + resumeProviderSession: { driverSessionId: String((opened.thread as { id: string }).id), providerSessionId: String((opened.thread as { sessionId: string }).sessionId) }, + adoptExistingRunner: { pid: runnerPid, processGroupId: runnerPid, startedAt, + isAlive: () => !dead(runnerPid!), signal: signal => { if (dead(runnerPid!)) return true; process.kill(-runnerPid!, signal); return true; } } }; + if (boundary === "queued") { + const noProof = createCapabilityRunnerdCodexTransport({ ...recoveryOptions, adoptExistingRunner: undefined }); + await expect(noProof.transport.request("thread/read", {})).rejects.toThrow("queued_attachment_requires_verified_live_source"); + await noProof.detachControllerForRestart().catch(() => undefined); + checkpoint.restoreSessionDirectory(next); + const wrongEndpoint = createCapabilityRunnerdCodexTransport({ ...recoveryOptions, + controlPlaneRegistration: async (core, identity) => { + const registered = await registration(core, identity); + if (identity?.runId !== target.runId) return registered; + return { ...registered, connection: mode === "listen" + ? { mode: "listen", listenAddress: "0.0.0.0", listenPort: guestPort + 1, listenPath: `/api/runner/v1/connect/${target.runId}` } + : { mode: "connect", connectUrl: `ws://127.0.0.1:${address.port}/changed` } }; + } }); + await expect(wrongEndpoint.transport.request("thread/read", {})).rejects.toThrow("registered_endpoint_mismatch"); + await wrongEndpoint.detachControllerForRestart().catch(() => undefined); + expect(noSpawn).not.toHaveBeenCalled(); + checkpoint.restoreSessionDirectory(next); + } + replacement = createCapabilityRunnerdCodexTransport(recoveryOptions); + const read = await within(replacement.transport.request("thread/read", {}), `recover ${boundary}`); + expect(read.thread).toMatchObject({ id: (opened.thread as { id: string }).id, + sessionId: (opened.thread as { sessionId: string }).sessionId }); + expect(dead(providerPid!)).toBe(false); + expect(execFileSync("ps", ["-p", String(providerPid), "-o", "lstart="], { encoding: "utf8" }).trim()).toBe(providerStart); + expect(replacement.evidence().runnerPid).toBe(runnerPid); + expect(currentCore.activeRunnerConnectionCount()).toBe(1); + expect(currentCore.store.state.identity).toEqual(target); + expect(currentCore.store.state.stagedRunAttachment).toBeDefined(); + expect(currentCore.markStagedTargetCheckpointPublished({ ...target, itemId: "wrong" })).toBe(false); + await currentCore.flushCheckpoint(); + const beforeMark = checkpoint; + expect(currentCore.markStagedTargetCheckpointPublished(target)).toBe(true); + await currentCore.flushCheckpoint(); expect(checkpoint).toBe(beforeMark); + await within(replacement.transport.request("thread/goal/get", {}), "post-checkpoint read"); + expect(currentCore.store.state.stagedRunAttachment).toBeUndefined(); + expect(noSpawn).not.toHaveBeenCalled(); + const effects = (await readFile(callLog, "utf8")).trim().split(/\r?\n/); + expect(effects.filter(line => line === "process-start")).toHaveLength(1); + expect(effects.filter(line => line === "thread/start")).toHaveLength(1); + expect(effects.filter(line => line === "turn/start")).toHaveLength(0); + } catch (error) { + const physical = JSON.parse(await readFile(join(directory, "runner/runner-state.json"), "utf8").catch(() => "{}")); + throw new Error(`${mode}/${boundary}: ${String(error)}; ${JSON.stringify({ + active: currentCore?.activeRunnerConnectionCount(), coreRun: currentCore?.store.state.identity.runId, + coreWarm: currentCore?.store.state.warmTransition?.phase, header: checkpoint && inspectStagedRunAttachmentHeader(checkpoint.records.get("header")!)?.phase, + runnerRun: physical.runId, runnerWarm: physical.warmTransition?.phase, lifecycle: physical.lifecycle, + failureCode: physical.recoverableFailure?.code, failureKeys: Object.keys(physical.recoverableFailure ?? {}), + })}`, { cause: error }); + } finally { + release(); + if (!runnerPid && !replacement) { await first.transport.close().catch(() => undefined); await first.retireProbeResources?.(1_000).catch(() => undefined); } + else await first.detachControllerForRestart().catch(() => undefined); + await replacement?.transport.close().catch(() => undefined); + for (const pid of [providerPid, runnerPid]) if (pid && !dead(-pid)) { try { process.kill(-pid, "SIGKILL"); } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; } } + for (const core of cores) await core.stop().catch(() => undefined); + for (const pid of [providerPid, runnerPid]) if (pid) await vi.waitFor(() => expect(dead(pid)).toBe(true)); + } + } + } finally { http.closeAllConnections(); await new Promise(done => http.close(() => done())); await rm(root, { recursive: true, force: true }); } +}, 90_000); diff --git a/scripts/benchmark-native-controller-recovery.ts b/scripts/benchmark-native-controller-recovery.ts new file mode 100644 index 00000000000..8681954b34c --- /dev/null +++ b/scripts/benchmark-native-controller-recovery.ts @@ -0,0 +1,996 @@ +import fs from "node:fs"; +import { syncBuiltinESMExports } from "node:module"; +import assert from "node:assert/strict"; +import { randomUUID, randomBytes, createHash } from "node:crypto"; +import { createRequire } from "node:module"; +import { mkdir, writeFile, readFile } from "node:fs/promises"; +import { monitorEventLoopDelay } from "node:perf_hooks"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { execFileSync } from "node:child_process"; + +// Run only against an explicitly disposable, otherwise idle Postgres cluster. +// pg_stat_statements must be preloaded; CPU/memory measurements use its cgroup. +const repoRoot = fileURLToPath(new URL("../", import.meta.url)); +function required(name: string) { + const value = process.env[name]; + assert(value, `Set ${name}`); + return value; +} +const baselineRoot = path.resolve(required("BENCH_BASELINE_ROOT")); +const root = path.resolve(required("BENCH_REPORT_ROOT")); +const url = required("BENCH_DATABASE_URL"); +const parsedUrl = new URL(url); +assert( + ["postgres:", "postgresql:"].includes(parsedUrl.protocol) && + ["127.0.0.1", "localhost", "[::1]"].includes(parsedUrl.hostname), + "Use an explicit loopback Postgres URL", +); +assert.equal(process.env.BENCH_DISPOSABLE_POSTGRES, "1", "The whole cluster must be owned by this benchmark"); +const mode = process.argv[2] ?? "steady"; +assert(["pilot", "steady", "burst"].includes(mode)); +const pgCgroup = required("BENCH_POSTGRES_CGROUP"); +const [cpuQuota, cpuPeriod] = fs.readFileSync(path.join(pgCgroup, "cpu.max"), "utf8").trim().split(/\s+/).map(Number); +assert.equal(cpuQuota / cpuPeriod, 2, "The reference workload requires two Postgres CPUs"); +assert.equal( + Number(fs.readFileSync(path.join(pgCgroup, "memory.max"), "utf8")), + 1536 * 1024 * 1024, + "The reference workload requires a 1536 MiB Postgres memory limit", +); +const load = (base: string, file: string) => import(pathToFileURL(path.join(base, file)).href); +const { + completionContracts, + createDb, + livenessDb, + checkpointDb, + closeRegisteredClients, + migratePostgresIfEmpty, + companies, + agents, + issues, + heartbeatRuns, + nativeRunFinalizations, +} = await load(repoRoot, "packages/db/src/index.ts"); +const { nativeRunnerCheckpointWriter: CandidateWriter } = await load( + repoRoot, + "server/src/services/native-runtime/native-runner-checkpoint.ts", +); +const { nativeRunnerCheckpointWriter: BaselineWriter } = await load( + baselineRoot, + "server/src/services/native-runtime/native-runner-checkpoint.ts", +); +const { startNativeSessionExecutionLeaseRenewal } = await load( + repoRoot, + "server/src/services/native-runtime/native-execution-lease.ts", +); +const { loadAgentRunAuthentication } = await load(repoRoot, "server/src/middleware/agent-run-authentication.ts"); +const { DurablePrpControlPlane: CandidateCore } = await load( + repoRoot, + "packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts", +); +const { DurablePrpControlPlane: BaselineCore } = await load( + baselineRoot, + "packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts", +); +const { PaperclipControlPlanePort: CandidatePort } = await load( + repoRoot, + "server/src/services/native-runtime/paperclip-control-plane-port.ts", +); +const { PaperclipControlPlanePort: BaselinePort } = await load( + baselineRoot, + "server/src/services/native-runtime/paperclip-control-plane-port.ts", +); +const require = createRequire(path.join(repoRoot, "server/package.json")); +const postgres = createRequire(path.join(repoRoot, "packages/db/package.json"))("postgres"); +const { and, eq, desc } = require("drizzle-orm"); +const pressure = (base: string, name: string) => { + const file = path.join(base, name + ".pressure"); + return fs.existsSync(file) ? fs.readFileSync(file, "utf8").trim() : null; +}; +const pgResources = () => { + const raw = fs.readFileSync(path.join(pgCgroup, "cpu.stat"), "utf8"); + return { + cpuUsec: Number(raw.match(/usage_usec (\d+)/)![1]), + throttledUsec: Number(raw.match(/throttled_usec (\d+)/)?.[1] ?? 0), + throttledPeriods: Number(raw.match(/nr_throttled (\d+)/)?.[1] ?? 0), + cpuPressure: pressure(pgCgroup, "cpu"), + memoryPressure: pressure(pgCgroup, "memory"), + memoryBytes: Number(fs.readFileSync(path.join(pgCgroup, "memory.current"), "utf8")), + }; +}; +const revision = (directory: string) => + execFileSync("git", ["rev-parse", "HEAD"], { + cwd: directory, + encoding: "utf8", + }).trim(); +const provenance = { + harnessSha256: createHash("sha256").update(fs.readFileSync(fileURLToPath(import.meta.url))).digest("hex"), + lockObserver: { intervalMs: 25, nominalQueriesPerSecond: 40, includedInPostgresCpu: true }, + baselineRevision: process.env.BENCH_BASELINE_REVISION ?? revision(baselineRoot), + candidateRevision: process.env.BENCH_CANDIDATE_REVISION ?? revision(repoRoot), + candidateDiff: + process.env.BENCH_CANDIDATE_DIFF ?? + execFileSync("git", ["diff", "HEAD", "--", "server/src/services/native-runtime", "packages/db/src"], { + cwd: repoRoot, + encoding: "utf8", + }), +}; +await mkdir(root); // Refuse to overwrite earlier measurements. +await writeFile(path.join(root, "provenance.json"), JSON.stringify(provenance, null, 2)); +const admin = postgres(url, { + max: 1, + onnotice: () => {}, + connection: { application_name: "benchmark-observer" }, +}); +const ownedPrefix = "native_recovery_bench_" + randomUUID().replaceAll("-", ""); +const templateName = ownedPrefix + "_template"; +const databaseUrl = (name: string) => { + const value = new URL(url); + value.pathname = "/" + name; + return value.toString(); +}; +process.env.PAPERCLIP_HOME = root + "/home"; +process.env.PAPERCLIP_SECRETS_MASTER_KEY = randomBytes(32).toString("hex"); +const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); +const summary = (a: number[]) => { + const b = [...a].sort((x, y) => x - y); + return { + n: b.length, + p50: b[Math.floor(b.length * 0.5)] ?? 0, + p95: b[Math.floor(b.length * 0.95)] ?? 0, + p99: b[Math.floor(b.length * 0.99)] ?? 0, + max: b.at(-1) ?? 0, + }; +}; +// Sampling reports bounded estimates for observed episodes, never exact lock time. +// A wait shorter than a polling gap can be missed entirely. Query text is excluded. +function observeLockWaits(monitor: any, intervalMs = 25) { + const active = new Map(); + const episodes: any[] = []; + const gaps: number[] = []; + const errors: string[] = []; + let previous = performance.now(), stopped = false; + let poll: Promise | undefined; + let samples = 0; + const collect = async () => { + const rows = await monitor`select a.pid,a.application_name,a.query_start::text as query_start,a.wait_event,l.waitstart::text as wait_start,extract(epoch from clock_timestamp()-l.waitstart)*1000 as wait_age_ms from pg_stat_activity a join pg_locks l on l.pid=a.pid and not l.granted and l.waitstart is not null where a.datname=current_database() and a.application_name!='benchmark-observer' and a.wait_event_type='Lock'`; + const now = performance.now(), gap = now - previous; + gaps.push(gap); + samples++; + const seen = new Set(); + for (const row of rows) { + const key = `${row.pid}/${row.query_start}/${row.wait_start}/${row.wait_event}`; + seen.add(key); + const old = active.get(key); + if (old) { old.lastMs = now; old.wait_age_ms = Number(row.wait_age_ms); } + else active.set(key, { ...row, firstMs: now, lastMs: now, precedingGapMs: gap }); + } + for (const [key, episode] of active) { + if (seen.has(key)) continue; + episodes.push({ ...episode, observedLowerMs: Number(episode.wait_age_ms), + estimatedUpperMs: Number(episode.wait_age_ms) + now - episode.lastMs, rightCensored: false }); + active.delete(key); + } + previous = now; + }; + const timer = setInterval(() => { + if (stopped || poll) return; + poll = collect().catch((e: any) => errors.push(e.message)).finally(() => { poll = undefined; }); + }, intervalMs); + return { async stop() { + stopped = true; + clearInterval(timer); + await poll; + await collect().catch((e: any) => errors.push(e.message)); + for (const episode of active.values()) episodes.push({ ...episode, + observedLowerMs: Number(episode.wait_age_ms), estimatedUpperMs: null, rightCensored: true }); + const byPool = Object.fromEntries(["benchmark-application", "paperclip-controller-lease", "paperclip-runner-checkpoint"].map(pool => { + const selected = episodes.filter(e => e.application_name === pool); + return [pool, { observedEpisodes: selected.length, + observedLowerMs: summary(selected.map(e => e.observedLowerMs)), + estimatedUpperMs: summary(selected.filter(e => e.estimatedUpperMs !== null).map(e => e.estimatedUpperMs)) }]; + })); + return { intervalMs, samples, gapsMs: summary(gaps), errors, byPool, episodes, + limits: "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." }; + } }; +} +async function setup() { + await admin.unsafe(`create database ${templateName}`); + const u = databaseUrl(templateName); + await migratePostgresIfEmpty(u); + const db = createDb(u); + await db.$client.unsafe("create extension if not exists pg_stat_statements"); + const companyId = randomUUID(), + agentId = randomUUID(); + await db.insert(companies).values({ + id: companyId, + name: "Synthetic normal-operation benchmark", + issuePrefix: "BENCH", + requireBoardApprovalForNewAgents: false, + }); + await db.insert(agents).values({ + id: agentId, + companyId, + name: "Benchmark fixture", + adapterType: "paperclip_runner", + adapterConfig: { provider: "codex" }, + status: "active", + }); + for (let batch = 0; batch < 20; batch++) + await db.insert(issues).values( + Array.from({ length: 500 }, (_, j) => ({ + companyId, + title: `Historical task ${batch * 500 + j}`, + status: j % 3 ? "done" : "backlog", + identifier: `BENCH-${batch * 500 + j + 1}`, + issueNumber: batch * 500 + j + 1, + description: "Synthetic context ".repeat(80), + })), + ); + await db.$client.unsafe("analyze"); + await writeFile(root + "/seed.json", JSON.stringify({ companyId, agentId })); + await closeRegisteredClients(u); +} +async function sampleStats(dbName: string) { + const [stat] = + await admin`select xact_commit,xact_rollback,deadlocks,blks_read,blks_hit,tup_inserted,tup_updated,tup_deleted,temp_bytes from pg_stat_database where datname=${dbName}`; + const [wal] = await admin`select pg_current_wal_insert_lsn()::text as lsn`; + const statements = + await admin`select query,calls,total_exec_time,rows,shared_blks_hit,shared_blks_read,wal_bytes from pg_stat_statements where dbid=(select oid from pg_database where datname=${dbName}) order by calls desc`; + return { stat, wal: wal.lsn, statements }; +} +async function run(variant: string, n: number, trial: number, durationMs: number) { + const label = `${String(trial).padStart(2, "0")}_${variant}_${n}`, + dbName = ownedPrefix + "_" + trial; + await admin.unsafe(`create database ${dbName} template ${templateName}`); + const u = databaseUrl(dbName), + db = createDb(u, { + maxConnections: 10, + applicationName: "benchmark-application", + }); + const seed = JSON.parse(await readFile(root + "/seed.json", "utf8")); + const owner = { + bootId: randomUUID(), + pid: process.pid, + processStartedAt: new Date(), + }; + const items: any[] = [], + metrics: any = { + label, + variant, + n, + durationMs, + setupMs: 0, + errors: [], + dropped: 0, + renewalLosses: 0, + controls: 0, + changedRows: 0, + changedBytes: 0, + checkpointTransactions: 0, + sqlCount: 0, + sqlCategories: {}, + normalRead: [], + normalWrite: [], + databaseAdmission: [], + controlLatency: [], + checkpointLatency: [], + normalSchedulingLag: [], + controlSchedulingLag: [], + normalOffered: 0, + normalCompleted: 0, + normalWritesOffered: 0, + normalWritesCompleted: 0, + admissionsOffered: 0, + admissionsCompleted: 0, + peakPending: 0, + samples: [], + }; + // postgres.js calls onexecute after selecting a connection and encoding BEGIN. + // Preserve that callback. Queue/admission and BEGIN response time are separate; + // SQL execution/lock waits are reported independently by Postgres statistics. + metrics.poolAdmission = []; + metrics.beginResponse = []; + metrics.poolQueueByPool = {}; + metrics.transactionLatencyByPool = {}; + for (const [pool, client] of [ + ["request", db.$client], + ["liveness", livenessDb(db).$client], + ["checkpoint", checkpointDb(db).$client], + ] as const) { + metrics.poolQueueByPool[pool] = []; + metrics.transactionLatencyByPool[pool] = []; + const originalBegin = client.begin, + originalUnsafe = client.unsafe; + let entering: { start: number; sent?: number } | null = null; + client.unsafe = (query, parameters, options) => { + const admission = entering; + if (!admission || !/^begin\b/i.test(query)) return originalUnsafe(query, parameters, options); + return originalUnsafe(query, parameters, { + ...options, + onexecute: (connection) => { + admission.sent = performance.now(); + if (measuring) metrics.poolQueueByPool[pool].push(admission.sent - admission.start); + return options.onexecute(connection); + }, + }); + }; + client.begin = (...args) => { + const admission = { + start: performance.now(), + sent: undefined as number | undefined, + }; + const index = args.length - 1, + work = args[index]; + args[index] = async (connection) => { + if (measuring) { + metrics.poolAdmission.push(performance.now() - admission.start); + if (admission.sent !== undefined) metrics.beginResponse.push(performance.now() - admission.sent); + } + return work(connection); + }; + entering = admission; + try { + return originalBegin(...args).finally(() => { + if (measuring) metrics.transactionLatencyByPool[pool].push(performance.now() - admission.start); + }); + } finally { + entering = null; + } + }; + } + metrics.snapshotLatency = []; + metrics.snapshotTransactions = 0; + let measuring = false; + const originalFsync = fs.fsyncSync; + metrics.fsyncDurations = []; + fs.fsyncSync = function (fd) { + const t = performance.now(); + try { + return originalFsync(fd); + } finally { + if (measuring) metrics.fsyncDurations.push(performance.now() - t); + } + }; + syncBuiltinESMExports(); + db.$client.options.debug = (_conn: any, query: string) => { + if (!measuring) return; + metrics.sqlCount++; + const c = /native_runner_checkpoints/.test(query) + ? "checkpoint" + : /native_run_finalizations/.test(query) + ? "ownership" + : /heartbeat_runs/.test(query) + ? "run" + : /issues/.test(query) + ? "task" + : "other"; + metrics.sqlCategories[c] = (metrics.sqlCategories[c] ?? 0) + 1; + }; + metrics.livenessSqlCount = 0; + metrics.livenessSqlKinds = {}; + { + const live = livenessDb(db); + live.$client.options.debug = (_connection, query) => { + if (!measuring) return; + metrics.livenessSqlCount++; + const kind = query.trim().split(/\s+/)[0].toLowerCase(); + metrics.livenessSqlKinds[kind] = (metrics.livenessSqlKinds[kind] ?? 0) + 1; + }; + } + metrics.checkpointSqlCount = 0; + { + checkpointDb(db).$client.options.debug = (_conn, query) => { + if (measuring) metrics.checkpointSqlCount++; + }; + } + const tSetup = performance.now(), + runNonce = randomUUID(); + for (let i = 0; i < n; i++) { + const runId = randomUUID(), + issueId = randomUUID(), + leaseOwner = randomUUID(), + runnerId = randomUUID(), + sessionId = randomUUID(), + contractId = randomUUID(); + await db.transaction(async (tx: any) => { + await tx.insert(issues).values({ + id: issueId, + companyId: seed.companyId, + title: "Active fixture " + i, + status: "in_progress", + assigneeAgentId: seed.agentId, + }); + await tx.insert(completionContracts).values({ + id: contractId, + companyId: seed.companyId, + issueId, + revision: 1, + schemaVersion: "fixture", + policyVersion: "fixture", + risk: "low", + completionAuthority: "agent", + incompleteCriteriaPolicy: "allow", + contractJson: {}, + canonicalSha256: "fixture", + createdByActorType: "system", + createdByActorId: "benchmark", + }); + await tx.insert(heartbeatRuns).values({ + id: runId, + companyId: seed.companyId, + agentId: seed.agentId, + status: "running", + runtimeMode: "native", + nativeIssueId: issueId, + nativeSessionId: sessionId, + runnerInstanceId: runnerId, + completionContractId: contractId, + completionContractSha256: "fixture", + contextSnapshot: { + issueId, + fixture: "normal-load", + prompt: "x".repeat(4096), + }, + }); + await tx.insert(nativeRunFinalizations).values({ + runId, + companyId: seed.companyId, + issueId, + phase: "observed", + attempt: 1, + leaseOwner, + controllerBootId: owner.bootId, + controllerPid: owner.pid, + controllerProcessStartedAt: owner.processStartedAt, + controllerGeneration: 1, + leaseExpiresAt: new Date(Date.now() + 1200000), + }); + await tx.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + }); + const stateDirectory = (process.env.BENCH_STATE_ROOT ?? root + "/states") + "/" + label + "/" + runNonce + "/" + i; + const Core = variant === "candidate" ? CandidateCore : BaselineCore; + const core = new Core({ + stateDirectory, + identity: { + runnerInstanceId: runnerId, + environmentLeaseId: randomUUID(), + runId, + normalizedSessionId: sessionId, + turnId: randomUUID(), + itemId: randomUUID(), + }, + expectedRunnerVersion: "fixture-v1", + expectedRunnerDigest: "sha256:" + "a".repeat(64), + }); + { + const writer = (variant === "candidate" ? CandidateWriter : BaselineWriter)(db, { + runId, + issueId, + companyId: seed.companyId, + authorityRunId: runId, + bootId: owner.bootId, + generation: 1, + leaseOwner, + }); + core.setCheckpointPersistence(async (cp: any) => { + const start = performance.now(); + await writer(cp); + if (measuring) { + metrics.checkpointTransactions++; + metrics.changedRows += cp.changedRecords.size; + metrics.changedBytes += [...cp.changedRecords.values()].reduce( + (s: number, v: any) => s + Buffer.byteLength(v), + 0, + ); + metrics.checkpointLatency.push(performance.now() - start); + } + }); + await core.flushCheckpoint(); + } + const identity = { + companyId: seed.companyId, + agentId: seed.agentId, + issueId, + runId, + sessionId, + }; + const snapshot = { + backendKind: "codex", + sessionId, + identity, + driverKind: "codex_app_server", + driverSessionId: "thread_" + runId, + providerSessionId: "thread_" + runId, + activeTurnId: null, + terminalTurns: [], + pendingRuntimeRequests: [], + }; + const Port = variant === "candidate" ? CandidatePort : BaselinePort; + const port = new Port( + db, + { + ...identity, + sourceInstanceId: runnerId, + controlPlaneSourceInstanceId: "controller", + completionContractId: contractId, + completionContractSha256: "fixture", + }, + variant === "candidate" + ? { + nativeRecovery: { + compatibility: { + configDigest: "sha256:" + "a".repeat(64), + processConfigDigest: null, + }, + owner: { + companyId: seed.companyId, + issueId, + runId, + authorityRunId: runId, + bootId: owner.bootId, + generation: 1, + leaseOwner, + }, + }, + } + : {}, + ); + await port.openRun({ + identity, + backendKind: "codex", + sourceInstanceId: runnerId, + }); + items.push({ + core, + port, + snapshot, + runId, + issueId, + leaseOwner, + companyId: seed.companyId, + agentId: seed.agentId, + }); + } + metrics.setupMs = performance.now() - tSetup; + const monitor = postgres(u, { + max: 1, + onnotice: () => {}, + connection: { application_name: "benchmark-observer" }, + }); + if (mode === "pilot") { + // Verify the observer against one deliberately held row lock before measured work. + const calibration = observeLockWaits(monitor); + let locked!: () => void, release!: () => void; + const acquired = new Promise(r => { locked = r; }); + const held = new Promise(r => { release = r; }); + const blocker = db.$client.begin(async (tx: any) => { + await tx`select 1 from native_run_finalizations where run_id=${items[0].runId} for update`; + locked(); + await held; + }); + await acquired; + const waiterStarted = performance.now(); + const waiter = checkpointDb(db).$client.begin(async (tx: any) => { + await tx`select 1 from native_run_finalizations where run_id=${items[0].runId} for update`; + }); + // postgres.js BEGIN returns a lazy Promise; observing it starts the contender. + const waiting = Promise.resolve(waiter); + await sleep(250); + const releasedAtMs = performance.now() - waiterStarted; + release(); + await Promise.all([blocker, waiting]); + await sleep(50); + const observed = await calibration.stop(); + metrics.lockCalibration = { intentionalHoldMs: 250, releasedAtMs, + transactionElapsedMs: performance.now() - waiterStarted, observed }; + assert(observed.episodes.some((e: any) => e.application_name === "paperclip-runner-checkpoint" && e.observedLowerMs >= 100), "Lock observer did not detect the deliberate row wait"); + } + await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() + 60000) }); + await db.$client.unsafe("analyze"); + // Warm normal queries before measuring, identical in each arm. + for (let i = 0; i < 100; i++) await loadAgentRunAuthentication(db, items[i % n]); + await admin.unsafe("select pg_stat_statements_reset()"); + const pgBefore = pgResources(); + const before = await sampleStats(dbName), + cpuBefore = process.cpuUsage(), + el = monitorEventLoopDelay({ resolution: 10 }); + el.enable(); + const pending = new Set>(), + renewals: any[] = [], + timers: any[] = []; + function offer(action: () => Promise, out: number[], scheduled = performance.now()) { + if (pending.size >= 500) { + metrics.dropped++; + return; + } + const task = Promise.resolve() + .then(action) + .then(() => out.push(performance.now() - scheduled)) + .catch((e) => metrics.errors.push({ message: e.message, code: e.code })) + .finally(() => pending.delete(task)); + pending.add(task); + metrics.peakPending = Math.max(metrics.peakPending, pending.size); + } + measuring = true; + const lockWaitObserver = observeLockWaits(monitor); + const started = performance.now(), + startedAt = new Date().toISOString(); + await writeFile( + root + "/phase.json", + JSON.stringify({ + label, + phase: "running", + startedAt, + n, + variant, + durationMs, + }), + ); + console.log( + JSON.stringify({ + phase: "started", + label, + setupMs: metrics.setupMs, + startedAt, + }), + ); + for (const item of items) { + const input = { db, ...item, attempt: 1, controller: owner }; + renewals.push( + startNativeSessionExecutionLeaseRenewal({ + ...input, + remote: true, + onLost: async () => { + metrics.renewalLosses++; + }, + }), + ); + } + // Both arms receive the same offered workload. No await-before-next-tick pacing. + let tick = 0; + const offerNormal = () => { + const due = Math.min(Math.floor(durationMs / 100), Math.floor((performance.now() - started) / 100)); + while (tick < due) { + tick++; + metrics.normalSchedulingLag.push(Math.max(0, performance.now() - started - tick * 100)); + for (let k = 0; k < (mode === "burst" ? 10 : 50); k++) { + const idx = (tick * 50 + k) % n, + item = items[idx]; + metrics.normalOffered++; + offer( + async () => { + if (k % 10 < 7) { + const rows = await loadAgentRunAuthentication(db, item); + assert.equal(rows.length, 1); + assert.equal(rows[0].status, "running"); + } else if (k % 10 < 9) { + const rows = await db + .select({ + id: issues.id, + title: issues.title, + status: issues.status, + }) + .from(issues) + .where(and(eq(issues.companyId, seed.companyId), eq(issues.status, "backlog"))) + .orderBy(desc(issues.createdAt)) + .limit(50); + assert.equal(rows.length, 50); + } else { + const rows = await db + .select({ + runId: nativeRunFinalizations.runId, + phase: nativeRunFinalizations.phase, + }) + .from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, item.runId)); + assert.equal(rows[0]?.phase, "observed"); + } + metrics.normalCompleted++; + }, + metrics.normalRead, + started + tick * 100, + ); + } + for (let k = 0; k < (mode === "burst" ? 2 : 10); k++) { + const item = items[(tick * 10 + k) % n]; + metrics.normalWritesOffered++; + offer( + async () => { + await db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, item.issueId)); + metrics.normalWritesCompleted++; + }, + metrics.normalWrite, + started + tick * 100, + ); + } + if (tick % 50 === 0) { + metrics.admissionsOffered++; + offer( + async () => { + const runId = randomUUID(), + issueId = randomUUID(); + await db.transaction(async (tx: any) => { + await tx.insert(issues).values({ + id: issueId, + companyId: seed.companyId, + title: "Fresh admission database probe", + status: "in_progress", + }); + await tx.insert(heartbeatRuns).values({ + id: runId, + companyId: seed.companyId, + agentId: seed.agentId, + status: "queued", + runtimeMode: "native", + nativeIssueId: issueId, + }); + await tx.insert(nativeRunFinalizations).values({ + runId, + companyId: seed.companyId, + issueId, + phase: "observed", + }); + await tx.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + }); + metrics.admissionsCompleted++; + }, + metrics.databaseAdmission, + started + tick * 100, + ); + } + } + }; + const normal = setInterval(offerNormal, 100); + timers.push(normal); + // Same aggregate control rate, spread evenly across sessions for normal operation. + let offeredControls = 0; + const offerControl = () => { + const elapsed = performance.now() - started; + const due = Math.min( + Math.floor((durationMs * n) / 5000), + mode === "burst" ? Math.floor(elapsed / 5000) * n : Math.floor((elapsed * n) / 5000), + ); + while (offeredControls < due) { + const index = offeredControls++, + item = items[index % n], + seq = Math.floor(index / n) + 1; + const scheduled = started + (mode === "burst" ? seq * 5000 : ((index + 1) * 5000) / n); + metrics.controlSchedulingLag.push(Math.max(0, performance.now() - scheduled)); + offer( + async () => { + item.core.queueCommand( + "semantic_tool.result", + { + callId: `fixture_call_${seq}`, + operationId: "fixture_read", + result: { body: "x".repeat(4096) }, + isError: false, + }, + `control_${seq}`, + ); + await item.core.flushCheckpoint(); + const snapshotStarted = performance.now(); + await item.port.checkpointSession(item.snapshot); + metrics.snapshotLatency.push(performance.now() - snapshotStarted); + metrics.snapshotTransactions++; + metrics.controls++; + }, + metrics.controlLatency, + scheduled, + ); + } + }; + const control = setInterval(offerControl, 10); + timers.push(control); + const sampler = setInterval(() => { + void (async () => { + const m = process.memoryUsage(); + const activity = + await monitor`select count(*)::int as connections,count(*) filter(where state='active')::int as active,count(*) filter(where wait_event_type='Lock')::int as locks from pg_stat_activity where datname=current_database() and application_name!='benchmark-observer'`; + metrics.samples.push({ + elapsedMs: performance.now() - started, + rss: m.rss, + heapUsed: m.heapUsed, + postgresMemoryBytes: pgResources().memoryBytes, + ...activity[0], + }); + })().catch((e) => metrics.errors.push({ observer: e.message })); + }, 1000); + timers.push(sampler); + await sleep(durationMs); + offerNormal(); + offerControl(); + for (const t of timers) clearInterval(t); + const drainStart = performance.now(); + await Promise.allSettled([...pending]); + await Promise.allSettled(renewals.map((r) => r.stop())); + metrics.drainMs = performance.now() - drainStart; + metrics.lockWait = await lockWaitObserver.stop(); + metrics.elapsedMs = performance.now() - started; + metrics.startedAt = startedAt; + metrics.endedAt = new Date().toISOString(); + measuring = false; + fs.fsyncSync = originalFsync; + syncBuiltinESMExports(); + metrics.fsyncTotalMs = metrics.fsyncDurations.reduce((a, b) => a + b, 0); + el.disable(); + metrics.nodeCpu = process.cpuUsage(cpuBefore); + metrics.eventLoopDelay = { + p95Ms: el.percentile(95) / 1e6, + p99Ms: el.percentile(99) / 1e6, + maxMs: el.max / 1e6, + }; + // Flush PG statistics after all application activity; observer SQL is tagged/excluded below. + await sleep(1100); + const after = await sampleStats(dbName); + const [wal] = await admin`select pg_wal_lsn_diff(${after.wal}::pg_lsn,${before.wal}::pg_lsn)::text as bytes`; + metrics.walBytes = Number(wal.bytes); + const pgAfter = pgResources(); + metrics.postgresCpuUsec = pgAfter.cpuUsec - pgBefore.cpuUsec; + metrics.postgresResources = { before: pgBefore, after: pgAfter, throttledUsec: pgAfter.throttledUsec - pgBefore.throttledUsec, throttledPeriods: pgAfter.throttledPeriods - pgBefore.throttledPeriods }; + metrics.postgresMemoryBytes = pgAfter.memoryBytes; + metrics.postgresPeakSampledMemoryBytes = Math.max( + pgBefore.memoryBytes, + pgAfter.memoryBytes, + ...metrics.samples.map((s) => s.postgresMemoryBytes), + ); + metrics.platform = { + node: process.version, + os: process.platform, + arch: process.arch, + }; + metrics.databaseStats = Object.fromEntries( + Object.keys(after.stat).map((k) => [k, Number(after.stat[k]) - Number(before.stat[k])]), + ); + metrics.statements = after.statements; + metrics.lockObserverSql = after.statements.filter((s: any) => /pg_locks.*waitstart|waitstart.*pg_locks/.test(s.query)); + metrics.tableStats = + await monitor`select relname,n_tup_ins,n_tup_upd,n_tup_del,n_dead_tup from pg_stat_user_tables where relname in ('native_runner_checkpoints','native_run_finalizations','heartbeat_runs','issues') order by relname`; + const [storage] = + await monitor`select count(*)::int as rows,coalesce(sum(pg_column_size(material)),0)::bigint::text as encrypted_bytes,pg_total_relation_size('native_runner_checkpoints')::text as relation_bytes from native_runner_checkpoints`; + metrics.checkpointStorage = storage; + for (const name of [ + "poolAdmission", + "beginResponse", + "snapshotLatency", + "normalRead", + "normalWrite", + "databaseAdmission", + "controlLatency", + "checkpointLatency", + "normalSchedulingLag", + "controlSchedulingLag", + "fsyncDurations", + ]) { + metrics[name + "Summary"] = summary(metrics[name]); + delete metrics[name]; + } + for (const pool of Object.keys(metrics.poolQueueByPool)) { + metrics.poolQueueByPool[pool] = summary(metrics.poolQueueByPool[pool]); + metrics.transactionLatencyByPool[pool] = summary(metrics.transactionLatencyByPool[pool]); + } + metrics.nodeCpuAverageOneCorePercent = + ((metrics.nodeCpu.user + metrics.nodeCpu.system) / 1000 / metrics.elapsedMs) * 100; + metrics.correctnessPassed = + metrics.errors.length === 0 && + metrics.lockWait.errors.length === 0 && + metrics.renewalLosses === 0 && + metrics.dropped === 0 && + metrics.normalCompleted === metrics.normalOffered && + metrics.normalWritesCompleted === metrics.normalWritesOffered && + metrics.admissionsCompleted === metrics.admissionsOffered; + metrics.absoluteGates = { + requestP99: metrics.normalReadSummary.p99 < 250, + drain: metrics.drainMs < 1000, + throughput: metrics.normalCompleted / metrics.normalOffered >= 0.99, + postgresCpu: metrics.postgresCpuUsec / 1000 / metrics.elapsedMs / 2 < 0.75, + postgresMemory: metrics.postgresPeakSampledMemoryBytes < 1382.4 * 1024 * 1024, + }; + metrics.passed = metrics.correctnessPassed && Object.values(metrics.absoluteGates).every(Boolean); + await writeFile(root + "/" + label + ".json", JSON.stringify(metrics, null, 2) + "\n"); + console.log( + JSON.stringify({ + phase: "complete", + label, + passed: metrics.passed, + reads: metrics.normalReadSummary, + writes: metrics.normalWriteSummary, + control: metrics.controlLatencySummary, + renewalLosses: metrics.renewalLosses, + dropped: metrics.dropped, + cpu: metrics.nodeCpuAverageOneCorePercent, + errors: metrics.errors.slice(0, 3), + }), + ); + await monitor.end(); + await closeRegisteredClients(u); + await admin.unsafe(`drop database ${dbName}`); + return metrics; +} +try { + await admin.unsafe("create extension if not exists pg_stat_statements"); + await setup(); + if (mode === "pilot") { + if (!(await run("candidate", 10, 9, 25000)).passed) process.exitCode = 1; + } else { + const profile = [ + ["baseline", 300], + ["candidate", 300], + ["candidate", 300], + ["baseline", 300], + ] as const; + const result = []; + for (const [variant, n] of profile) result.push(await run(variant, n, result.length + 1, 65000)); + const report = { + schema: "paperclip.normal-recovery-load/v1", + ...provenance, + scope: + "Paired baseline/candidate, including real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", + hardware: { + postgresCpuLimit: 2, + postgresMemoryLimitMiB: 1536, + mainPool: 10, + livenessPool: 2, + candidateCheckpointPool: 3, + }, + workload: { + mode, + normalReadsPerSecond: mode === "burst" ? 100 : 500, + normalWritesPerSecond: mode === "burst" ? 20 : 100, + databaseAdmissionEverySeconds: 5, + checkpointControlEverySeconds: 5, + controlPayloadBytes: 4096, + phaseSeconds: 65, + snapshotTransactionsPerControl: 1, + baselineCheckpointPool: 3, + baselinePoolPolicy: "Controlled baseline uses the common current DB factory's checkpoint cap of 3; this is not the unmodified historical cap-2 baseline.", + baselineRenewalIntervalSeconds: 20, + candidateRenewalIntervalSeconds: 20, + historicalIssues: 10000, + }, + results: result, + }; + const perOperation = (r: any, key: string) => + r[key] / (r.normalCompleted + r.normalWritesCompleted + r.controls + r.admissionsCompleted); + const average = (values: number[]) => values.reduce((a, b) => a + b, 0) / values.length; + const comparable = + result.every((r) => r.correctnessPassed) && + result.every( + (r) => + r.controls === result[0].controls && + r.normalCompleted === result[0].normalCompleted && + r.normalWritesCompleted === result[0].normalWritesCompleted, + ); + const relative = {}; + for (const key of ["postgresCpuUsec", "walBytes", "requestP99"]) { + const value = (r: any) => (key === "requestP99" ? r.normalReadSummary.p99 : perOperation(r, key)); + const baseline = average(result.filter((r) => r.variant === "baseline").map(value)); + const candidate = average(result.filter((r) => r.variant === "candidate").map(value)); + relative[key] = { + baseline, + candidate, + increasePercent: comparable ? 100 * (candidate / baseline - 1) : null, + passed: comparable ? candidate <= baseline * 1.1 : null, + }; + } + Object.assign(report, { + relative, + comparable, + passed: result.every((r) => r.passed) && Object.values(relative).every((r: any) => r.passed), + measurementLimits: [ + "Database request latency is not HTTP latency", + "Pool queue includes client encoding; BEGIN response includes network time", + "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed", + "Peak memory is sampled once a second", + "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide", + ], + }); + await writeFile(root + "/report.json", JSON.stringify(report, null, 2) + "\n"); + if (!(report as typeof report & { passed: boolean }).passed) process.exitCode = 1; + } +} finally { + await admin.end({ timeout: 1 }); +} +// On failure, retain owned databases for diagnosis. Successful trials are dropped; +// the random template remains available for fixture inspection until cluster cleanup. diff --git a/server/src/__tests__/heartbeat-execution-configuration.test.ts b/server/src/__tests__/heartbeat-execution-configuration.test.ts index 2cd52f84849..a38d0151087 100644 --- a/server/src/__tests__/heartbeat-execution-configuration.test.ts +++ b/server/src/__tests__/heartbeat-execution-configuration.test.ts @@ -2,18 +2,19 @@ import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js"; import { createHash, randomUUID } from "node:crypto"; import { trace } from "@opentelemetry/api"; import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace"; -import { access, chmod, mkdir, readdir, readFile, mkdtemp, rm, stat } from "node:fs/promises"; +import { access, chmod, mkdir, readdir, readFile, mkdtemp, rm, stat, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; import { eq } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; -import { agents, authUsers, companies, companyMemberships, companySkills, companySkillTestRuns, createDb, heartbeatRunEvents, heartbeatRuns, issues, projects, plugins, environments } from "@paperclipai/db"; +import { agents, authUsers, companies, companyMemberships, companySkills, companySkillTestRuns, createDb, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, issues, projects, plugins, environments } from "@paperclipai/db"; import { companySkillService } from "../services/company-skills.js"; import { withSkillVersionCacheLock } from "../services/skill-version-cache.js"; import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js"; import { HttpError } from "../errors.js"; import { heartbeatService } from "../services/heartbeat.js"; import { bindHeartbeatRunnerHomeMode, prepareHeartbeatExecutionConfiguration, type HeartbeatExecutionConfigurationInput } from "../services/heartbeat/execution-configuration.js"; +import { buildEffectiveRunSessionConfigMetadata, resolveTaskSessionConfigFreshness } from "../services/heartbeat/workspaces.js"; import { aiRuntimeConnectionBindingSchema } from "@paperclipai/shared"; import { secretService } from "../services/secrets.js"; import { instanceSettingsService } from "../services/instance-settings.js"; @@ -24,6 +25,12 @@ import * as assignedMcp from "../services/native-runtime/assigned-mcp-tools.js"; import * as sentry from "../sentry.js"; import { waitForPendingRunFailureReports } from "../services/run-failure-report.js"; import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js"; +import { selectHeartbeatEnvironment, type HeartbeatEnvironmentSelectionInput } from "../services/heartbeat/environment-selection.js"; +import { environmentService } from "../services/environments.js"; +import { executionWorkspaceService } from "../services/execution-workspaces.js"; +import { buildExecutionWorkspaceAdapterConfig } from "../services/execution-workspace-policy.js"; +import { buildNativeExecutionInput } from "../services/native-runtime/native-execution-input.js"; +import { nativeRuntimeContextFixture } from "../services/native-runtime/runtime-context.test-fixture.js"; function deferred() { let resolve!: (value: T) => void; @@ -35,6 +42,44 @@ const skillFailures = vi.hoisted(() => new Map()); const skillWaits = vi.hoisted(() => new Map Promise>()); describe("Runner home mode during recovery preparation", () => { + const fingerprint = (effectiveAdapterConfig: Record) => buildEffectiveRunSessionConfigMetadata({ + adapterType: "paperclip_runner", effectiveAdapterConfig, agentRuntimeConfig: {}, issueOverrides: null, + workspaceConfig: {}, environment: null, environmentEnv: null, projectEnv: null, routineEnv: null, runtimeSkills: [], + }); + it.each([ + { schema: "paperclip.native-execution-input.v5" }, + { schema: "paperclip.native-execution-input.v7", homeMode: "isolated" }, + ])("preserves the implicit isolated fingerprint through recovery and the next run: %j", async saved => { + const config = { provider: "codex", model: "fixture-model" }; + // This is the effective configuration shape built by preparation, including + // its optional property when no explicit home mode was configured. + const prepare = (input: unknown) => ({ ...config, homeMode: bindHeartbeatRunnerHomeMode(config, input).homeMode }); + const fresh = await fingerprint(prepare(null)); + const recovered = await fingerprint(prepare(saved)); + const next = await fingerprint(prepare(null)); + expect(recovered.fingerprint).toBe(fresh.fingerprint); + expect(next.fingerprint).toBe(recovered.fingerprint); + expect(resolveTaskSessionConfigFreshness({ hasTaskSession: true, configuredModel: config.model, + taskSessionParams: { __paperclipConfiguredModel: config.model, __paperclipConfigFingerprint: recovered.fingerprint, + __paperclipConfigFingerprintVersion: recovered.version, __paperclipConfigCategoryFingerprints: recovered.categoryFingerprints }, + configMetadata: next })).toMatchObject({ reset: false, changedCategories: [] }); + expect(config).not.toHaveProperty("homeMode"); + }); + + it("pins the admitted isolated mode but invalidates a genuinely preconfigured next run", async () => { + const config = { provider: "codex", model: "fixture-model", homeMode: "preconfigured" }; + const recoveredConfig = bindHeartbeatRunnerHomeMode(config, { schema: "paperclip.native-execution-input.v5" }); + expect(recoveredConfig.homeMode).toBe("isolated"); + expect(bindHeartbeatRunnerHomeMode(config, null).homeMode).toBe("preconfigured"); + const recovered = await fingerprint(recoveredConfig); + const next = await fingerprint(config); + expect(next.fingerprint).not.toBe(recovered.fingerprint); + expect(resolveTaskSessionConfigFreshness({ hasTaskSession: true, configuredModel: config.model, + taskSessionParams: { __paperclipConfiguredModel: config.model, __paperclipConfigFingerprint: recovered.fingerprint, + __paperclipConfigFingerprintVersion: recovered.version, __paperclipConfigCategoryFingerprints: recovered.categoryFingerprints }, + configMetadata: next })).toMatchObject({ reset: true, changedCategories: ["adapterConfig"] }); + }); + it.each([ ["preconfigured", { schema: "paperclip.native-execution-input.v5" }, "isolated"], ["preconfigured", { schema: "paperclip.native-execution-input.v7", homeMode: "isolated" }, "isolated"], @@ -395,6 +440,118 @@ describePostgres("heartbeat execution configuration through adapter dispatch", ( expect(initial.configSnapshot).toEqual(reused.configSnapshot); }); + it("keeps shared provider workspace configuration across native recovery and the next turn", async () => { + const f = await seed(); + const selectedSkills = []; + for (const name of ["Alphabetically first long skill", "B longer selected skill", "C", "D small"]) { + selectedSkills.push(await companySkillService(db).createLocalSkill(f.companyId, { name, markdown: "# Keep the saved result\n" })); + } + const instructionsRootPath = path.join(home, f.companyId); + await mkdir(instructionsRootPath); + await writeFile(path.join(instructionsRootPath, "AGENTS.md"), "Preserve the workspace across controller replacement.\n"); + const config = { provider: "codex", model: "fixture-model", lifecycleMode: "warm", idleTimeoutMs: 300_000, + codexPermissionMode: "full", paperclipSkillSync: { desiredSkills: selectedSkills.map(skill => skill.key) }, + instructionsRootPath, instructionsFilePath: path.join(instructionsRootPath, "AGENTS.md"), + instructionsEntryFile: "AGENTS.md", instructionsBundleMode: "managed", env: { OPENAI_API_KEY: "fixture-direct-key" } }; + const [environment] = await db.insert(environments).values({ name: `Directory ${f.companyId}`, driver: "plugin", + config: { pluginKey: "test.directory", driverKey: "tasks" } }).returning(); + const [agent] = await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: config, + defaultEnvironmentId: environment.id }).where(eq(agents.id, f.agentId)).returning(); + const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, + issueId: f.issueId, scopeKind: "issue", status: "running", runtimeMode: "native", responsibleUserId: f.userId, contextSnapshot: {} }).returning(); + const workspaceId = run.id; + const [workspace] = await db.insert(executionWorkspaces).values({ id: workspaceId, companyId: f.companyId, + projectId: issue.projectId, sourceIssueId: issue.id, name: "Shared task workspace", mode: "shared_workspace", + strategyType: "project_primary", providerType: "adapter_managed", cwd: `/workspaces/${workspaceId}` }).returning(); + const issueSettings = { mode: "shared_workspace" as const }; + const sourceConfig = (current: Record) => buildExecutionWorkspaceAdapterConfig({ agentConfig: current, + projectPolicy: null, issueSettings, mode: "shared_workspace", legacyUseProjectWorkspace: null }); + const selectionInput: HeartbeatEnvironmentSelectionInput = { + run, agent, task: { issueRef: { ...issue, executionWorkspaceId: null }, issueContext: issue, issueId: issue.id, + executionProjectId: issue.projectId, workspaceProjectContext: { hasWorkspace: false }, boundSourceWorkspace: null, + hasSelectedWorkspaceSource: false, taskWorkspaceIntent: null, responsibleUserId: f.userId, context: {}, + freshWorkspace: null }, + policy: { trustPreset: { kind: "standard", preset: "standard", boundary: null, sourcePresets: {} }, + requestedExecutionWorkspaceMode: "shared_workspace", isolatedWorkspacesEnabled: true, + projectExecutionWorkspacePolicy: null, issueExecutionWorkspaceSettings: issueSettings }, + config: { config, sourceWorkspaceConfig: sourceConfig(config), issueAssigneeOverrides: null }, + services: { environmentsSvc: environmentService(db), executionWorkspacesSvc: executionWorkspaceService(db), + instanceSettings: instanceSettingsService(db), findSharedWorkspaceHolder: vi.fn(async () => null), + findExecutionWorkspaceHolder: vi.fn(async () => null) }, + }; + const prepare = async () => { + const selected = await selectHeartbeatEnvironment(db, selectionInput); + const result = await prepareHeartbeatExecutionConfiguration(db, { run, agent, + task: { issueId: issue.id, context: {}, responsibleUserId: f.userId, acceptedPlanContinuationWake: false, + acceptedPlanWakeRoutingDecision: null, taskProjectContext: null, + routineEnvContext: { routineId: null, env: null, responsibleUserId: null } }, + config: { sourceWorkspaceConfig: selectionInput.config.sourceWorkspaceConfig, aiBinding: undefined }, + policy: { trustPreset: selectionInput.policy.trustPreset, useIsolatedTaskDirectory: selected.useIsolatedTaskDirectory, + explicitTaskDirectory: selected.explicitTaskDirectory, requestedExecutionWorkspaceMode: selectionInput.policy.requestedExecutionWorkspaceMode }, + environment: selected, services: { secretsSvc: secretService(db), companySkills: companySkillService(db), + appendRunEvent: vi.fn(), finalizeAiConnectionBusyDeferral: vi.fn() }, + onFailureSecretsResolved: () => {}, onManagedAiRuntimeAcquired: () => {}, onIdentityRedactorPrepared: () => {} }); + if (!result.prepared) throw new Error("Configuration was not prepared"); + const metadata = await buildEffectiveRunSessionConfigMetadata({ adapterType: "paperclip_runner", + effectiveAdapterConfig: result.runtimeConfig, agentRuntimeConfig: {}, issueOverrides: null, workspaceConfig: {}, + environment: null, environmentEnv: null, projectEnv: null, routineEnv: null, runtimeSkills: result.runtimeSkillEntries }); + return { selected, result, metadata }; + }; + const first = await prepare(); + expect(first.selected.useIsolatedTaskDirectory).toBe(false); + expect(first.result.runtimeConfig.workspaceStrategy).toBeUndefined(); + // Preserve the pre-fix fresh name/key query order, including exact versions. + expect(first.result.runtimeConfig.paperclipSkillSync).toEqual({ desiredSkills: selectedSkills.map(skill => ({ + key: skill.key, versionId: skill.currentVersionId, + })) }); + // Read the actual JSONB snapshot rather than reusing the pre-persistence + // JavaScript map: PostgreSQL key order differs for these four real keys. + const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + run.contextSnapshot = saved.contextSnapshot; + selectionInput.task.freshWorkspace = null; + selectionInput.task.issueRef!.executionWorkspaceId = workspaceId; + selectionInput.task.issueRef!.executionWorkspacePreference = "reuse_existing"; + await db.update(issues).set({ executionWorkspaceId: workspaceId, executionWorkspacePreference: "reuse_existing" }).where(eq(issues.id, issue.id)); + selectionInput.run.runnerProfileJson = { nativeExecutionInput: buildNativeExecutionInput({ + companyId: f.companyId, agentId: agent.id, runId: run.id, normalizedSessionId: randomUUID(), provider: "codex", + issue: { id: issue.id, identifier: "CFG-1", title: issue.title, description: null, workMode: "standard" }, + taskPrompt: "Continue the admitted work", workspace: { id: workspaceId, cwd: workspace.cwd!, repoUrl: null, repoRef: null, branchName: null }, + completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", + contract: { revision: "1", objective: "Retain the directory", criteria: [{ id: "objective", requirement: "Preserve the workspace" }] } }, + runtimeContext: nativeRuntimeContextFixture(), + }) }; + const recovered = await prepare(); + expect(recovered.selected.nativeRecoveryExecutionWorkspaceId).toBe(workspaceId); + expect(recovered.selected.useIsolatedTaskDirectory).toBe(false); + expect(recovered.result.runtimeSkillEntries).toEqual(first.result.runtimeSkillEntries); + expect(resolveTaskSessionConfigFreshness({ hasTaskSession: true, configuredModel: config.model, + taskSessionParams: { __paperclipConfiguredModel: config.model, __paperclipConfigFingerprint: first.metadata.fingerprint, + __paperclipConfigFingerprintVersion: first.metadata.version, __paperclipConfigCategoryFingerprints: first.metadata.categoryFingerprints }, + configMetadata: recovered.metadata })).toMatchObject({ reset: false, changedCategories: [] }); + expect(recovered.result.runtimeConfig).toEqual(first.result.runtimeConfig); + expect(recovered.metadata.fingerprint).toBe(first.metadata.fingerprint); + selectionInput.run.runnerProfileJson = {}; + run.contextSnapshot = {}; + const next = await prepare(); + expect(next.result.runtimeConfig).toEqual(first.result.runtimeConfig); + expect(next.metadata.fingerprint).toBe(first.metadata.fingerprint); + + selectionInput.config.sourceWorkspaceConfig = sourceConfig({ ...config, + paperclipSkillSync: { desiredSkills: selectedSkills.slice(1).map(skill => skill.key) } }); + const fewerSkills = await prepare(); + expect(fewerSkills.result.runtimeSkillEntries).toHaveLength(3); + expect(fewerSkills.metadata.fingerprint).not.toBe(first.metadata.fingerprint); + + // A real model change must still invalidate the next admission. + selectionInput.config.config = { ...config, model: "changed-model" }; + selectionInput.config.sourceWorkspaceConfig = sourceConfig(selectionInput.config.config); + const changed = await prepare(); + expect(changed.selected.useIsolatedTaskDirectory).toBe(false); + expect(changed.result.runtimeConfig.model).toBe("changed-model"); + expect(changed.metadata.fingerprint).not.toBe(first.metadata.fingerprint); + }, 30_000); + it.each([ { mode: "shared_workspace" as const, config: { workspaceStrategy: { type: "git_worktree", baseRef: "main" } } }, { mode: "agent_default" as const, config: {} }, diff --git a/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts b/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts index c7a3301ceee..3e952c66126 100644 --- a/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts +++ b/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts @@ -2,6 +2,9 @@ import { createDb } from "@paperclipai/db"; import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js"; import { dockerCommandRunner } from "./docker-command-runner.js"; +// Separate TypeScript fixture import time from actual controller admission. +process.send?.({ ready: true }); + process.once("message", async (input: { connectionString: string; container: string; diff --git a/server/src/__tests__/native-harness-backup-expiration.test.ts b/server/src/__tests__/native-harness-backup-expiration.test.ts index 2437d9711b5..cd242bd6392 100644 --- a/server/src/__tests__/native-harness-backup-expiration.test.ts +++ b/server/src/__tests__/native-harness-backup-expiration.test.ts @@ -412,6 +412,15 @@ describeEmbeddedPostgres("native harness backup expiration", () => { expect(await exists(session.current)).toBe(false); }); + it("retains incomplete native snapshot publications even after the issue closes", async () => { + const session = await seedSession({ finalization: { phase: "committed" } }); + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeControllerRecovery: { + schema: "paperclip.native-controller-recovery.v1", checkpoint: null, + } } }).where(eq(heartbeatRuns.id, session.runId)); + expect(await service().tick()).toMatchObject({ expired: 0, skipped: { native_recovery_incomplete: 1 } }); + expect(await exists(session.current)).toBe(true); + }); + it.each([ ["committed", "resuming_session"], ["terminal_failure", "blocked"], diff --git a/server/src/__tests__/native-session-resumption.test.ts b/server/src/__tests__/native-session-resumption.test.ts index 03ae5e6691c..47b8e01dfdc 100644 --- a/server/src/__tests__/native-session-resumption.test.ts +++ b/server/src/__tests__/native-session-resumption.test.ts @@ -9,12 +9,14 @@ import { eq, inArray } from "drizzle-orm"; import { activityLog, agentWakeupRequests, + agentTaskSessions, agents, companies, authUsers, companyMemberships, completionContracts, createDb, + closeRegisteredClients, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, @@ -78,6 +80,9 @@ vi.mock("../sentry.js", async () => { return { ...actual, captureRunFailure: mockCaptureRunFailure }; }); +import { createIsolatedTestHome } from "./helpers/isolated-test-home.js"; +import { issueService } from "../services/issues.js"; +import * as heartbeatWorkspaces from "../services/heartbeat/workspaces.js"; import { heartbeatService } from "../services/heartbeat.js"; import { persistHeartbeatRunProcessMetadata } from "../services/heartbeat/run-lifecycle.js"; import { instanceSettingsService } from "../services/instance-settings.js"; @@ -1512,4 +1517,165 @@ describe.each(["unchanged", "newer_active", "stale_idle"] as const)( ); + +describe("goal-less native task-session publication", () => { + let database: Awaited> | undefined; + let db: ReturnType; + beforeAll(async () => { + const external = process.env.PAPERCLIP_TEST_DATABASE_URL; + if (external) db = createDb(external); + else { + database = await startEmbeddedPostgresTestDatabase("paperclip-native-pointer-"); + db = createDb(database.connectionString); + } + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + afterAll(async () => { + await closeRegisteredClients(process.env.PAPERCLIP_TEST_DATABASE_URL ?? database!.connectionString); + await database?.cleanup(); + }); + + it.each(["retained", "reset"] as const)("publishes the completed task session before provider cleanup and resumes an ordinary comment (%s)", async mode => { + const home = createIsolatedTestHome(); + const previousHome = process.env.PAPERCLIP_HOME; + process.env.PAPERCLIP_HOME = home.home; + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(); + const issuePrefix = `NPT${companyId.slice(0, 6).toUpperCase()}`; + const projectId = randomUUID(), projectWorkspaceId = randomUUID(), workProductId = randomUUID(); + let providerSessionId: string | undefined; + const repoRoot = fileURLToPath(new URL("../../../", import.meta.url)); + const evidenceRef = `work_product:${workProductId}`; + const inputs: NativeExecutionInput[] = []; + const configs: Record[] = []; + const buildMetadata = heartbeatWorkspaces.buildEffectiveRunSessionConfigMetadata; + const configuration = vi.spyOn(heartbeatWorkspaces, "buildEffectiveRunSessionConfigMetadata").mockImplementation(input => { + configs.push(structuredClone(input.effectiveAdapterConfig)); + return buildMetadata(input); + }); + const openSession = vi.fn(), recoverSession = vi.fn(), startTurn = vi.fn(); + let releaseClose!: () => void, notifyClose!: () => void; + const closeHeld = new Promise(resolve => { releaseClose = resolve; }); + const closeEntered = new Promise(resolve => { notifyClose = resolve; }); + let firstRunId: string | undefined; + const backendFactory = (execution: NativeExecutionInput): NativeSessionBackend => { + inputs.push(structuredClone(execution)); + firstRunId ??= execution.binding.runId; + const currentRunId = execution.binding.runId; + const currentProviderSessionId = `provider-${execution.session.normalizedSessionId}`; + providerSessionId ??= currentProviderSessionId; + const turnId = `turn-${currentRunId}`; + const identity = { companyId, agentId, issueId, runId: currentRunId, sessionId: execution.session.normalizedSessionId }; + const result = structuredClone(CONTROL_PLANE_CONFORMANCE_RESULT); + result.completionClaim.contractRevision = execution.completionContract.contract.revision; + result.completionClaim.criteria = execution.completionContract.contract.criteria.map(criterion => ({ + ...result.completionClaim.criteria[0]!, criterionId: criterion.id, evidenceRefs: [evidenceRef], + })); + result.evidence = [{ kind: "work_product", ref: evidenceRef }]; + result.verification[0]!.artifactRef = evidenceRef; + let started!: () => void; + const turnStarted = new Promise(resolve => { started = resolve; }); + const session: NativeSession = { + identity: () => identity, + async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; }, + async *events() { + await turnStarted; + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, currentRunId)); + yield { schema: "paperclip.prp.event.v1", sourceEventId: `${currentRunId}:terminal`, sourceSeq: 1, + sourceInstanceId: run.runnerInstanceId!, sourceKind: "runner", runId: currentRunId, + normalizedSessionId: identity.sessionId, turnId, eventType: "turn.completed", schemaVersion: 1, + priority: 0, emittedAt: new Date().toISOString(), payload: {} } satisfies PrpEvent; + }, + async startTurn(input) { startTurn(input); started(); return { turnId }; }, + async result() { await turnStarted; return { result, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId }; }, + async snapshot() { return { backendKind: "mock", sessionId: "driver-retained", identity, providerSessionId: currentProviderSessionId, + activeTurnId: null, pendingRuntimeRequests: [], lineage: [], cursor: "1" }; }, + async close() { if (currentRunId === firstRunId) { notifyClose(); await closeHeld; } }, + }; + return { + async descriptor() { return { kind: "mock", name: "task-pointer-backend", version: "1", + capabilities: { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }, + runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } }; }, + async openSession(input) { openSession(input); return session; }, + async recoverSession(snapshot) { recoverSession(snapshot); expect(snapshot.providerSessionId).toBe(currentProviderSessionId); + return { recovered: true, session }; }, + }; + }; + const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: backendFactory }); + try { + await instanceSettingsService(db).updateExperimental({ enableNativeRunner: true, enableIsolatedWorkspaces: true }); + await db.insert(authUsers).values({ id: "pointer-user", name: "Pointer user", email: "pointer@example.test", createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing(); + await db.insert(companies).values({ id: companyId, name: "Native pointer", issuePrefix, status: "active", defaultResponsibleUserId: "pointer-user" }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "pointer-user", status: "active", membershipRole: "member" }); + await db.insert(projects).values({ id: projectId, companyId, name: "Pointer project", status: "active" }); + await db.insert(projectWorkspaces).values({ id: projectWorkspaceId, companyId, projectId, name: "Pointer workspace", cwd: repoRoot, isPrimary: true }); + await db.insert(agents).values({ id: agentId, companyId, name: "Pointer agent", adapterType: "paperclip_runner", status: "active", + adapterConfig: { provider: "codex", workspaceStrategy: { type: "project_primary" } }, + runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }); + await db.insert(issues).values({ id: issueId, companyId, projectId, projectWorkspaceId, issueNumber: 1, + identifier: `${issuePrefix}-1`, title: "Deliver the saved artifact", status: "in_progress", assigneeAgentId: agentId, workMode: "standard", + // Declare the shared mode up front, as the normal profile does. The + // first workspace otherwise materializes this setting and legitimately + // changes the effective adapter strategy before the next admission. + executionWorkspaceSettings: { mode: "shared_workspace" } }); + await db.insert(issueWorkProducts).values({ id: workProductId, companyId, issueId, type: "artifact", provider: "paperclip", + title: "Saved artifact", status: "ready_for_review", reviewState: "approved" }); + const first = await heartbeat.wakeup(agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented", + requestedByActorType: "user", requestedByActorId: "pointer-user", payload: { issueId }, contextSnapshot: { issueId, taskId: issueId, skipIssueComment: true } }); + expect(first).not.toBeNull(); + await closeEntered; + const [persisted] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, first!.id)); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, first!.id))).toHaveLength(1); + expect(persisted.runnerProfileJson?.sessionCheckpoint).toMatchObject({ identity: { sessionId: persisted.nativeSessionId }, + providerSessionId, semanticResult: expect.any(Object), terminal: { runTerminalState: "succeeded" } }); + const pointer = await db.select().from(agentTaskSessions).where(eq(agentTaskSessions.agentId, agentId)); + expect(pointer).toEqual([expect.objectContaining({ lastRunId: first!.id, + sessionParamsJson: expect.objectContaining({ sessionId: persisted.nativeSessionId }), sessionDisplayId: providerSessionId })]); + if (mode === "reset") { + expect(await heartbeat.resetRuntimeSession(agentId, { taskKey: issueId })).toMatchObject({ clearedTaskSessions: 1 }); + expect(await heartbeat.listTaskSessions(agentId)).toEqual([]); + } + releaseClose(); + await heartbeat.drainActiveRunExecutions(); + if (mode === "reset") expect(await heartbeat.listTaskSessions(agentId)).toEqual([]); + // The normal comment route reopens a completed task before its user wake. + // Use those public services; a raw comment on a still-done task is stale. + const issuesSvc = issueService(db); + await issuesSvc.update(issueId, { status: "todo", actorUserId: "pointer-user" }); + const comment = await issuesSvc.addComment(issueId, "Continue using the saved artifact.", { userId: "pointer-user" }); + const second = await heartbeat.wakeup(agentId, { source: "automation", triggerDetail: "system", reason: "issue_reopened_via_comment", + requestedByActorType: "user", requestedByActorId: "pointer-user", payload: { issueId, commentId: comment.id, mutation: "comment", reopenedFrom: "done" }, + contextSnapshot: { issueId, taskId: issueId, wakeCommentId: comment.id, reopenedFrom: "done", skipIssueComment: true } }); + expect(second).not.toBeNull(); + await heartbeat.drainActiveRunExecutions(); + const after = await heartbeat.getRun(second!.id); + expect(after?.status).toBe("succeeded"); + expect(inputs).toHaveLength(2); + if (mode === "retained") { + const changedKeys = [...new Set(configs.flatMap(config => Object.keys(config)))].filter(key => + JSON.stringify(configs[0]?.[key]) !== JSON.stringify(configs.at(-1)?.[key])); + expect(changedKeys).toEqual([]); + expect(after).toMatchObject({ nativeSessionId: persisted.nativeSessionId, sessionIdAfter: providerSessionId }); + expect(openSession).toHaveBeenCalledOnce(); + expect(recoverSession).toHaveBeenCalledOnce(); + } else { + expect(after?.nativeSessionId).not.toBe(persisted.nativeSessionId); + expect(after?.sessionIdAfter).not.toBe(providerSessionId); + expect(openSession).toHaveBeenCalledTimes(2); + expect(recoverSession).not.toHaveBeenCalled(); + } + expect(startTurn).toHaveBeenCalledTimes(2); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, second!.id))).toHaveLength(1); + } finally { + releaseClose(); + try { + await heartbeat.drainActiveRunExecutions(); + } finally { + configuration.mockRestore(); + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + home.cleanup(); + } + } + }, 30_000); +}); + enableRunFailureReporting(); diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index 8ba95a27e99..c2f0d519b3e 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -6149,9 +6149,14 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (!versionId || (selectedKeys && !selectedKeys.has(key))) continue; const [version] = await db.select({ skillId: companySkillVersions.companySkillId }).from(companySkillVersions) .where(and(eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.id, versionId))); - skills = skills.filter(skill => skill.key !== key); + // A saved JSONB selection map has a different key order from the fresh + // name/key query. Replace existing rows in place so pinning versions does + // not reorder the server-produced preference (and its session fingerprint). + const index = skills.findIndex(skill => skill.key === key); + if (index !== -1) skills.splice(index, 1); const owner = version ? await getById(companyId, version.skillId) : null; - if (owner) skills.push({ ...owner, key, slug: owner.key === key ? owner.slug : key.split("/").at(-1)! }); + if (owner) skills.splice(index === -1 ? skills.length : index, 0, + { ...owner, key, slug: owner.key === key ? owner.slug : key.split("/").at(-1)! }); } await packages.prefetch(companyId, skills.flatMap(skill => { const id = options.versionSelections?.get(skill.key) ?? skill.currentVersionId; diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 20b8ebe11c4..4b449d2477d 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { NATIVE_RETAINED_OWNER_WAIT } from "./native-runtime/native-controller-recovery.js"; import { externalConversationSettlementWork } from "./external-conversation-settlement-work.js"; import { nativeRecoveryNoticeWork } from "./native-runtime/native-recovery-notice-work.js"; import { nativeRunProjectionWork } from "./native-runtime/native-run-projection-work.js"; @@ -606,6 +607,9 @@ const nativeSessionResumeDispatchTimers = new Map< string, ReturnType >(); +// Routes and the scheduler can drain/resume through different service facades. +// Rearm through the original owner so its database and dispatch mode survive. +const nativeRecoveryDeadlines = new Map void }>(); // Shared with HTTP admission so an idle hold fences work before inspection. export { computeTaskDrain, applyTaskDrain, startTaskDrain, stopTaskDrain } from "./task-admission.js"; @@ -1334,6 +1338,9 @@ export function heartbeatService( } function resumeQueuedRunsAfterDrain() { + for (const pending of nativeRecoveryDeadlines.values()) { + pending.rearm(); + } const promise = heartbeatQueue.resumeQueuedRunsAfterDrain(); activeWakeupPromises.add(promise); void promise.then( @@ -1709,10 +1716,15 @@ export function heartbeatService( } } + // Native ownership waits survive a temporary drain without adding a polling + // loop. A process restart rediscovers the same deadlines from coordinator rows. function scheduleNativeSessionResumeDispatch( runId: string, nextAttemptAt: Date, + mode?: "restart", ) { + const pending = { rearm: () => scheduleNativeSessionResumeDispatch(runId, nextAttemptAt, mode) }; + nativeRecoveryDeadlines.set(runId, pending); const prior = nativeSessionResumeDispatchTimers.get(runId); if (prior) clearTimeout(prior); const delayMs = Math.max(0, nextAttemptAt.getTime() - Date.now()); @@ -1721,6 +1733,12 @@ export function heartbeatService( nativeSessionResumeDispatchTimers.delete(runId); void (async () => { if ((await getSchedulingSuppression()).suppressed) return; + if (nativeRecoveryDeadlines.get(runId) !== pending) return; + nativeRecoveryDeadlines.delete(runId); + if (mode === "restart") { + await recoverNativeRunsAfterRestart(new Date(), [runId]); + return; + } await dispatchNativeSessionResumptions({ db, runnerInstanceId: @@ -1742,6 +1760,9 @@ export function heartbeatService( }, }); })().catch((error) => { + if (!nativeRecoveryDeadlines.has(runId)) { + scheduleNativeSessionResumeDispatch(runId, new Date(Date.now() + 30_000), mode); + } logger.error( { err: error, runId }, "failed to dispatch scheduled native session resume", @@ -3532,7 +3553,8 @@ export function heartbeatService( err.original instanceof Error ? err.original.message : String(err.original ?? ""); - const retryReasonCode = /native_finalization_missing/i.test( + const waitingForRetainedOwner = retryMessage === NATIVE_RETAINED_OWNER_WAIT; + const retryReasonCode = waitingForRetainedOwner ? NATIVE_RETAINED_OWNER_WAIT : /native_finalization_missing/i.test( retryMessage, ) ? "semantic_result_missing" @@ -3552,7 +3574,8 @@ export function heartbeatService( stream: "system", level: "warn", message: - retryReasonCode === "semantic_result_missing" + waitingForRetainedOwner ? "native continuation waiting for the retained controller lease" + : retryReasonCode === "semantic_result_missing" ? "provider turn completed without a semantic result; same-run disposition recovery persisted" : "native session transport interrupted; same-run resume persisted", payload: { diff --git a/server/src/services/heartbeat/execution-configuration.ts b/server/src/services/heartbeat/execution-configuration.ts index fb9626b932f..6d88d5d7795 100644 --- a/server/src/services/heartbeat/execution-configuration.ts +++ b/server/src/services/heartbeat/execution-configuration.ts @@ -49,7 +49,11 @@ export function bindHeartbeatRunnerHomeMode(config: Record, per const saved = parseObject(persistedExecution); if (Object.keys(saved).length === 0) return config; // Execution schemas before v7 always used isolated homes. - return { ...config, homeMode: saved.homeMode ?? "isolated" }; + const homeMode = saved.homeMode ?? "isolated"; + // Omission already means isolated. Recovery must not add a fingerprinted + // configuration property merely to restate that unchanged default. + if (homeMode === "isolated" && config.homeMode === undefined) return config; + return { ...config, homeMode }; } export interface HeartbeatExecutionConfigurationInput { diff --git a/server/src/services/heartbeat/native-recovery-deadlines.test.ts b/server/src/services/heartbeat/native-recovery-deadlines.test.ts new file mode 100644 index 00000000000..e39daad1b52 --- /dev/null +++ b/server/src/services/heartbeat/native-recovery-deadlines.test.ts @@ -0,0 +1,102 @@ +import type { Db } from "@paperclipai/db"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { heartbeatService, startTaskDrain, stopTaskDrain } from "../heartbeat.js"; +import type { HeartbeatRecoveryDependencies } from "./recovery.js"; + +const hooks = vi.hoisted(() => ({ + schedule: undefined as HeartbeatRecoveryDependencies["scheduleNativeSessionResumeDispatch"] | undefined, + restart: vi.fn(async () => []), + resume: vi.fn(async () => []), + queue: vi.fn(async () => undefined), +})); + +vi.mock("./recovery.js", async original => { + const module = await original(); + return { ...module, createHeartbeatRecovery: (db: Db, dependencies: HeartbeatRecoveryDependencies) => { + hooks.schedule = dependencies.scheduleNativeSessionResumeDispatch; + return { ...module.createHeartbeatRecovery(db, dependencies), recoverNativeRunsAfterRestart: hooks.restart }; + } }; +}); +vi.mock("./queue.js", async original => { + const module = await original(); + return { ...module, createHeartbeatQueue: (...args: Parameters) => ({ + ...module.createHeartbeatQueue(...args), resumeQueuedRunsAfterDrain: hooks.queue, + }) }; +}); +vi.mock("../native-runtime/index.js", async original => ({ + ...await original(), + dispatchNativeSessionResumptions: hooks.resume, +})); + +describe("native deadline drain and resume", () => { + let heartbeat: ReturnType; + beforeEach(() => { + vi.clearAllMocks(); + vi.useFakeTimers(); + stopTaskDrain(); + const db = new Proxy({}, { get: () => { throw new Error("Unexpected deadline database access"); } }) as Db; + heartbeat = heartbeatService(db, { runtimeEnv: {} }); + }); + afterEach(async () => { + stopTaskDrain(); + await heartbeat?.resumeQueuedRunsAfterDrain(); + await vi.runOnlyPendingTimersAsync(); + await heartbeat?.drainActiveRunExecutions(); + vi.useRealTimers(); + }); + + it.each([undefined, "restart"] as const)("retains an overdue %s wait through drain with its original dispatch mode", async mode => { + const deadline = new Date(Date.now() + 100); + startTaskDrain({}); + hooks.schedule!("waiting-run", deadline, mode); + await vi.advanceTimersByTimeAsync(100); + expect(hooks.restart).not.toHaveBeenCalled(); + expect(hooks.resume).not.toHaveBeenCalled(); + stopTaskDrain(); + await heartbeat.resumeQueuedRunsAfterDrain(); + await vi.advanceTimersByTimeAsync(0); + const called = mode === "restart" ? hooks.restart : hooks.resume; + const other = mode === "restart" ? hooks.resume : hooks.restart; + expect(called).toHaveBeenCalledOnce(); + expect(other).not.toHaveBeenCalled(); + await heartbeat.resumeQueuedRunsAfterDrain(); + await vi.advanceTimersByTimeAsync(60_000); + expect(called).toHaveBeenCalledOnce(); + expect(vi.getTimerCount()).toBe(0); + }); + + it("preserves a future follow-up deadline when draining cancels its timer", async () => { + const deadline = new Date(Date.now() + 100); + hooks.schedule!("future-run", deadline); + await heartbeat.drainActiveRunExecutions(); + await heartbeat.resumeQueuedRunsAfterDrain(); + await vi.advanceTimersByTimeAsync(99); + expect(hooks.resume).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(1); + expect(hooks.resume).toHaveBeenCalledOnce(); + expect(hooks.restart).not.toHaveBeenCalled(); + }); + + it("rearms the original owner when another service facade ends the drain", async () => { + hooks.schedule!("other-facade-run", new Date(Date.now() + 100)); + const db = new Proxy({}, { get: () => { throw new Error("Unexpected replacement facade database access"); } }) as Db; + const other = heartbeatService(db, { runtimeEnv: {} }); + await other.drainActiveRunExecutions(); + await vi.advanceTimersByTimeAsync(100); + await other.resumeQueuedRunsAfterDrain(); + await vi.advanceTimersByTimeAsync(0); + expect(hooks.resume).toHaveBeenCalledOnce(); + expect(hooks.restart).not.toHaveBeenCalled(); + }); + + it("retries a failed follow-up obligation without changing its dispatch mode", async () => { + hooks.resume.mockRejectedValueOnce(new Error("temporary database failure")); + hooks.schedule!("failed-follow-up", new Date()); + await vi.advanceTimersByTimeAsync(0); + expect(hooks.resume).toHaveBeenCalledOnce(); + await vi.advanceTimersByTimeAsync(30_000); + expect(hooks.resume).toHaveBeenCalledTimes(2); + expect(hooks.restart).not.toHaveBeenCalled(); + expect(vi.getTimerCount()).toBe(0); + }); +}); diff --git a/server/src/services/heartbeat/recovery.test.ts b/server/src/services/heartbeat/recovery.test.ts index 893a1326816..a92ceff5823 100644 --- a/server/src/services/heartbeat/recovery.test.ts +++ b/server/src/services/heartbeat/recovery.test.ts @@ -1158,11 +1158,12 @@ describePostgres("heartbeat recovery module ownership", () => { await Promise.all([...deps.activeRunExecutionPromises]); }); - it("rearms the persisted native retry deadline through the service callback", async () => { + it.each(["future", "overdue-owner"])("rearms the persisted native retry deadline through the service callback: %s", async scenario => { const run = await seedRun(); - const nextAttemptAt = new Date(Date.now() + 60_000); + const nextAttemptAt = new Date(Date.now() + (scenario === "future" ? 60_000 : -1_000)); await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, status: "failed" }).where(eq(heartbeatRuns.id, run.id)); - await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!, phase: "retryable_failure", nextAttemptAt }); + await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!, phase: "retryable_failure", nextAttemptAt, + failureCode: scenario === "overdue-owner" ? "native_controller_recovery_waiting_for_owner" : null }); const deps = callbacks(db); const result = await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart(); expect(result.scheduledRetryRunIds).toEqual([run.id]); @@ -1173,6 +1174,16 @@ describePostgres("heartbeat recovery module ownership", () => { await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); }); + it("retains a live remote owner's actual lease deadline without dispatching competing work", async () => { + const deps = callbacks(db); + const runId = randomUUID(), retryAt = new Date(Date.now() + 60_000); + native.claims.mockResolvedValue([{ kind: "awaiting_evidence", runId, reason: "remote_controller_lease_live", retryAt }]); + const result = await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart(); + expect(result.awaitingEvidenceRunIds).toEqual([runId]); + expect(deps.scheduleNativeSessionResumeDispatch).toHaveBeenCalledWith(runId, retryAt, "restart"); + expect(deps.executeRun).not.toHaveBeenCalled(); + }); + it("retains shutdown ownership until background native cleanup physically settles", async () => { const deps = callbacks(db); let settle!: () => void; diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index fd5969a2088..d4e4d65c599 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -1,3 +1,4 @@ +import { NATIVE_RETAINED_OWNER_WAIT } from "../native-runtime/native-controller-recovery.js"; import { nativeEnvironmentSettlementDisposition, nativeEnvironmentSettlementWork } from "../native-runtime/native-environment-settlement-work.js"; import { nativeWorkspaceFinalizationWork } from "../native-runtime/native-workspace-finalization-work.js"; import { attemptNativeWorkspaceFinalization } from "../native-runtime/native-workspace-finalizer.js"; @@ -126,7 +127,7 @@ export interface HeartbeatRecoveryDependencies extends Pick Promise; }) => Promise; - scheduleNativeSessionResumeDispatch: (runId: string, nextAttemptAt: Date) => void; + scheduleNativeSessionResumeDispatch: (runId: string, nextAttemptAt: Date, mode?: "restart") => void; cancelHeartbeatNativeRun: (input: { db: Db; runId: string; reason: string; runtimeMode: string | null; }) => Promise; @@ -956,7 +957,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD inArray(heartbeatRuns.status, ["running", "failed"]), isNull(nativeRunFinalizations.resultId), eq(nativeRunFinalizations.phase, "retryable_failure"), - gt(nativeRunFinalizations.nextAttemptAt, now), + or(gt(nativeRunFinalizations.nextAttemptAt, now), eq(nativeRunFinalizations.failureCode, NATIVE_RETAINED_OWNER_WAIT)), ), ); for (const scheduled of scheduledNativeRetries) { @@ -1004,6 +1005,9 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD disposition.kind === "bootstrap_incomplete", ); for (const disposition of dispositions) { + if (disposition.kind === "awaiting_evidence" && disposition.retryAt) { + scheduleNativeSessionResumeDispatch(disposition.runId, disposition.retryAt, "restart"); + } const run = await getRun(disposition.runId); if (run) { const isClaim = diff --git a/server/src/services/heartbeat/run-completion.test.ts b/server/src/services/heartbeat/run-completion.test.ts index 62ce16df460..9b8a9810b2a 100644 --- a/server/src/services/heartbeat/run-completion.test.ts +++ b/server/src/services/heartbeat/run-completion.test.ts @@ -309,9 +309,16 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)( it("retains native completion metadata when reconciliation already committed the same terminal status", async () => { const finishedAt = new Date("2026-10-09T12:00:00Z"); const f = await fixture({ runtimeMode: "native", status: "succeeded", finishedAt, resultJson: { durable: "accepted" }, usageJson: { durableReceipt: "native" } }); - f.input.adapterResult.nativeFinalization = { schema: "paperclip.native-finalization.v1", runtimeMode: "native", runId: f.run.id, companyId: f.run.companyId, issueId: "unused", result: {}, terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", runTerminalState: "succeeded", reportedWorkDisposition: "done" }, turnId: null, sourceInstanceId: "test", normalizedSessionId: "native-session", providerSessionId: null, driverKind: "test", driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, workspaceFinalizeStatus: "succeeded" }; + await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, f.agent.id)); + f.agent.adapterType = "paperclip_runner"; + f.input.sessionCodec = getAdapterSessionCodec(f.agent.adapterType); + const nativeSessionId = randomUUID(), providerSessionId = randomUUID(); + f.input.adapterResult.sessionId = nativeSessionId; + delete f.input.adapterResult.sessionParams; + f.input.adapterResult.sessionDisplayId = providerSessionId; + f.input.adapterResult.nativeFinalization = { schema: "paperclip.native-finalization.v1", runtimeMode: "native", runId: f.run.id, companyId: f.run.companyId, issueId: "unused", result: {}, terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", runTerminalState: "succeeded", reportedWorkDisposition: "done" }, turnId: null, sourceInstanceId: "test", normalizedSessionId: nativeSessionId, providerSessionId, driverKind: "test", driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, workspaceFinalizeStatus: "succeeded" }; await f.completion.completeRun(f.input); - expect(await read(f.run.id)).toMatchObject({ status: "succeeded", finishedAt, sessionIdAfter: "provider-session", resultJson: { durable: "accepted" }, usageJson: { durableReceipt: "native", inputTokens: 20 } }); + expect(await read(f.run.id)).toMatchObject({ status: "succeeded", finishedAt, sessionIdAfter: providerSessionId, resultJson: { durable: "accepted" }, usageJson: { durableReceipt: "native", inputTokens: 20 } }); expect(f.deps.updateRuntimeState).toHaveBeenCalledOnce(); }); diff --git a/server/src/services/heartbeat/run-completion.ts b/server/src/services/heartbeat/run-completion.ts index c82816f384b..9a0696a1d11 100644 --- a/server/src/services/heartbeat/run-completion.ts +++ b/server/src/services/heartbeat/run-completion.ts @@ -1,4 +1,5 @@ import { preserveAcceptedRunUsageProjectionSql } from "../usage-receipts.js"; +import { nativeTaskSessionPublicationHandled } from "../native-runtime/native-task-session-publication.js"; import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js"; import { measureRunPhase } from "../execution-preparation-timing.js"; import { PROJECT_ACCESS_DENIED, ProjectAccessDeniedError } from "../project-access-denial.js"; @@ -489,6 +490,13 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC normalizedUsage, ); if (!taskKey) return; + if (nativeTaskSessionPublicationHandled(finalizedRun.runnerProfileJson, { + runId: finalizedRun.id, + companyId: finalizedRun.companyId, + agentId: agent.id, + taskKey, + adapterType: agent.adapterType, + })) return; if ( adapterResult.clearSession || (!nextSessionState.params && !nextSessionState.displayId) diff --git a/server/src/services/heartbeat/runtime-execution.test.ts b/server/src/services/heartbeat/runtime-execution.test.ts index 4f5b9c349dc..f39b180fbde 100644 --- a/server/src/services/heartbeat/runtime-execution.test.ts +++ b/server/src/services/heartbeat/runtime-execution.test.ts @@ -1,3 +1,5 @@ +import { NativeControllerRecoveryWaitingError } from "../native-runtime/native-controller-recovery.js"; + import { createNativeUsageDelivery } from "../native-runtime/native-usage-delivery.js"; import type { PrpEvent } from "../../vendor/paperclip-runner/index.js"; import { randomUUID } from "node:crypto"; @@ -10,7 +12,7 @@ import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vites import { trace } from "@opentelemetry/api"; import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace"; import { createRunPreparationTiming, measureRunPhase, measureWorkspaceSyncPhase, withRunExecutionTiming } from "../execution-preparation-timing.js"; -import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { recoveryPostgresSupport as support, startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js"; import * as pluginWorkspaces from "../plugin-workspaces.js"; import * as legacyRecovery from "../legacy-execution-recovery.js"; import { createAgentIdentityRedactor } from "../agent-identity-redaction.js"; @@ -43,15 +45,14 @@ vi.mock("./run-preparation.js", async original => ({ revokeHeartbeatRunGatewayTokens: vi.fn(async () => {}), })); -const support = await getEmbeddedPostgresTestSupport(); describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () => { - let database: Awaited>; + let database: Awaited>; let db: Db; let home: string; beforeAll(async () => { home = await mkdtemp(join(tmpdir(), "paperclip-runtime-execution-")); vi.stubEnv("PAPERCLIP_HOME", home); - database = await startEmbeddedPostgresTestDatabase("paperclip-runtime-execution"); + database = await startRecoveryPostgres("paperclip-runtime-execution"); db = createDb(database.connectionString); await db.execute(sql`set client_min_messages = warning`); }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); @@ -283,6 +284,18 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () = expectCleanup(input); }); + it("preserves the retained sandbox while native admission waits for its current owner", async () => { + const { input } = await fixture(true); + const error = new NativeControllerRecoveryWaitingError(new Date(Date.now() + 60_000)); + vi.mocked(executePaperclipNativeSession).mockRejectedValueOnce(error); + await expect(executeHeartbeatRuntime(db, input)).rejects.toMatchObject({ + name: "NativeSessionResumeScheduledError", original: error, + }); + expect(input.effects.onNativeSessionResumeScheduled).toHaveBeenCalledOnce(); + expect(input.effects.onProviderResourceDisposition).not.toHaveBeenCalledWith("stop_and_retain"); + expect(await finalizations()).toHaveLength(0); + }); + it("records native restoration inside the workspace finalization operation", async () => { const { input, run } = await fixture(true); const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT; diff --git a/server/src/services/heartbeat/runtime-execution.ts b/server/src/services/heartbeat/runtime-execution.ts index b5b64abc948..092639894cf 100644 --- a/server/src/services/heartbeat/runtime-execution.ts +++ b/server/src/services/heartbeat/runtime-execution.ts @@ -1,3 +1,5 @@ +import { NativeControllerRecoveryWaitingError } from "../native-runtime/native-controller-recovery.js"; + import { usesNativeUsageDelivery } from "../native-runtime/native-usage-accounting.js"; import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; import { writeExecutionControl } from "../execution-control-notifications.js"; @@ -559,6 +561,20 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe sessionGoalControl.requestId, )); if (completedGoalControl) sessionGoalControl = null; + // Goal checkpoints already publish their task session. Goal-less turns + // need the same durable pointer before cleanup can delay host completion. + const taskSessionPublication = taskKey && !sessionGoalControl && + context.resumeSessionGoalHeartbeat !== true && !completedGoalControl + ? { + taskKey, + adapterType: agent.adapterType, + sessionParamsJson: attachPaperclipSessionMetadataToSessionParams( + { ...runtimeSessionParamsForAdapter, cwd: executionWorkspace.cwd }, + configuredModel, + sessionConfigMetadata, + ), + } + : undefined; const nativeDispatchAtMs = Date.now(); const runCreatedAtMs = run.createdAt.getTime(); const runStartedAtMs = (run.startedAt ?? run.createdAt).getTime(); @@ -608,6 +624,7 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe resumeSessionGoalHeartbeat: context.resumeSessionGoalHeartbeat === true || completedGoalControl, + taskSessionPublication, onGoalCheckpoint: async (snapshot) => { if (!taskKey) return; const params = @@ -1026,6 +1043,10 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe preserveRunGatewayTokens = true; throw adapterErr; } + if (adapterErr instanceof NativeControllerRecoveryWaitingError) { + input.effects.onNativeSessionResumeScheduled(); + throw new NativeSessionResumeScheduledError(adapterErr); + } if (adapterErr instanceof NativeRunnerOwnershipUnverifiedError) { input.effects.onNativeOwnershipHeld(); throw adapterErr; diff --git a/server/src/services/heartbeat/upgrade-readiness.ts b/server/src/services/heartbeat/upgrade-readiness.ts index 1994ce5555e..a3036a4ee5f 100644 --- a/server/src/services/heartbeat/upgrade-readiness.ts +++ b/server/src/services/heartbeat/upgrade-readiness.ts @@ -4,6 +4,7 @@ import { inspectNativeRunnerUpgradeHeader } from "../native-runtime/native-runne import { readNativeWorkspaceSyncReference } from "../native-runtime/native-workspace-sync.js"; import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "../native-runtime/native-runner-recovery-contract.js"; import { currentNativeControllerIdentity } from "../native-runtime/native-restart-recovery.js"; +import { readNativeControllerRecoveryInventory } from "../native-runtime/native-controller-recovery-inventory.js"; const MAX_UPGRADE_RUNS = 1_000; type Counters = { activeRuns: number; pendingWakes: number; draining: boolean }; @@ -60,8 +61,16 @@ export async function readUpgradeReadiness(db: Db, input: { && identity.runnerInstanceId === row.runnerId && identity.normalizedSessionId === row.sessionId && identity.environmentLeaseId === (row.pluginLeaseId ?? workspace?.workspaceId)) }; }); - return summarizeUpgradeReadiness({ rows: candidates, before, after: input.counters(), + // Additive evidence does not strengthen or redefine the existing ready flag. + const retained = await readNativeControllerRecoveryInventory(db).catch(() => ({ + version: 1 as const, observedAt: new Date().toISOString(), coverage: "unknown" as const, + reasons: { inventory_unavailable: 1 }, fullPayloadsValidated: false as const, + })); + const summary = summarizeUpgradeReadiness({ rows: candidates, before, after: input.counters(), activeRunIds: [...new Set([...beforeIds, ...input.activeRunIds])], controllerBootId: owner.bootId }); + return { ...summary, nativeControllerRecovery: { activeExecutionAdoption: { + recoverableRuns: summary.recoverableRuns, blockingRuns: summary.blockingRuns, truncated: summary.truncated, + }, retained } }; } export function summarizeUpgradeReadiness(input: { diff --git a/server/src/services/native-runtime/native-controller-recovery-inventory.ts b/server/src/services/native-runtime/native-controller-recovery-inventory.ts new file mode 100644 index 00000000000..88f0077f9bd --- /dev/null +++ b/server/src/services/native-runtime/native-controller-recovery-inventory.ts @@ -0,0 +1,61 @@ +import type { Db } from "@paperclipai/db"; +import { sql } from "drizzle-orm"; +import { inspectNativeRunnerUpgradeHeader } from "./native-runner-checkpoint.js"; + +/** An explicit, paginated diagnostic. Never called by an idle/background scan. + * This inventories selected retained task-session pointers, not all history. + * Header evidence is preliminary: full snapshot/journal validation still occurs + * only when ordinary admission selects its exact continuation source. */ +export async function readNativeControllerRecoveryInventory(db: Db, input: { afterId?: string; limit?: number } = {}) { + if (input.limit !== undefined && !Number.isFinite(input.limit)) throw new Error("native_recovery_inventory_limit_invalid"); + const limit = Math.min(128, Math.max(1, Math.floor(input.limit ?? 128))); + const rows = await db.transaction(async tx => { + await tx.execute(sql`set local statement_timeout = '2s'`); + return tx.execute<{ + id: string; runId: string | null; companyId: string; issueId: string | null; sessionId: string | null; + runnerId: string | null; metadata: Record | null; material: Record | null; + pending: boolean; + }>(sql`with selected as ( + select id, company_id, agent_id, last_run_id from agent_task_sessions + where adapter_type = 'paperclip_runner' ${input.afterId ? sql`and id > ${input.afterId}::uuid` : sql``} + order by id limit ${limit + 1} + ) select s.id, s.company_id as "companyId", r.id as "runId", r.native_issue_id as "issueId", + r.native_session_id as "sessionId", r.runner_instance_id as "runnerId", + case when pg_column_size(r.runner_profile_json->'nativeControllerRecovery') <= 4096 + then r.runner_profile_json->'nativeControllerRecovery' else null end as metadata, + case when pg_column_size(c.material) <= 262144 and sum(pg_column_size(c.material)) over () <= 8388608 + then c.material else null end as material, + (r.finished_at is null or r.status not in ('succeeded', 'failed', 'cancelled', 'interrupted', 'timed_out') + or f.run_id is null or f.lease_owner is not null or f.recovery_state is not null + or f.phase not in ('committed', 'terminal_failure')) is true as pending + from selected s left join heartbeat_runs r on r.id = s.last_run_id and r.company_id = s.company_id and r.agent_id = s.agent_id + left join native_run_finalizations f on f.run_id = r.id and f.company_id = s.company_id + left join native_runner_checkpoints c on c.run_id = r.id and c.company_id = s.company_id + and c.issue_id = r.native_issue_id and c.part = 'header' + order by s.id`); + }, { accessMode: "read only" }); + const truncated = rows.length > limit; + const page = rows.slice(0, limit); + const reasons: Record = {}; + let selectedRetainedEvidence = 0, outstandingObligations = 0; + for (const row of page) { + let reason: string | null = null; + if (row.pending) { outstandingObligations++; reason = "persistence_or_handoff_pending"; } + else if (!row.runId || !row.issueId || !row.sessionId) reason = "selected_binding_missing"; + else if (row.metadata?.schema !== "paperclip.native-controller-recovery.v1") reason = "metadata_unsupported_or_missing"; + else { + const checkpoint = row.metadata.checkpoint as Record | null; + const header = row.material ? inspectNativeRunnerUpgradeHeader({ companyId: row.companyId, issueId: row.issueId, runId: row.runId }, row.material) : null; + if (!checkpoint || checkpoint.runId !== row.runId || checkpoint.runnerInstanceId !== row.runnerId + || checkpoint.normalizedSessionId !== row.sessionId || !header + || header.runnerInstanceId !== row.runnerId || header.normalizedSessionId !== row.sessionId) reason = "checkpoint_binding_missing"; + else selectedRetainedEvidence++; + } + if (reason) reasons[reason] = (reasons[reason] ?? 0) + 1; + } + return { version: 1 as const, observedAt: new Date().toISOString(), inspected: page.length, + coverage: truncated || input.afterId ? "partial" as const : "selected_task_session_pointers" as const, + truncated, nextAfterId: truncated ? page.at(-1)!.id : null, + selectedRetainedEvidence, outstandingObligations, reasons, + fullPayloadsValidated: false as const }; +} diff --git a/server/src/services/native-runtime/native-controller-recovery.integration.test.ts b/server/src/services/native-runtime/native-controller-recovery.integration.test.ts new file mode 100644 index 00000000000..2accfe2a6ca --- /dev/null +++ b/server/src/services/native-runtime/native-controller-recovery.integration.test.ts @@ -0,0 +1,165 @@ +import { randomUUID } from "node:crypto"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agentTaskSessions, agents, companies, createDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults } from "@paperclipai/db"; +import { DurablePrpControlPlane, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; +import { startRecoveryPostgres, recoveryPostgresSupport } from "../../__tests__/helpers/recovery-postgres.js"; +import { createIsolatedTestHome } from "../../__tests__/helpers/isolated-test-home.js"; +import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; +import { nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js"; +import { createNativeControllerRecovery, deferNativeControllerRecovery, importNativeControllerRecovery, NATIVE_RETAINED_OWNER_WAIT, resolveNativeControllerRecovery } from "./native-controller-recovery.js"; +import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js"; +import { claimNativeRestartRecoveries } from "./native-restart-recovery.js"; +import { readNativeControllerRecoveryInventory } from "./native-controller-recovery-inventory.js"; + +describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable discovery and import", () => { + let database: Awaited>; + let db: ReturnType; + let home: ReturnType; + let root: string; + const previousHome = process.env.PAPERCLIP_HOME; + const companyId = randomUUID(), agentId = randomUUID(); + const compatibility = { configDigest: `sha256:${"a".repeat(64)}`, processConfigDigest: null }; + beforeAll(async () => { + home = createIsolatedTestHome(); + process.env.PAPERCLIP_HOME = home.home; + root = await mkdtemp(join(tmpdir(), "native-controller-discovery-")); + database = await startRecoveryPostgres("native-controller-discovery-"); + db = createDb(database.connectionString); + await db.insert(companies).values({ id: companyId, name: "Controller discovery", issuePrefix: "NCD" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Runner", adapterType: "paperclip_runner", adapterConfig: { provider: "codex" } }); + }, 60_000); + afterAll(async () => { + await database?.cleanup(); + if (root) await rm(root, { recursive: true, force: true }); + home?.cleanup(); + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + }); + + async function seed() { + const issueId = randomUUID(), runId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Continue retained execution", assigneeAgentId: agentId, status: "in_progress" }); + const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: issueId }).returning(); + const native = await prepareNativeHeartbeatRun({ db, run, issue: { id: issueId, title: "Continue retained execution", description: null, reviewPolicy: null }, environmentLeaseId: randomUUID() }); + const owner = { companyId, issueId, runId, authorityRunId: runId, bootId: randomUUID(), generation: 1, leaseOwner: randomUUID() }; + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "observed", controllerBootId: owner.bootId, + controllerGeneration: 1, leaseOwner: owner.leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000) }); + const core = new DurablePrpControlPlane({ stateDirectory: join(root, runId), + identity: { runId, normalizedSessionId: native.normalizedSessionId, runnerInstanceId: native.runnerInstanceId, + environmentLeaseId: native.environmentLeaseId, turnId: native.turnId, itemId: native.itemId }, + expectedRunnerVersion: "discovery-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + core.setCheckpointPersistence(nativeRunnerCheckpointWriter(db, owner)); + await core.flushCheckpoint(); + await core.stop(); + const snapshot: PersistedNativeSession = { backendKind: "mock", sessionId: native.normalizedSessionId, + identity: { companyId, agentId, issueId, runId, sessionId: native.normalizedSessionId } }; + const [prepared] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const [result] = await db.insert(nativeRunResults).values({ companyId, issueId, runId, + completionContractId: prepared.completionContractId!, serverFingerprint: "fixture", schemaStatus: "accepted", + resultJson: {}, canonicalSha256: "fixture" }).returning(); + await db.update(nativeRunFinalizations).set({ phase: "committed", resultId: result.id, leaseOwner: null, leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, runId)); + const profile = { ...prepared.runnerProfileJson, sessionCheckpoint: snapshot }; + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date(), runnerProfileJson: profile }).where(eq(heartbeatRuns.id, runId)); + const scope = { companyId, agentId, issueId, runId, normalizedSessionId: native.normalizedSessionId }; + return { scope, native, profile, snapshot, owner, + supervisor: { schema: "paperclip.native-session-supervisor.v1", ...compatibility, snapshot } }; + } + + it("imports a retained source once, rejects stale input, and resolves it without any local projection", async () => { + const fixture = await seed(); + const input = { ...fixture.scope, expectedConfigDigest: compatibility.configDigest, supervisor: fixture.supervisor }; + expect(await importNativeControllerRecovery(db, { ...input, supervisor: { ...input.supervisor, snapshot: { ...fixture.snapshot, providerSessionId: "stale" } } })) + .toEqual({ kind: "missing_evidence", reason: "stale_supervisor_snapshot" }); + expect(await importNativeControllerRecovery(db, input)).toEqual({ kind: "imported" }); + expect(await importNativeControllerRecovery(db, input)).toEqual({ kind: "unchanged" }); + await rm(join(root, fixture.scope.runId), { recursive: true }); + const currentRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: currentRunId, companyId, agentId, nativeIssueId: fixture.scope.issueId, + nativeSessionId: fixture.scope.normalizedSessionId, status: "running", runtimeMode: "native" }); + const recovered = await resolveNativeControllerRecovery(db, { ...fixture.scope, currentRunId }); + expect(recovered.kind).toBe("recoverable"); + if (recovered.kind !== "recoverable") throw new Error(JSON.stringify(recovered)); + expect(recovered.snapshot).toEqual(fixture.snapshot); + expect(recovered.checkpoint.recoveryEvidence.runnerInstanceId).toBe(fixture.native.runnerInstanceId); + const deadline = new Date(Date.now() + 60_000); + await db.update(nativeRunFinalizations).set({ leaseOwner: "other-controller", leaseExpiresAt: deadline }).where(eq(nativeRunFinalizations.runId, fixture.scope.runId)); + expect(await importNativeControllerRecovery(db, input)).toEqual({ kind: "waiting_for_owner", deadline }); + expect(await resolveNativeControllerRecovery(db, { ...fixture.scope, currentRunId })).toEqual({ kind: "waiting_for_owner", deadline }); + }); + + it("holds mismatched snapshots and runner bindings instead of falling back to fresh execution", async () => { + const fixture = await seed(), currentRunId = randomUUID(); + const metadata = createNativeControllerRecovery(fixture.snapshot, compatibility, { runId: fixture.scope.runId, + runnerInstanceId: fixture.native.runnerInstanceId, normalizedSessionId: fixture.scope.normalizedSessionId }); + await db.insert(heartbeatRuns).values({ id: currentRunId, companyId, agentId, nativeIssueId: fixture.scope.issueId, + nativeSessionId: fixture.scope.normalizedSessionId, status: "running", runtimeMode: "native" }); + for (const envelope of [{ ...metadata, snapshotSha256: "wrong" }, { ...metadata, checkpoint: { ...metadata.checkpoint!, runnerInstanceId: randomUUID() } }]) { + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...fixture.profile, nativeControllerRecovery: envelope } }).where(eq(heartbeatRuns.id, fixture.scope.runId)); + expect(await resolveNativeControllerRecovery(db, { ...fixture.scope, currentRunId })) + .toMatchObject({ kind: "missing_evidence", reason: "snapshot_digest_or_checkpoint_binding" }); + } + }); + + it("keeps an unstarted follow-up on the same run until the retained owner's deadline", async () => { + const fixture = await seed(), runId = randomUUID(); + const scope = { ...fixture.scope, runId }; + const deadline = new Date(Date.now() + 60_000); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: scope.issueId, + nativeSessionId: scope.normalizedSessionId, status: "running", runtimeMode: "native", + // The admission boundary has already stored its input; the deferral + // and resumption owners never execute this transport-independent fixture. + runnerProfileJson: { nativeExecutionInput: { binding: scope } } }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, scope.issueId)); + await db.insert(nativeRunFinalizations).values({ companyId, issueId: scope.issueId, runId, phase: "observed" }); + await deferNativeControllerRecovery(db, { ...scope, retryAt: deadline }); + const read = () => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + expect((await read())[0]).toMatchObject({ phase: "retryable_failure", attempt: 0, leaseOwner: null, + nextAttemptAt: deadline, failureCode: NATIVE_RETAINED_OWNER_WAIT }); + expect(await claimNativeSessionResumptions({ db, runnerInstanceId: "successor", runIds: [runId] })).toEqual([]); + // Even after expiry, restart adoption cannot reinterpret this pre-provider + // wait as a crashed provider. The ordinary resumption owner consumes it. + expect(await claimNativeRestartRecoveries({ db, runIds: [runId], restartKind: "hard", now: deadline })).toEqual([]); + const claims = await claimNativeSessionResumptions({ db, runnerInstanceId: "successor", runIds: [runId], now: deadline }); + expect(claims).toHaveLength(1); + const nextDeadline = new Date(deadline.getTime() + 60_000); + await deferNativeControllerRecovery(db, { ...scope, leaseOwner: claims[0].leaseOwner, retryAt: nextDeadline }); + expect((await read())[0]).toMatchObject({ attempt: 0, leaseOwner: null, nextAttemptAt: nextDeadline }); + await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, runId)); + expect(await claimNativeSessionResumptions({ db, runnerInstanceId: "successor", runIds: [runId], now: nextDeadline })).toEqual([]); + }); + + it.each(["live-owner", "expired-owner", "provider-attempt", "stopped", "operator-stop"])("does not publish an admission wait after authority changes: %s", async scenario => { + const fixture = await seed(), runId = randomUUID(); + const scope = { ...fixture.scope, runId }; + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: scope.issueId, + status: scenario === "stopped" ? "cancelled" : "running", runtimeMode: "native", + resultJson: scenario === "operator-stop" ? { cancellation: { source: "operator", expected: true, + initiator: { type: "user" }, reason: "Stop", recordedAt: new Date().toISOString() } } : null, + runnerProfileJson: { nativeExecutionInput: { binding: scope } } }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, scope.issueId)); + await db.insert(nativeRunFinalizations).values({ companyId, issueId: scope.issueId, runId, phase: "observed", + attempt: scenario === "provider-attempt" ? 1 : 0, + leaseOwner: scenario.endsWith("-owner") ? "other-controller" : null, + leaseExpiresAt: scenario === "expired-owner" ? new Date(Date.now() - 1_000) : null }); + await expect(deferNativeControllerRecovery(db, { ...scope, + leaseOwner: scenario === "expired-owner" ? "other-controller" : undefined, + retryAt: new Date(Date.now() + 60_000) })) + .rejects.toThrow("wait_authority_changed"); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0].phase).toBe("observed"); + }); + + it("reports bounded retained-pointer coverage without loading historical payloads", async () => { + const fixture = await seed(); + await importNativeControllerRecovery(db, { ...fixture.scope, expectedConfigDigest: compatibility.configDigest, supervisor: fixture.supervisor }); + await db.insert(agentTaskSessions).values([0, 1].map(index => ({ companyId, agentId, taskKey: `${fixture.scope.issueId}-${index}`, + adapterType: "paperclip_runner", lastRunId: fixture.scope.runId }))); + const first = await readNativeControllerRecoveryInventory(db, { limit: 1 }); + expect(first).toMatchObject({ inspected: 1, coverage: "partial", truncated: true, selectedRetainedEvidence: 1, fullPayloadsValidated: false }); + const next = await readNativeControllerRecoveryInventory(db, { limit: 1, afterId: first.nextAfterId! }); + expect(next).toMatchObject({ inspected: 1, coverage: "partial", truncated: false, selectedRetainedEvidence: 1 }); + }); +}); diff --git a/server/src/services/native-runtime/native-controller-recovery.test.ts b/server/src/services/native-runtime/native-controller-recovery.test.ts new file mode 100644 index 00000000000..261763192fb --- /dev/null +++ b/server/src/services/native-runtime/native-controller-recovery.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, it } from "vitest"; +import { createNativeControllerRecovery, readNativeControllerRecovery } from "./native-controller-recovery.js"; +import { nativeSha256 } from "./canonical.js"; +import { redactRegisteredSecretValues } from "../run-secret-redaction.js"; +import type { PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; + +const snapshot: PersistedNativeSession = { backendKind: "mock", sessionId: "provider", + identity: { companyId: "company", agentId: "agent", issueId: "issue", runId: "run", sessionId: "session" } }; +const compatibility = { configDigest: `sha256:${"a".repeat(64)}`, processConfigDigest: null }; +const binding = { runId: "run", runnerInstanceId: "runner", normalizedSessionId: "session" }; + +describe("native controller recovery envelope", () => { + it("binds the stored snapshot, including redaction, without serializing its contents twice", () => { + const stored = { ...snapshot, providerSessionId: "[REDACTED]" }; + const envelope = createNativeControllerRecovery(stored, compatibility, binding); + expect(envelope.snapshotSha256).toBe(nativeSha256(stored)); + expect(readNativeControllerRecovery(envelope, JSON.parse(JSON.stringify(stored)))).toEqual(envelope); + expect(readNativeControllerRecovery(envelope, { ...stored, providerSessionId: "secret-before-redaction" })).toBeNull(); + expect(JSON.stringify(envelope)).not.toContain("[REDACTED]"); + expect(Buffer.byteLength(JSON.stringify(envelope))).toBeLessThan(4096); + }); + + it("does not certify a missing PRP publication or unknown format", () => { + const pending = createNativeControllerRecovery(snapshot, compatibility, null); + expect(readNativeControllerRecovery(pending, snapshot)).toBeNull(); + const published = createNativeControllerRecovery(snapshot, compatibility, binding); + expect(readNativeControllerRecovery({ ...published, schema: "future" }, snapshot)).toBeNull(); + expect(readNativeControllerRecovery({ ...published, processConfigDigest: "unverified" }, snapshot)).toBeNull(); + }); + + it("enforces the metadata byte bound on both write and read", () => { + expect(() => createNativeControllerRecovery(snapshot, { ...compatibility, configDigest: "x".repeat(4096) }, binding)) + .toThrow("metadata_too_large"); + expect(readNativeControllerRecovery({ ...createNativeControllerRecovery(snapshot, compatibility, binding), extra: "x".repeat(4096) }, snapshot)) + .toBeNull(); + }); + + it("keeps compatibility metadata out of public run projections", () => { + expect(redactRegisteredSecretValues({ runnerProfileJson: { preserved: true, + nativeControllerRecovery: createNativeControllerRecovery(snapshot, compatibility, binding) } }, [])) + .toEqual({ runnerProfileJson: { preserved: true } }); + }); +}); diff --git a/server/src/services/native-runtime/native-controller-recovery.ts b/server/src/services/native-runtime/native-controller-recovery.ts new file mode 100644 index 00000000000..4c5fd4ac419 --- /dev/null +++ b/server/src/services/native-runtime/native-controller-recovery.ts @@ -0,0 +1,270 @@ +import { and, eq, ne, sql } from "drizzle-orm"; +import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db"; +import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; +import type { DurablePrpCheckpoint, PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; +import { readRunCancellation } from "../run-cancellation.js"; +import { nativeSha256 } from "./canonical.js"; +import { findNativeSessionResumeRun, isNativeSessionId, isUnusedNativeSessionBootstrap, nativeSessionProviderEvidence } from "./native-session-resume.js"; +import { loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; +import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js"; + +export const NATIVE_CONTROLLER_RECOVERY_MAX_BYTES = 4 * 1024; + +/** Private run-profile metadata. No credentials or controller-local paths. */ +export interface NativeControllerRecovery { + schema: "paperclip.native-controller-recovery.v1"; + snapshotSha256: string; + configDigest: string; + processConfigDigest: string | null; + checkpoint: { + runId: string; + runnerInstanceId: string; + normalizedSessionId: string; + } | null; +} + +export type NativeControllerCompatibility = Pick; + +export function createNativeControllerRecovery( + snapshot: PersistedNativeSession, + compatibility: NativeControllerCompatibility, + checkpoint: NativeControllerRecovery["checkpoint"], +): NativeControllerRecovery { + const value: NativeControllerRecovery = { + schema: "paperclip.native-controller-recovery.v1", + snapshotSha256: nativeSha256(snapshot), + configDigest: compatibility.configDigest, + processConfigDigest: compatibility.processConfigDigest, + checkpoint, + }; + if (Buffer.byteLength(JSON.stringify(value)) > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES) { + throw new Error("native_controller_recovery_metadata_too_large"); + } + return value; +} + +function record(value: unknown): Record { + return value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} + +export function readNativeControllerRecovery(value: unknown, snapshot: unknown): NativeControllerRecovery | null { + const envelope = record(value), checkpoint = record(envelope.checkpoint); + if (Buffer.byteLength(JSON.stringify(value) ?? "null") > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES + || envelope.schema !== "paperclip.native-controller-recovery.v1" + || typeof envelope.snapshotSha256 !== "string" || envelope.snapshotSha256 !== nativeSha256(snapshot) + || typeof envelope.configDigest !== "string" || !/^sha256:[a-f0-9]{64}$/.test(envelope.configDigest) + || !(envelope.processConfigDigest === null || (typeof envelope.processConfigDigest === "string" && /^[a-f0-9]{64}$/.test(envelope.processConfigDigest))) + || ![checkpoint.runId, checkpoint.runnerInstanceId, checkpoint.normalizedSessionId].every(v => typeof v === "string" && v.length > 0)) return null; + return envelope as unknown as NativeControllerRecovery; +} + +export type NativeControllerRecoveryOutcome = + | { kind: "recoverable"; snapshot: PersistedNativeSession; metadata: NativeControllerRecovery; checkpoint: DurablePrpCheckpoint } + | { kind: "waiting_for_owner"; deadline: Date } + | { kind: "missing_evidence" | "unsupported" | "dependency_unavailable"; reason: string; cause?: unknown } + | { kind: "no_prior_execution" }; + +export const NATIVE_RETAINED_OWNER_WAIT = "native_controller_recovery_waiting_for_owner"; + +export class NativeControllerRecoveryWaitingError extends Error { + constructor(readonly retryAt: Date) { + super(NATIVE_RETAINED_OWNER_WAIT); + this.name = "NativeControllerRecoveryWaitingError"; + } +} + +/** Admission has not crossed the provider boundary. Retain this same business + * run on the existing retry owner; never turn an ownership wait into a failure + * or invent a new provider attempt. The usual resumption gates recheck Stop, + * permissions and task ownership when this deadline arrives. */ +export async function deferNativeControllerRecovery(db: Db, input: { + companyId: string; agentId: string; issueId: string; runId: string; + leaseOwner?: string; retryAt: Date; +}): Promise { + await db.transaction(async tx => { + const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues) + .where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update"); + const [owner] = await tx.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), + eq(nativeRunFinalizations.runId, input.runId))).for("update"); + const [run] = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId))).for("update"); + const [authority] = await tx.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints) + .where(and(eq(nativeRunnerCheckpoints.companyId, input.companyId), eq(nativeRunnerCheckpoints.runId, input.runId))).limit(1); + const progress = await nativeSessionProviderEvidence(tx as unknown as Db, [input.runId]); + const cancellation = record(run?.resultJson?.nativeCancellation); + const stop = readRunCancellation(run?.resultJson); + const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`); + if (!run || run.runtimeMode !== "native" || run.status !== "running" || task?.executionRunId !== input.runId + || !owner || owner.phase !== "observed" || owner.resultId || owner.attempt !== 0 + || owner.controllerGeneration !== 0 || owner.leaseOwner !== (input.leaseOwner ?? null) + || (input.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt <= new Date(clock.now))) + || run.processPid || run.processGroupId || run.processStartedAt || authority || progress.size + || !record(run.runnerProfileJson).nativeExecutionInput || run.resultJson?.startupCancellation + || (stop?.source === "operator" && stop.expected) + || (["run", "issue"].includes(String(cancellation.scope)) && ["pending", "acknowledged"].includes(String(cancellation.dispatchState)))) { + throw new Error("native_controller_recovery_wait_authority_changed"); + } + await notifyNativeFinalizationWork(tx, input); + await tx.update(nativeRunFinalizations).set({ + phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null, + nextAttemptAt: input.retryAt, recoveryState: "resuming_session", failureCode: NATIVE_RETAINED_OWNER_WAIT, + failureDetail: { originalFailureCode: NATIVE_RETAINED_OWNER_WAIT, + message: "Waiting for the retained native controller owner to release its lease.", + recoveryMode: record(run.runnerProfileJson).sessionCheckpoint ? "same_session_resume" : "bootstrap_retry", + providerSessionEstablished: false, providerEventsExist: false }, + updatedAt: new Date(), + }).where(eq(nativeRunFinalizations.runId, input.runId)); + }); +} + +/** Bind a read of retained Runner lifecycle to the surviving sandbox. The + * ordinary admission owner still checks current permissions before execution. */ +export async function assertNativeRetainedSandboxBinding(db: Db, input: { + companyId: string; issueId: string; runId: string; + target: Extract; +}): Promise { + const { target } = input; + if (target.transport !== "sandbox" || target.sandboxLeaseAcquisition?.outcome !== "resumed" + || !target.providerKey || !target.environmentId || !target.leaseId) { + throw new Error("native_controller_recovery_dependency_unavailable:retained_sandbox"); + } + const [owner] = await db.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), + eq(nativeRunFinalizations.runId, input.runId))); + const leases = await db.select({ metadata: environmentLeases.metadata, status: environmentLeases.status }).from(environmentLeases).where(and( + eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.issueId, input.issueId), + eq(environmentLeases.heartbeatRunId, input.runId), eq(environmentLeases.environmentId, target.environmentId), + eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, target.sandboxLeaseAcquisition.providerLeaseId))); + if (!owner || owner.leaseOwner || owner.phase !== "committed" || !owner.resultId || owner.recoveryState + || leases.length !== 1 || leases[0].metadata?.remoteCwd !== target.remoteCwd + || !["active", "released", "expired"].includes(leases[0].status)) { + throw new Error("native_controller_recovery_missing_evidence:retained_sandbox_binding"); + } +} + +/** Only the existing bounded selector chooses history. Never search backwards + * again when its selected record is incomplete or incompatible. This operation + * reads evidence; it grants neither command nor acknowledgement authority. */ +export async function resolveNativeControllerRecovery(db: Db, input: { + companyId: string; agentId: string; issueId: string; normalizedSessionId: string; currentRunId: string; +}): Promise { + if (!isNativeSessionId(input.normalizedSessionId)) return { kind: "missing_evidence", reason: "session_binding" }; + const [current] = await db.select({ createdAt: heartbeatRuns.createdAt, processPid: heartbeatRuns.processPid, + processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt, + runnerProfileJson: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.currentRunId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId), + eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))); + if (!current) return { kind: "missing_evidence", reason: "current_binding" }; + const source = await findNativeSessionResumeRun(db, { ...input, beforeCreatedAt: current.createdAt, includeProgressBarrier: true }); + if (!source) { + // An exhausted candidate window is unknown history, not fresh-bootstrap + // proof. Read only IDs, and retain the selector's existing 32-row boundary. + const prior = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + ne(heartbeatRuns.id, input.currentRunId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId), + eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).limit(33); + if (prior.length > 32) return { kind: "missing_evidence", reason: "candidate_window_exhausted" }; + const progress = await nativeSessionProviderEvidence(db, [input.currentRunId]); + if (!isUnusedNativeSessionBootstrap(current, progress.has(input.currentRunId))) { + return { kind: "missing_evidence", reason: "current_execution_progress" }; + } + // The normal admission and remote untouched-session claim still execute. + return { kind: "no_prior_execution" }; + } + const [owner] = await db.select({ leaseOwner: nativeRunFinalizations.leaseOwner, leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt }).from(nativeRunFinalizations) + .where(and(eq(nativeRunFinalizations.runId, source.id), eq(nativeRunFinalizations.companyId, input.companyId), + eq(nativeRunFinalizations.issueId, input.issueId))); + if (owner?.leaseOwner && owner.leaseExpiresAt && owner.leaseExpiresAt.getTime() > Date.now()) { + return { kind: "waiting_for_owner", deadline: owner.leaseExpiresAt }; + } + if (source.runtimeMode !== "native" || !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(source.status) + || source.createdAt > current.createdAt) return { kind: "missing_evidence", reason: "newer_progress_barrier" }; + const profile = record(source.runnerProfileJson), snapshot = record(profile.sessionCheckpoint); + const identity = record(snapshot.identity); + if (identity.companyId !== input.companyId || identity.agentId !== input.agentId + || identity.issueId !== input.issueId || identity.runId !== source.id || identity.sessionId !== input.normalizedSessionId) { + return { kind: "missing_evidence", reason: "snapshot_binding" }; + } + if (record(profile.nativeControllerRecovery).schema !== "paperclip.native-controller-recovery.v1") { + return { kind: "unsupported", reason: "metadata_format" }; + } + const metadata = readNativeControllerRecovery(profile.nativeControllerRecovery, snapshot); + if (!metadata?.checkpoint || metadata.checkpoint.runId !== source.id + || metadata.checkpoint.normalizedSessionId !== input.normalizedSessionId + || metadata.checkpoint.runnerInstanceId !== source.runnerInstanceId) { + return { kind: "missing_evidence", reason: "snapshot_digest_or_checkpoint_binding" }; + } + let checkpoint: DurablePrpCheckpoint | null; + try { + checkpoint = await loadNativeRunnerCheckpoint(db, { companyId: input.companyId, issueId: input.issueId, runId: source.id }); + } catch (cause) { + return { kind: "dependency_unavailable", reason: "checkpoint_authority_unreadable", cause }; + } + if (!checkpoint || checkpoint.recoveryEvidence.runnerInstanceId !== metadata.checkpoint.runnerInstanceId + || checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId) { + return { kind: "missing_evidence", reason: "checkpoint_binding" }; + } + return { kind: "recoverable", snapshot: snapshot as unknown as PersistedNativeSession, metadata, checkpoint }; +} + +/** Bounded internal migration of ONE retained source while its disk is available. + * The migration owner supplies compatibility computed from the admitted input. + * This never repairs/replaces the saved snapshot or claims a live execution. */ +export async function importNativeControllerRecovery(db: Db, input: { + companyId: string; agentId: string; issueId: string; runId: string; normalizedSessionId: string; + expectedConfigDigest: string; + supervisor: unknown; +}): Promise<{ kind: "imported" | "unchanged" } | Exclude> { + const supervisor = record(input.supervisor); + if (supervisor.schema !== "paperclip.native-session-supervisor.v1" || supervisor.configDigest !== input.expectedConfigDigest) { + return { kind: "unsupported", reason: "supervisor_compatibility" }; + } + return db.transaction(async tx => { + const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues) + .where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("share"); + const [owner] = await tx.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), + eq(nativeRunFinalizations.runId, input.runId))).for("update"); + if (!task || !owner) return { kind: "missing_evidence" as const, reason: "owner_binding" }; + if (owner.leaseOwner || task.executionRunId) { + return owner.leaseExpiresAt && owner.leaseExpiresAt > new Date() + ? { kind: "waiting_for_owner" as const, deadline: owner.leaseExpiresAt } + : { kind: "missing_evidence" as const, reason: "normal_ownership_recovery_required" }; + } + if (owner.phase !== "committed" || !owner.resultId || owner.recoveryState) { + return { kind: "missing_evidence" as const, reason: "pending_finalization" }; + } + const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), + eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId), + eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).for("update"); + const profile = record(run?.runnerProfileJson), snapshot = record(profile.sessionCheckpoint), identity = record(snapshot.identity); + if (run?.runtimeMode !== "native" || run.status !== "succeeded" || !run.finishedAt + || identity.runId !== input.runId || identity.companyId !== input.companyId || identity.issueId !== input.issueId + || identity.agentId !== input.agentId || identity.sessionId !== input.normalizedSessionId + || nativeSha256(snapshot) !== nativeSha256(redactRunnerApiPayloads(supervisor.snapshot))) { + return { kind: "missing_evidence" as const, reason: "stale_supervisor_snapshot" }; + } + const checkpoint = await loadNativeRunnerCheckpoint(tx, input); + if (!checkpoint || checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId + || checkpoint.recoveryEvidence.runnerInstanceId !== run.runnerInstanceId) { + return { kind: "missing_evidence" as const, reason: "checkpoint_binding" }; + } + const metadata = createNativeControllerRecovery(snapshot as unknown as PersistedNativeSession, { + configDigest: input.expectedConfigDigest, + processConfigDigest: typeof supervisor.processConfigDigest === "string" ? supervisor.processConfigDigest : null, + }, { runId: input.runId, runnerInstanceId: checkpoint.recoveryEvidence.runnerInstanceId, normalizedSessionId: input.normalizedSessionId }); + if (!readNativeControllerRecovery(metadata, snapshot)) return { kind: "unsupported" as const, reason: "metadata_format" }; + if (profile.nativeControllerRecovery) { + return nativeSha256(profile.nativeControllerRecovery) === nativeSha256(metadata) + ? { kind: "unchanged" as const } + : { kind: "missing_evidence" as const, reason: "existing_metadata_conflict" }; + } + await tx.update(heartbeatRuns).set({ runnerProfileJson: { ...profile, nativeControllerRecovery: metadata }, updatedAt: new Date() }) + .where(eq(heartbeatRuns.id, input.runId)); + return { kind: "imported" as const }; + }); +} diff --git a/server/src/services/native-runtime/native-harness-backup-expiration.ts b/server/src/services/native-runtime/native-harness-backup-expiration.ts index 8ea9a242629..34402b8ad2d 100644 --- a/server/src/services/native-runtime/native-harness-backup-expiration.ts +++ b/server/src/services/native-runtime/native-harness-backup-expiration.ts @@ -181,6 +181,8 @@ async function sessionRetentionBlocker( id: heartbeatRuns.id, status: heartbeatRuns.status, finishedAt: heartbeatRuns.finishedAt, + incompleteRecovery: sql`(${heartbeatRuns.runnerProfileJson}->'nativeControllerRecovery'->>'schema' = 'paperclip.native-controller-recovery.v1' + and ${heartbeatRuns.runnerProfileJson}->'nativeControllerRecovery'->'checkpoint' = 'null'::jsonb) is true`, issueId: sql`coalesce(${heartbeatRuns.nativeIssueId}, ${heartbeatRuns.issueId})`, }) .from(heartbeatRuns) @@ -194,6 +196,7 @@ async function sessionRetentionBlocker( if (runs.some((run) => LIVE_RUN_STATUSES.includes(run.status) || !run.finishedAt)) { return { reason: "live_run" }; } + if (runs.some(run => run.incompleteRecovery)) return { reason: "native_recovery_incomplete" }; if (input.checkpoint && runs.some(run => run.finishedAt! >= input.closedBefore)) return { reason: "recent_run" }; const issueIds = [...new Set(runs.map((run) => run.issueId).filter((id): id is string => Boolean(id)))]; if (issueIds.length === 0) return { reason: "issue_unknown" }; diff --git a/server/src/services/native-runtime/native-recovery-usage-delivery.test.ts b/server/src/services/native-runtime/native-recovery-usage-delivery.test.ts new file mode 100644 index 00000000000..c2fdaec2e21 --- /dev/null +++ b/server/src/services/native-runtime/native-recovery-usage-delivery.test.ts @@ -0,0 +1,92 @@ +import { randomUUID } from "node:crypto"; +import { PgDialect } from "drizzle-orm/pg-core"; +import type { SQL } from "drizzle-orm"; +import type { Db } from "@paperclipai/db"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import type { DurableRecoveryIdentity, PrpEvent } from "../../vendor/paperclip-runner/index.js"; +import { buildNativeExecutionInput } from "./native-execution-input.js"; +import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; +import { createNativeRecoveryUsageDelivery } from "./native-recovery-usage-delivery.js"; + +const accounting = vi.hoisted(() => ({ open: vi.fn(), capture: vi.fn() })); +vi.mock("./native-usage-delivery.js", () => ({ createNativeUsageContinuationDelivery: accounting.open })); +beforeEach(() => { vi.clearAllMocks(); accounting.capture.mockResolvedValue(undefined); accounting.open.mockResolvedValue({ capture: accounting.capture }); }); + +function fixture() { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), sessionId = randomUUID(); + const source: DurableRecoveryIdentity = { runId: randomUUID(), runnerInstanceId: randomUUID(), environmentLeaseId: randomUUID(), + normalizedSessionId: sessionId, turnId: randomUUID(), itemId: randomUUID() }; + const build = (runId: string) => buildNativeExecutionInput({ companyId, agentId, runId, normalizedSessionId: sessionId, + provider: "codex", issue: { id: issueId, identifier: "REC-1", title: "Continue", description: null, workMode: "standard" }, + taskPrompt: "Continue once", workspace: { id: randomUUID(), cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null }, + completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", + contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Preserve the result" }] } }, + runtimeContext: nativeRuntimeContextFixture() }); + const execution = build(randomUUID()), previous = build(source.runId); + const row = { run: { status: "succeeded", finishedAt: new Date() as Date | null, runtimeMode: "native", runnerInstanceId: source.runnerInstanceId, + nativeSessionId: sessionId, runnerProfileJson: { nativeExecutionInput: previous } }, + owner: { phase: "committed", resultId: randomUUID() as string | null, leaseOwner: null as string | null, companyId, issueId } }; + const where = vi.fn(async (_predicate: SQL) => [row]); + const select = vi.fn(() => ({ from: () => ({ innerJoin: () => ({ where }) }) })); + const db = { select } as unknown as Db; + const matchesExecution = vi.fn(() => true); + const deliver = () => createNativeRecoveryUsageDelivery({ db, execution, sourceIdentity: { ...source }, matchesExecution }); + const event: PrpEvent = { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", sourceInstanceId: source.runnerInstanceId, + sourceEventId: "source-usage", sourceSeq: 1, runId: source.runId, normalizedSessionId: sessionId, turnId: source.turnId, + eventType: "usage.reported", priority: 0, emittedAt: new Date().toISOString(), payload: {} }; + return { source, execution, previous, row, select, where, matchesExecution, deliver, event }; +} + +describe("source accounting recovery authority", () => { + it("pins the relational source tuple and hydrates once without binding target accounting", async () => { + const f = fixture(), deliver = f.deliver(); + await deliver(f.event, f.source); + await deliver({ ...f.event, sourceEventId: "source-terminal", sourceSeq: 2, eventType: "turn.completed" }, f.source); + expect(f.select).toHaveBeenCalledOnce(); + expect(new PgDialect().sqlToQuery(f.where.mock.calls[0][0]).params).toEqual([ + f.source.runId, f.execution.binding.companyId, f.execution.binding.agentId, f.execution.binding.issueId, + ]); + expect(f.matchesExecution).toHaveBeenCalledWith(f.previous); + expect(accounting.open).toHaveBeenCalledExactlyOnceWith(expect.anything(), { + companyId: f.execution.binding.companyId, runId: f.source.runId, runnerInstanceId: f.source.runnerInstanceId, provider: f.previous.provider, + }); + expect(accounting.capture).toHaveBeenCalledTimes(2); + expect(accounting.capture).toHaveBeenCalledWith(f.event); + }); + + it.each(["runId", "runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "turnId", "itemId"] as const)( + "rejects a changed pinned %s before source hydration", async key => { + const f = fixture(); + await expect(f.deliver()(f.event, { ...f.source, [key]: randomUUID() })).rejects.toThrow("native_recovery_usage_source_unproven"); + expect(f.select).not.toHaveBeenCalled(); + expect(accounting.open).not.toHaveBeenCalled(); + }); + + it.each(["runId", "sourceInstanceId", "normalizedSessionId"] as const)("rejects mismatched event %s before source hydration", async key => { + const f = fixture(); + await expect(f.deliver()({ ...f.event, [key]: randomUUID() }, f.source)).rejects.toThrow("native_recovery_usage_source_unproven"); + expect(f.select).not.toHaveBeenCalled(); + }); + + it.each(["nonterminal", "unfinished", "uncommitted", "no-result", "held", "company", "issue", "scope"])( + "keeps an unproven source held: %s", async change => { + const f = fixture(); + if (change === "nonterminal") f.row.run.status = "running"; + if (change === "unfinished") f.row.run.finishedAt = null; + if (change === "uncommitted") f.row.owner.phase = "observed"; + if (change === "no-result") f.row.owner.resultId = null; + if (change === "held") f.row.owner.leaseOwner = "live-owner"; + if (change === "company") f.row.owner.companyId = randomUUID(); + if (change === "issue") f.row.owner.issueId = randomUUID(); + if (change === "scope") f.matchesExecution.mockReturnValue(false); + await expect(f.deliver()(f.event, f.source)).rejects.toThrow("native_recovery_usage_source_unproven"); + expect(accounting.open).not.toHaveBeenCalled(); + expect(accounting.capture).not.toHaveBeenCalled(); + }); + + it("keeps a durable source receipt failure unacknowledged", async () => { + const f = fixture(); + accounting.capture.mockRejectedValue(new Error("native_usage_after_finalization_requires_reconciliation")); + await expect(f.deliver()(f.event, f.source)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation"); + }); +}); diff --git a/server/src/services/native-runtime/native-recovery-usage-delivery.ts b/server/src/services/native-runtime/native-recovery-usage-delivery.ts new file mode 100644 index 00000000000..9401c72dd71 --- /dev/null +++ b/server/src/services/native-runtime/native-recovery-usage-delivery.ts @@ -0,0 +1,47 @@ +import { and, eq } from "drizzle-orm"; +import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { parseNativeExecutionInput, type DurableRecoveryIdentity, type NativeExecutionInput, type PrpEvent } from "../../vendor/paperclip-runner/index.js"; +import { createNativeUsageContinuationDelivery } from "./native-usage-delivery.js"; + +const identityKeys = ["runId", "runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "turnId", "itemId"] as const; + +/** A cold handoff may discover accounting that belongs to its completed source. + * Hydrate that producer once; never route its receipts through the new run. */ +export function createNativeRecoveryUsageDelivery(input: { + db: Db; + execution: NativeExecutionInput; + sourceIdentity: Record; + matchesExecution(previous: NativeExecutionInput): boolean; +}) { + let delivery: ReturnType | undefined; + const deny = (): never => { throw new Error("native_recovery_usage_source_unproven"); }; + const open = async (source: DurableRecoveryIdentity) => { + const { binding } = input.execution; + const [prior] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations }) + .from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) + .where(and(eq(heartbeatRuns.id, source.runId), eq(heartbeatRuns.companyId, binding.companyId), + eq(heartbeatRuns.agentId, binding.agentId), eq(heartbeatRuns.nativeIssueId, binding.issueId))); + if (!prior || prior.run.status !== "succeeded" || prior.run.finishedAt === null || + prior.run.runtimeMode !== "native" || prior.run.runnerInstanceId !== source.runnerInstanceId || + prior.run.nativeSessionId !== source.normalizedSessionId || prior.owner.phase !== "committed" || + !prior.owner.resultId || prior.owner.leaseOwner || prior.owner.companyId !== binding.companyId || + prior.owner.issueId !== binding.issueId) return deny(); + const previous = parseNativeExecutionInput(prior.run.runnerProfileJson?.nativeExecutionInput); + if (previous.binding.companyId !== binding.companyId || previous.binding.agentId !== binding.agentId || + previous.binding.issueId !== binding.issueId || previous.binding.runId !== source.runId || + !input.matchesExecution(previous)) return deny(); + return createNativeUsageContinuationDelivery(input.db, { + companyId: binding.companyId, runId: source.runId, + runnerInstanceId: source.runnerInstanceId, provider: previous.provider, + }); + }; + return async (event: PrpEvent, source: DurableRecoveryIdentity) => { + if (source.runId === input.execution.binding.runId || + identityKeys.some(key => typeof input.sourceIdentity[key] !== "string" || + !input.sourceIdentity[key] || source[key] !== input.sourceIdentity[key]) || + event.runId !== source.runId || event.sourceInstanceId !== source.runnerInstanceId || + event.normalizedSessionId !== source.normalizedSessionId) return deny(); + delivery ??= open(source); + await (await delivery).capture(event); + }; +} diff --git a/server/src/services/native-runtime/native-restart-recovery.test.ts b/server/src/services/native-runtime/native-restart-recovery.test.ts index f032829c639..fd79245b6e5 100644 --- a/server/src/services/native-runtime/native-restart-recovery.test.ts +++ b/server/src/services/native-runtime/native-restart-recovery.test.ts @@ -8,6 +8,18 @@ import { } from "./native-restart-recovery.js"; describe("native restart recovery classification", () => { + it.each([false, true])("recognizes verified staged remote evidence with target snapshot=%s", hasCheckpoint => { + const evidence = { remoteSandbox: true, runnerPidAlive: false, runnerGroupAlive: false, + processStartMatches: false, hasCheckpoint, hasProviderEvidence: false }; + expect(classifyNativeRunnerRecoveryEvidence(evidence).claimKind).toBeNull(); + expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, stagedAttachmentVerified: true })).toEqual({ + claimKind: "reattach_remote_runner", reason: "staged_remote_attachment_requires_sandbox_verification", + }); + expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, stagedAttachmentVerified: true, + checkpointFailed: true }).claimKind).toBeNull(); + expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, remoteSandbox: false, + stagedAttachmentVerified: true }).claimKind).toBe(hasCheckpoint ? null : "bootstrap_incomplete"); + }); it.each([true, false])("uses the remote fenced lease instead of the old container PID (live=%s)", async live => { const now = new Date(); const probe = vi.fn(() => { throw new Error("must not probe another container's PID"); }); diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index 8ac4d9bd918..53ccc1b07f7 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -1,3 +1,4 @@ +import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js"; import { notifyNativeEnvironmentSettlement } from "../native-environment-settlement-notifications.js"; import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js"; import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; @@ -6,7 +7,8 @@ import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-pre import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { preserveNativeFailedWorkspace } from "./native-failed-workspace-recovery.js"; -import { hasNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; +import { hasNativeRunnerCheckpoint, loadNativeRunnerCheckpointHeader } from "./native-runner-checkpoint.js"; +import { resolveNativeStagedAttachmentRecovery, type NativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js"; import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js"; import { activeNativeExecutionRunIds, NATIVE_PREPARATION_FENCING_VERSION } from "./native-active-runs.js"; import { @@ -15,7 +17,7 @@ import { recordNativeLocalProcessStop, } from "../native-local-process-stop.js"; import { randomUUID } from "node:crypto"; -import { and, desc, eq, inArray, isNotNull, isNull, lte, or, sql } from "drizzle-orm"; +import { and, desc, eq, inArray, isNotNull, isNull, lte, ne, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { agents, @@ -68,6 +70,8 @@ export type NativeRestartRecoveryClaim = recoveryRequestId: string | null; remote: { kind?: "sandbox"; providerLeaseId: string; remoteCwd: string } | { kind: "plugin_task"; leaseId: string }; + /** Discovery evidence only; executor revalidates the encrypted header. */ + stagedAttachment?: Pick; } | { kind: "resume_dead_runner"; @@ -94,6 +98,7 @@ export type NativeRestartRecoveryDisposition = kind: "awaiting_evidence" | "blocked" | "already_finalized"; runId: string; reason: string; + retryAt?: Date; }; export function nextNativeProviderAttempt( @@ -320,6 +325,7 @@ export function classifyNativeRunnerRecoveryEvidence(input: { checkpointIdentityMatches?: boolean; hasProviderEvidence: boolean; checkpointFailed?: boolean; + stagedAttachmentVerified?: boolean; providerAttempt?: number; }): { claimKind: NativeRestartRecoveryClaim["kind"] | null; @@ -334,6 +340,9 @@ export function classifyNativeRunnerRecoveryEvidence(input: { // inspection; the executor must prove the original lease and authenticate // the surviving runner before it can resume delivery. if (input.remoteSandbox) { + if (input.stagedAttachmentVerified === true) { + return { claimKind: "reattach_remote_runner", reason: "staged_remote_attachment_requires_sandbox_verification" }; + } if (input.unclaimedBootstrap === true && !input.hasCheckpoint && !input.hasProviderEvidence) { return { claimKind: "bootstrap_incomplete", reason: "remote_execution_never_claimed" }; } @@ -514,6 +523,9 @@ export async function claimNativeRestartRecoveries(input: { eq(agents.adapterType, "paperclip_runner"), inArray(heartbeatRuns.status, ["running", "failed"]), isNull(nativeRunFinalizations.resultId), + // A pre-provider retained-owner wait belongs to normal same-run + // resumption, including when its deadline elapsed while we were down. + or(isNull(nativeRunFinalizations.failureCode), ne(nativeRunFinalizations.failureCode, NATIVE_RETAINED_OWNER_WAIT)), or( eq(nativeRunFinalizations.phase, "observed"), and( @@ -668,9 +680,21 @@ export async function claimNativeRestartRecoveries(input: { kind: "awaiting_evidence", runId: row.run.id, reason: takeover.reason, + ...(durableAuthority && row.coordinator.leaseExpiresAt ? { retryAt: row.coordinator.leaseExpiresAt } : {}), } as const; } + // Only the classic remote path stages attachments. After the owner is + // eligible for takeover, inspect its bounded header even if activation + // already published a runtime snapshot. Never scan command/event payloads. + const stagedHeader = remoteWorkspace && !pluginTask && durableAuthority + ? await loadNativeRunnerCheckpointHeader(tx, { + companyId: row.run.companyId, issueId: row.coordinator.issueId, runId: row.run.id, + }) : null; + const stagedAttachment = stagedHeader ? await resolveNativeStagedAttachmentRecovery({ + db: tx, currentRun: row.run, header: stagedHeader, + }) : null; + const runnerPidAlive = !remoteProcess && processIsAlive(row.run.processPid); const runnerGroupAlive = @@ -859,6 +883,7 @@ export async function claimNativeRestartRecoveries(input: { knownProviderProcessIdentityAmbiguous: providerProcesses.ambiguousLivePids.length > 0, hasCheckpoint, + stagedAttachmentVerified: stagedAttachment !== null, checkpointIdentityMatches: checkpointIdentityMatches && (!remoteSandbox || remoteWorkspace !== null), @@ -1109,6 +1134,10 @@ export async function claimNativeRestartRecoveries(input: { return { kind: claimKind, ...common, + ...(stagedAttachment ? { stagedAttachment: { + sourceIdentity: stagedAttachment.sourceIdentity, targetIdentity: stagedAttachment.targetIdentity, + templateDigest: stagedAttachment.templateDigest, + } } : {}), remote: pluginTask ? { kind: "plugin_task", leaseId: pluginTask.id } : { providerLeaseId: remoteWorkspace!.providerLeaseId, remoteCwd: remoteWorkspace!.remoteCwd, diff --git a/server/src/services/native-runtime/native-runner-checkpoint.ts b/server/src/services/native-runtime/native-runner-checkpoint.ts index 3cdef6040e2..00859cfd3f0 100644 --- a/server/src/services/native-runtime/native-runner-checkpoint.ts +++ b/server/src/services/native-runtime/native-runner-checkpoint.ts @@ -5,7 +5,7 @@ import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provid import { legacyControllerBootId } from "../legacy-controller-lease.js"; type Binding = { companyId: string; issueId: string; runId: string }; -type Owner = Binding & { bootId: string; leaseOwner: string; generation: number; authorityRunId: string }; +export type NativeRunnerCheckpointOwner = Binding & { bootId: string; leaseOwner: string; generation: number; authorityRunId: string }; function encryptParts(binding: Binding, records: ReadonlyMap) { if (!records.size) return []; const cipher = createLocalEncryptedCipher("write"); @@ -25,49 +25,79 @@ function decryptPart(cipher: ReturnType, bind return envelope.contents; } -export function nativeRunnerCheckpointWriter(db: Db, owner: Owner) { +type CheckpointTransaction = Parameters[0]>[0]; + +/** A dependent run-lock query can consume this CTE to preserve coordinator-first + * lock order without adding a second database round trip. */ +export function nativeRunnerCheckpointOwnerLock(tx: CheckpointTransaction, owner: NativeRunnerCheckpointOwner) { + return tx.$with("checkpoint_owner").as(tx.select({ + leaseOwner: nativeRunFinalizations.leaseOwner, + leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt, + resultId: nativeRunFinalizations.resultId, + }).from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, owner.companyId), eq(nativeRunFinalizations.issueId, owner.issueId), + eq(nativeRunFinalizations.runId, owner.authorityRunId), eq(nativeRunFinalizations.controllerBootId, owner.bootId), + eq(nativeRunFinalizations.controllerGeneration, owner.generation))).for("update")); +} + +/** Locks the coordinator before callers lock a run. A caller that subsequently + * waits for another lock must recheck the fence before writing. */ +export async function lockNativeRunnerCheckpointOwner( + tx: CheckpointTransaction, owner: NativeRunnerCheckpointOwner, +) { + // A locking CTE acquires ownership before the outer projection samples + // wall time. Sampling in the locking SELECT can precede its lock wait. + // Keep both steps in one statement without another database round trip. + const locked = nativeRunnerCheckpointOwnerLock(tx, owner); + const [owned] = await tx.with(locked).select({ leaseOwner: locked.leaseOwner, + leaseExpiresAt: locked.leaseExpiresAt, resultId: locked.resultId, now: sql`clock_timestamp()` }).from(locked); + if (!owned) throw new Error("native_runner_checkpoint_owner_lost"); + return owned; +} + +export async function validateNativeRunnerCheckpointOwner( + tx: CheckpointTransaction, owner: NativeRunnerCheckpointOwner, + owned: Awaited>, newWork: boolean, +): Promise { + const current = owned; + const activeOwner = current.leaseOwner === owner.leaseOwner + && current.leaseExpiresAt !== null && current.leaseExpiresAt > new Date(owned.now); + if (!activeOwner) { + const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues) + .where(and(eq(issues.companyId, owner.companyId), eq(issues.id, owner.issueId))); + const [successor] = task?.executionRunId && task.executionRunId !== owner.runId + ? await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.runId, task.executionRunId), eq(nativeRunFinalizations.companyId, owner.companyId), + eq(nativeRunFinalizations.issueId, owner.issueId), eq(nativeRunFinalizations.controllerBootId, owner.bootId), + gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`))) : []; + const [preparingSuccessor] = !successor && task?.executionRunId && task.executionRunId !== owner.runId + ? await tx.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, task.executionRunId), eq(heartbeatRuns.companyId, owner.companyId), + eq(heartbeatRuns.issueId, owner.issueId), eq(heartbeatRuns.runtimeMode, "legacy"), + eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.executionStage, "preparing"), + eq(heartbeatRuns.controllerBootId, legacyControllerBootId), + gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`), + notExists(tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, heartbeatRuns.id))))) : []; + if (!current.resultId || newWork || !task || (task.executionRunId && task.executionRunId !== owner.runId && !successor && !preparingSuccessor)) { + throw new Error("native_runner_checkpoint_owner_lost"); + } + } +} + +export async function assertNativeRunnerCheckpointOwner(tx: CheckpointTransaction, owner: NativeRunnerCheckpointOwner, newWork: boolean) { + const locked = await lockNativeRunnerCheckpointOwner(tx, owner); + await validateNativeRunnerCheckpointOwner(tx, owner, locked, newWork); +} + +export function nativeRunnerCheckpointWriter(db: Db, owner: NativeRunnerCheckpointOwner) { const persistenceDb = checkpointDb(db); let previousWork = new Set(); return async (checkpoint: DurablePrpCheckpoint): Promise => { if (checkpoint.runId !== owner.runId) throw new Error("native_runner_checkpoint_binding_mismatch"); const changed = encryptParts(owner, checkpoint.changedRecords); await persistenceDb.transaction(async tx => { - // A locking CTE acquires ownership before the outer projection samples - // wall time. Sampling in the locking SELECT can precede its lock wait. - // Keep both steps in one statement without another database round trip. - const locked = tx.$with("checkpoint_owner").as(tx.select().from(nativeRunFinalizations) - .where(and(eq(nativeRunFinalizations.companyId, owner.companyId), - eq(nativeRunFinalizations.issueId, owner.issueId), eq(nativeRunFinalizations.runId, owner.authorityRunId), - eq(nativeRunFinalizations.controllerBootId, owner.bootId), - eq(nativeRunFinalizations.controllerGeneration, owner.generation))).for("update")); - const [owned] = await tx.with(locked).select({ leaseOwner: locked.leaseOwner, - leaseExpiresAt: locked.leaseExpiresAt, resultId: locked.resultId, now: sql`clock_timestamp()` }).from(locked); - if (!owned) throw new Error("native_runner_checkpoint_owner_lost"); - const current = owned; - const activeOwner = current.leaseOwner === owner.leaseOwner - && current.leaseExpiresAt !== null && current.leaseExpiresAt > new Date(owned.now); - if (!activeOwner) { - const newWork = checkpoint.workCommandIds.some(id => !previousWork.has(id)); - const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues) - .where(and(eq(issues.companyId, owner.companyId), eq(issues.id, owner.issueId))); - const [successor] = task?.executionRunId && task.executionRunId !== owner.runId - ? await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and( - eq(nativeRunFinalizations.runId, task.executionRunId), eq(nativeRunFinalizations.companyId, owner.companyId), - eq(nativeRunFinalizations.issueId, owner.issueId), eq(nativeRunFinalizations.controllerBootId, owner.bootId), - gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`))) : []; - const [preparingSuccessor] = !successor && task?.executionRunId && task.executionRunId !== owner.runId - ? await tx.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).where(and( - eq(heartbeatRuns.id, task.executionRunId), eq(heartbeatRuns.companyId, owner.companyId), - eq(heartbeatRuns.issueId, owner.issueId), eq(heartbeatRuns.runtimeMode, "legacy"), - eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.executionStage, "preparing"), - eq(heartbeatRuns.controllerBootId, legacyControllerBootId), - gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`), - notExists(tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations) - .where(eq(nativeRunFinalizations.runId, heartbeatRuns.id))))) : []; - if (!current.resultId || newWork || !task || (task.executionRunId && task.executionRunId !== owner.runId && !successor && !preparingSuccessor)) { - throw new Error("native_runner_checkpoint_owner_lost"); - } - } + await assertNativeRunnerCheckpointOwner(tx, owner, checkpoint.workCommandIds.some(id => !previousWork.has(id))); if (checkpoint.predecessor) { const [prior] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( eq(heartbeatRuns.id, checkpoint.predecessor.runId), eq(heartbeatRuns.companyId, owner.companyId), @@ -150,7 +180,17 @@ export async function registerNativeRunnerCheckpoint(input: { await input.authority.flushCheckpoint(); } -export async function loadNativeRunnerCheckpoint(db: Db, binding: Binding): Promise { +/** Exact-run bounded metadata read; never loads command/event payload records. */ +export async function loadNativeRunnerCheckpointHeader(db: Pick, binding: Binding): Promise { + const [row] = await db.select({ material: nativeRunnerCheckpoints.material }).from(nativeRunnerCheckpoints).where(and( + eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId), + eq(nativeRunnerCheckpoints.runId, binding.runId), eq(nativeRunnerCheckpoints.part, "header"))).limit(1); + if (!row) return null; + try { return decryptPart(createLocalEncryptedCipher("read"), binding, "header", row.material); } + catch { return null; } +} + +export async function loadNativeRunnerCheckpoint(db: Pick, binding: Binding): Promise { const rows = await db.select().from(nativeRunnerCheckpoints).where(and( eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId), eq(nativeRunnerCheckpoints.runId, binding.runId))); diff --git a/server/src/services/native-runtime/native-runner-execution-driver.test.ts b/server/src/services/native-runtime/native-runner-execution-driver.test.ts index 28bc0b34db1..51c79577e02 100644 --- a/server/src/services/native-runtime/native-runner-execution-driver.test.ts +++ b/server/src/services/native-runtime/native-runner-execution-driver.test.ts @@ -6,9 +6,12 @@ import { prepareNativeRemoteRunner, createNativeRunnerExecutionDriver } from "./ const transport = vi.hoisted(() => ({ register: vi.fn(), connect: vi.fn(), + checkpoint: vi.fn(), + checkpointRegistration: vi.fn(), })); vi.mock("../../realtime/runner-prp-ws.js", () => ({ registerRunnerPrpAuthority: transport.register })); vi.mock("../../realtime/runner-prp-outbound.js", () => ({ connectRunnerPrpIngress: transport.connect })); +vi.mock("./native-runner-checkpoint.js", () => ({ nativeRunnerCheckpointRegistration: transport.checkpointRegistration })); const binding = { companyId: "company", issueId: "issue", agentId: "agent", runId: "run-a" }; const authority = {} as Parameters["controlPlaneRegistration"]>>[0]; @@ -30,9 +33,41 @@ beforeEach(() => { vi.clearAllMocks(); transport.register.mockResolvedValue({ connectUrl: "ws://127.0.0.1/core", release: vi.fn() }); transport.connect.mockReturnValue({ ready: Promise.resolve(), close: vi.fn(async () => {}) }); + transport.checkpointRegistration.mockReturnValue(transport.checkpoint); + transport.checkpoint.mockResolvedValue(undefined); }); describe("shared Runner execution driver", () => { + it("installs both staged checkpoint writers under the business lease without acquiring endpoints", async () => { + const prepare = vi.fn(); + const persistence = { db: {} as never, bootId: "boot-c", authorityRunId: "run-c" }; + const driver = createNativeRunnerExecutionDriver({ target: sandbox(), binding: { ...binding, runId: "run-c" }, + controllerPersistence: persistence, prepareRemote: prepare }); + const sourceIdentity = { runnerInstanceId: "runner", environmentLeaseId: "lease", normalizedSessionId: "session", + runId: "run-a", turnId: "turn-a", itemId: "item-a" }; + const targetIdentity = { ...sourceIdentity, runId: "run-c", turnId: "turn-c", itemId: "item-c" }; + await driver.onStagedRunAttachment!(authority, { sourceIdentity, targetIdentity, templateDigest: `sha256:${"a".repeat(64)}` }); + expect(transport.checkpointRegistration).toHaveBeenCalledExactlyOnceWith({ ...persistence, + binding: { companyId: binding.companyId, issueId: binding.issueId, runId: "run-c" } }); + expect(transport.checkpoint.mock.calls).toEqual([[authority, sourceIdentity], [authority, targetIdentity]]); + expect(prepare).not.toHaveBeenCalled(); + expect(transport.register).not.toHaveBeenCalled(); + expect(transport.connect).not.toHaveBeenCalled(); + }); + + it("installs the target writer first when restoring an already persisted stage", async () => { + const sourceIdentity = { runnerInstanceId: "runner", environmentLeaseId: "lease", normalizedSessionId: "session", + runId: "run-a", turnId: "turn-a", itemId: "item-a" }; + const targetIdentity = { ...sourceIdentity, runId: "run-c", turnId: "turn-c", itemId: "item-c" }; + const restored = { store: { state: { stagedRunAttachment: { sourceIdentity, targetIdentity } } } } as unknown as typeof authority; + const driver = createNativeRunnerExecutionDriver({ target: sandbox(), binding: { ...binding, runId: "run-c" }, + controllerPersistence: { db: {} as never, bootId: "boot-c", authorityRunId: "run-c" } }); + await driver.onStagedRunAttachment!(restored, { sourceIdentity, targetIdentity, templateDigest: `sha256:${"a".repeat(64)}` }); + expect(transport.checkpoint.mock.calls).toEqual([[restored, targetIdentity], [restored, sourceIdentity]]); + expect(transport.register).not.toHaveBeenCalled(); + expect(transport.connect).not.toHaveBeenCalled(); + }); + it("keeps local execution on the registered loopback route", async () => { const driver = createNativeRunnerExecutionDriver({ target: { kind: "local" }, binding }); const connection = await driver.controlPlaneRegistration!(authority); diff --git a/server/src/services/native-runtime/native-runner-execution-driver.ts b/server/src/services/native-runtime/native-runner-execution-driver.ts index 6bf3b44ded5..0929d99d82a 100644 --- a/server/src/services/native-runtime/native-runner-execution-driver.ts +++ b/server/src/services/native-runtime/native-runner-execution-driver.ts @@ -92,6 +92,20 @@ export async function prepareNativeRemoteRunner(input: RemotePreparation) { export function createNativeRunnerExecutionDriver(input: NativeRunnerExecutionDriverInput): RunnerExecutionDriver { return { ...input.process, + onStagedRunAttachment: input.controllerPersistence && input.binding.issueId + ? async (authority, proof) => { + // Both journals use the current business lease. Install persistence + // before staging; endpoint acquisition still belongs to attachment. + const register = nativeRunnerCheckpointRegistration({ ...input.controllerPersistence!, + binding: { companyId: input.binding.companyId, issueId: input.binding.issueId!, runId: input.binding.runId }, + }); + // A restored stage already publishes under the target. Its writer + // must exist before registration flushes that restored revision. + const identities = authority.store?.state.stagedRunAttachment + ? [proof.targetIdentity, proof.sourceIdentity] : [proof.sourceIdentity, proof.targetIdentity]; + for (const identity of identities) await register(authority, identity); + } + : undefined, controlPlaneRegistration: async (authority, attachmentIdentity) => { await input.beforeConnect?.(attachmentIdentity, authority); if (input.controllerPersistence && input.binding.issueId) { diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 3dbcd10ab35..6711939a601 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -60,6 +60,7 @@ import { readProcessStartedAt } from "../hot-restart.js"; import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; import { buildNativeExecutionInput } from "./native-execution-input.js"; +import { buildNativeExecutionWithCheckpoint, findNativeSessionResumeRun, nativeToolContractFingerprintForTarget } from "./native-session-resume.js"; import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; import { materializeAsset } from "./runtime-context.js"; import { createRunnerTestTlsTunnel, dockerCommandRunner } from "../../__tests__/helpers/docker-command-runner.js"; @@ -321,8 +322,10 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( }; } - it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "legacy", "changed", "same-server"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { + it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "legacy", "changed", "same-server", "diskless", "diskless-per-turn", "diskless-per-turn-read-retry"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { const hardRestart = !["warm", "same-server"].includes(scenario); + const diskless = scenario.startsWith("diskless"); + const perTurn = scenario.startsWith("diskless-per-turn"); const container = process.env.PAPERCLIP_TEST_RUNNER_CONTAINER!; const tunnel = await createRunnerTestTlsTunnel({ port: Number(process.env.PAPERCLIP_TEST_RUNNER_PORT), key: process.env.PAPERCLIP_TEST_RUNNER_TLS_KEY!, cert: process.env.NODE_EXTRA_CA_CERTS! }); @@ -346,12 +349,13 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( taskPrompt: "Complete the first turn.", workspace: { id: randomUUID(), cwd: workspace, repoUrl: null, repoRef: null, branchName: null }, normalizedSessionId: fixture.native.normalizedSessionId, - lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 }, + lifecyclePolicy: perTurn ? { mode: "per_turn", idleTimeoutMs: null } : { mode: "warm", idleTimeoutMs: 60_000 }, runtimeContext: context, completionContract: await boundContract(fixture.runId), }); await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...fixture.preparedProfile, nativeExecutionInput: execution, + nativeToolContractFingerprint: nativeToolContractFingerprintForTarget("remote", execution.provider.kind), } }).where(eq(heartbeatRuns.id, fixture.runId)); const remoteCwd = `/tmp/paperclip-continuity-${fixture.issueId}`; expect((await runner.execute({ command: "mkdir", args: ["-p", remoteCwd] })).exitCode).toBe(0); @@ -385,6 +389,26 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( onLog: async (_stream: "stdout" | "stderr", text: string) => { logs.push(text); }, }; let controller: ReturnType | undefined; + const executeInController = (executionOptions: Parameters[0]) => new Promise<{ exitCode: number | null }>((resolveResult, reject) => { + controller = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), resolve("server/src/__tests__/helpers/remote-warm-controller.test-worker.ts")], + { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env }); + let timer = setTimeout(() => { controller?.kill("SIGKILL"); reject(new Error("Controller fixture imports exceeded 120 seconds")); }, 120_000); + controller.stdout?.on("data", data => logs.push(String(data))); + controller.stderr?.on("data", data => logs.push(String(data))); + controller.on("error", error => { clearTimeout(timer); reject(error); }); + controller.on("exit", (code, signal) => { clearTimeout(timer); reject(new Error(`Warm controller exited: ${code}/${signal}: ${logs.join("").slice(-3_000)}`)); }); + controller.on("message", (message: { ready?: boolean; log?: { text: string }; result?: { exitCode: number | null }; error?: string }) => { + if (message.ready) { + clearTimeout(timer); + timer = setTimeout(() => { controller?.kill("SIGKILL"); reject(new Error(`Warm controller admission exceeded 60 seconds: ${logs.join("").slice(-3_000)}`)); }, 60_000); + controller!.send?.({ connectionString: temporary.connectionString, container, ingressUrl: `wss://127.0.0.1:${tunnel.port}`, + options: { ...executionOptions, db: undefined, onLog: undefined, runnerExecutionTarget: { ...executionOptions.runnerExecutionTarget, runner: undefined } } }); + } + if (message.log) logs.push(message.log.text); + if (message.error) { clearTimeout(timer); reject(new Error(message.error)); } + if (message.result) { clearTimeout(timer); resolveResult(message.result); } + }); + }); try { const firstOptions = { ...options, execution, runnerExecutionTarget: target }; const executeFirst = async () => { @@ -454,28 +478,13 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( expect(nextProcess.stdout).toBe(priorProcess.stdout); return result; }; - const first = hardRestart ? await new Promise<{ exitCode: number | null }>((resolveResult, reject) => { - controller = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), resolve("server/src/__tests__/helpers/remote-warm-controller.test-worker.ts")], - { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env }); - const timer = setTimeout(() => { controller?.kill("SIGKILL"); reject(new Error("Warm controller startup exceeded 60 seconds")); }, 60_000); - controller.stdout?.on("data", data => logs.push(String(data))); - controller.stderr?.on("data", data => logs.push(String(data))); - controller.on("error", error => { clearTimeout(timer); reject(error); }); - controller.on("exit", (code, signal) => { clearTimeout(timer); reject(new Error(`Warm controller exited: ${code}/${signal}: ${logs.join("").slice(-3_000)}`)); }); - controller.on("message", (message: { log?: { text: string }; result?: { exitCode: number | null }; error?: string }) => { - if (message.log) logs.push(message.log.text); - if (message.error) { clearTimeout(timer); reject(new Error(message.error)); } - if (message.result) { clearTimeout(timer); resolveResult(message.result); } - }); - controller.send?.({ connectionString: temporary.connectionString, container, ingressUrl: `wss://127.0.0.1:${tunnel.port}`, - options: { ...firstOptions, db: undefined, onLog: undefined, runnerExecutionTarget: { ...target, runner: undefined } } }); - }) : await executeFirst(); + const first = hardRestart ? await executeInController(firstOptions) : await executeFirst(); expect(first.exitCode).toBe(0); await finalizeNativeRun({ db: fixture.db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); const snapshot = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] }); expect(snapshot.exitCode).toBe(0); const saved = JSON.parse(snapshot.stdout); - expect(saved.lifecycle).toBe("ready"); + expect(saved.lifecycle).toBe(perTurn ? "suspended" : "ready"); const processBefore = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); expect(processBefore.exitCode).toBe(0); const historyBefore = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/filesystem/codex-home/fake-codex-state.json`] }); @@ -502,12 +511,11 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( runnerExecutionTarget: { ...target, leaseId: nextLeaseId, sandboxLeaseAcquisition: { outcome: "resumed" as const, providerLeaseId: container } }, }; if (controller) { - await expect(executePaperclipNativeSession(nextOptions)).rejects.toThrow("prior_owner_terminal_state_indeterminate"); const exited = new Promise(resolveExit => controller!.once("exit", () => resolveExit())); - controller.kill("SIGKILL"); + (controller as ReturnType).kill("SIGKILL"); await exited; controller = undefined; - await expect(executePaperclipNativeSession({ ...nextOptions, + if (!perTurn) await expect(executePaperclipNativeSession({ ...nextOptions, runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget, sandboxLeaseAcquisition: { outcome: "replacement", providerLeaseId: "unrelated-sandbox" } }, })).rejects.toThrow("prior_owner_terminal_state_indeterminate"); @@ -526,10 +534,43 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( } expect(changed).toBe(1); } - const [nextPrepared] = await fixture.db.select({ profile: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(eq(heartbeatRuns.id, nextRunId)); - await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...nextPrepared.profile, - nativeExecutionInput: nextOptions.execution } }).where(eq(heartbeatRuns.id, nextRunId)); - const second = await executePaperclipNativeSession(nextOptions); + if (diskless) { + // Preserve sandbox/provider storage and workspace dependencies. Delete + // both native controller projections; admission must discover the source. + await rm(resolve(resolvePaperclipInstanceRoot(), "runtime", "paperclip-runner"), { recursive: true, force: true }); + } + // Exercise the existing admission selector/rebinder, which supplies per-turn + // runtime snapshots. It does not restore PRP or supervisor files. + const admitContinuation = async (current: typeof execution) => { + const [row] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, current.binding.runId)); + const selected = await findNativeSessionResumeRun(fixture.db, { ...current.binding, currentRunId: row.id, + normalizedSessionId: fixture.native.normalizedSessionId, beforeCreatedAt: row.createdAt }); + const admission = buildNativeExecutionWithCheckpoint({ previousRun: selected, + normalizedSessionId: fixture.native.normalizedSessionId, executionTargetKind: "remote", buildExecution: () => current }); + if (perTurn) expect(admission.checkpoint).not.toBeNull(); + await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...row.runnerProfileJson, + nativeExecutionInput: current, nativeToolContractFingerprint: nativeToolContractFingerprintForTarget("remote", current.provider.kind), + ...(perTurn ? { sessionCheckpoint: admission.checkpoint } : {}) } }).where(eq(heartbeatRuns.id, row.id)); + }; + await admitContinuation(nextOptions.execution); + if (scenario === "diskless-per-turn-read-retry") { + let rejectedReads = 0; + await expect(executePaperclipNativeSession({ ...nextOptions, + runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget, runner: { ...runner, + execute: async request => { + if (request.args?.some(arg => arg.includes("base64 <") && arg.includes("runner-state.json"))) { + rejectedReads++; + throw new Error("fixture_remote_state_timeout"); + } + return runner.execute(request); + }, + } }, + })).rejects.toThrow("fixture_remote_state_timeout"); + expect(rejectedReads).toBe(1); + // Retry with the partial PRP projection left by the failed read. No + // fixture writes a suspended receipt or changes the surviving Runner. + } + const second = diskless ? await executeInController(nextOptions) : await executePaperclipNativeSession(nextOptions); expect(second.exitCode).toBe(0); const historyAfter = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/filesystem/codex-home/fake-codex-state.json`] }); const providerAfter = JSON.parse(historyAfter.stdout); @@ -540,12 +581,19 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( expect(calls.stdout.split("\n").filter(line => line === "thread/start")).toHaveLength(1); const providerStarts = calls.stdout.split("\n").filter(line => line === "process-start").length; expect(providerStarts).toBeGreaterThanOrEqual(1); - expect(providerStarts).toBeLessThanOrEqual(["legacy", "changed"].includes(scenario) ? 3 : hardRestart ? 2 : 1); + expect(providerStarts).toBeLessThanOrEqual(perTurn || ["legacy", "changed"].includes(scenario) ? 3 : hardRestart ? 2 : 1); const processAfter = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); expect(processAfter.exitCode).toBe(0); - if (["legacy", "changed"].includes(scenario)) expect(processAfter.stdout).not.toBe(processBefore.stdout); + if (perTurn || ["legacy", "changed"].includes(scenario)) expect(processAfter.stdout).not.toBe(processBefore.stdout); else expect(processAfter.stdout).toBe(processBefore.stdout); await finalizeNativeRun({ db: fixture.db, runId: nextRunId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + if (diskless && controller) { + const exited = new Promise(resolveExit => controller!.once("exit", () => resolveExit())); + (controller as ReturnType).kill("SIGKILL"); + await exited; + controller = undefined; + await rm(resolve(resolvePaperclipInstanceRoot(), "runtime", "paperclip-runner"), { recursive: true, force: true }); + } const thirdRunId = randomUUID(); const thirdLeaseId = randomUUID(); const [thirdRun] = await fixture.db.insert(heartbeatRuns).values({ @@ -562,9 +610,11 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( issue: { id: fixture.issueId, title: "Continue the same conversation", description: null, reviewPolicy: null }, environmentLeaseId: thirdLeaseId }); await fixture.db.insert(nativeRunFinalizations).values({ runId: thirdRunId, companyId, issueId: fixture.issueId, phase: "observed" }); + const thirdExecution = { ...nextOptions.execution, completionContract: await boundContract(thirdRunId), + binding: { ...execution.binding, runId: thirdRunId }, task: { ...execution.task, prompt: "Complete the third conversation turn." } }; + await admitContinuation(thirdExecution); const third = await executePaperclipNativeSession({ ...nextOptions, runnerInstanceId: thirdNative.runnerInstanceId, - execution: { ...nextOptions.execution, completionContract: await boundContract(thirdRunId), - binding: { ...execution.binding, runId: thirdRunId }, task: { ...execution.task, prompt: "Complete the third conversation turn." } }, + execution: thirdExecution, runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget, leaseId: thirdLeaseId }, }); expect(third.exitCode).toBe(0); @@ -573,10 +623,11 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( const finalCalls = await runner.execute({ command: "cat", args: [`${remoteCwd}/provider-calls.log`] }); expect(finalCalls.stdout.split("\n").filter(line => line === "turn/start")).toHaveLength(3); expect(finalCalls.stdout.split("\n").filter(line => line === "thread/start")).toHaveLength(1); - expect(finalCalls.stdout.split("\n").filter(line => line === "process-start")).toHaveLength(providerStarts); + expect(finalCalls.stdout.split("\n").filter(line => line === "process-start")).toHaveLength(providerStarts + (perTurn ? 1 : 0)); const finalProcess = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); - expect(finalProcess.stdout).toBe(processAfter.stdout); - expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0 }); + if (perTurn) expect(finalProcess.stdout).not.toBe(processAfter.stdout); + else expect(finalProcess.stdout).toBe(processAfter.stdout); + expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: perTurn ? 0 : 1, busy: 0, failed: 0 }); const suspended = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] }); expect(suspended.exitCode).toBe(0); expect(JSON.parse(suspended.stdout).lifecycle).toBe("suspended"); @@ -588,7 +639,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( await closeIdleWarmNativeSessionsForRestart(); await tunnel.close(); } - }, 180_000); + }, 360_000); async function persistRestartEvidence(input: { fixture: Awaited>; @@ -1277,10 +1328,11 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( } finally { await authority.stop(); } }); - it.each(["idle", "preparing_successor", "unclaimed_successor", "claimed_successor", "expired_preparation", "dispatching_successor", "foreign_preparation", "foreign_native_preparation", "wrong_issue", "new_work", "replaced_controller"] as const)( - "settles a completed warm checkpoint during early retirement without granting work: %s", - async scenario => { - const fixture = await seedRun(`WARM-EARLY-RETIREMENT-${scenario}`); + it.each((["idle", "preparing_successor", "unclaimed_successor", "claimed_successor", "expired_preparation", "dispatching_successor", "foreign_preparation", "foreign_native_preparation", "wrong_issue", "new_work", "replaced_controller", "unaccepted"] as const).flatMap(scenario => + (["runner.suspend", "turn.stop"] as const).map(command => ({ scenario, command }))))( + "settles a completed warm checkpoint during early retirement without granting work: $scenario/$command", + async ({ scenario, command }) => { + const fixture = await seedRun(`WARM-EARLY-RETIREMENT-${scenario}-${command}`); const binding = { companyId, issueId: fixture.issueId, runId: fixture.runId }; const identity = transportOptions(fixture, runtimeRoot).prpIdentity; const authority = new DurablePrpControlPlane({ @@ -1305,7 +1357,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( }); await fixture.db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }) .where(eq(heartbeatRuns.id, fixture.runId)); - await fixture.db.update(nativeRunFinalizations).set({ phase: "committed", resultId, + await fixture.db.update(nativeRunFinalizations).set({ phase: "committed", resultId: scenario === "unaccepted" ? null : resultId, leaseOwner: null, leaseExpiresAt: null, ...(scenario === "replaced_controller" ? { controllerBootId: randomUUID(), controllerGeneration: 2 } : {}), }).where(eq(nativeRunFinalizations.runId, fixture.runId)); @@ -1314,7 +1366,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( if (scenario !== "idle") { nextRunId = randomUUID(); const preparing = scenario !== "claimed_successor" && scenario !== "unclaimed_successor"; - const otherIssue = scenario === "wrong_issue" ? (await seedRun("WARM-OTHER-ISSUE")).issueId : fixture.issueId; + const otherIssue = scenario === "wrong_issue" ? (await seedRun(`WARM-OTHER-ISSUE-${command}`)).issueId : fixture.issueId; await fixture.db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, issueId: otherIssue, scopeKind: "issue", contextSnapshot: { issueId: otherIssue }, ...(preparing ? { runtimeMode: "legacy", executionStage: scenario === "dispatching_successor" ? "dispatching" : "preparing", @@ -1331,7 +1383,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( } await fixture.db.update(issues).set({ executionRunId: nextRunId }).where(eq(issues.id, fixture.issueId)); const before = (await loadNativeRunnerCheckpoint(fixture.db, binding))!; - authority.queueCommand(scenario === "new_work" ? "turn.start" : "runner.suspend", {}, `early-retirement-${scenario}`); + authority.queueCommand(scenario === "new_work" ? "turn.start" : command, {}, `early-retirement-${scenario}`); if (!["idle", "preparing_successor", "claimed_successor"].includes(scenario)) { await expect(authority.flushCheckpoint()).rejects.toThrow("native_runner_checkpoint_owner_lost"); expect((await loadNativeRunnerCheckpoint(fixture.db, binding))?.records).toEqual(before.records); @@ -1342,7 +1394,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( saved.restoreSessionDirectory(restoredRoot); const restored = JSON.parse(await readFile(resolve(restoredRoot, "control-plane", "control-plane-state.json"), "utf8")); expect(restored.identity).toEqual(identity); - expect(restored.commands).toContainEqual(expect.objectContaining({ type: "runner.suspend", status: "pending" })); + expect(restored.commands).toContainEqual(expect.objectContaining({ type: command, status: "pending" })); } const [task] = await fixture.db.select({ executionRunId: issues.executionRunId }).from(issues) .where(eq(issues.id, fixture.issueId)); @@ -1351,6 +1403,104 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( }, ); + realProcessIt("durably retires an accepted warm Runner while its same-controller successor prepares", async () => { + const fixture = await seedRun("REAL-WARM-RETIREMENT"); + const binding = { companyId, issueId: fixture.issueId, runId: fixture.runId }; + const root = resolve(runtimeRoot, fixture.runId, "real-retirement"); + const options = transportOptions(fixture, root); + const retirementController = await currentNativeControllerIdentity(); + const diagnostics: string[] = [], checkpointFailures: unknown[] = []; + let authority!: DurablePrpControlPlane; + await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: retirementController.bootId, + controllerGeneration: 1, leaseOwner: "real-retirement-owner", leaseExpiresAt: new Date(Date.now() + 60_000), + }).where(eq(nativeRunFinalizations.runId, fixture.runId)); + const transport = createRunnerdCodexTransport({ ...options, + codexArgs: [...options.codexArgs, "--linger-after-turn-start"], + onDiagnostic: message => diagnostics.push(message), + controlPlaneRegistration: async core => { + authority = core; + const write = nativeRunnerCheckpointWriter(fixture.db, { ...binding, authorityRunId: fixture.runId, + bootId: retirementController.bootId, generation: 1, leaseOwner: "real-retirement-owner" }); + core.setCheckpointPersistence(async checkpoint => { + try { await write(checkpoint); } + catch (error) { checkpointFailures.push(error); throw error; } + }); + return registerRunnerPrpAuthority({ ...binding, authority: core }); + }, + }); + let runnerPid: number | null = null, providerPid: number | null = null; + try { + await transport.transport.request("thread/start", { cwd: root, dynamicTools: [] }); + await transport.transport.request("turn/start", { input: [{ type: "text", text: "Keep the provider alive after accepting its result." }] }); + runnerPid = transport.evidence().runnerPid; + providerPid = transport.evidence().providerPid; + expect(runnerPid).toEqual(expect.any(Number)); + expect(providerPid).toEqual(expect.any(Number)); + await waitForCondition("active real provider before accepted-result retirement", () => + authority.store.state.committedEvents.some(event => event.eventType === "turn.started")); + await authority.flushCheckpoint(); + const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const resultId = randomUUID(); + await fixture.db.insert(nativeRunResults).values({ id: resultId, ...binding, turnId: options.prpIdentity.turnId, + completionContractId: run.completionContractId!, serverFingerprint: "real-retirement-accepted", + schemaStatus: "accepted", canonicalSha256: "a".repeat(64), + resultJson: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "Accepted before provider cleanup." }, + }); + await fixture.db.update(nativeRunFinalizations).set({ phase: "committed", resultId, leaseOwner: null, leaseExpiresAt: null }) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + await fixture.db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }) + .where(eq(heartbeatRuns.id, fixture.runId)); + const nextRunId = randomUUID(); + const [nextRun] = await fixture.db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, + issueId: fixture.issueId, nativeIssueId: fixture.issueId, scopeKind: "issue", status: "running", runtimeMode: "native", + invocationSource: "assignment", contextSnapshot: { issueId: fixture.issueId } }).returning(); + const next = await prepareNativeHeartbeatRun({ db: fixture.db, run: nextRun, + issue: { id: fixture.issueId, title: "Next conversation", description: null }, environmentLeaseId: randomUUID() }); + await fixture.db.insert(nativeRunFinalizations).values({ runId: nextRunId, companyId, issueId: fixture.issueId, phase: "observed" }); + await fixture.db.update(issues).set({ executionRunId: nextRunId }).where(eq(issues.id, fixture.issueId)); + const [contract] = await fixture.db.select().from(completionContracts).where(eq(completionContracts.id, run.completionContractId!)); + const nextExecution = buildNativeExecutionInput({ companyId, agentId, runId: nextRunId, + issue: { id: fixture.issueId, identifier: "NRR-NATIVE-RETIREMENT", title: "Next conversation", description: null, workMode: "standard" }, + taskPrompt: "Start only after the previous provider durably suspends.", + workspace: { id: randomUUID(), cwd: root, repoUrl: null, repoRef: null, branchName: null }, + normalizedSessionId: next.normalizedSessionId, runtimeContext: nativeRuntimeContextFixture(), + completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: contract.schemaVersion, contract: contract.contractJson as never }, + }); + const fixtureStoppedAfterRealClaim = new Error("fixture stopped before next provider dispatch"); + await expect(executePaperclipNativeSession({ db: fixture.db, execution: nextExecution, runnerInstanceId: next.runnerInstanceId, + onExecutionLeaseAcquired: async () => { + const [claimed] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, nextRunId)); + expect(claimed.controllerBootId).toBe(retirementController.bootId); + expect(claimed.leaseOwner).toEqual(expect.any(String)); + expect(claimed.leaseExpiresAt!.getTime()).toBeGreaterThan(Date.now()); + await expect(transport.transport.close(), diagnostics.join("\n")).resolves.toBeUndefined(); + throw fixtureStoppedAfterRealClaim; + }, + })).rejects.toBe(fixtureStoppedAfterRealClaim); + expect(checkpointFailures).toEqual([]); + const state = JSON.parse(await readFile(resolve(root, "runner", "runner-state.json"), "utf8")); + expect(state).toMatchObject({ lifecycle: "suspended", runId: fixture.runId, + normalizedSessionId: options.prpIdentity.normalizedSessionId, runnerInstanceId: options.prpIdentity.runnerInstanceId }); + const saved = (await loadNativeRunnerCheckpoint(fixture.db, binding))!; + const commands = [...saved.records].filter(([key]) => key.startsWith("command:")).map(([, value]) => JSON.parse(value)); + for (const type of ["turn.stop", "runner.drain", "runner.suspend"]) { + expect(commands, diagnostics.join("\n")).toContainEqual(expect.objectContaining({ type, status: "completed" })); + } + expect(commands.filter(command => command.type === "turn.start")).toHaveLength(1); + const [task] = await fixture.db.select().from(issues).where(eq(issues.id, fixture.issueId)); + expect(task.executionRunId).toBe(nextRunId); + } catch (error) { + throw new Error(`${String(error)}; owned retirement diagnostics ${JSON.stringify({ + checkpointFailures: checkpointFailures.map(String), diagnostics, + commands: authority?.store.state.commands.map(({ type, status }) => ({ type, status })), + })}`); + } finally { + await transport.transport.close().catch(() => undefined); + if (providerPid && processAlive(providerPid)) process.kill(providerPid, "SIGKILL"); + await stopOwnedProcessGroup(runnerPid, root).catch(() => undefined); + } + }, 45_000); + it("applies checkpoint deltas in Postgres without rewriting authority and removes only evicted events", async () => { const fixture = await seedRun("CHECKPOINT-DELTA"); const root = resolve(runtimeRoot, fixture.runId); @@ -1463,8 +1613,8 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( realProcessIt("keeps checkpoint SQL independent of streamed output volume with a real Postgres writer", async () => { const samples: Array<{ chunks: number; checkpointWrites: number; transactions: number; changedRows: number; checkpointBytes: number; walBytes: number; elapsedMs: number; idleSql: number; - streamSql: number; streamTransactions: number; streamChangedRows: number }> = []; - for (const chunks of [8, 512]) { + streamSql: number; streamTransactions: number; streamChangedRows: number; renewalSql: number }> = []; + for (const chunks of [8, 512, 8192]) { const fixture = await seedRun(`CHECKPOINT-COST-${chunks}`); const root = resolve(runtimeRoot, fixture.runId); const releaseOutput = resolve(root, "release-output"); @@ -1472,19 +1622,31 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( const owner = { ...binding, authorityRunId: fixture.runId, bootId: successor.bootId, generation: 1, leaseOwner: "cost-controller" }; await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: owner.bootId, - controllerGeneration: 1, leaseOwner: owner.leaseOwner, leaseExpiresAt: new Date(Date.now() + 120_000), + controllerPid: successor.pid, controllerProcessStartedAt: successor.processStartedAt, + controllerGeneration: 1, leaseOwner: owner.leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000), }).where(eq(nativeRunFinalizations.runId, fixture.runId)); let authority!: DurablePrpControlPlane; let measuring = false; let transactions = 0, changedRows = 0, checkpointBytes = 0; const statements: string[] = []; - const capturedClients = [fixture.db.$client, checkpointDb(fixture.db).$client]; + const renewalStatements: string[] = []; + let leaseLost = false; + const renewal = startNativeSessionExecutionLeaseRenewal({ db: fixture.db, ...binding, leaseOwner: owner.leaseOwner, + attempt: 0, controller: successor, remote: true, onLost: async () => { leaseLost = true; } }); + const capturedClients = [fixture.db.$client, checkpointDb(fixture.db).$client, livenessDb(fixture.db).$client]; const priorDebug = capturedClients.map(client => client.options.debug); // Capture statements only, never SQL parameters (encrypted state or secrets). - for (const client of capturedClients) { - client.options.debug = (_connection, query) => { if (measuring) statements.push(query); }; - } + capturedClients.forEach((client, index) => { + client.options.debug = (_connection, query) => { + if (!measuring) return; + // Renewals are existing liveness traffic, reported separately. A + // checkpoint query on any pool still counts against the zero oracle. + if (index === 2 && !query.includes("native_runner_checkpoints")) renewalStatements.push(query); + else statements.push(query); + }; + }); const write = nativeRunnerCheckpointWriter(fixture.db, owner); + let costFailure: unknown; const transport = createRunnerdCodexTransport({ ...transportOptions(fixture, root), codexArgs: ["--state-file", resolve(root, "provider.json"), "--linger-after-turn-start", "--held-output", "--held-output-count", String(chunks), "--held-output-release-file", releaseOutput], @@ -1516,7 +1678,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( const beforeStream = { sql: statements.length, transactions, changedRows }; await writeFile(releaseOutput, "release\n", { flag: "wx" }); await waitForCondition(`the final scripted text chunk of ${chunks}`, () => authority.store.state.committedEvents - .some(event => event.eventType === "item.delta" && JSON.stringify(event.envelope).includes(`still computing ${chunks - 1} `)), 60_000) + .some(event => event.eventType === "item.delta" && JSON.stringify(event.envelope).includes(`still computing ${chunks - 1} `)), chunks > 4096 ? 480_000 : 60_000) .catch(error => { console.info("checkpoint-cost-stream-diagnostic", JSON.stringify({ chunks, cursor: authority.store.state.ackedSourceSeq, receivedTextEvents: authority.store.state.committedEvents.filter(event => event.eventType === "item.delta").length })); throw error; }); @@ -1528,13 +1690,14 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( && query.includes("native_runner_checkpoints")).length; expect(checkpointWrites).toBeGreaterThan(0); // The collector must see real control writes. const snapshot = { chunks, checkpointWrites, transactions, changedRows, checkpointBytes, - walBytes: Number(walEnd.bytes), elapsedMs, idleSql: 0, + walBytes: Number(walEnd.bytes), elapsedMs, idleSql: 0, renewalSql: renewalStatements.length, streamSql: statements.length - beforeStream.sql, streamTransactions: transactions - beforeStream.transactions, streamChangedRows: changedRows - beforeStream.changedRows }; // Setup control messages may coalesce differently. The released text-only // interval must add exactly zero work, independent of that scheduling. expect(snapshot).toMatchObject({ streamSql: 0, streamTransactions: 0, streamChangedRows: 0 }); + expect(leaseLost).toBe(false); statements.length = 0; measuring = true; await new Promise(done => setTimeout(done, 100)); @@ -1547,20 +1710,34 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( expect(statements.some(query => /insert into "native_runner_checkpoints"/i.test(query))).toBe(true); measuring = false; samples.push(snapshot); + console.info("native-checkpoint-cost-sample", JSON.stringify(snapshot)); + } catch (error) { + costFailure = error; + throw error; } finally { measuring = false; capturedClients.forEach((client, index) => { client.options.debug = priorDebug[index]; }); - await transport.transport.close(); + try { await transport.transport.close(); } + catch (error) { + if (costFailure) throw new AggregateError([costFailure, error], "Checkpoint cost measurement and cleanup failed"); + throw error; + } + finally { + await renewal.stop().catch(error => { + if (costFailure) throw new AggregateError([costFailure, error], "Checkpoint cost measurement and lease renewal failed"); + throw error; + }); + } } } const report = { schema: "paperclip.checkpoint-cost/v1", samples, scope: "isolated Postgres; control operations included; WAL is diagnostic; no production capacity claim" }; if (process.env.PAPERCLIP_TEST_CHECKPOINT_COST_REPORT) await writeFile(process.env.PAPERCLIP_TEST_CHECKPOINT_COST_REPORT, `${JSON.stringify(report, null, 2)}\n`, { flag: "wx", mode: 0o600 }); - // Both stream sizes were checked independently; total setup counts remain + // Each stream size was checked independently; total setup counts remain // diagnostic because control-message batching is scheduling-dependent. console.info("native-checkpoint-cost", JSON.stringify(report)); - }, 120_000); + }, 540_000); realProcessIt.each([false, true])("adopts the same runner from Postgres after losing the controller filesystem (prepared task: %s)", async preparedTask => { const fixture = await seedRun(`EMPTY-CONTROLLER-DISK-${preparedTask}`); @@ -1579,9 +1756,10 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( heartbeatRunId: fixture.runId, issueId: fixture.issueId, status: "active", leasePolicy: "ephemeral", provider: "plugin", providerLeaseId: "saved-task", metadata: { agentId, runnerTaskLease: true, runnerTaskDispatched: true } }); } + const originalLeaseExpiresAt = new Date(Date.now() + 60_000); await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: bootId, controllerGeneration: 1, controllerPid: process.pid, controllerProcessStartedAt: new Date(), - leaseOwner: "original-controller", leaseExpiresAt: new Date(Date.now() + 60_000), + leaseOwner: "original-controller", leaseExpiresAt: originalLeaseExpiresAt, }).where(eq(nativeRunFinalizations.runId, fixture.runId)); const options = { ...transportOptions(fixture, controllerRoot), @@ -1647,9 +1825,13 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( await new Promise(resolveOutage => setTimeout(resolveOutage, outageMs)); expect(processAlive(runnerPid)).toBe(true); expect(processAlive(providerPid)).toBe(true); - await fixture.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1) }) - .where(eq(nativeRunFinalizations.runId, fixture.runId)); bootId = randomUUID(); + if (originalLeaseExpiresAt.getTime() > Date.now()) { + const [waiting] = await claimNativeRestartRecoveries({ db: fixture.db, controller: { ...successor, bootId }, + restartKind: "hard", runIds: [fixture.runId] }); + expect(waiting).toMatchObject({ kind: "awaiting_evidence", reason: "remote_controller_lease_live", retryAt: originalLeaseExpiresAt }); + await new Promise(resolveExpiry => setTimeout(resolveExpiry, Math.max(0, originalLeaseExpiresAt.getTime() - Date.now()) + 20)); + } const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: { ...successor, bootId }, restartKind: "hard", runIds: [fixture.runId] }); expect(claim).toMatchObject({ kind: "reattach_remote_runner", providerAttempt: 0, controllerGeneration: 2 }); diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 96732286a2d..fea3271f66d 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -1,6 +1,8 @@ import { onRetainedNativeSessionCleanupRelease } from "../retained-native-session-cleanup-notifications.js"; import { readIdleLocalWork, markIdleStartupComplete, markIdleIngressTracked } from "../idle-local-work.js"; import * as runnerCheckpoint from "./native-runner-checkpoint.js"; +import * as stagedAttachmentRecovery from "./native-staged-attachment-recovery.js"; +import * as controllerRecovery from "./native-controller-recovery.js"; import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js"; import * as environmentTaskRuntime from "../environment-task-runtime.js"; import * as pluginTaskRuntime from "./plugin-runner-task.js"; @@ -419,6 +421,10 @@ import { import { onNativeRunnerEvidence } from "./native-runner-evidence-signal.js"; beforeEach(() => { + // These transport-wiring fixtures use a partial Db double and synthetic IDs. + // The real selector/bootstrap proof is exercised with Postgres by the durable + // discovery and diskless admission suites, not by this backend unit boundary. + vi.spyOn(controllerRecovery, "resolveNativeControllerRecovery").mockResolvedValue({ kind: "no_prior_execution" }); state.drainMaintenance.mockReset().mockResolvedValue(undefined); usageRecorder.capture.mockReset().mockImplementation(async receipt => receipt); usageRecorder.create.mockReset().mockResolvedValue({ capture: usageRecorder.capture, @@ -9294,6 +9300,44 @@ describe("native warm session supervision", () => { expect(state.execute).toHaveBeenCalledTimes(2); }); + it("claims native execution authority before retiring a superseded warm identity", async () => { + const { base, next, run } = fixture("identity-native-claim-order"); + const updates: Array<{ table: unknown; values: Record }> = []; + let inspected = false; + const close = vi.fn(async () => { + if (inspected) return; + inspected = true; + expect(updates.some(update => update.table === nativeRunFinalizations + && update.values.leaseOwner && update.values.controllerBootId)).toBe(true); + }); + state.execute.mockReset().mockImplementationOnce(async options => { + options.onSession?.({ close }); return result; + }).mockImplementationOnce(async () => result); + await run(base); + // Normal runtime selection already converted preparation to native and + // inserted an unclaimed observed coordinator before calling this facade. + await executePaperclipNativeSession({ db: leaseDb(next, {}, {}, updates), + execution: next, runnerInstanceId: "runner", useRunnerd: true }); + expect(close).toHaveBeenCalledOnce(); + expect(state.execute).toHaveBeenCalledTimes(2); + }); + + it.each(["cancelled", "foreign-owner"] as const)("does not retire or stage a warm successor after claim denial: %s", async denial => { + const { base, next, run } = fixture(`identity-native-claim-denied-${denial}`); + const close = vi.fn(async () => undefined); + state.execute.mockReset().mockImplementationOnce(async options => { + options.onSession?.({ close }); return result; + }); + await run(base); + state.stageNativeRunnerWakeAttachments.mockClear(); + const db = leaseDb(next, denial === "foreign-owner" ? { leaseOwner: "different-controller" } : {}, + {}, [], {}, denial === "cancelled" ? "cancelled" : "running"); + await expect(executePaperclipNativeSession({ db, execution: next, runnerInstanceId: "runner", useRunnerd: true })).rejects.toThrow(); + expect(close).not.toHaveBeenCalled(); + expect(state.stageNativeRunnerWakeAttachments).not.toHaveBeenCalled(); + expect(state.execute).toHaveBeenCalledTimes(1); + }); + it("does not retire an active turn when a new session identity arrives", async () => { const { base, next, run } = fixture("identity-active"); let finish!: () => void; @@ -12274,6 +12318,55 @@ describe("runnerd provider runtime wiring", () => { ).toThrow("scope_closed"); }); + it.each(["attached", "changed-admission", "changed-template", "changed-template-digest", "changed-preparation"] as const)( + "verifies frozen staged admission after normal wake attachment preparation: %s", async mode => { + const current = scopedNativeExecution(`staged-attachment-${mode}`); + current.task.prompt = "Inspect attachment ref:original"; + const root = await initializeScopedRunnerRoot(current, "attachment-runner"); + const sourceIdentity = { runnerInstanceId: "attachment-runner", environmentLeaseId: "attachment-lease", + runId: "attachment-source", normalizedSessionId: current.session.normalizedSessionId!, turnId: "source-turn", itemId: "source-item" }; + const targetIdentity = { ...sourceIdentity, runId: current.binding.runId, turnId: "target-turn", itemId: "target-item" }; + const controller = new DurablePrpControlPlane({ stateDirectory: join(root, "control-plane"), identity: sourceIdentity, + expectedRunnerVersion: "fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + controller.stageRunAttachment(targetIdentity, { provider: { kind: "codex", instructions: "prepared attachment grant" } }); + const path = join(root, "control-plane/control-plane-state.json"); + const saved = JSON.parse(await readFile(path, "utf8")); + const canonicalTemplateDigest = saved.stagedRunAttachment.templateDigest; + if (mode === "changed-template" || mode === "changed-template-digest") { + saved.stagedRunAttachment.runAttachTemplate.provider.instructions = "substituted grant"; + if (mode === "changed-template-digest") saved.stagedRunAttachment.templateDigest = `sha256:${nativeSha256(saved.stagedRunAttachment.runAttachTemplate)}`; + await writeFile(path, JSON.stringify(saved)); + } + state.stageNativeRunnerWakeAttachments.mockImplementationOnce(async () => { + if (mode === "changed-preparation") current.task.prompt = "Inspect attachment ref:mutated-during-preparation"; + return { attachments: [{ id: "attachment-original" }], cleanup: async () => undefined } as never; + }); + state.renderNativeRunnerStagedAttachmentPrompt.mockReturnValueOnce("Materialized attachment: .paperclip-inbound/current/original"); + const admission = structuredClone(current); + if (mode === "changed-admission") admission.task.prompt = "Inspect attachment ref:changed"; + const reached = new Error("verified staged admission reached recheck"); + const restore = vi.spyOn(runnerCheckpoint, "restoreNativeRunnerCheckpoint").mockResolvedValue(null); + const header = vi.spyOn(runnerCheckpoint, "loadNativeRunnerCheckpointHeader").mockResolvedValue("staged header"); + const verify = vi.spyOn(stagedAttachmentRecovery, "resolveNativeStagedAttachmentRecovery") + .mockResolvedValueOnce({ currentExecution: admission, sourceIdentity, targetIdentity, + templateDigest: canonicalTemplateDigest } as never) + .mockRejectedValueOnce(reached); + try { + const resume = executePaperclipNativeSession({ db: leaseDb(current), execution: current, useRunnerd: true, + runnerInstanceId: "attachment-runner", restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, + leaseOwner: "controller", controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, + remote: { kind: "sandbox", leaseId: "attachment-lease", providerLeaseId: "sandbox", remoteCwd: "/workspace" } } as never, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", leaseId: "attachment-lease", remoteCwd: "/workspace", + runner: { execute: vi.fn() }, sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never }); + if (mode === "attached") { await expect(resume).rejects.toBe(reached); expect(verify).toHaveBeenCalledTimes(2); } + else await expect(resume).rejects.toThrow(mode === "changed-template" ? "staged_attachment_binding_mismatch" + : mode === "changed-preparation" ? "native_attachment_admission_changed" : "staged_attachment_recovery_unproven"); + expect(state.stageNativeRunnerWakeAttachments).toHaveBeenCalledWith(expect.objectContaining({ binding: expect.objectContaining({ + companyId: current.binding.companyId, issueId: current.binding.issueId, agentId: current.binding.agentId, runId: current.binding.runId, + }) })); + } finally { restore.mockRestore(); header.mockRestore(); verify.mockRestore(); await controller.stop(); } + }); + it("passes the run checkpoint active turn into restart recovery", async () => { state.createBackend.mockClear(); state.createTransport.mockClear(); @@ -12304,6 +12397,27 @@ describe("runnerd provider runtime wiring", () => { ); }); + it.each(["warm", "per_turn"] as const)("defers a %s retained-owner wait before claiming a provider attempt", async mode => { + const fresh = scopedNativeExecution(`retained-owner-${mode}`); + fresh.session.lifecyclePolicy = mode === "warm" + ? { mode: "warm", idleTimeoutMs: 300_000 } : { mode: "per_turn", idleTimeoutMs: null }; + const deadline = new Date(Date.now() + 60_000); + vi.mocked(controllerRecovery.resolveNativeControllerRecovery).mockResolvedValueOnce({ kind: "waiting_for_owner", deadline }); + const defer = vi.spyOn(controllerRecovery, "deferNativeControllerRecovery").mockResolvedValueOnce(); + const onExecutionLeaseAcquired = vi.fn(); + const updates: Array<{ table: unknown; values: Record }> = []; + const db = leaseDb(fresh, {}, {}, updates); + await expect(executePaperclipNativeSession({ db, execution: fresh, runnerInstanceId: "waiting-controller", + useRunnerd: true, onExecutionLeaseAcquired, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", + environmentId: "retained-environment", leaseId: "retained-lease" }, + })).rejects.toMatchObject({ name: "NativeControllerRecoveryWaitingError", retryAt: deadline }); + expect(defer).toHaveBeenCalledWith(db, expect.objectContaining({ ...fresh.binding, retryAt: deadline })); + expect(onExecutionLeaseAcquired).not.toHaveBeenCalled(); + expect(updates.some(update => update.table === nativeRunFinalizations && update.values.attempt !== undefined)).toBe(false); + expect(state.createBackend).not.toHaveBeenCalled(); + }); + it("rejects overlapping runs for the same runnerd provider session scope", async () => { const first = { ...execution, @@ -12664,7 +12778,7 @@ describe("runnerd provider runtime wiring", () => { } as NativeExecutionInputV1; const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!, runnerInstanceId: `admitted-${provider}`, environmentLeaseId: "admitted-lease", - turnId: `turn-${current.binding.runId}`, itemId: `item-${current.binding.runId}` }; + turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` }; const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId); await mkdir(join(root, "control-plane"), { recursive: true }); await writeFile(join(root, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity))); @@ -12694,9 +12808,56 @@ describe("runnerd provider runtime wiring", () => { runnerArtifact: artifact, adoptExistingRunner: { pid: 123, startedAt: "2026-09-19T10:00:00.000Z" } }); expect(state.resolveRemoteRunnerBinary).not.toHaveBeenCalled(); expect(execute.mock.calls.some(([request]) => request.args.includes("paperclip-runner-launch") || request.args.includes("--build-metadata"))).toBe(false); + const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise }; + await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined(); + expect(execute.mock.calls.some(([request]) => request.args.includes("paperclip-runner-launch") + || request.args.includes("paperclip-runner-claim-unstarted-session"))).toBe(false); }); - it.each(["claude", "codex"].flatMap(provider => (["per_turn", "warm"] as const).flatMap(policy => ["canonical", "local-receipt", "local-receipt-missing-terminal", "local-receipt-pending-command", "quarantined", "empty-runner", "empty-runner-quarantined", "nonempty-runner", "wrong-identity", "active", "foreign-scope", "no-stop-proof", "not-suspended", "pending-outbox", "pending-command", "missing-terminal", "ambiguous", "contradictory"].map(mode => [provider, policy, mode] as const))))( + it.each(["verified", "no-proof", "dead", "stale"] as const)("reuses only verified idle warm process during actual remote preparation: %s", async mode => { + const previous = scopedNativeExecution(`warm-live-preparation-${mode}`); + previous.session.lifecyclePolicy = { mode: "warm", idleTimeoutMs: 300_000 }; + const current = { ...previous, binding: { ...previous.binding, runId: `${previous.binding.runId}-next` } }; + const identity = { runId: previous.binding.runId, normalizedSessionId: previous.session.normalizedSessionId!, + runnerInstanceId: "retained-warm-runner", environmentLeaseId: "retained-lease", turnId: "prior-turn", itemId: "prior-item" }; + const root = await initializeScopedRunnerRoot(previous, identity.runnerInstanceId); + await writeRunnerAuthority(root, identity, "suspended"); + const projections = vi.spyOn(controllerRecovery, "resolveNativeControllerRecovery").mockResolvedValue({ kind: "no_prior_execution" }); + onTestFinished(() => projections.mockRestore()); + const db = leaseDb(current), ordinarySelect = db.select.bind(db); + db.select = ((fields?: Record) => fields?.status === heartbeatRuns.status && fields?.runnerProfileJson === heartbeatRuns.runnerProfileJson + ? priorNativeRunDb(previous).select(fields as never) : ordinarySelect(fields as never)) as Db["select"]; + let alive = true; + const execute = vi.fn(async (request: { args: string[] }) => ({ exitCode: request.args[2] === "paperclip-runner-recovery-check" && !alive ? 45 : 0, + timedOut: false, stderr: "", stdout: request.args[0] === "--build-metadata" ? JSON.stringify({ + schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd", + packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2, + durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"], + }) : request.args[0] === "--version" ? "codex-cli 0.160.0" : request.args[1]?.includes("base64") + ? Buffer.from(JSON.stringify({ ...identity, lifecycle: "ready", outbox: [], pendingTerminalDelivery: null })).toString("base64") + : request.args[1]?.includes("cat --") ? `nonce\n123\n2026-09-19T10:00:00.000Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : "" })); + const revalidate = vi.fn(async () => undefined); + const syncIn = vi.fn(async () => undefined); + await createRunnerdBackend({ db, execution: current, runnerInstanceId: identity.runnerInstanceId, + runnerIngressAuthorized: true, + ...(mode === "no-proof" ? {} : { remoteWarmRecovery: { identity, reuseProcess: true, previousExecution: previous, + checkpoint: {} as never, remote: { providerLeaseId: "retained-sandbox", remoteCwd: "/workspace" }, revalidate } }), + runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", environmentId: "environment", + leaseId: identity.environmentLeaseId, providerKey: "daytona", effectiveCapabilities: { runnerWebSocketIngress: true }, + sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "retained-sandbox" }, runner: { execute, syncIn } } as never }); + state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); + const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise }; + if (mode === "dead") alive = false; + if (mode === "stale") revalidate.mockRejectedValueOnce(new Error("fixture_warm_owner_changed")); + if (mode === "verified") await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined(); + else await expect(options.prepareExternalRunnerState()).rejects.toThrow(mode === "no-proof" ? "resumed_not_suspended" + : mode === "dead" ? "runner_remote_process_identity_unavailable" : "fixture_warm_owner_changed"); + expect(syncIn).not.toHaveBeenCalled(); + expect(execute.mock.calls.some(([request]) => request.args.includes("paperclip-runner-launch") + || request.args.includes("paperclip-runner-claim-unstarted-session"))).toBe(false); + }); + + it.each(["claude", "codex"].flatMap(provider => (["per_turn", "warm"] as const).flatMap(policy => ["canonical", "resumed", "resumed-timeout", "local-receipt", "local-receipt-missing-terminal", "local-receipt-pending-command", "quarantined", "empty-runner", "empty-runner-quarantined", "nonempty-runner", "wrong-identity", "active", "foreign-scope", "no-stop-proof", "not-suspended", "pending-outbox", "pending-command", "missing-terminal", "ambiguous", "contradictory"].filter(mode => policy === "per_turn" || !mode.startsWith("resumed")).map(mode => [provider, policy, mode] as const))))( "recovers a blocked %s %s plugin checkpoint on a replacement lease: %s", async (provider, policy, mode) => { const { permissionPolicy: _permissionPolicy, ...claudeProfile } = QUALIFIED_ACPX_PROFILES.claude; const first = { ...scopedNativeExecution(`plugin-${provider}-${policy}-${mode}`), @@ -12744,8 +12905,13 @@ describe("runnerd provider runtime wiring", () => { if (mode === "no-stop-proof") throw new Error("Plugin Runner retained session is not authorized"); return "old-task"; }); + let timedOut = false; const task = vi.fn(async (operation: any) => { environmentTaskOperationSchema.parse(operation); + if (mode === "resumed-timeout" && operation.kind === "read_state" && !timedOut) { + timedOut = true; + throw new Error("fixture_remote_state_timeout"); + } if (operation.kind === "read_state") return { kind: "state", taskId: "replacement-task", state: { ...durableRunnerState(identity, mode === "not-suspended" ? "ready" : "suspended"), outbox: mode === "pending-outbox" ? [{}] : [], pendingTerminalDelivery: null, @@ -12764,7 +12930,7 @@ describe("runnerd provider runtime wiring", () => { managedAiCredentialHome: home, managedAiCredentialIdentity: "revision", managedAiAccountIdentity: "account", runnerEnvironment: { ANTHROPIC_API_KEY: "synthetic-fixture" }, runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", leaseId: "replacement-lease", runnerTask: task, - sandboxLeaseAcquisition: { outcome: "replacement", providerLeaseId: "replacement-task" } } as never }; + sandboxLeaseAcquisition: { outcome: mode.startsWith("resumed") ? "resumed" : "replacement", providerLeaseId: "replacement-task" } } as never }; if (throughExecutor) { const validated = validatePrpStructuredRunResult({ schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "done", completionClaim: { contractRevision: next.completionContract.contract.revision, objectiveSatisfied: true, criteria: [], remainingWork: [] }, @@ -12774,12 +12940,19 @@ describe("runnerd provider runtime wiring", () => { normalizedSessionId: next.session.normalizedSessionId, providerSessionId: "original-provider-history", driverKind: next.session.driverKind, driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null }); } + if (mode === "resumed-timeout") { + await expect(createRunnerdBackend(launch)).rejects.toThrow("fixture_remote_state_timeout"); + expect(existsSync(join(scoped, "plugin-runner-state.json"))).toBe(false); + expect(await readFile(join(scoped, "control-plane/control-plane-state.json"), "utf8")).toBe(journal); + } const backend = throughExecutor ? executePaperclipNativeSession({ ...launch, useRunnerd: true }) : createRunnerdBackend(launch); - if (["canonical", "local-receipt", "quarantined", "empty-runner", "empty-runner-quarantined"].includes(mode)) { + if (["canonical", "resumed", "resumed-timeout", "local-receipt", "quarantined", "empty-runner", "empty-runner-quarantined"].includes(mode)) { await expect(backend).resolves.toBeDefined(); expect(state.createBackend).toHaveBeenCalledOnce(); - expect(owner).toHaveBeenCalledWith(expect.objectContaining({ resumeFrom: "old-task", priorIdentity: identity })); - expect(resume).toHaveBeenCalledWith(db, expect.objectContaining({ leaseId: "replacement-lease", priorIdentity: identity, inspectCheckpoint: true })); + expect(owner).toHaveBeenCalledWith(expect.objectContaining({ priorIdentity: identity, + ...(!mode.startsWith("resumed") ? { resumeFrom: "old-task" } : {}) })); + if (!mode.startsWith("resumed")) expect(resume).toHaveBeenCalledWith(db, expect.objectContaining({ leaseId: "replacement-lease", priorIdentity: identity, inspectCheckpoint: true })); + else expect(task).toHaveBeenCalledWith({ kind: "read_state", identity }); expect(JSON.parse(await readFile(join(scoped, "plugin-runner-state.json"), "utf8"))).toMatchObject({ ...identity, lifecycle: "suspended" }); expect(await readFile(join(scoped, "control-plane/control-plane-state.json"), "utf8")).toBe(journal); } else { diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index cf183108d3a..604bff67972 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,5 +1,6 @@ import { rehydratePinnedSkillAssets } from "./runtime-context.js"; import { createNativeUsageDelivery } from "./native-usage-delivery.js"; +import { createNativeRecoveryUsageDelivery } from "./native-recovery-usage-delivery.js"; import { createNativeTurnAccounting, nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage, usesNativeUsageDelivery, isNativeUsageAccountingEvent } from "./native-usage-accounting.js"; export { nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage } from "./native-usage-accounting.js"; import { measureRunnerOperation, recordRunnerTimingReceipt } from "./runner-operation-timing.js"; @@ -105,8 +106,9 @@ import { } from "node:fs"; import { tmpdir, userInfo } from "node:os"; import { DatabaseSync } from "node:sqlite"; -import { nativeRunnerCheckpointPersistence, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; +import { loadNativeRunnerCheckpointHeader, nativeRunnerCheckpointPersistence, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; import { REMOTE_CONTROLLER_LEASE_MS, REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runner-recovery-contract.js"; +import type { NativeTaskSessionPublicationContext } from "./native-task-session-publication.js"; import { basename, dirname, join, posix, resolve } from "node:path"; import type { AdapterExecutionResult, @@ -139,6 +141,7 @@ import { createRunnerdCodexTransport, executeNativeSession, applyNativeSessionGoalControl, + inspectStagedRunAttachment, inspectWarmRunTransition, parseNativeExecutionInput, parsePaperclipQuestionSet, @@ -185,6 +188,8 @@ import { nativeRunResults, } from "@paperclipai/db"; import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js"; +import { assertNativeRetainedSandboxBinding, deferNativeControllerRecovery, NativeControllerRecoveryWaitingError, resolveNativeControllerRecovery } from "./native-controller-recovery.js"; +import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js"; import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { appendNativeCredentialUpdate } from "./native-credential-updates.js"; import { nativeSha256 } from "./canonical.js"; @@ -210,6 +215,7 @@ import { } from "./remote-runner-recovery-diagnostics.js"; import { readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS } from "./remote-runner-recovery-probe.js"; import { inspectRemoteWarmSessionRecovery, type RemoteWarmSessionRecovery } from "./remote-warm-session-recovery.js"; +import { resolveNativeStagedAttachmentRecovery, type NativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js"; import { emitStartupReuse, environmentStartupReuse } from "@paperclipai/adapter-utils/startup-reuse"; import { persistActivity, publishActivity } from "../activity-log.js"; import { commitNativeStatusDecision } from "./status-decision-committer.js"; @@ -561,6 +567,7 @@ export type NativeInstructionWorkingCopy = { type WarmNativeSession = { onReleased: () => void; + sessionCheckpointPersisted?: (snapshot: PersistedNativeSession) => void; pluginTaskId?: string; observabilityCompatibilityKey?: string; runId: string; @@ -1802,6 +1809,14 @@ class SessionAuthorityEpoch { this.#assertCurrent(); } + async acceptRecoveryAccountingEvent(event: PrpEvent, capture: () => Promise) { + if (!isNativeUsageAccountingEvent(event)) return; + this.#assertCurrent(); + if (event.runId === this.runId) throw new Error("native_recovery_usage_source_unproven"); + await capture(); + this.#assertCurrent(); + } + definitions() { this.#assertCurrent(); return this.#authority.definitions(); @@ -5076,6 +5091,7 @@ async function recoverPluginRunnerCheckpoint(input: { async function migrateRunnerdStateRootForExecution(input: { db: Db; execution: NativeExecutionInput; + preparedExecutionAdmission?: NativePreparedExecutionAdmission; allowVerifiedBackup: boolean; retainedWarmOwnerRunId?: string; allowLocalRecovery: boolean; @@ -5111,6 +5127,12 @@ async function migrateRunnerdStateRootForExecution(input: { if (!isSafeNativeStateDirectory(scoped)) { throw new Error("runner_state_directory_unsafe"); } + const staged = await verifyStagedAttachmentRestart(input); + if (staged) { + // Physical inspection belongs to backend admission after lease renewal + // starts. This phase only validates the immutable source/target proof. + return; + } if (hasRetainedWarmTransitionEvidence(scoped)) { // A copied receipt is not restart/process authority. Keep both valid // unsupported and invalid forward evidence out of destructive legacy @@ -5623,6 +5645,11 @@ function readRunnerdDurableIdentity( ).toString("utf8"), ), ); + if (state.schema === "paperclip.runner.durable.control-plane-state.staged-attachment.v1") { + const staged = inspectStagedRunAttachment(state); + if (!staged) throw new Error("native_runner_staged_attachment_binding_mismatch"); + return { ...staged.sourceIdentity }; + } if (state.schema !== RUNNERD_CONTROL_PLANE_STATE_SCHEMA && !(allowExternal && state.schema === "paperclip.runner.external-control-plane-state.v1")) return null; return record(state.identity); @@ -5631,6 +5658,62 @@ function readRunnerdDurableIdentity( } } +function readStagedRunnerdAttachment(root: string) { + const path = resolve(root, "control-plane", "control-plane-state.json"); + if (!existsSync(path)) return null; + let state: Record; + try { + state = record(JSON.parse(readBoundedNativeFile(path, NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, + "runner_durable_identity_too_large").toString("utf8"))); + } catch { + // Detection must not preempt the existing forward-evidence fence or + // unreadable legacy identity quarantine. Neither path admits this state. + return null; + } + if (state.stagedRunAttachment === undefined && + state.schema !== "paperclip.runner.durable.control-plane-state.staged-attachment.v1") return null; + const staged = inspectStagedRunAttachment(state); + if (!staged) throw new Error("native_runner_staged_attachment_binding_mismatch"); + return staged; +} + +/** A staged attachment has not published a target session. Re-establish its + * exact source and target from private durable evidence before any remote RPC. */ +type NativePreparedExecutionAdmission = Readonly<{ admissionSha256: string; preparedSha256: string }>; + +async function verifyStagedAttachmentRestart(input: { + db: Db; execution: NativeExecutionInput; restartRecovery?: NativeRestartRecoveryClaim; + preparedExecutionAdmission?: NativePreparedExecutionAdmission; + runnerExecutionTarget?: AdapterExecutionTarget | null; +}, expected?: NativeStagedAttachmentRecovery): Promise { + const root = scopedRunnerdStateRoot(input.execution); + const localStage = readStagedRunnerdAttachment(root); + const staged = expected ?? localStage; + if (!staged) return null; + const deny = (): never => { throw new Error("native_runner_staged_attachment_recovery_unproven"); }; + const target = input.runnerExecutionTarget; + if (input.restartRecovery?.kind !== "reattach_remote_runner" || input.restartRecovery.remote.kind === "plugin_task" || + target?.kind !== "remote" || target.transport !== "sandbox" || target.runnerTask || + target.sandboxLeaseAcquisition?.outcome !== "resumed") return deny(); + const [current] = await input.db.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.execution.binding.runId), eq(heartbeatRuns.companyId, input.execution.binding.companyId), + eq(heartbeatRuns.agentId, input.execution.binding.agentId), eq(heartbeatRuns.nativeIssueId, input.execution.binding.issueId))); + const header = await loadNativeRunnerCheckpointHeader(input.db, input.execution.binding); + if (!current || current.status !== "running" || !header) return deny(); + const verified = await resolveNativeStagedAttachmentRecovery({ db: input.db, currentRun: current, header, + expectedTarget: staged.targetIdentity, + matchesExecution: (previous, execution) => nativeSessionScopeKey(previous) === nativeSessionScopeKey(execution) && + nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(execution, "remote"), + }); + const executionSha256 = nativeSha256(input.execution); + if (!verified || nativeSha256(verified.currentExecution) !== (input.preparedExecutionAdmission?.admissionSha256 ?? executionSha256) || + (input.preparedExecutionAdmission && executionSha256 !== input.preparedExecutionAdmission.preparedSha256) || + verified.templateDigest !== staged.templateDigest || + nativeSha256(verified.sourceIdentity) !== nativeSha256(staged.sourceIdentity) || + (localStage && nativeSha256(readStagedRunnerdAttachment(root)) !== nativeSha256(localStage))) return deny(); + return verified; +} + function durableIdentityMatchesExecution( identity: Record | null, execution: NativeExecutionInput, @@ -5784,12 +5867,43 @@ async function restorePerTurnRunnerdController(input: { const root = scopedRunnerdStateRoot(input.execution); const identity = readRunnerdDurableIdentity(root); const reference = identity ?? readRunnerdDurableIdentity(root, true); - if (!durableIdentityMatchesSession(reference, input.execution)) return; - if (identity?.runId === input.execution.binding.runId) return; - const restored = await restoreNativeRunnerCheckpoint(input.db, - { companyId: input.execution.binding.companyId, issueId: input.execution.binding.issueId, runId: reference.runId }, - root, nativeSessionKey(input.execution)); - if (!restored && !identity) throw new Error("native_runner_checkpoint_missing"); + if (!reference) { + if (!await restoreNativeControllerProjections(input.db, input.execution, root)) return; + } else { + if (!durableIdentityMatchesSession(reference, input.execution)) return; + if (identity?.runId === input.execution.binding.runId) return; + const restored = await restoreNativeRunnerCheckpoint(input.db, + { companyId: input.execution.binding.companyId, issueId: input.execution.binding.issueId, runId: reference.runId }, + root, nativeSessionKey(input.execution)); + if (!restored && !identity) throw new Error("native_runner_checkpoint_missing"); + } + const target = input.runnerExecutionTarget; + const prior = readRunnerdDurableIdentity(root); + // PRP and lifecycle evidence have different owners. A failed remote read can + // leave a valid PRP projection, so its presence must not suppress this retry. + // Never overwrite contradictory local Runner evidence with a remote receipt. + if (target.transport !== "sandbox" || target.sandboxLeaseAcquisition?.outcome !== "resumed" + || !durableIdentityMatchesSession(prior, input.execution) || prior.runId === input.execution.binding.runId + || runnerdAuthorityLifecycle(root, prior) !== "absent") return; + let state: Record; + if (target.runnerTask) { + // The admitted task owner binds this read to the surviving physical task; + // its suspended identity still belongs to the previous business turn. + const result = environmentTaskResultSchema.parse(await target.runnerTask({ kind: "read_state", identity: prior })); + if (result.kind !== "state") throw new Error("Plugin Runner checkpoint unavailable"); + state = result.state; + } else { + if (!target.runner) throw new Error("native_controller_recovery_dependency_unavailable:runner_state_reader"); + await assertNativeRetainedSandboxBinding(input.db, { ...input.execution.binding, runId: prior.runId, target }); + const stateDirectory = posix.join(target.remoteCwd, ".paperclip-runtime", "paperclip-runner", "sessions", + createHash("sha256").update(nativeSessionKey(input.execution)).digest("hex"), "runner"); + state = await readRemoteRunnerState({ runner: target.runner, stateDirectory }); + } + if (state.lifecycle !== "suspended" || !Array.isArray(state.outbox) || state.outbox.length !== 0 + || state.pendingTerminalDelivery != null || state.pendingProviderCleanup != null || state.warmTransition != null) { + throw new Error("native_controller_recovery_missing_evidence:runner_not_settled"); + } + savePluginRunnerCheckpoint(root, state, prior); } /** Materialize opaque controller state before ordinary warm admission reads it. @@ -5805,6 +5919,9 @@ async function restoreWarmRunnerdController(input: { const root = scopedRunnerdStateRoot(input.execution); const identity = readRunnerdDurableIdentity(root); const reference = identity ?? readRunnerdDurableIdentity(root, true); + if (!idleWarmNativeSessionOwnerRunId(input) && (!reference || !existsSync(nativeSessionCheckpointPath(input.execution)))) { + if (await restoreNativeControllerProjections(input.db, input.execution, root)) return; + } if (!durableIdentityMatchesSession(reference, input.execution)) return; const retainedRunId = idleWarmNativeSessionOwnerRunId(input); let runId = retainedRunId ?? reference.runId; @@ -5831,6 +5948,23 @@ async function restoreWarmRunnerdController(input: { if (!restored && !identity) throw new Error("native_runner_checkpoint_missing"); } +async function restoreNativeControllerProjections(db: Db, execution: NativeExecutionInput, root: string): Promise { + const recovered = await resolveNativeControllerRecovery(db, { + ...execution.binding, currentRunId: execution.binding.runId, normalizedSessionId: nativeSessionKey(execution), + }); + if (recovered.kind === "no_prior_execution") return false; + if (recovered.kind === "waiting_for_owner") { + throw new NativeControllerRecoveryWaitingError(recovered.deadline); + } + if (recovered.kind !== "recoverable") throw new Error(`native_controller_recovery_${recovered.kind}:${recovered.reason}`, { cause: recovered.cause }); + recovered.checkpoint.restoreSessionDirectory(root); + if (execution.session.lifecyclePolicy.mode === "warm") { + persistWarmNativeCheckpoint(execution, recovered.metadata.configDigest, recovered.snapshot, + recovered.metadata.processConfigDigest ?? undefined); + } + return true; +} + function nativeHarnessEnvironmentFingerprint( execution: NativeExecutionInput, ): string { @@ -7830,6 +7964,8 @@ export async function executePaperclipNativeSession(input: { refreshTools?: boolean; db: Db; execution: NativeExecutionInput; + /** Private server preparation proof; never persisted or accepted from a tool. */ + preparedExecutionAdmission?: NativePreparedExecutionAdmission; runnerInstanceId: string; /** Trusted task identity from the heartbeat orchestration. */ conversationMode?: boolean; @@ -7860,6 +7996,8 @@ export async function executePaperclipNativeSession(input: { /** Persist task-level continuity before a durable goal can outlive this run. */ onUsage?: (receipt: AdapterUsageCheckpoint) => Promise; onGoalCheckpoint?: (snapshot: PersistedNativeSession) => Promise; + /** Prepared continuity metadata; published with the accepted terminal checkpoint. */ + taskSessionPublication?: NativeTaskSessionPublicationContext; sessionGoalControl?: NativeSessionGoalControl | null; resumeSessionGoalHeartbeat?: boolean; preparationSpans?: NativeRunHistoricalSpan[]; @@ -7957,7 +8095,7 @@ export async function executePaperclipNativeSession(input: { throw new Error("native_session_supervisor_busy"); } executingNativeOwnerScopes.set(ownerScope, ownerToken); - await retireSupersededWarmNativeSessions( + if (!input.useRunnerd) await retireSupersededWarmNativeSessions( ownerScope, nativeSessionScopeKey(input.execution), nativeSessionKey(input.execution), ); if ((input.execution.schema === "paperclip.native-execution-input.v6" || input.execution.schema === "paperclip.native-execution-input.v8") && isCloudManagedInstance() @@ -7990,53 +8128,65 @@ export async function executePaperclipNativeSession(input: { } const workspaceRoot = input.execution.workspace.cwd; if (workspaceRoot === null) throw new Error("native_workspace_required"); - const targetKind = input.runnerExecutionTarget?.kind ?? "local"; - const chatAttachmentReadScope = new NativeChatAttachmentReadScope({ - db: input.db, - binding: input.execution.binding, - workspaceRoot, - executionTargetKind: targetKind, - }); - preparedInput = { ...input, chatAttachmentReadScope }; - cleanupStagedAttachments = () => chatAttachmentReadScope.close(); - const attachmentStage = await stageNativeRunnerWakeAttachments({ - db: input.db, - binding: { - companyId: input.execution.binding.companyId, - issueId: input.execution.binding.issueId, - runId: input.execution.binding.runId, - agentId: input.execution.binding.agentId, + const needsWarmRetirement = [...warmNativeSessions].some(([scope, entry]) => + scope !== sessionScopeId && entry.ownerScope === ownerScope); + const prepareAttachments = async () => { + const admissionSha256 = nativeSha256(input.execution); + const targetKind = input.runnerExecutionTarget?.kind ?? "local"; + const chatAttachmentReadScope = new NativeChatAttachmentReadScope({ + db: input.db, + binding: input.execution.binding, workspaceRoot, executionTargetKind: targetKind, - }, - }); - cleanupStagedAttachments = async () => { - const cleanupResults = await Promise.allSettled([ - chatAttachmentReadScope.close(), - attachmentStage.cleanup(), - ]); - if (cleanupResults.some((result) => result.status === "rejected")) { - throw new Error("paperclip_runner_attachment_staging_cleanup_failed"); - } - }; - const stagedPrompt = renderNativeRunnerStagedAttachmentPrompt( - attachmentStage.attachments, - ); - if (stagedPrompt) { - preparedInput = { - ...preparedInput, - execution: parseNativeExecutionInput({ - ...input.execution, - task: { - ...input.execution.task, - prompt: `${input.execution.task.prompt}\n\n${stagedPrompt}`, - }, - }), + }); + preparedInput = { ...input, chatAttachmentReadScope }; + cleanupStagedAttachments = () => chatAttachmentReadScope.close(); + const attachmentStage = await stageNativeRunnerWakeAttachments({ + db: input.db, + binding: { + companyId: input.execution.binding.companyId, + issueId: input.execution.binding.issueId, + runId: input.execution.binding.runId, + agentId: input.execution.binding.agentId, + workspaceRoot, + executionTargetKind: targetKind, + }, + }); + cleanupStagedAttachments = async () => { + const cleanupResults = await Promise.allSettled([ + chatAttachmentReadScope.close(), + attachmentStage.cleanup(), + ]); + if (cleanupResults.some((result) => result.status === "rejected")) { + throw new Error("paperclip_runner_attachment_staging_cleanup_failed"); + } }; - } - return await executePaperclipNativeSessionWithinScope(preparedInput); + const stagedPrompt = renderNativeRunnerStagedAttachmentPrompt( + attachmentStage.attachments, + ); + if (nativeSha256(input.execution) !== admissionSha256) throw new Error("native_attachment_admission_changed"); + if (stagedPrompt) { + const preparedExecution = parseNativeExecutionInput({ + ...input.execution, + task: { ...input.execution.task, prompt: `${input.execution.task.prompt}\n\n${stagedPrompt}` }, + }); + preparedInput = { + ...preparedInput, + execution: preparedExecution, + preparedExecutionAdmission: { admissionSha256, preparedSha256: nativeSha256(preparedExecution) }, + }; + } + return preparedInput; + }; + return needsWarmRetirement + ? await executePaperclipNativeSessionWithinScope(input, prepareAttachments) + : await executePaperclipNativeSessionWithinScope(await prepareAttachments()); } catch (error) { executionFailure = error; + if (error instanceof NativeControllerRecoveryWaitingError) { + await deferNativeControllerRecovery(input.db, { ...input.execution.binding, + leaseOwner: input.leaseOwner, retryAt: error.retryAt }); + } const remoteRunnerRecovery = readRemoteRunnerRecoveryError(error); if (remoteRunnerRecovery) { await input.onEvent?.({ @@ -8097,6 +8247,7 @@ export async function executePaperclipNativeSession(input: { async function executePaperclipNativeSessionWithinScope( input: Parameters[0], + prepareAfterClaim?: () => Promise[0]>, ): Promise { if (input.execution.provider.kind === "acpx" && input.execution.provider.agent === "antigravity") { const candidate = resolveAntigravityCandidateAdmission({ @@ -8120,6 +8271,8 @@ async function executePaperclipNativeSessionWithinScope( const restoredController = input.restartRecovery?.kind === "reattach_remote_runner" ? await restoreNativeRunnerCheckpoint(input.db, input.execution.binding, scopedRunnerdStateRoot(input.execution)) : null; + const stagedRestart = input.restartRecovery?.kind === "reattach_remote_runner" + ? await verifyStagedAttachmentRestart(input) : null; if (input.useRunnerd) await restoreWarmRunnerdController(input); const identityRedactor = createAgentIdentityRedactor(input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY); const redactIdentityText = (text: string) => redactSensitiveText( @@ -8240,6 +8393,7 @@ async function executePaperclipNativeSessionWithinScope( retainedTransition = await migrateRunnerdStateRootForExecution({ db: input.db, execution: input.execution, + preparedExecutionAdmission: input.preparedExecutionAdmission, allowVerifiedBackup: input.runnerExecutionTarget?.kind === "remote", retainedWarmOwnerRunId: idleWarmNativeSessionOwnerRunId(input) ?? remoteWarmRecovery?.identity.runId, allowLocalRecovery: input.runnerExecutionTarget?.kind !== "remote", @@ -8274,6 +8428,7 @@ async function executePaperclipNativeSessionWithinScope( const executionLeaseTtlMs = input.runnerExecutionTarget?.kind === "remote" ? REMOTE_CONTROLLER_LEASE_MS : NATIVE_SESSION_EXECUTION_LEASE_TTL_MS; let attempt: number; + let controllerGeneration = 0; try { attempt = await trace.measure( "native.coordinator.claim", @@ -8454,6 +8609,7 @@ async function executePaperclipNativeSessionWithinScope( .returning({ runId: nativeRunFinalizations.runId }) .then((rows) => rows[0] ?? null); if (!claimed) throw new Error("native_session_lease_lost"); + controllerGeneration = nextControllerGeneration; await tx .update(heartbeatRuns) .set({ @@ -8488,6 +8644,15 @@ async function executePaperclipNativeSessionWithinScope( && nativeRunsDetachingForRestart.get(input.execution.binding.runId) === startup.priorRestartDetach) { nativeRunsDetachingForRestart.delete(input.execution.binding.runId); } + // Runtime selection has already retired the legacy preparation lease. + // Claim current native authority before the old warm checkpoint writer + // must prove its same-controller successor. Shared staging stays after + // strict predecessor suspension; it cannot run on a failed claim/Stop. + if (input.useRunnerd) await retireSupersededWarmNativeSessions( + nativeSessionOwnerScope(input.execution, input.runnerExecutionTarget?.environmentId ?? null), + nativeSessionScopeKey(input.execution), nativeSessionKey(input.execution), + ); + if (prepareAfterClaim) input = await prepareAfterClaim(); await trace.end(taskPrepareScope); } catch (error) { await trace.end(taskPrepareScope, { outcome: "failed" }); @@ -8675,6 +8840,10 @@ async function executePaperclipNativeSessionWithinScope( && nativeSessionStartups.get(input.execution.binding.runId) === startupForAdmission) return; assertControllerActive(); }; + const controllerCompatibility = { + configDigest: nativeSessionConfigDigest(input.execution, input.runnerExecutionTarget?.kind ?? "local"), + processConfigDigest: warmNativeProcessConfigDigest(input) ?? null, + }; const controlPlane = new PaperclipControlPlanePort( input.db, { @@ -8692,6 +8861,16 @@ async function executePaperclipNativeSessionWithinScope( privateKeyPem: input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY, assertControllerActive, assertRunAdmissionActive, + taskSessionPublication: input.taskSessionPublication, + taskSessionPublicationOwner: input.taskSessionPublication ? { + ...input.execution.binding, authorityRunId: input.execution.binding.runId, + bootId: controller.bootId, leaseOwner, generation: controllerGeneration, + } : undefined, + nativeRecovery: input.useRunnerd && !input.backend && input.runnerExecutionTarget?.kind === "remote" ? { + compatibility: controllerCompatibility, + owner: { ...input.execution.binding, authorityRunId: input.execution.binding.runId, + bootId: controller.bootId, leaseOwner, generation: controllerGeneration }, + } : undefined, onResultCommitted: async () => { const at=Date.now(); await trace.record({name:"completion.result_committed",parentName:"native.session.execute",startedAtMs:at,endedAtMs:at}); @@ -8988,10 +9167,7 @@ async function executePaperclipNativeSessionWithinScope( : null; const warmConfigDigest = lifecyclePolicy.mode === "warm" - ? nativeSessionConfigDigest( - input.execution, - input.runnerExecutionTarget?.kind ?? "local", - ) + ? controllerCompatibility.configDigest : null; const configuredEnvironmentDigest = createHash("sha256") .update(JSON.stringify(configuredEnvironment(input.runnerEnvironment))) @@ -9004,7 +9180,8 @@ async function executePaperclipNativeSessionWithinScope( let managedCredentialSession: NativeSession | undefined; let githubAccess: NativeGitHubAccess | undefined; let releaseGitHubRun: (() => void) | undefined; - let persistedWarmSession: PersistedNativeSession | null | undefined; + let persistedWarmSession: PersistedNativeSession | null | undefined = stagedRestart && !stagedRestart.hasTargetCheckpoint + ? stagedRestart.checkpoint : undefined; if (warmSessionId !== null && warmConfigDigest !== null && !input.restartRecovery) { const entry = warmNativeSessions.get(warmSessionId); if (entry) { @@ -9264,6 +9441,13 @@ async function executePaperclipNativeSessionWithinScope( persistedWarmSession = null; } else { await recovered.session.close({ reason: "recovered warm session requires current process configuration" }); + // Closing an externally persisted controller leaves an identity marker, + // not its attachment template. The next backend must rotate the saved + // PRP authority from the committed suspension checkpoint. + const suspendedController = await restoreNativeRunnerCheckpoint(input.db, + remoteWarmRecovery.previousExecution.binding, scopedRunnerdStateRoot(input.execution), + nativeSessionKey(input.execution)); + if (!suspendedController) throw new Error("native_runner_checkpoint_missing_after_suspension"); persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution, nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote"), await recovered.session.snapshot()); persistedWarmSession = loadWarmNativeCheckpoint(input.execution, warmConfigDigest!, "remote"); @@ -9379,17 +9563,24 @@ async function executePaperclipNativeSessionWithinScope( requireSessionCloseBeforeReturn: runnerdBackend !== null || input.instructionWorkingCopy !== undefined, onSessionClosed: input.instructionWorkingCopy?.collectStopped, onCheckpoint: async (snapshot) => { + // The runtime invokes this only after its existing database + // snapshot transaction commits. Retirement coalesces into the + // next PRP command; this notification performs no persistence. + const checkpointOwner = warmSessionId === null ? undefined + : warmNativeSessions.get(warmSessionId)?.sessionCheckpointPersisted; + (checkpointOwner ?? runnerdBackend?.sessionCheckpointPersisted)?.(snapshot); if (snapshot.governedWait) { const startup = nativeSessionStartups.get(input.execution.binding.runId); if (startup) startup.governedWait = true; } snapshot = identityRedactor.redact(snapshot); + if (input.runnerExecutionTarget?.kind === "remote") snapshot = redactRunnerApiPayloads(snapshot); if (warmSessionId !== null && warmConfigDigest !== null) { await persistWarmNativeCheckpoint( input.execution, warmConfigDigest, snapshot, - warmNativeProcessConfigDigest(input), + controllerCompatibility.processConfigDigest ?? undefined, ); } if ( @@ -9474,11 +9665,15 @@ async function executePaperclipNativeSessionWithinScope( if (existing.ownerToken !== warmSessionOwnerToken) { throw new Error("native_session_supervisor_busy"); } + if (existing.session !== session) { + existing.sessionCheckpointPersisted = runnerdBackend?.sessionCheckpointPersisted; + } existing.session = session; existing.runId = input.execution.binding.runId; existing.issueId = input.execution.binding.issueId; } else warmNativeSessions.set(warmSessionId, { + sessionCheckpointPersisted: runnerdBackend?.sessionCheckpointPersisted, onReleased: () => releaseRetainedNativeSessionCleanup(input.db, { companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId, }), @@ -11501,6 +11696,62 @@ export async function verifyRemoteRunnerReattachment(input: { }; } +/** A crash before source bootstrap leaves a suspended Runner; a crash during + * its drain leaves the same live source. Neither permits a replacement until + * the physical process and all six persisted identity fields are proven. */ +async function verifyStagedRemoteRunner(input: { + recovery: NativeStagedAttachmentRecovery; + execution: NativeExecutionInput; + target: AdapterExecutionTarget | null | undefined; + claim: Extract; +}) { + const { target } = input; + if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask) { + throw new Error("native_runner_staged_attachment_profile_unsupported"); + } + const sourceIdentity = input.recovery.sourceIdentity; + const stateDirectory = posix.join(target.remoteCwd, ".paperclip-runtime", "paperclip-runner", "sessions", + createHash("sha256").update(sourceIdentity.normalizedSessionId).digest("hex"), "runner"); + const read = () => readRemoteRunnerState({ runner: target.runner!, stateDirectory }); + const state = await read(); + const control = record(JSON.parse(readBoundedNativeFile(resolve(scopedRunnerdStateRoot(input.execution), + "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, + "runner_durable_identity_too_large").toString("utf8"))); + const warm = inspectWarmRunTransition({ controlPlaneState: control, runnerState: state, + expectedNewIdentity: input.recovery.targetIdentity, expectedRunnerVersion: input.recovery.runnerVersion, + expectedRunnerDigest: input.recovery.runnerDigest }); + if (!warm && (control.warmTransition !== undefined || state.warmTransition !== undefined)) { + throw new Error("native_runner_staged_attachment_recovery_unproven"); + } + const identity = warm?.runnerIdentity ?? input.recovery.protocolIdentity; + const sameIdentity = (value: Record) => + Object.entries(identity).every(([key, expected]) => value[key] === expected); + if (!sameIdentity(state)) throw new Error("runner_remote_recovery_identity_mismatch"); + const claim = { ...input.claim, runId: identity.runId }; + const verify = () => verifyRemoteRunnerReattachment({ claim, target, identity: { ...identity }, + runId: identity.runId, normalizedSessionId: identity.normalizedSessionId }); + if (state.lifecycle !== "suspended") return verify(); + const settled = (value: Record) => sameIdentity(value) && value.lifecycle === "suspended" && + Array.isArray(value.outbox) && value.outbox.length === 0 && value.pendingTerminalDelivery == null && + value.pendingProviderCleanup == null && value.warmTransition == null; + if (!settled(state)) throw new Error("native_runner_staged_attachment_source_unsettled"); + try { + const retiring = await verify(); + const deadline = Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS; + while (await retiring.isAlive()) { + if (Date.now() >= deadline) throw new Error("native_runner_staged_attachment_source_retiring"); + await new Promise(resolveWait => setTimeout(resolveWait, 100)); + } + } catch (error) { + const diagnostic = readRemoteRunnerRecoveryError(error); + if (diagnostic?.stage !== "process_verification" || diagnostic.reason !== "process_not_running") throw error; + } + // An absent/unsafe marker or a changed process generation never proves exit. + // The package rechecks this same suspended source before launching it. + if (!settled(await read())) throw new Error("native_runner_staged_attachment_source_unsettled"); + return undefined; +} + async function waitForRemoteRunnerProcessIdentity(input: { runner: CommandManagedRuntimeRunner; identityPath: string; @@ -11880,6 +12131,7 @@ export async function createRunnerdBackend(input: { controllerPersistence?: { bootId: string; runnerArtifact?: { version: string; digest: string } }; onAccountingEvent?: (event: PrpEvent) => Promise; execution: NativeExecutionInput; + preparedExecutionAdmission?: NativePreparedExecutionAdmission; runnerInstanceId: string; chatAttachmentReadScope?: NativeChatAttachmentReadScope; restartRecovery?: NativeRestartRecoveryClaim; @@ -11929,7 +12181,7 @@ export async function createRunnerdBackend(input: { contextSnapshot: Record; }, ) => Promise; -}): Promise { +}): Promise { const sessionScopeId = nativeSessionScopeKey(input.execution); const scopeOwner = executingRunnerdSessionScopes.get(sessionScopeId); if (scopeOwner && scopeOwner !== input.execution.binding.runId) { @@ -11954,6 +12206,7 @@ export async function createRunnerdBackend(input: { retainedTransition = await migrateRunnerdStateRootForExecution({ db: input.db, execution: input.execution, + preparedExecutionAdmission: input.preparedExecutionAdmission, allowVerifiedBackup: input.runnerExecutionTarget?.kind === "remote", retainedWarmOwnerRunId: undefined, allowLocalRecovery: input.runnerExecutionTarget?.kind !== "remote", @@ -12079,11 +12332,16 @@ async function prepareRemoteExternalRunner({ input, target, runner, root, identi return { runnerBinary, readProviderState, ...executionDriver }; } +type ManagedRunnerdBackend = NativeSessionBackend & { + bindManagedSession(session: NativeSession): NativeSession; + sessionCheckpointPersisted?(snapshot: PersistedNativeSession): void; +}; + async function createRunnerdBackendWithinSessionClaim( input: Parameters[0], sessionScopeId: string, retainedTransition?: VerifiedWarmTransitionBinding, -): Promise { +): Promise { const timeController = createRunPreparationTiming(input.execution.binding.runId, "native.controller"); let recoveryPending = retainedTransition !== undefined; const target = input.runnerExecutionTarget ?? { kind: "local" as const }; @@ -12216,6 +12474,17 @@ async function createRunnerdBackendWithinSessionClaim( : durableIdentityMatchesSession(durableIdentity, input.execution) ? durableIdentity : null; + const stagedRecovery = input.restartRecovery?.kind === "reattach_remote_runner" + ? await verifyStagedAttachmentRestart(input) : null; + let stagedRecoveryPending = stagedRecovery !== null; + const recoveryUsageIdentity = stagedRecovery?.sourceIdentity ?? input.remoteWarmRecovery?.identity ?? durableIdentity; + const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity + ? createNativeRecoveryUsageDelivery({ + db: input.db, execution: input.execution, sourceIdentity: { ...recoveryUsageIdentity }, + matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) && + nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind), + }) + : undefined; const effectiveRunnerInstanceId = durableBinding?.runnerInstanceId ?? input.runnerInstanceId; const effectiveEnvironmentLeaseId = @@ -13048,7 +13317,20 @@ async function createRunnerdBackendWithinSessionClaim( bytesTransferred: 0, }, }); - if (sandboxLeaseAcquisition?.outcome === "resumed") { + if (sandboxLeaseAcquisition?.outcome === "resumed" && adoptedProcess && + (input.restartRecovery?.kind === "reattach_remote_runner" || input.remoteWarmRecovery)) { + // Active recovery and a verified idle warm handoff adopt the + // exact surviving process, rather than restoring a suspended + // per-turn backup. Identity/lease/PID generation were verified + // above; refresh that proof after any preparation wait. The + // transport still authenticates the durable PRP authority. + await measureNativeRunnerSpan(input.trace, "sandbox.lease.resume", async () => { + await input.remoteWarmRecovery?.revalidate(); + if (!await adoptedProcess.isAlive()) { + throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "process_not_running" }); + } + }); + } else if (sandboxLeaseAcquisition?.outcome === "resumed") { const reuseStartedAtMs = Date.now(); const state = await measureNativeRunnerSpan( input.trace, @@ -13606,7 +13888,9 @@ async function createRunnerdBackendWithinSessionClaim( ? input.restartRecovery.process : null; const adoptedProcess = - input.remoteWarmRecovery + stagedRecovery && input.restartRecovery?.kind === "reattach_remote_runner" + ? await verifyStagedRemoteRunner({ recovery: stagedRecovery, execution: input.execution, target, claim: input.restartRecovery }) + : input.remoteWarmRecovery ? await (async () => { await input.remoteWarmRecovery!.revalidate(); return verifyRemoteRunnerReattachment({ @@ -13641,6 +13925,10 @@ async function createRunnerdBackendWithinSessionClaim( ) => { const current = sessionToolAuthorityEpochs.get(sessionScopeId); if (!current) throw new Error("native_session_tool_authority_unavailable"); + const attachmentRunId = runnerControlPlane?.store?.state.identity.runId; + if (attachmentRunId && attachmentRunId !== current.runId) { + throw new Error("native_runner_source_semantic_reconciliation_required"); + } return current.execute(call); }; if (requiresRemoteProviderPack) { @@ -13688,8 +13976,23 @@ async function createRunnerdBackendWithinSessionClaim( onAccountingEvent: input.onAccountingEvent ? async event => { const current = sessionToolAuthorityEpochs.get(sessionScopeId); if (!current) throw new Error("native_session_authority_unavailable"); + if (input.remoteWarmRecovery && event.runId === input.remoteWarmRecovery.identity.runId) { + if (!recoveryUsage) throw new Error("native_recovery_usage_source_unproven"); + const source = input.remoteWarmRecovery.identity; + await current.acceptRecoveryAccountingEvent(event, () => recoveryUsage(event, { + runnerInstanceId: source.runnerInstanceId, environmentLeaseId: source.environmentLeaseId, + runId: source.runId, normalizedSessionId: source.normalizedSessionId, + turnId: source.turnId, itemId: source.itemId, + })); + return; + } await current.acceptAccountingEvent(event); } : undefined, + onRecoveryAccountingEvent: input.onAccountingEvent ? async (event, source) => { + const current = sessionToolAuthorityEpochs.get(sessionScopeId); + if (!current || !recoveryUsage) throw new Error("native_recovery_usage_source_unproven"); + await current.acceptRecoveryAccountingEvent(event, () => recoveryUsage(event, source)); + } : undefined, onCredentialsUpdated: (event) => appendNativeCredentialUpdate(input.db, { companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId, @@ -13866,13 +14169,23 @@ async function createRunnerdBackendWithinSessionClaim( input.execution.interactionResponses.length > 0 ? "allow_replacement_after_governed_wait" : undefined), - prpIdentity: input.remoteWarmRecovery ? { + prpIdentity: stagedRecovery ? stagedRecovery.targetIdentity : input.remoteWarmRecovery ? { runnerInstanceId: input.remoteWarmRecovery.identity.runnerInstanceId, environmentLeaseId: input.remoteWarmRecovery.identity.environmentLeaseId, runId: input.remoteWarmRecovery.identity.runId, normalizedSessionId: input.remoteWarmRecovery.identity.normalizedSessionId, turnId: input.remoteWarmRecovery.identity.turnId, itemId: input.remoteWarmRecovery.identity.itemId, + } : input.restartRecovery?.kind === "reattach_remote_runner" ? { + // Reattachment verified all six durable fields against the surviving + // Runner. Warm handoff uses opaque attachment turn/item IDs; deriving + // replacements from runId would request a different same-run authority. + runnerInstanceId: durableIdentity!.runnerInstanceId as string, + environmentLeaseId: durableIdentity!.environmentLeaseId as string, + runId: durableIdentity!.runId as string, + normalizedSessionId: durableIdentity!.normalizedSessionId as string, + turnId: durableIdentity!.turnId as string, + itemId: durableIdentity!.itemId as string, } : { runnerInstanceId: effectiveRunnerInstanceId, environmentLeaseId: effectiveEnvironmentLeaseId, @@ -13883,10 +14196,19 @@ async function createRunnerdBackendWithinSessionClaim( turnId: `turn-${input.execution.binding.runId}`, itemId: `item-${input.execution.binding.runId}`, }, - warmTransitionRegistrationMode: retainedTransition - ? "routed_connect" - : undefined, - authorizeWarmTransitionRecovery: retainedTransition + warmTransitionRegistrationMode: stagedRecovery && remoteTarget?.transport === "sandbox" && + remoteTarget.effectiveCapabilities?.runnerWebSocketIngress === true + ? "routed_same_endpoint" + : retainedTransition || stagedRecovery ? "routed_connect" : undefined, + authorizeWarmTransitionRecovery: stagedRecovery + ? async () => { + if (!stagedRecoveryPending) return; + const current = await verifyStagedAttachmentRestart(input, stagedRecovery); + if (!current || current.templateDigest !== stagedRecovery.templateDigest) { + throw new Error("native_runner_staged_attachment_recovery_unproven"); + } + } + : retainedTransition ? async ( stage: "before_bootstrap" | "before_spawn" | "before_authentication", @@ -13961,6 +14283,13 @@ async function createRunnerdBackendWithinSessionClaim( beforeConnect: async (_identity, authority) => { runnerControlPlane = authority; await input.remoteWarmRecovery?.revalidate(); + if (stagedRecovery && authority.store.state.stagedRunAttachment) { + const current = await verifyStagedAttachmentRestart(input, stagedRecovery); + if (!current || current.templateDigest !== stagedRecovery.templateDigest || + nativeSha256(current.targetIdentity) !== nativeSha256(stagedRecovery.targetIdentity)) { + throw new Error("native_runner_staged_attachment_recovery_unproven"); + } + } if (retainedTransition && recoveryPending) { const current = await verifyWarmTransitionRestart(input); if (current.stateFingerprint !== retainedTransition.stateFingerprint) { @@ -14052,6 +14381,15 @@ async function createRunnerdBackendWithinSessionClaim( }; return { bindManagedSession: wrapManagedSession, + sessionCheckpointPersisted: snapshot => { + const staged = runnerControlPlane?.store.state.stagedRunAttachment; + if (staged && snapshot.identity.runId === staged.targetIdentity.runId && + snapshot.identity.sessionId === staged.targetIdentity.normalizedSessionId) { + if (runnerControlPlane!.markStagedTargetCheckpointPublished(staged.targetIdentity)) { + stagedRecoveryPending = false; + } + } + }, descriptor: () => backend.descriptor(), openSession: async (sessionInput) => { const session = await backend.openSession(sessionInput); @@ -14074,7 +14412,7 @@ async function createRunnerdBackendWithinSessionClaim( await recordAdmittedProviderPack(); return wrapManagedSession(session); }, - } satisfies NativeSessionBackend & { bindManagedSession(session: NativeSession): NativeSession }; + } satisfies ManagedRunnerdBackend; } export function createRemoteNativeArtifactPreparation(input: { diff --git a/server/src/services/native-runtime/native-session-resume-projection.test.ts b/server/src/services/native-runtime/native-session-resume-projection.test.ts index 08242f2a690..502bcaf68f3 100644 --- a/server/src/services/native-runtime/native-session-resume-projection.test.ts +++ b/server/src/services/native-runtime/native-session-resume-projection.test.ts @@ -5,6 +5,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import { findNativeSessionResumeRun } from "./native-session-resume.js"; +import { resolveNativeControllerRecovery } from "./native-controller-recovery.js"; const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TEST_DATABASE_URL"); const supported = Boolean(externalUrl) || (await getEmbeddedPostgresTestSupport()).supported; @@ -58,7 +59,7 @@ const supported = Boolean(externalUrl) || (await getEmbeddedPostgresTestSupport( const saved = source(); await db.insert(heartbeatRuns).values([saved, run(1), run(2)]); const result = await select(); - expect(result).toEqual({ ...saved, runnerProfileJson: { nativeExecutionInput, sessionCheckpoint: checkpoint, nativeToolContractFingerprint: "fingerprint" }, + expect(result).toEqual({ ...saved, runnerProfileJson: { nativeExecutionInput, sessionCheckpoint: checkpoint, nativeToolContractFingerprint: "fingerprint", nativeControllerRecovery: null }, contextSnapshot: { paperclipWake, paperclipIssue } }); const stored = await db.select({ profile: heartbeatRuns.runnerProfileJson, context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(eq(heartbeatRuns.id, saved.id)); expect(stored[0].profile).toEqual(profile); @@ -80,6 +81,16 @@ const supported = Boolean(externalUrl) || (await getEmbeddedPostgresTestSupport( } }); + it("does not call missing or newer incompatible recovery material a fresh session", async () => { + const saved = source(); + await db.insert(heartbeatRuns).values([saved, run(30, { id: currentRunId, status: "running" })]); + const { beforeCreatedAt: _before, ...binding } = input; + expect(await resolveNativeControllerRecovery(db, binding)).toEqual({ kind: "missing_evidence", reason: "snapshot_binding" }); + // The newest progress barrier remains authoritative even without a snapshot. + await db.insert(heartbeatRuns).values(run(10, { runtimeMode: "legacy" })); + expect(await resolveNativeControllerRecovery(db, binding)).toEqual({ kind: "missing_evidence", reason: "newer_progress_barrier" }); + }); + it("skips only unused SQL-null, JSON-null, scalar, array and checkpoint-null profiles", async () => { const saved = source(), newer = run(1, { runnerProfileJson: null }); await db.insert(heartbeatRuns).values([saved, newer]); diff --git a/server/src/services/native-runtime/native-session-resume.ts b/server/src/services/native-runtime/native-session-resume.ts index 25463b68c78..f297fa95057 100644 --- a/server/src/services/native-runtime/native-session-resume.ts +++ b/server/src/services/native-runtime/native-session-resume.ts @@ -223,6 +223,8 @@ export async function findNativeSessionResumeRun( normalizedSessionId: string; currentRunId: string; beforeCreatedAt: Date; + /** Recovery discovery must retain an ineligible newest progress barrier. */ + includeProgressBarrier?: boolean; }, ) { if (!isNativeSessionId(input.normalizedSessionId)) return null; @@ -287,10 +289,12 @@ export async function findNativeSessionResumeRun( nativeToolContractFingerprint: unknown; nativeExecutionInput: unknown; sessionCheckpoint: unknown; + nativeControllerRecovery: unknown; }>`jsonb_build_object( 'nativeToolContractFingerprint', ${heartbeatRuns.runnerProfileJson}->'nativeToolContractFingerprint', 'nativeExecutionInput', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput', - 'sessionCheckpoint', ${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint' + 'sessionCheckpoint', ${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint', + 'nativeControllerRecovery', ${heartbeatRuns.runnerProfileJson}->'nativeControllerRecovery' )`, contextSnapshot: sql<{ paperclipWake: unknown; @@ -306,14 +310,16 @@ export async function findNativeSessionResumeRun( .from(heartbeatRuns) .innerJoin(barrier, eq(heartbeatRuns.id, barrier.id)) .where(and( - eq(barrier.runtimeMode, "native"), - inArray(barrier.status, terminalStatuses), - eq(barrier.hasCheckpoint, true), + ...(input.includeProgressBarrier ? [] : [ + eq(barrier.runtimeMode, "native"), + inArray(barrier.status, terminalStatuses), + eq(barrier.hasCheckpoint, true), + sql`date_trunc('milliseconds', ${barrier.createdAt}) <= ${input.beforeCreatedAt.toISOString()}::timestamptz`, + ]), sql`${heartbeatRuns.companyId}::text = ${input.companyId}`, sql`${heartbeatRuns.agentId}::text = ${input.agentId}`, sql`${heartbeatRuns.nativeIssueId}::text = ${input.issueId}`, sql`${heartbeatRuns.nativeSessionId}::text = ${input.normalizedSessionId}`, - sql`date_trunc('milliseconds', ${barrier.createdAt}) <= ${input.beforeCreatedAt.toISOString()}::timestamptz`, )); return run ?? null; } diff --git a/server/src/services/native-runtime/native-staged-attachment-recovery.test.ts b/server/src/services/native-runtime/native-staged-attachment-recovery.test.ts new file mode 100644 index 00000000000..1c872bba7ab --- /dev/null +++ b/server/src/services/native-runtime/native-staged-attachment-recovery.test.ts @@ -0,0 +1,187 @@ +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { Db } from "@paperclipai/db"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { DurablePrpControlPlane, type DurablePrpCheckpoint } from "../../vendor/paperclip-runner/index.js"; +import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provider.js"; +import { buildNativeExecutionInput } from "./native-execution-input.js"; +import { nativeToolContractFingerprintForTarget } from "./native-session-resume.js"; +import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; +import { resolveNativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js"; + +const companyId = "10000000-0000-4000-8000-000000000001"; +const issueId = "20000000-0000-4000-8000-000000000002"; +const agentId = "30000000-0000-4000-8000-000000000003"; +const sourceRunId = "40000000-0000-4000-8000-000000000004"; +const currentRunId = "50000000-0000-4000-8000-000000000005"; +const normalizedSessionId = "60000000-0000-4000-8000-000000000006"; +const runnerInstanceId = "70000000-0000-4000-8000-000000000007"; +const sourceIdentity = { runId: sourceRunId, normalizedSessionId, runnerInstanceId, + environmentLeaseId: "environment-1", turnId: "opaque-source-turn", itemId: "opaque-source-item" }; +const targetIdentity = { ...sourceIdentity, runId: currentRunId, turnId: "opaque-target-turn", itemId: "opaque-target-item" }; +const roots: string[] = []; +afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); }); + +function execution(runId: string) { + return buildNativeExecutionInput({ companyId, runId, agentId, normalizedSessionId, + issue: { id: issueId, identifier: "TEST-1", title: "Continue", description: null, workMode: "standard" }, + taskPrompt: "Continue normally", provider: "codex", workspace: { id: "workspace-1", cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null }, + completionContract: { id: "80000000-0000-4000-8000-000000000008", sha256: `sha256:${"a".repeat(64)}`, + schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: "Continue", criteria: [{ id: "done", requirement: "Answer" }] } }, + runtimeContext: nativeRuntimeContextFixture(), + }); +} + +async function fixture() { + const root = await mkdtemp(join(tmpdir(), "native-staged-proof-")); roots.push(root); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity: sourceIdentity, + expectedRunnerVersion: "0.0.0", expectedRunnerDigest: `sha256:${"f".repeat(64)}` }); + let saved!: DurablePrpCheckpoint; + core.setCheckpointPersistence(async checkpoint => { saved = checkpoint; }); + core.persistRunAttachTemplate({ provider: { provider: "codex", command: "codex" }, authorizedTools: {} }); + await core.flushCheckpoint(); + const sourceHeader = saved.records.get("header")!; + core.stageRunAttachment(targetIdentity, { provider: { provider: "codex", command: "codex" }, authorizedTools: {} }); + await core.flushCheckpoint(); + const header = saved.records.get("header")!; + const base = { companyId, agentId, issueId, nativeIssueId: issueId, runnerInstanceId, + nativeSessionId: normalizedSessionId, runtimeMode: "native", status: "running", finishedAt: null as Date | null }; + const currentRun = { ...base, id: currentRunId, runnerProfileJson: { nativeExecutionInput: execution(currentRunId) } }; + const sourceRun = { ...base, id: sourceRunId, status: "succeeded", finishedAt: new Date(), runnerProfileJson: { + nativeExecutionInput: execution(sourceRunId), nativeToolContractFingerprint: nativeToolContractFingerprintForTarget("remote", "codex"), + sessionCheckpoint: { sessionId: "provider-thread-1", providerSessionId: "provider-thread-1", identity: { + companyId, issueId, agentId, runId: sourceRunId, sessionId: normalizedSessionId }, + }, + } }; + const source = { run: sourceRun, coordinator: { runId: sourceRunId, companyId, issueId, + phase: "committed", resultId: "result-1", leaseOwner: null, leaseExpiresAt: null }, resultId: "result-1", resultStatus: "accepted" }; + function database(sourceRows = [source], contents = sourceHeader, envelope = {}) { + const material = createLocalEncryptedCipher("write").encrypt(JSON.stringify({ companyId, issueId, runId: sourceRunId, part: "header", contents, ...envelope })); + const batches: unknown[][] = [sourceRows, [{ material }]]; + const select = vi.fn(() => { + const chain = { from: () => chain, innerJoin: () => chain, where: () => chain, + limit: vi.fn(async () => batches.shift() ?? []) }; + return chain; + }); + return { db: { select } as unknown as Pick, select }; + } + return { header, sourceHeader, source, sourceRun, currentRun, database }; +} + +describe("private staged attachment restart evidence", () => { + it("proves both frozen run bindings before the target has a session snapshot", async () => { + const f = await fixture(); const { db, select } = f.database(); + const matchesExecution = vi.fn(() => true); + const result = await resolveNativeStagedAttachmentRecovery({ db, header: f.header, currentRun: f.currentRun, + expectedTarget: targetIdentity, matchesExecution }); + expect(result).toMatchObject({ sourceIdentity, targetIdentity, sourceRun: { id: sourceRunId }, hasTargetCheckpoint: false, targetCheckpoint: null, + protocolIdentity: sourceIdentity, phase: "source_bootstrap", receipt: null }); + expect(matchesExecution).toHaveBeenCalledOnce(); + expect(select).toHaveBeenCalledTimes(2); // one source point lookup and one encrypted header point lookup + expect(f.currentRun.runnerProfileJson).not.toHaveProperty("sessionCheckpoint"); + }); + + it("retains staged evidence when the target runtime has already published its snapshot", async () => { + const f = await fixture(); + const snapshot = structuredClone(f.sourceRun.runnerProfileJson.sessionCheckpoint); + snapshot.identity.runId = currentRunId; + Object.assign(snapshot, { cursor: "target-cursor", activeTurnId: "target-current-turn", pendingRuntimeRequests: [{ requestId: "target-request" }] }); + Object.assign(f.currentRun.runnerProfileJson, { sessionCheckpoint: snapshot }); + const result = await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun }); + expect(result).toMatchObject({ sourceIdentity, targetIdentity, checkpoint: { identity: { runId: currentRunId } }, + hasTargetCheckpoint: true, targetCheckpoint: snapshot }); + expect(f.currentRun.runnerProfileJson).toHaveProperty("sessionCheckpoint", snapshot); + }); + + it("returns the codec's queued phase without assuming the Runner has activated the target", async () => { + const f = await fixture(); const header = JSON.parse(f.header); + // Header inspection deliberately does not certify separate command records. + // Full restoration/authentication owns that later gate; no RPC runs here. + header.state.stagedRunAttachment.commandId = "staged-attach-command"; + const result = await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: JSON.stringify(header), currentRun: f.currentRun }); + expect(result).toMatchObject({ commandId: "staged-attach-command", protocolIdentity: sourceIdentity, + phase: "attachment_queued", receipt: null, targetIdentity }); + }); + + it.each(["runId", "companyId", "issueId", "agentId", "sessionId", "providerSessionId"])( + "rejects a present target checkpoint with changed %s instead of replaying the source", async key => { + const f = await fixture(); const snapshot = structuredClone(f.sourceRun.runnerProfileJson.sessionCheckpoint); + snapshot.identity.runId = currentRunId; + if (key === "providerSessionId") snapshot.providerSessionId = "foreign"; + else Object.assign(snapshot.identity, { [key]: "foreign" }); + Object.assign(f.currentRun.runnerProfileJson, { sessionCheckpoint: snapshot }); + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull(); + }); + + it.each(["runId", "runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "turnId", "itemId"] as const)( + "rejects a different expected target %s before querying the source", async key => { + const f = await fixture(); const { db, select } = f.database(); + expect(await resolveNativeStagedAttachmentRecovery({ db, header: f.header, currentRun: f.currentRun, + expectedTarget: { ...targetIdentity, [key]: "foreign" } })).toBeNull(); + expect(select).not.toHaveBeenCalled(); + }); + + it.each(["agentId", "companyId", "nativeIssueId", "nativeSessionId", "runnerInstanceId"] as const)( + "rejects mixed source DB %s", async key => { + const f = await fixture(); Object.assign(f.source.run, { [key]: "foreign" }); + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull(); + }); + + it.each(["binding", "schema", "runner", "session"])("rejects current frozen evidence mismatch: %s", async kind => { + const f = await fixture(); const { db, select } = f.database(); + if (kind === "binding") f.currentRun.runnerProfileJson.nativeExecutionInput.binding.agentId = sourceRunId; + if (kind === "schema") Object.assign(f.currentRun.runnerProfileJson.nativeExecutionInput, { schema: "unsupported" }); + if (kind === "runner") f.currentRun.runnerInstanceId = sourceRunId; + if (kind === "session") f.currentRun.nativeSessionId = sourceRunId; + expect(await resolveNativeStagedAttachmentRecovery({ db, header: f.header, currentRun: f.currentRun })).toBeNull(); + expect(select).not.toHaveBeenCalled(); + }); + + it.each(["phase", "leaseOwner", "resultStatus", "status", "finishedAt", "frozenRun", "workspace", "provider", "sourceTuple", "artifactVersion", "artifactDigest"])( + "rejects unsupported or stale source evidence: %s", async kind => { + const f = await fixture(); let sourceHeader = f.sourceHeader; + if (kind === "phase") f.source.coordinator.phase = "observed"; + if (kind === "leaseOwner") Object.assign(f.source.coordinator, { leaseOwner: "another-owner", leaseExpiresAt: new Date(0) }); + if (kind === "resultStatus") f.source.resultStatus = "rejected"; + if (kind === "status") f.source.run.status = "running"; + if (kind === "finishedAt") Object.assign(f.source.run, { finishedAt: null }); + if (kind === "frozenRun") f.source.run.runnerProfileJson.nativeExecutionInput.binding.runId = currentRunId; + if (kind === "workspace") f.source.run.runnerProfileJson.nativeExecutionInput.binding.executionWorkspaceId = "foreign"; + if (kind === "provider") Object.assign(f.source.run.runnerProfileJson.nativeExecutionInput.provider, { model: "another-model" }); + if (kind === "sourceTuple") { const h = JSON.parse(sourceHeader); h.state.identity.turnId = "foreign"; sourceHeader = JSON.stringify(h); } + if (kind === "artifactVersion") { const h = JSON.parse(sourceHeader); h.runnerVersion = "other-version"; sourceHeader = JSON.stringify(h); } + if (kind === "artifactDigest") { const h = JSON.parse(sourceHeader); h.runnerDigest = `sha256:${"d".repeat(64)}`; sourceHeader = JSON.stringify(h); } + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database([f.source], sourceHeader), header: f.header, currentRun: f.currentRun })).toBeNull(); + }); + + it("keeps private executor scope/config verification mandatory when supplied", async () => { + const f = await fixture(); + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun, + matchesExecution: () => false })).toBeNull(); + }); + + it.each(["companyId", "issueId", "runId", "part"])("rejects encrypted source header with another %s", async key => { + const f = await fixture(); + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database([f.source], f.sourceHeader, { [key]: "foreign" }), + header: f.header, currentRun: f.currentRun })).toBeNull(); + }); + + it("does not accept a source lacking the current tool contract or provider identity", async () => { + const f = await fixture(); + f.source.run.runnerProfileJson.nativeToolContractFingerprint = "old-contract"; + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull(); + f.source.run.runnerProfileJson.nativeToolContractFingerprint = nativeToolContractFingerprintForTarget("remote", "codex"); + f.source.run.runnerProfileJson.sessionCheckpoint.providerSessionId = ""; + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull(); + }); + + it("rejects corrupted or ordinary headers instead of promoting them to staged evidence", async () => { + const f = await fixture(); + for (const header of [f.sourceHeader, "{}", "not-json"]) { + const { db, select } = f.database(); + expect(await resolveNativeStagedAttachmentRecovery({ db, header, currentRun: f.currentRun })).toBeNull(); + expect(select).not.toHaveBeenCalled(); + } + }); +}); diff --git a/server/src/services/native-runtime/native-staged-attachment-recovery.ts b/server/src/services/native-runtime/native-staged-attachment-recovery.ts new file mode 100644 index 00000000000..e64bdf3402c --- /dev/null +++ b/server/src/services/native-runtime/native-staged-attachment-recovery.ts @@ -0,0 +1,124 @@ +import { isDeepStrictEqual } from "node:util"; +import { and, eq } from "drizzle-orm"; +import { heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { + inspectDurablePrpCheckpointHeader, + inspectStagedRunAttachmentHeader, + parseNativeExecutionInput, + type DurableRecoveryIdentity, + type NativeExecutionInput, + type PersistedNativeSession, + type StagedRunAttachment, +} from "../../vendor/paperclip-runner/index.js"; +import { loadNativeRunnerCheckpointHeader } from "./native-runner-checkpoint.js"; +import { rebindNativeSessionCheckpoint } from "./native-session-resume.js"; + +type FrozenRun = Pick; + +type StagedHeaderProof = NonNullable>; +export type NativeStagedAttachmentRecovery = Omit & + Pick & { + sourceRun: FrozenRun; + sourceExecution: NativeExecutionInput; + currentExecution: NativeExecutionInput; + /** In-memory input only. It is not a published/resumable target snapshot. */ + checkpoint: PersistedNativeSession; + /** A published target snapshot takes precedence over the source rebound. */ + hasTargetCheckpoint: boolean; + targetCheckpoint: PersistedNativeSession | null; + runnerVersion: string; + runnerDigest: string; +}; + +function record(value: unknown): Record { + return value !== null && typeof value === "object" && !Array.isArray(value) + ? value as Record : {}; +} + +function boundTargetCheckpoint(run: FrozenRun, source: PersistedNativeSession): PersistedNativeSession | null { + const snapshot = record(run.runnerProfileJson?.sessionCheckpoint), identity = record(snapshot.identity); + if (typeof snapshot.sessionId !== "string" || !snapshot.sessionId || identity.runId !== run.id || + identity.companyId !== run.companyId || identity.issueId !== run.nativeIssueId || identity.agentId !== run.agentId || + identity.sessionId !== run.nativeSessionId) return null; + // A target checkpoint may advance cursor/goal/tool progress, but cannot change + // the provider session retained by this attachment. Do not rebound it again. + if ((source.providerSessionId != null && source.providerSessionId !== snapshot.providerSessionId) || + (source.providerIdentity != null && !isDeepStrictEqual(source.providerIdentity, snapshot.providerIdentity))) return null; + return structuredClone(snapshot) as unknown as PersistedNativeSession; +} + +function frozenExecution(run: FrozenRun): NativeExecutionInput | null { + try { + const execution = parseNativeExecutionInput(run.runnerProfileJson?.nativeExecutionInput); + return run.runtimeMode === "native" && execution.binding.runId === run.id && + execution.binding.companyId === run.companyId && execution.binding.agentId === run.agentId && + execution.binding.issueId === run.nativeIssueId && execution.session.normalizedSessionId === run.nativeSessionId + ? execution : null; + } catch { return null; } +} + +function matchesRun(identity: DurableRecoveryIdentity, run: FrozenRun): boolean { + return identity.runId === run.id && identity.runnerInstanceId === run.runnerInstanceId && + identity.normalizedSessionId === run.nativeSessionId; +} + +/** Staging proves which prior attachment may be inspected, not that it is alive + * or that work may be dispatched. Classification reads only two indexed point + * lookups: the named source/result and its encrypted header. The executor must + * re-read this proof with its private scope/config matcher before any RPC, then + * independently prove the physical Runner and authenticate the target tuple. */ +export async function resolveNativeStagedAttachmentRecovery(input: { + db: Pick; + currentRun: FrozenRun; + header: string; + expectedTarget?: DurableRecoveryIdentity; + matchesExecution?: (previous: NativeExecutionInput, current: NativeExecutionInput) => boolean; +}): Promise { + const proof = inspectStagedRunAttachmentHeader(input.header); + const current = frozenExecution(input.currentRun); + if (!proof || !current || current.provider.kind !== "codex" || !["running", "failed"].includes(input.currentRun.status) || + !matchesRun(proof.attachment.targetIdentity, input.currentRun) || + (input.expectedTarget && !isDeepStrictEqual(proof.attachment.targetIdentity, input.expectedTarget))) return null; + const { attachment } = proof; + const [source] = await input.db.select({ run: heartbeatRuns, coordinator: nativeRunFinalizations, + resultId: nativeRunResults.id, resultStatus: nativeRunResults.schemaStatus }).from(heartbeatRuns) + .innerJoin(nativeRunFinalizations, and(eq(nativeRunFinalizations.runId, heartbeatRuns.id), + eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId), eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId))) + .innerJoin(nativeRunResults, and(eq(nativeRunResults.id, nativeRunFinalizations.resultId), + eq(nativeRunResults.runId, heartbeatRuns.id), eq(nativeRunResults.companyId, heartbeatRuns.companyId), + eq(nativeRunResults.issueId, heartbeatRuns.nativeIssueId))) + .where(and(eq(heartbeatRuns.id, attachment.sourceIdentity.runId), eq(heartbeatRuns.companyId, input.currentRun.companyId), + eq(heartbeatRuns.agentId, input.currentRun.agentId), eq(heartbeatRuns.nativeIssueId, current.binding.issueId))) + .limit(1); + if (!source || source.run.companyId !== current.binding.companyId || source.run.agentId !== current.binding.agentId || + source.run.nativeIssueId !== current.binding.issueId || !matchesRun(attachment.sourceIdentity, source.run) || + source.run.status !== "succeeded" || source.run.finishedAt === null || source.coordinator.phase !== "committed" || + source.coordinator.companyId !== current.binding.companyId || source.coordinator.issueId !== current.binding.issueId || + source.coordinator.runId !== source.run.id || !source.coordinator.resultId || + source.coordinator.resultId !== source.resultId || source.resultStatus !== "accepted" || + source.coordinator.leaseOwner !== null || source.coordinator.leaseExpiresAt !== null) return null; + const previous = frozenExecution(source.run); + if (!previous || (input.matchesExecution && !input.matchesExecution(previous, current))) return null; + // Reuse the existing provider, tool-contract, workspace, runtime-context and + // normalized-session compatibility owner; no second set of digest formulas. + const checkpoint = rebindNativeSessionCheckpoint({ previousRun: source.run, currentExecution: current, executionTargetKind: "remote" }); + if (!checkpoint || !(typeof checkpoint.providerSessionId === "string" && checkpoint.providerSessionId.length > 0 || + checkpoint.providerIdentity && Object.keys(checkpoint.providerIdentity).length > 0)) return null; + const hasTargetCheckpoint = input.currentRun.runnerProfileJson?.sessionCheckpoint != null; + const targetCheckpoint = hasTargetCheckpoint ? boundTargetCheckpoint(input.currentRun, checkpoint) : null; + if (hasTargetCheckpoint && !targetCheckpoint) return null; + const sourceHeader = await loadNativeRunnerCheckpointHeader(input.db, { + companyId: source.run.companyId, issueId: current.binding.issueId, runId: source.run.id, + }); + if (!sourceHeader || !isDeepStrictEqual(inspectDurablePrpCheckpointHeader(sourceHeader), attachment.sourceIdentity)) return null; + // The codec above has validated this bounded header's artifact fields. A + // staged target cannot silently select different bytes from its source. + const sourceArtifact = JSON.parse(sourceHeader) as { runnerVersion: string; runnerDigest: string }; + if (sourceArtifact.runnerVersion !== proof.runnerVersion || sourceArtifact.runnerDigest !== proof.runnerDigest) return null; + return { ...attachment, sourceRun: source.run, sourceExecution: previous, currentExecution: current, + checkpoint, hasTargetCheckpoint, targetCheckpoint, + protocolIdentity: proof.protocolIdentity, phase: proof.phase, receipt: proof.receipt, + runnerVersion: proof.runnerVersion, runnerDigest: proof.runnerDigest }; +} diff --git a/server/src/services/native-runtime/native-task-session-publication.test.ts b/server/src/services/native-runtime/native-task-session-publication.test.ts new file mode 100644 index 00000000000..4ab4c73c059 --- /dev/null +++ b/server/src/services/native-runtime/native-task-session-publication.test.ts @@ -0,0 +1,319 @@ +import { createRequire } from "node:module"; +import { randomUUID } from "node:crypto"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { eq, sql } from "drizzle-orm"; +import { drizzle } from "drizzle-orm/postgres-js"; +import { agents, agentTaskSessions, companies, completionContracts, createDb, heartbeatRuns, issues, + nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { CONTROL_PLANE_CONFORMANCE_OPEN, CONTROL_PLANE_CONFORMANCE_RESULT, + CONTROL_PLANE_CONFORMANCE_TERMINAL } from "../../vendor/paperclip-runner/testing.js"; +import type { PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; +import { startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js"; +import { redactRegisteredSecretValues } from "../run-secret-redaction.js"; +import { nativeSha256 } from "./canonical.js"; +import { PaperclipControlPlanePort, type PaperclipControlPlaneBinding } from "./paperclip-control-plane-port.js"; +import { nativeTaskSessionPublicationHandled, publishNativeTaskSessionInTransaction, NATIVE_TASK_SESSION_PUBLICATION_KEY, + type NativeTaskSessionPublicationContext } from "./native-task-session-publication.js"; + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_PUBLICATION_TEST_DATABASE_URL"); +describe("accepted native task session publication", () => { + let temporary: Awaited> | null = null; + let db: Db; + let measured: Db; + let client: Db["$client"]; + const statements: string[] = []; + beforeAll(async () => { + if (!external) temporary = await startRecoveryPostgres("native-publication-"); + const url = external ?? temporary!.connectionString; + db = createDb(url, { maxConnections: 2 }); + // Test-only wire observer. Production still uses the existing Db/pools. + const postgres = createRequire(import.meta.resolve("@paperclipai/db"))("postgres") as + (url: string, options: object) => Db["$client"]; + client = postgres(url, { max: 1, prepare: false, connection: { application_name: "native-publication-observer" }, + debug: (_id: number, query: string) => { statements.push(query); } }); + measured = drizzle(client) as unknown as Db; + }); + afterAll(async () => { + await client?.end({ timeout: 1 }); + await db?.$client.end({ timeout: 1 }); + await temporary?.cleanup(); + }); + + async function fixture(source: "runner" | "control" = "runner", accepted = true) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + const sessionId = randomUUID(), contractId = randomUUID(), runner = randomUUID(), control = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Native pointer", issuePrefix: `P${companyId.slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Native pointer", adapterType: "codex_local", status: "running" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Native pointer", status: "in_progress", assigneeAgentId: agentId }); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "standalone-v1", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: "standalone-v1", objective: "Conformance", criteria: [{ id: "objective", requirement: "Complete" }] }, + canonicalSha256: "publication-contract", createdByActorType: "system", createdByActorId: "test" }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", + nativeIssueId: issueId, nativeSessionId: sessionId, runnerInstanceId: runner, completionContractId: contractId, + completionContractSha256: "publication-contract", startedAt: new Date(Date.now() - 2_000), runnerProfileJson: {} }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const owner = { companyId, issueId, runId, authorityRunId: runId, bootId: "publication-boot", generation: 1, leaseOwner: "publication-owner" }; + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", + controllerBootId: owner.bootId, controllerGeneration: owner.generation, leaseOwner: owner.leaseOwner, + leaseExpiresAt: new Date(Date.now() + 60_000) }); + const binding: PaperclipControlPlaneBinding = { companyId, agentId, issueId, runId, sessionId, + completionContractId: contractId, completionContractSha256: "publication-contract", sourceInstanceId: runner, + controlPlaneSourceInstanceId: control }; + const context: NativeTaskSessionPublicationContext = { taskKey: issueId, adapterType: "codex_local", + sessionParamsJson: { cwd: "/working", paperclipSessionConfigFingerprint: "prepared-once", opaque: "private-prepared-param" } }; + const snapshot: PersistedNativeSession = { backendKind: "mock", sessionId, identity: { companyId, agentId, issueId, runId, sessionId }, + providerSessionId: "provider-thread", activeTurnId: null, pendingRuntimeRequests: [], goal: null, + semanticResult: CONTROL_PLANE_CONFORMANCE_RESULT, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL }; + const port = new PaperclipControlPlanePort(measured, binding, { taskSessionPublication: context, taskSessionPublicationOwner: owner }); + await port.openRun({ ...CONTROL_PLANE_CONFORMANCE_OPEN, identity: snapshot.identity, sourceInstanceId: runner }); + if (accepted) await new PaperclipControlPlanePort(db, binding).completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, + terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId: "publication-turn" }); + await db.insert(agentTaskSessions).values({ companyId, agentId, adapterType: context.adapterType, taskKey: issueId, + goalSourceId: `${source === "runner" ? runner : control}:${runId}`, goalSourceCursor: 4 }); + const read = async () => { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const [session] = await db.select().from(agentTaskSessions).where(eq(agentTaskSessions.taskKey, issueId)); + return { run, session }; + }; + return { binding, owner, context, snapshot, port, read }; + } + + it.each(["runner", "control"] as const)("publishes from the exact accepted %s epoch in one checkpoint transaction", async source => { + const f = await fixture(source); statements.length = 0; + await f.port.checkpointSession(f.snapshot); + const queries = [...statements], { run, session } = await f.read(); + expect(session).toMatchObject({ sessionParamsJson: { ...f.context.sessionParamsJson, sessionId: f.binding.sessionId }, + sessionDisplayId: "provider-thread", lastRunId: f.binding.runId }); + expect(nativeTaskSessionPublicationHandled(run.runnerProfileJson, { ...f.binding, ...f.context })).toBe(true); + expect(queries.filter(query => /^begin\b/i.test(query.trim()))).toHaveLength(1); + expect(queries.filter(query => /^commit\b/i.test(query.trim()))).toHaveLength(1); + expect(queries.filter(query => /native_run_results/i.test(query))).toHaveLength(1); + expect(queries.filter(query => /^select .*from "agent_task_sessions"/i.test(query.trim()))).toHaveLength(1); + expect(queries.filter(query => /^update "agent_task_sessions"/i.test(query.trim()))).toHaveLength(1); + expect(queries).toHaveLength(7); + expect(JSON.stringify(run.runnerProfileJson)).not.toContain("private-prepared-param"); + expect(JSON.stringify({ [NATIVE_TASK_SESSION_PUBLICATION_KEY]: run.runnerProfileJson![NATIVE_TASK_SESSION_PUBLICATION_KEY] }).length).toBeLessThan(4_096); + expect(redactRegisteredSecretValues(run.runnerProfileJson, [])).not.toHaveProperty(NATIVE_TASK_SESSION_PUBLICATION_KEY); + expect(nativeTaskSessionPublicationHandled(run.runnerProfileJson, { ...f.binding, ...f.context, companyId: randomUUID() })).toBe(false); + expect(nativeTaskSessionPublicationHandled(run.runnerProfileJson, { ...f.binding, ...f.context, agentId: randomUUID() })).toBe(false); + const malformed = structuredClone(run.runnerProfileJson!); + (malformed[NATIVE_TASK_SESSION_PUBLICATION_KEY] as Record).resultId = "not-an-id"; + expect(nativeTaskSessionPublicationHandled(malformed, { ...f.binding, ...f.context })).toBe(false); + }); + + it("does not recreate a reset pointer or query task state on repeated terminal maintenance", async () => { + const f = await fixture(); await f.port.checkpointSession(f.snapshot); + await db.delete(agentTaskSessions).where(eq(agentTaskSessions.taskKey, f.binding.issueId)); statements.length = 0; + await f.port.checkpointSession(f.snapshot); + const repeated = [...statements]; + expect((await f.read()).session).toBeUndefined(); + expect(repeated.filter(query => /agent_task_sessions|native_run_results/i.test(query))).toHaveLength(0); + expect(repeated.filter(query => /^begin\b/i.test(query.trim()))).toHaveLength(1); + expect(repeated.filter(query => /^update /i.test(query.trim()))).toHaveLength(1); + }); + + it("persists a pre-acceptance terminal observation and publishes only at the existing accepted checkpoint", async () => { + const f = await fixture("runner", false); + // Driver event/startup checkpoints can expose a semantic terminal observation + // before the runtime calls completeRun. That observation is not acceptance. + await f.port.checkpointSession(f.snapshot); + const observed = await f.read(); + expect(observed.run.runnerProfileJson?.sessionCheckpoint).toMatchObject({ semanticResult: CONTROL_PLANE_CONFORMANCE_RESULT }); + expect(observed.run.runnerProfileJson).not.toHaveProperty(NATIVE_TASK_SESSION_PUBLICATION_KEY); + expect(observed.session?.sessionParamsJson).toBeNull(); + expect(nativeTaskSessionPublicationHandled(observed.run.runnerProfileJson, { ...f.binding, ...f.context })).toBe(false); + await f.port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, + terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId: "publication-turn" }); + expect((await f.read()).session?.sessionParamsJson).toBeNull(); + await f.port.checkpointSession(f.snapshot); + const completed = await f.read(); + expect(completed.session?.lastRunId).toBe(f.binding.runId); + expect(nativeTaskSessionPublicationHandled(completed.run.runnerProfileJson, { ...f.binding, ...f.context })).toBe(true); + }); + + it("does not hold the issue parent lock while a result replay waits for the status coordinator", async () => { + const f = await fixture(); + let committed = 0; + const port = new PaperclipControlPlanePort(db, f.binding, { nativeRecovery: { owner: f.owner, + compatibility: { configDigest: nativeSha256("config"), processConfigDigest: null } }, + onResultCommitted: async () => { committed++; } }); + await port.openRun({ ...CONTROL_PLANE_CONFORMANCE_OPEN, identity: f.snapshot.identity, sourceInstanceId: f.binding.sourceInstanceId }); + let replay: Promise | undefined; + let statusLockError: unknown = null; + try { + await db.transaction(async tx => { + // This is commitNativeStatusDecision's actual lock prefix. Hold its + // coordinator first, then overlap an identical remote result replay. + await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.binding.runId)).for("update"); + const [backend] = await tx.execute(sql`select pg_backend_pid() as pid`); + replay = port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, + terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId: "publication-turn" }).then(() => null, error => error); + let waiting = false; + for (let attempt = 0; attempt < 100; attempt++) { + const [state] = await measured.execute(sql`select exists ( + select 1 from pg_stat_activity where ${backend.pid} = any(pg_blocking_pids(pid))) as waiting`); + if (state.waiting) { waiting = true; break; } + await new Promise(resolve => setTimeout(resolve, 10)); + } + expect(waiting).toBe(true); + // A bounded test-local timeout reports the inverse-order cycle without + // relying on which transaction PostgreSQL chooses as deadlock victim. + await tx.execute(sql`set local lock_timeout = '500ms'`); + await tx.select().from(issues).where(eq(issues.id, f.binding.issueId)).for("update"); + }); + } catch (error) { statusLockError = error; } + const replayError = await replay; + expect(statusLockError).toBeNull(); + expect(replayError).toBeNull(); + expect(committed).toBe(0); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.binding.runId))).toHaveLength(1); + await db.update(nativeRunFinalizations).set({ controllerGeneration: 2 }).where(eq(nativeRunFinalizations.runId, f.binding.runId)); + await expect(port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, + terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId: "publication-turn" })).rejects.toThrow("checkpoint_owner_lost"); + }); + + it.each(["missing", "foreign_epoch", "newer_pointer", "newer_execution", "generation_reset", "stop"])("fences %s and records a handled suppression", async kind => { + const f = await fixture(); + if (kind === "missing") await db.delete(agentTaskSessions).where(eq(agentTaskSessions.taskKey, f.binding.issueId)); + if (kind === "foreign_epoch") await db.update(agentTaskSessions).set({ goalSourceId: `${randomUUID()}:${f.binding.runId}` }).where(eq(agentTaskSessions.taskKey, f.binding.issueId)); + if (kind === "newer_pointer" || kind === "newer_execution") { + const newer = randomUUID(); + await db.insert(heartbeatRuns).values({ id: newer, companyId: f.binding.companyId, agentId: f.binding.agentId, + status: "running", issueId: f.binding.issueId, startedAt: new Date() }); + if (kind === "newer_pointer") await db.update(agentTaskSessions).set({ lastRunId: newer, sessionParamsJson: { sessionId: "newer" } }).where(eq(agentTaskSessions.taskKey, f.binding.issueId)); + else await db.update(issues).set({ executionRunId: newer }).where(eq(issues.id, f.binding.issueId)); + } + if (kind === "generation_reset") await db.update(issues).set({ conversationAgentId: f.binding.agentId, + conversationUserId: "publication-test-user", conversationState: "active", conversationSessionGeneration: 2 }).where(eq(issues.id, f.binding.issueId)); + if (kind === "stop") await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.binding.runId)); + await f.port.checkpointSession(f.snapshot); + const { run, session } = await f.read(); + expect(nativeTaskSessionPublicationHandled(run.runnerProfileJson, { ...f.binding, ...f.context })).toBe(true); + expect(session?.sessionParamsJson?.sessionId).not.toBe(f.binding.sessionId); + }); + + it("keeps unsupported task keys and missing remote recovery headers outside handled publication", async () => { + const f = await fixture(); + const context = { ...f.context, taskKey: "custom-task" }; + const port = new PaperclipControlPlanePort(measured, f.binding, { taskSessionPublication: context, taskSessionPublicationOwner: f.owner }); + await port.openRun({ ...CONTROL_PLANE_CONFORMANCE_OPEN, identity: f.snapshot.identity, sourceInstanceId: f.binding.sourceInstanceId }); + await port.checkpointSession(f.snapshot); + const { run, session } = await f.read(); + expect(session?.sessionParamsJson).toBeNull(); + expect(nativeTaskSessionPublicationHandled(run.runnerProfileJson, { ...f.binding, ...context })).toBe(false); + const remote = await fixture(); + const remotePort = new PaperclipControlPlanePort(measured, remote.binding, { taskSessionPublication: remote.context, + taskSessionPublicationOwner: remote.owner, nativeRecovery: { owner: remote.owner, + compatibility: { configDigest: nativeSha256("config"), processConfigDigest: null } } }); + await remotePort.openRun({ ...CONTROL_PLANE_CONFORMANCE_OPEN, identity: remote.snapshot.identity, sourceInstanceId: remote.binding.sourceInstanceId }); + await remotePort.checkpointSession(remote.snapshot); + expect((await remote.read()).session?.sessionParamsJson).toBeNull(); + expect(nativeTaskSessionPublicationHandled((await remote.read()).run.runnerProfileJson, { ...remote.binding, ...remote.context })).toBe(false); + }); + + it("rejects stale owners and a different accepted result without partially publishing", async () => { + const stale = await fixture(); + await db.update(nativeRunFinalizations).set({ controllerGeneration: 2 }).where(eq(nativeRunFinalizations.runId, stale.binding.runId)); + await expect(stale.port.checkpointSession(stale.snapshot)).rejects.toThrow("checkpoint_owner_lost"); + expect((await stale.read()).session?.sessionParamsJson).toBeNull(); + const wrong = await fixture(); + await expect(wrong.port.checkpointSession({ ...wrong.snapshot, semanticResult: { ...CONTROL_PLANE_CONFORMANCE_RESULT, summary: "different" } })).rejects.toThrow("publication_result_mismatch"); + expect((await wrong.read()).session?.sessionParamsJson).toBeNull(); + const foreign = await fixture(); + const foreignResult = await fixture(); + const [otherCoordinator] = await db.select({ resultId: nativeRunFinalizations.resultId }).from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, foreignResult.binding.runId)); + await expect(db.update(nativeRunFinalizations).set({ resultId: otherCoordinator.resultId }) + .where(eq(nativeRunFinalizations.runId, foreign.binding.runId))) + .rejects.toMatchObject({ cause: { constraint_name: "native_run_finalizations_result_owner_fk" } }); + expect((await foreign.read()).session?.sessionParamsJson).toBeNull(); + await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, foreign.binding.runId)); + // The CP defers preacceptance snapshots, while the publication helper itself + // still requires accepted-result authority if called directly. + await expect(publishNativeTaskSessionInTransaction({} as never, { binding: foreign.binding, + context: foreign.context, snapshot: foreign.snapshot, run: (await foreign.read()).run, + issue: null, resultId: null })).rejects.toThrow("publication_result_missing"); + }); + + it("adds no publication SQL for nonterminal or goal snapshots", async () => { + const f = await fixture(); statements.length = 0; + await f.port.checkpointSession({ ...f.snapshot, semanticResult: null, terminal: null }); + expect(statements.some(query => /agent_task_sessions|native_run_results|publication_issue/i.test(query))).toBe(false); + statements.length = 0; + await f.port.checkpointSession({ ...f.snapshot, goal: {} as NonNullable }); + expect(statements.some(query => /agent_task_sessions|native_run_results|publication_issue/i.test(query))).toBe(false); + expect((await f.read()).session?.sessionParamsJson).toBeNull(); + }); + + it.each(["completed_predecessor", "expired_accepted_lease"])("preserves legitimate %s maintenance", async kind => { + const f = await fixture(); + if (kind === "completed_predecessor") { + const prior = randomUUID(); + await db.insert(heartbeatRuns).values({ id: prior, companyId: f.binding.companyId, agentId: f.binding.agentId, + issueId: f.binding.issueId, status: "succeeded", finishedAt: new Date(Date.now() - 5_000) }); + await db.update(agentTaskSessions).set({ lastRunId: prior, sessionParamsJson: { sessionId: "prior" } }) + .where(eq(agentTaskSessions.taskKey, f.binding.issueId)); + } else { + await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) }) + .where(eq(nativeRunFinalizations.runId, f.binding.runId)); + } + await f.port.checkpointSession(f.snapshot); + expect((await f.read()).session?.lastRunId).toBe(f.binding.runId); + }); + + it("does not block completeRun's issue KEY SHARE while publishing", async () => { + const f = await fixture(); let release!: () => void; let locked!: () => void; + const ready = new Promise(resolve => { locked = resolve; }); + const hold = new Promise(resolve => { release = resolve; }); + const blocker = db.transaction(async tx => { + await tx.select().from(issues).where(eq(issues.id, f.binding.issueId)).for("key share"); locked(); await hold; + }); + await ready; + try { await f.port.checkpointSession(f.snapshot); } + finally { release(); await blocker; } + expect((await f.read()).session?.lastRunId).toBe(f.binding.runId); + }); + + it("rechecks the owner clock and issue authority after an issue lock wait", async () => { + const f = await fixture(); let release!: () => void; let locked!: () => void; + await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() + 500) }).where(eq(nativeRunFinalizations.runId, f.binding.runId)); + const ready = new Promise(resolve => { locked = resolve; }); + const hold = new Promise(resolve => { release = resolve; }); + const newer = randomUUID(); + await db.insert(heartbeatRuns).values({ id: newer, companyId: f.binding.companyId, agentId: f.binding.agentId, + issueId: f.binding.issueId, status: "running", startedAt: new Date() }); + const blocker = db.transaction(async tx => { + await tx.select().from(issues).where(eq(issues.id, f.binding.issueId)).for("no key update"); + await tx.update(issues).set({ executionRunId: newer }).where(eq(issues.id, f.binding.issueId)); + locked(); await hold; + }); + await ready; + const publication = f.port.checkpointSession(f.snapshot); + const rejected = expect(publication).rejects.toThrow("checkpoint_owner_lost"); + try { + let waiting = false; + for (let attempt = 0; attempt < 100; attempt++) { + const rows = await db.execute(sql`select 1 from pg_stat_activity where application_name = 'native-publication-observer' + and wait_event_type = 'Lock' and query like '%publication_issue%'`); + if (rows.length) { waiting = true; break; } + await new Promise(resolve => setTimeout(resolve, 10)); + } + expect(waiting).toBe(true); + await new Promise(resolve => setTimeout(resolve, 600)); + } finally { release(); await blocker; } + await rejected; + expect((await f.read()).session?.sessionParamsJson).toBeNull(); + }); + + it("rolls back the pointer if combined private metadata exceeds its existing bound", async () => { + const f = await fixture(); + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeControllerRecovery: { padding: "x".repeat(4_096) } } }).where(eq(heartbeatRuns.id, f.binding.runId)); + await expect(f.port.checkpointSession(f.snapshot)).rejects.toThrow("publication_metadata_too_large"); + const { run, session } = await f.read(); + expect(session?.sessionParamsJson).toBeNull(); + expect(run.runnerProfileJson).not.toHaveProperty(NATIVE_TASK_SESSION_PUBLICATION_KEY); + }); +}); diff --git a/server/src/services/native-runtime/native-task-session-publication.ts b/server/src/services/native-runtime/native-task-session-publication.ts new file mode 100644 index 00000000000..ee61fc3e74e --- /dev/null +++ b/server/src/services/native-runtime/native-task-session-publication.ts @@ -0,0 +1,145 @@ +import { and, eq } from "drizzle-orm"; +import { agentTaskSessions, heartbeatRuns, nativeRunResults, type Db, type issues } from "@paperclipai/db"; +import { validatePrpStructuredRunResult, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; +import { nativeSha256 } from "./canonical.js"; +import type { PaperclipControlPlaneBinding } from "./paperclip-control-plane-port.js"; +import { readRunCancellation } from "../run-cancellation.js"; + +/** Prepared once by heartbeat; never copied into recovery metadata or receipts. */ +export interface NativeTaskSessionPublicationContext { + taskKey: string; + adapterType: string; + sessionParamsJson: Record | null; +} + +export const NATIVE_TASK_SESSION_PUBLICATION_KEY = "nativeTaskSessionPublication"; +type Transaction = Parameters[0]>[0]; +type Outcome = "published" | "suppressed_reset" | "suppressed_newer" | "suppressed_authority" | "missing_stub" | "unsupported"; +interface Receipt { + schema: "paperclip.native-task-session-publication.v1"; + companyId: string; + issueId: string; + agentId: string; + runId: string; + taskKey: string; + adapterType: string; + sessionId: string; + resultId: string; + outcome: Outcome; +} + +function record(value: unknown): Record { + return value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} +const uuid = (value: unknown): value is string => typeof value === "string" && /^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(value); +const outcomes: readonly string[] = ["published", "suppressed_reset", "suppressed_newer", "suppressed_authority", "missing_stub", "unsupported"]; + +function receiptFromProfile(profile: unknown): Receipt | null { + const root = record(profile), value = record(root[NATIVE_TASK_SESSION_PUBLICATION_KEY]); + if (value.schema !== "paperclip.native-task-session-publication.v1" + || ![value.companyId, value.issueId, value.agentId, value.runId, value.sessionId, value.resultId].every(uuid) + || typeof value.taskKey !== "string" || value.taskKey.length > 512 + || typeof value.adapterType !== "string" || !value.adapterType.length || value.adapterType.length > 128 + || !outcomes.includes(String(value.outcome)) || Object.keys(value).length !== 10) return null; + const snapshot = record(root.sessionCheckpoint), identity = record(snapshot.identity); + if (identity.runId !== value.runId || identity.companyId !== value.companyId || identity.issueId !== value.issueId + || identity.agentId !== value.agentId || identity.sessionId !== value.sessionId) return null; + return value as unknown as Receipt; +} + +/** Only the exact server-written outcome may suppress heartbeat's late fallback. */ +export function nativeTaskSessionPublicationHandled(profile: unknown, binding: { + companyId: string; agentId: string; runId: string; taskKey: string; adapterType: string; +}): boolean { + const receipt = receiptFromProfile(profile); + return !!receipt && receipt.companyId === binding.companyId && receipt.agentId === binding.agentId + && receipt.runId === binding.runId && receipt.taskKey === binding.taskKey + && receipt.adapterType === binding.adapterType && ["published", "suppressed_reset", "suppressed_newer", "suppressed_authority", "missing_stub"].includes(receipt.outcome); +} + +export function eligibleNativeTaskSessionPublication(snapshot: PersistedNativeSession): boolean { + return snapshot.goal == null && snapshot.semanticResult != null && snapshot.terminal?.runTerminalState === "succeeded" + && snapshot.activeTurnId == null && !(snapshot.pendingRuntimeRequests?.length); +} + +export type NativeTaskSessionPublicationRun = Pick; +export type NativeTaskSessionPublicationIssue = Pick; + +/** Caller holds coordinator → issue NO KEY UPDATE → run. This helper owns only + * the task-row lock and UPDATE; never starts a transaction or creates a stub. */ +export async function publishNativeTaskSessionInTransaction(tx: Transaction, input: { + binding: PaperclipControlPlaneBinding; + context: NativeTaskSessionPublicationContext; + snapshot: PersistedNativeSession; + run: NativeTaskSessionPublicationRun; + issue: NativeTaskSessionPublicationIssue | null; + resultId: string | null; + recoveryReady?: boolean; +}): Promise { + const { binding, context, snapshot, run, issue } = input; + const existingReceipt = receiptFromProfile(run.runnerProfileJson); + if (existingReceipt) { + if (existingReceipt.companyId !== binding.companyId || existingReceipt.issueId !== binding.issueId + || existingReceipt.agentId !== binding.agentId || existingReceipt.runId !== binding.runId + || existingReceipt.sessionId !== snapshot.identity.sessionId || existingReceipt.taskKey !== context.taskKey + || existingReceipt.adapterType !== context.adapterType || existingReceipt.resultId !== input.resultId) { + throw new Error("native_task_session_publication_receipt_mismatch"); + } + return existingReceipt; + } + if (record(run.runnerProfileJson)[NATIVE_TASK_SESSION_PUBLICATION_KEY] !== undefined) { + throw new Error("native_task_session_publication_receipt_invalid"); + } + if (!input.resultId) throw new Error("native_task_session_publication_result_missing"); + const [accepted] = await tx.select({ id: nativeRunResults.id, canonicalSha256: nativeRunResults.canonicalSha256, + turnId: nativeRunResults.turnId }).from(nativeRunResults).where(and( + eq(nativeRunResults.id, input.resultId), eq(nativeRunResults.runId, binding.runId), + eq(nativeRunResults.companyId, binding.companyId), eq(nativeRunResults.issueId, binding.issueId), + eq(nativeRunResults.completionContractId, binding.completionContractId), eq(nativeRunResults.schemaStatus, "accepted"), + )).limit(1); + const validated = validatePrpStructuredRunResult(snapshot.semanticResult); + if (!accepted || !validated.ok || accepted.canonicalSha256 !== nativeSha256({ + binding, result: validated.result, terminal: snapshot.terminal, turnId: accepted.turnId ?? null, + })) throw new Error("native_task_session_publication_result_mismatch"); + const receipt = (outcome: Outcome): Receipt => ({ schema: "paperclip.native-task-session-publication.v1", + companyId: binding.companyId, issueId: binding.issueId, agentId: binding.agentId, runId: binding.runId, + taskKey: context.taskKey, adapterType: context.adapterType, sessionId: snapshot.identity.sessionId, + resultId: accepted.id, outcome }); + if (context.taskKey !== binding.issueId || input.recoveryReady === false) return receipt("unsupported"); + const cancellation = record(run.resultJson?.nativeCancellation), stop = readRunCancellation(run.resultJson); + if (!issue || ["cancelled", "interrupted", "failed", "timed_out"].includes(run.status) || issue.companyId !== binding.companyId + || run.resultJson?.startupCancellation || (stop?.source === "operator" && stop.expected) + || (["run", "issue"].includes(String(cancellation.scope)) && ["pending", "acknowledged"].includes(String(cancellation.dispatchState))) + || (issue.conversationAgentId && issue.conversationAgentId !== binding.agentId)) return receipt("suppressed_authority"); + if (issue.conversationAgentId + && run.contextSnapshot?.conversationSessionGeneration !== issue.conversationSessionGeneration) return receipt("suppressed_reset"); + if (issue.executionRunId && issue.executionRunId !== binding.runId) return receipt("suppressed_newer"); + const [session] = await tx.select({ id: agentTaskSessions.id, goalSourceId: agentTaskSessions.goalSourceId, + lastRunId: agentTaskSessions.lastRunId }).from(agentTaskSessions).where(and( + eq(agentTaskSessions.companyId, binding.companyId), eq(agentTaskSessions.agentId, binding.agentId), + eq(agentTaskSessions.adapterType, context.adapterType), eq(agentTaskSessions.taskKey, context.taskKey), + )).for("update").limit(1); + // Supported native turns synchronously project a stub before this terminal + // snapshot. Absence can be an operator deletion; never recreate it later. + if (!session) return receipt("missing_stub"); + const boundSources = [binding.sourceInstanceId, binding.controlPlaneSourceInstanceId] + .filter((value): value is string => typeof value === "string" && value.length > 0) + .map(source => `${source}:${binding.runId}`); + if (!session.goalSourceId || !boundSources.includes(session.goalSourceId)) return receipt("suppressed_newer"); + if (session.lastRunId && session.lastRunId !== binding.runId) { + const [prior] = await tx.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: heartbeatRuns.issueId, nativeIssueId: heartbeatRuns.nativeIssueId, finishedAt: heartbeatRuns.finishedAt, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, session.lastRunId)).limit(1); + if (!prior || prior.companyId !== binding.companyId || prior.agentId !== binding.agentId + || (prior.nativeIssueId ?? prior.issueId) !== binding.issueId || !prior.finishedAt || !run.startedAt + || prior.finishedAt > run.startedAt) return receipt("suppressed_newer"); + } + await tx.update(agentTaskSessions).set({ + sessionParamsJson: { ...context.sessionParamsJson, sessionId: snapshot.identity.sessionId }, + sessionDisplayId: snapshot.providerSessionId ?? snapshot.sessionId, lastRunId: binding.runId, + lastError: null, updatedAt: new Date(), + }).where(eq(agentTaskSessions.id, session.id)); + return receipt("published"); +} diff --git a/server/src/services/native-runtime/native-usage-delivery.integration.test.ts b/server/src/services/native-runtime/native-usage-delivery.integration.test.ts index 21eab93cb2c..e59ccb2a965 100644 --- a/server/src/services/native-runtime/native-usage-delivery.integration.test.ts +++ b/server/src/services/native-runtime/native-usage-delivery.integration.test.ts @@ -12,8 +12,8 @@ import { agents, budgetPolicies, companies, costEvents, createDb, heartbeatRuns, import { type PrpEvent } from "../../vendor/paperclip-runner/index.js"; import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { accountRunCost } from "../run-cost-accounting.js"; -import { usageReceiptSpoolPath } from "../usage-receipts.js"; -import { createNativeUsageDelivery } from "./native-usage-delivery.js"; +import { usageReceiptSpoolPath, type UsageReceiptEnvelope } from "../usage-receipts.js"; +import { createNativeUsageContinuationDelivery, createNativeUsageDelivery } from "./native-usage-delivery.js"; const runnerRoot = fileURLToPath(new URL("../../../../packages/paperclip-runner/runner/", import.meta.url)); const fixtureProfile = process.env.CI ? "release" : "debug"; @@ -44,6 +44,89 @@ describe("producer-owned native usage", () => { payload: type === "usage.reported" ? { runDeltaAvailable: true, runDeltaBasis: "run", runDelta: { inputTokens: 120, outputTokens: 10, providerCostUsd: amount } } : {}, } as PrpEvent); + const finalizedSource = async () => { + const input = await seed(); + await db.update(heartbeatRuns).set({ runnerInstanceId: input.runnerInstanceId }).where(eq(heartbeatRuns.id, input.runId)); + const delivery = await createNativeUsageDelivery(db, input); + const report = event(input, 1, "usage.reported"), done = event(input, 2, "turn.completed"); + await delivery.capture(report); + await delivery.capture(done); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, input.runId)); + expect(await accountRunCost(db, input.runId)).toBe(true); + const { billing: _billing, ...source } = input; + return { input, source, report, done }; + }; + + it("continues exact finalized source receipts without acquiring ownership or resetting accounting", async () => { + const { input, source, report, done } = await finalizedSource(); + const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)); + await expect(createNativeUsageDelivery(db, input)).rejects.toThrow("native_usage_run_not_available"); + const continuation = await createNativeUsageContinuationDelivery(db, source); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before); + await continuation.capture(report); + await continuation.capture(done); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before); + expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId))).toHaveLength(2); + expect(await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, input.runId))).toHaveLength(1); + await expect(continuation.capture({ ...report, sourceInstanceId: randomUUID() })).rejects.toThrow("native_usage_source_mismatch"); + await expect(continuation.capture({ ...report, payload: { ...report.payload as object, runDeltaAvailable: false } })) + .rejects.toThrow("native_usage_source_conflict"); + }); + + it("journals an equal finalized source tail using the source billing without repricing", async () => { + const { input, source } = await finalizedSource(); + const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)); + // Even a structurally supplied extra current-run classification cannot own + // the old receipt: the continuation derives billing from saved source data. + const withCurrentBilling = { ...source, billing: { provider: "anthropic", biller: "current-grant", billingType: "subscription_included" } }; + const continuation = await createNativeUsageContinuationDelivery(db, withCurrentBilling); + await continuation.capture(event(input, 3, "usage.reported")); + const receipts = await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId)); + expect(receipts).toHaveLength(3); + expect(receipts.find(row => row.sequence === 3)?.receiptJson.receipt).toMatchObject(billing); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before); + expect((await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, input.runId)))[0].costCents).toBe(25); + }); + + it.each(["tokens", "cost", "incomplete", "unpriceable", "basis"])( + "holds a differing finalized source tail before acknowledgement: %s", async change => { + const { input, source } = await finalizedSource(); + const continuation = await createNativeUsageContinuationDelivery(db, source); + const late = event(input, 3, "usage.reported"); + if (change === "incomplete") (late.payload as Record).runDeltaAvailable = false; + if (change === "basis") delete (late.payload as Record).runDeltaBasis; + if (["tokens", "cost", "unpriceable"].includes(change)) (late.payload as Record).runDelta = { + inputTokens: change === "tokens" ? 150 : change === "unpriceable" ? "unknown" : 120, + outputTokens: 10, providerCostUsd: change === "cost" ? 0.5 : 0.25, + }; + await expect(continuation.capture(late)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation"); + await expect(continuation.capture(late)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation"); + expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId))).toHaveLength(2); + expect((await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, input.runId)))[0].costCents).toBe(25); + }); + + it.each(["company", "run", "runner", "source"])("rejects a changed continuation binding without mutation: %s", async change => { + const { input, source } = await finalizedSource(); + if (change === "source") await db.update(heartbeatRuns).set({ usageJson: sql`${heartbeatRuns.usageJson} || ${JSON.stringify({ accountingReceiptSourceId: randomUUID() })}::jsonb` }) + .where(eq(heartbeatRuns.id, input.runId)); + const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)); + await expect(createNativeUsageContinuationDelivery(db, { ...source, + ...(change === "company" ? { companyId: randomUUID() } : {}), + ...(change === "run" ? { runId: randomUUID() } : {}), + ...(change === "runner" ? { runnerInstanceId: randomUUID() } : {}), + })).rejects.toThrow("native_usage_continuation_source_mismatch"); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before); + }); + + it("rejects changed immutable source receipt facts during hydration", async () => { + const { input, source } = await finalizedSource(); + const [receipt] = await db.select().from(runUsageReceipts).where(and(eq(runUsageReceipts.runId, input.runId), eq(runUsageReceipts.sequence, 1))); + const changed = structuredClone(receipt.receiptJson) as UsageReceiptEnvelope; + changed.producerEvent!.usage!.runDelta = { inputTokens: 999, outputTokens: 10 }; + await db.update(runUsageReceipts).set({ receiptJson: changed }).where(eq(runUsageReceipts.id, receipt.id)); + await expect(createNativeUsageContinuationDelivery(db, source)).rejects.toThrow("native_usage_source_conflict"); + }); + it("retains the same Runner report through a failed Postgres receipt and an empty replacement receiver", async () => { const input = await seed(); expect(existsSync(usageReceiptSpoolPath())).toBe(false); diff --git a/server/src/services/native-runtime/native-usage-delivery.ts b/server/src/services/native-runtime/native-usage-delivery.ts index 2bbf52d0d1b..0d6f05d7de8 100644 --- a/server/src/services/native-runtime/native-usage-delivery.ts +++ b/server/src/services/native-runtime/native-usage-delivery.ts @@ -16,11 +16,13 @@ const uuid = (identity: string) => { return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-5${hex.slice(13, 16)}-a${hex.slice(17, 20)}-${hex.slice(20, 32)}`; }; -export async function createNativeUsageDelivery(db: Db, input: { +type NativeUsageDeliveryInput = { companyId: string; runId: string; runnerInstanceId: string; provider: NativeExecutionInput["provider"]; billing: Pick; -}) { +}; + +export async function createNativeUsageDelivery(db: Db, input: NativeUsageDeliveryInput) { const sourceId = uuid(`${input.runId}:${input.runnerInstanceId}`); const [bound] = await db.update(heartbeatRuns).set({ usageJson: sql` coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || '{"accountingReceiptOwner":"native_producer"}'::jsonb || @@ -29,9 +31,39 @@ export async function createNativeUsageDelivery(db: Db, input: { }).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId), isNull(heartbeatRuns.costAccountedAt))).returning({ id: heartbeatRuns.id }); if (!bound) throw new Error("native_usage_run_not_available"); - const saved = await db.select({ receipt: runUsageReceipts.receiptJson }).from(runUsageReceipts).where(and( + return createNativeUsageReceiver(db, input, sourceId, await loadNativeUsageReceipts(db, input, sourceId)); +} + +/** Drain the authenticated previous producer's accounting without acquiring or + * resetting its ownership. Finalized sources retain normal receipt equality + * and reconciliation rules; this grants no business/tool execution authority. */ +export async function createNativeUsageContinuationDelivery(db: Db, input: Omit) { + const sourceId = uuid(`${input.runId}:${input.runnerInstanceId}`); + const [source] = await db.select({ runnerInstanceId: heartbeatRuns.runnerInstanceId, usage: heartbeatRuns.usageJson }) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))); + if (source?.runnerInstanceId !== input.runnerInstanceId || source.usage?.accountingReceiptOwner !== "native_producer" + || source.usage.accountingReceiptSourceId !== sourceId) throw new Error("native_usage_continuation_source_mismatch"); + const saved = await loadNativeUsageReceipts(db, input, sourceId); + if (!saved.length) throw new Error("native_usage_continuation_receipts_missing"); + const classification = (receipt: AdapterUsageCheckpoint) => ({ + provider: receipt.provider, biller: receipt.biller, billingType: receipt.billingType, + }); + const receipts = saved.map(row => (row.receipt as UsageReceiptEnvelope).receipt as AdapterUsageCheckpoint); + const billing = classification(receipts[0]); + if (receipts.some(receipt => receiptFingerprint(classification(receipt)) !== receiptFingerprint(billing) + || receipt.model !== (input.provider.model ?? "unknown"))) throw new Error("native_usage_source_conflict"); + return createNativeUsageReceiver(db, { ...input, billing }, sourceId, saved, true); +} + +function loadNativeUsageReceipts(db: Db, input: Pick, sourceId: string) { + return db.select({ receipt: runUsageReceipts.receiptJson, hash: runUsageReceipts.receiptHash, + sequence: runUsageReceipts.sequence }).from(runUsageReceipts).where(and( eq(runUsageReceipts.companyId, input.companyId), eq(runUsageReceipts.runId, input.runId), eq(runUsageReceipts.sourceId, sourceId))).orderBy(asc(runUsageReceipts.sequence)); +} + +function createNativeUsageReceiver(db: Db, input: NativeUsageDeliveryInput, sourceId: string, + saved: Awaited>, verifySaved = false) { const accepted = new Map(); const turnAccounting = ["opencode", "acpx"].includes(input.provider.kind) ? createNativeTurnAccounting(input.provider) : null; let observedUsage: Record | null = null; @@ -60,6 +92,12 @@ export async function createNativeUsageDelivery(db: Db, input: { for (const row of saved) { const envelope = row.receipt as UsageReceiptEnvelope; if (!envelope.producerEvent) throw new Error("native_usage_receipt_missing_producer"); + if (verifySaved && (envelope.companyId !== input.companyId || envelope.runId !== input.runId + || envelope.sourceId !== sourceId || envelope.sequence !== row.sequence || envelope.adapterType !== "paperclip_runner" + || envelope.id !== uuid(`${input.runId}:${input.runnerInstanceId}:${envelope.producerEvent.sourceEventId}`) + || receiptFingerprint(envelope) !== row.hash || accepted.has(envelope.producerEvent.sourceEventId))) { + throw new Error("native_usage_source_conflict"); + } accepted.set(envelope.producerEvent.sourceEventId, envelope); observe(envelope); } diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts index 2edc3e1b718..ec45dca6437 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts @@ -48,6 +48,7 @@ import { import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js"; import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js"; +import { nativeSha256 } from "./canonical.js"; import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js"; import { finalizeNativeRun, repairCommittedNativeChatResponse } from "./native-run-finalizer.js"; import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; @@ -313,6 +314,63 @@ describe("PaperclipControlPlanePort conformance", () => { } }); + it("publishes native recovery metadata in the existing snapshot transaction and fences stale generations", async () => { + const identity = { ...CONTROL_PLANE_CONFORMANCE_OPEN.identity, runId: randomUUID(), sessionId: randomUUID() }; + const runnerId = randomUUID(); + const owner = { ...identity, authorityRunId: identity.runId, bootId: randomUUID(), generation: 1, leaseOwner: "snapshot-owner" }; + await db.insert(heartbeatRuns).values({ id: identity.runId, companyId: identity.companyId, agentId: identity.agentId, + status: "running", runtimeMode: "native", nativeIssueId: identity.issueId, nativeSessionId: identity.sessionId, + runnerInstanceId: runnerId, completionContractId: contractId, completionContractSha256: contractSha, + runnerProfileJson: { unrelated: "preserve" } }); + await db.insert(nativeRunFinalizations).values({ runId: identity.runId, companyId: identity.companyId, issueId: identity.issueId, + phase: "observed", + controllerBootId: owner.bootId, controllerGeneration: owner.generation, leaseOwner: owner.leaseOwner, + leaseExpiresAt: new Date(Date.now() + 60_000) }); + const port = new PaperclipControlPlanePort(db, { ...identity, completionContractId: contractId, + completionContractSha256: contractSha, sourceInstanceId: runnerId, controlPlaneSourceInstanceId: "snapshot-control" }, { + nativeRecovery: { owner, compatibility: { configDigest: `sha256:${"a".repeat(64)}`, processConfigDigest: null } }, + }); + await port.openRun({ identity, backendKind: "mock", sourceInstanceId: runnerId }); + const snapshot = { backendKind: "mock" as const, sessionId: identity.sessionId, identity }; + const statements: string[] = []; + const previousDebug = db.$client.options.debug; + db.$client.options.debug = (_connection, statement) => { statements.push(statement); }; + try { await port.checkpointSession(snapshot); } + finally { db.$client.options.debug = previousDebug; } + expect(statements.filter(statement => /^begin\b/i.test(statement.trim()))).toHaveLength(1); + expect(statements.filter(statement => /^update\s+"heartbeat_runs"/i.test(statement.trim()))).toHaveLength(1); + expect(statements.filter(statement => /^(?:with|select)\b/i.test(statement.trim()))).toHaveLength(1); + const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, identity.runId)); + expect(saved.runnerProfileJson).toMatchObject({ unrelated: "preserve", sessionCheckpoint: snapshot, + nativeControllerRecovery: { schema: "paperclip.native-controller-recovery.v1", snapshotSha256: nativeSha256(saved.runnerProfileJson?.sessionCheckpoint), + checkpoint: null, processConfigDigest: null } }); + // Missing PRP is an explicit incomplete publication, never fabricated authority. + await db.update(nativeRunFinalizations).set({ controllerGeneration: 2 }).where(eq(nativeRunFinalizations.runId, identity.runId)); + await expect(port.checkpointSession({ ...snapshot, providerSessionId: "stale" })).rejects.toThrow("checkpoint_owner_lost"); + await expect(port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL })) + .rejects.toThrow("checkpoint_owner_lost"); + const [unchanged] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, identity.runId)); + expect(unchanged.runnerProfileJson).toEqual(saved.runnerProfileJson); + // A valid coordinator lock does not certify a lease that expires while the + // subsequent run-row lock waits. Recheck time at the actual write boundary. + await db.update(nativeRunFinalizations).set({ controllerGeneration: 1, leaseExpiresAt: new Date(Date.now() + 150) }) + .where(eq(nativeRunFinalizations.runId, identity.runId)); + let locked!: () => void, release!: () => void; + const acquired = new Promise(resolve => { locked = resolve; }); + const held = new Promise(resolve => { release = resolve; }); + const blocker = db.transaction(async tx => { + await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, identity.runId)).for("update"); + locked(); + await held; + }); + await acquired; + const waiting = port.checkpointSession(snapshot); + const rejected = expect(waiting).rejects.toThrow("checkpoint_owner_lost"); + try { await new Promise(resolve => setTimeout(resolve, 200)); } + finally { release(); await blocker; } + await rejected; + }); + it("permits only read-only admission while an existing startup drains", async () => { const identity = { ...CONTROL_PLANE_CONFORMANCE_OPEN.identity, runId: randomUUID(), sessionId: randomUUID() }; const runnerId = randomUUID(); diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.ts b/server/src/services/native-runtime/paperclip-control-plane-port.ts index 2ee3446f568..f7bac1622d3 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.ts @@ -1,7 +1,7 @@ import { notifyNativeResultCommit } from "../native-result-commit-notifications.js"; import { notifyNativeWorkspaceFinalization } from "../native-workspace-finalization-notifications.js"; import { createAgentIdentityRedactor } from "../agent-identity-redaction.js"; -import { and, asc, eq, gt, or } from "drizzle-orm"; +import { and, asc, eq, gt, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { heartbeatRunEvents, @@ -9,6 +9,7 @@ import { issues, nativeRunFinalizations, nativeRunResults, + nativeRunnerCheckpoints, } from "@paperclipai/db"; import type { CompleteControlPlaneRunInput, @@ -30,6 +31,10 @@ import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { publishLiveEvent } from "../live-events.js"; import { nativeSha256 } from "./canonical.js"; import { redactRunnerApiEvent, redactRunnerApiPayloads } from "./runner-api-log-redaction.js"; +import { createNativeControllerRecovery, NATIVE_CONTROLLER_RECOVERY_MAX_BYTES, type NativeControllerCompatibility } from "./native-controller-recovery.js"; +import { eligibleNativeTaskSessionPublication, publishNativeTaskSessionInTransaction, NATIVE_TASK_SESSION_PUBLICATION_KEY, + type NativeTaskSessionPublicationContext, type NativeTaskSessionPublicationIssue } from "./native-task-session-publication.js"; +import { lockNativeRunnerCheckpointOwner, nativeRunnerCheckpointOwnerLock, validateNativeRunnerCheckpointOwner, inspectNativeRunnerUpgradeHeader, type NativeRunnerCheckpointOwner } from "./native-runner-checkpoint.js"; export interface PaperclipControlPlaneBinding { companyId: string; @@ -90,6 +95,9 @@ export class PaperclipControlPlanePort implements ControlPlanePort { readonly #onEventPersistence?: (event: PrpEvent, startedAtMs: number, endedAtMs: number) => Promise; readonly #onCommittedEvent?: (event: PrpEvent, cursor: number) => Promise; readonly #onDuplicateEvent?: (event: PrpEvent, cursor: number) => Promise; + readonly #nativeRecovery?: { compatibility: NativeControllerCompatibility; owner: NativeRunnerCheckpointOwner }; + readonly #taskSessionPublication?: NativeTaskSessionPublicationContext; + readonly #taskSessionPublicationOwner?: NativeRunnerCheckpointOwner; constructor( db: Db, @@ -105,6 +113,9 @@ export class PaperclipControlPlanePort implements ControlPlanePort { onEventPersistence?: (event: PrpEvent, startedAtMs: number, endedAtMs: number) => Promise; onCommittedEvent?: (event: PrpEvent, cursor: number) => Promise; onDuplicateEvent?: (event: PrpEvent, cursor: number) => Promise; + nativeRecovery?: { compatibility: NativeControllerCompatibility; owner: NativeRunnerCheckpointOwner }; + taskSessionPublication?: NativeTaskSessionPublicationContext; + taskSessionPublicationOwner?: NativeRunnerCheckpointOwner; } = {}, ) { this.#db = db; @@ -116,6 +127,9 @@ export class PaperclipControlPlanePort implements ControlPlanePort { this.#onEventPersistence = options.onEventPersistence; this.#onResultCommitted = options.onResultCommitted; this.#onDuplicateEvent = options.onDuplicateEvent; + this.#nativeRecovery = options.nativeRecovery; + this.#taskSessionPublication = options.taskSessionPublication ? structuredClone(options.taskSessionPublication) : undefined; + this.#taskSessionPublicationOwner = options.taskSessionPublicationOwner; } #assertActive(options?: { signal: AbortSignal }): void { @@ -123,7 +137,9 @@ export class PaperclipControlPlanePort implements ControlPlanePort { options?.signal.throwIfAborted(); } - #matchesPersistedBinding(run: typeof heartbeatRuns.$inferSelect): boolean { + #matchesPersistedBinding(run: Pick): boolean { return run.companyId === this.#binding.companyId && run.agentId === this.#binding.agentId && run.runtimeMode === "native" @@ -200,19 +216,105 @@ export class PaperclipControlPlanePort implements ControlPlanePort { || identity.agentId !== this.#binding.agentId || this.#sessionId !== identity.sessionId ) throw new Error("native_session_checkpoint_binding_mismatch"); + const publication = this.#taskSessionPublication && eligibleNativeTaskSessionPublication(snapshot) + ? this.#taskSessionPublication : null; + if (publication && !this.#taskSessionPublicationOwner) throw new Error("native_task_session_publication_owner_required"); await this.#db.transaction(async (tx) => { - const run = await tx.select().from(heartbeatRuns) - .where(eq(heartbeatRuns.id, this.#binding.runId)).for("update").limit(1) + const recovery = this.#nativeRecovery; + const owner = recovery?.owner ?? (publication ? this.#taskSessionPublicationOwner : undefined); + if (publication && recovery && nativeSha256(recovery.owner) !== nativeSha256(this.#taskSessionPublicationOwner)) { + throw new Error("native_task_session_publication_owner_mismatch"); + } + const ownerLock = owner ? nativeRunnerCheckpointOwnerLock(tx, owner) : null; + // NO KEY UPDATE serializes reset/admission fields while remaining compatible + // with completeRun's parent KEY SHARE. Keep coordinator → issue → run order. + const publicationIssue = publication ? tx.$with("publication_issue").as(tx.select({ + id: issues.id, companyId: issues.companyId, executionRunId: issues.executionRunId, + conversationAgentId: issues.conversationAgentId, conversationSessionGeneration: issues.conversationSessionGeneration, + }).from(issues).where(and(eq(issues.id, this.#binding.issueId), eq(issues.companyId, this.#binding.companyId), + ownerLock ? sql`exists (select 1 from ${ownerLock})` : undefined)).for("no key update")) : null; + const lockedRun = tx.$with("snapshot_run").as(tx.select({ + id: heartbeatRuns.id, status: heartbeatRuns.status, + contextSnapshot: (publication ? sql>`jsonb_build_object('conversationSessionGeneration', ${heartbeatRuns.contextSnapshot}->'conversationSessionGeneration')` : sql`null`).as("publication_context"), + resultJson: (publication ? sql>`jsonb_build_object('nativeCancellation', ${heartbeatRuns.resultJson}->'nativeCancellation', + 'cancellation', ${heartbeatRuns.resultJson}->'cancellation', 'startupCancellation', ${heartbeatRuns.resultJson}->'startupCancellation')` : sql`null`).as("publication_cancellation"), + startedAt: heartbeatRuns.startedAt, createdAt: heartbeatRuns.createdAt, + companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, nativeIssueId: heartbeatRuns.nativeIssueId, + nativeSessionId: heartbeatRuns.nativeSessionId, runnerInstanceId: heartbeatRuns.runnerInstanceId, + completionContractId: heartbeatRuns.completionContractId, + completionContractSha256: heartbeatRuns.completionContractSha256, + runnerProfileJson: heartbeatRuns.runnerProfileJson, + }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.id, this.#binding.runId), ownerLock + ? sql`exists (select 1 from ${ownerLock})` : undefined, + publicationIssue ? sql`exists (select 1 from ${publicationIssue})` : undefined)).for("update")); + // Project the locking CTE's tuples, including any update committed while + // the lock waited. A second scan of heartbeat_runs could see an old MVCC tuple. + // The run CTE depends on the coordinator CTE: both locks precede the + // outer clock sample, including when either lock waits. + const selected = await tx.with(...(ownerLock ? [ownerLock] : []), ...(publicationIssue ? [publicationIssue] : []), lockedRun).select({ run: { + id: lockedRun.id, status: lockedRun.status, contextSnapshot: lockedRun.contextSnapshot, resultJson: lockedRun.resultJson, + startedAt: lockedRun.startedAt, createdAt: lockedRun.createdAt, + companyId: lockedRun.companyId, agentId: lockedRun.agentId, + runtimeMode: lockedRun.runtimeMode, nativeIssueId: lockedRun.nativeIssueId, + nativeSessionId: lockedRun.nativeSessionId, runnerInstanceId: lockedRun.runnerInstanceId, + completionContractId: lockedRun.completionContractId, + completionContractSha256: lockedRun.completionContractSha256, + runnerProfileJson: lockedRun.runnerProfileJson, + }, now: sql`clock_timestamp()`, ...(ownerLock ? { + leaseOwner: sql`(select ${ownerLock.leaseOwner} from ${ownerLock})`, + leaseExpiresAt: sql`(select ${ownerLock.leaseExpiresAt} from ${ownerLock})`.mapWith(nativeRunFinalizations.leaseExpiresAt), + resultId: sql`(select ${ownerLock.resultId} from ${ownerLock})`, + } : {}), ...(recovery ? { + checkpointHeader: sql | null>`(select material from ${nativeRunnerCheckpoints} + where ${nativeRunnerCheckpoints.runId} = ${this.#binding.runId} + and ${nativeRunnerCheckpoints.companyId} = ${this.#binding.companyId} + and ${nativeRunnerCheckpoints.issueId} = ${this.#binding.issueId} + and ${nativeRunnerCheckpoints.part} = 'header')`, + } : {}), ...(publicationIssue ? { publicationIssue: sql`(select jsonb_build_object( + 'id', ${publicationIssue.id}, 'companyId', ${publicationIssue.companyId}, 'executionRunId', ${publicationIssue.executionRunId}, + 'conversationAgentId', ${publicationIssue.conversationAgentId}, 'conversationSessionGeneration', ${publicationIssue.conversationSessionGeneration}) from ${publicationIssue})` } : {}), + }).from(lockedRun).limit(1) .then((rows) => rows[0] ?? null); + const run = selected?.run; this.#assertActive(options); + if (owner && !selected) throw new Error("native_runner_checkpoint_owner_lost"); if (!run || !this.#matchesPersistedBinding(run)) { throw new Error("native_session_checkpoint_binding_mismatch"); } - await tx.update(heartbeatRuns).set({ - runnerProfileJson: { + // The run row lock may have waited beyond our lease. Generation cannot + // change while we hold the coordinator, but wall-clock expiry still can. + if (owner && selected) await validateNativeRunnerCheckpointOwner(tx, owner, + { leaseOwner: selected.leaseOwner ?? null, leaseExpiresAt: selected.leaseExpiresAt ?? null, + resultId: selected.resultId ?? null, now: selected.now }, !snapshot.semanticResult); + const header = recovery && selected?.checkpointHeader + ? inspectNativeRunnerUpgradeHeader(this.#binding, selected.checkpointHeader) : null; + if (header && (header.normalizedSessionId !== identity.sessionId || header.runnerInstanceId !== this.#binding.sourceInstanceId)) { + throw new Error("native_session_checkpoint_prp_binding_mismatch"); + } + const profile: Record = { ...record(run.runnerProfileJson), sessionCheckpoint: structuredClone(snapshot) as unknown as Record, - }, + ...(recovery ? { nativeControllerRecovery: createNativeControllerRecovery(snapshot, recovery.compatibility, + header ? { runId: this.#binding.runId, runnerInstanceId: header.runnerInstanceId, + normalizedSessionId: header.normalizedSessionId } : null) } : {}), + }; + // Driver/live snapshots may already describe a terminal result before + // completeRun accepts it. Persist that observation without a handled + // receipt; the existing post-acceptance checkpoint publishes the pointer. + if (publication && selected?.resultId) { + const receipt = await publishNativeTaskSessionInTransaction(tx, { binding: this.#binding, context: publication, + snapshot, run, issue: selected.publicationIssue ?? null, resultId: selected.resultId ?? null, + recoveryReady: !recovery || header !== null }); + profile[NATIVE_TASK_SESSION_PUBLICATION_KEY] = receipt; + if (Buffer.byteLength(JSON.stringify({ nativeControllerRecovery: profile.nativeControllerRecovery, + [NATIVE_TASK_SESSION_PUBLICATION_KEY]: receipt })) > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES) { + throw new Error("native_task_session_publication_metadata_too_large"); + } + } + await tx.update(heartbeatRuns).set({ + runnerProfileJson: profile, sessionIdAfter: snapshot.providerSessionId ?? snapshot.sessionId, nativePhase: snapshot.semanticResult ? "workspace_finalizing" : "observed", nativePhaseUpdatedAt: new Date(), @@ -386,9 +488,12 @@ export class PaperclipControlPlanePort implements ControlPlanePort { }); const insertedResult = await this.#db.transaction(async (tx) => { - // Result insertion checks the task foreign key. Acquire that parent lock - // before the run, matching task mutations that subsequently update a run. - // Otherwise concurrent chat/status writes can deadlock after generation. + // Status commit and snapshot publication lock coordinator → issue → run. + // A replay must follow that order too: holding the issue while waiting + // for the coordinator deadlocks a status commit's issue UPDATE lock. + const recovery = this.#nativeRecovery; + const owner = recovery ? await lockNativeRunnerCheckpointOwner(tx, recovery.owner) : null; + // Result insertion checks the task FK. Keep its parent lock before run. await tx.select({ id: issues.id }).from(issues) .where(and(eq(issues.id, this.#binding.issueId), eq(issues.companyId, this.#binding.companyId))) .for("key share"); @@ -399,6 +504,10 @@ export class PaperclipControlPlanePort implements ControlPlanePort { if (!run || !this.#matchesPersistedBinding(run)) { throw new Error("native_result_binding_mismatch"); } + if (recovery && owner) { + const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`); + await validateNativeRunnerCheckpointOwner(tx, recovery.owner, { ...owner, now: clock.now }, false); + } const callerConditions = [eq(nativeRunResults.serverFingerprint, serverFingerprint)]; if (value.callerResultId) callerConditions.push(eq(nativeRunResults.callerResultId, value.callerResultId)); @@ -411,6 +520,7 @@ export class PaperclipControlPlanePort implements ControlPlanePort { if (existing.canonicalSha256 !== canonicalSha256) throw new Error("structured_result_replay_conflict"); return false; } + if (owner?.resultId) throw new Error("structured_result_replay_conflict"); const resultJson = { result: validated.result, terminal: value.terminal, diff --git a/server/src/services/native-runtime/remote-warm-session-recovery.test.ts b/server/src/services/native-runtime/remote-warm-session-recovery.test.ts new file mode 100644 index 00000000000..91317d2af4b --- /dev/null +++ b/server/src/services/native-runtime/remote-warm-session-recovery.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, it } from "vitest"; +import { isSettledRemoteWarmControlState } from "./remote-warm-session-recovery.js"; + +const identity = { runId: "run", normalizedSessionId: "session", runnerInstanceId: "runner", environmentLeaseId: "lease", turnId: "turn", itemId: "item" }; +const terminal = { eventType: "run.terminal", envelope: { ...identity } }; +const goal = { schema: "paperclip.session_goal.snapshot.v1", goal: null, workingNow: false }; +const notification = { schema: "paperclip.prp.event.v1", eventType: "session.goal.snapshot", runId: identity.runId, + normalizedSessionId: identity.normalizedSessionId, sourceInstanceId: "runner", sourceKind: "runner", payload: goal }; +const snapshot = { eventType: "session.goal.snapshot", envelope: { ...identity, payload: notification } }; +const state = (events: unknown[]) => ({ identity, commands: [{ status: "completed" }], committedEvents: events }); + +describe("settled remote warm control evidence", () => { + it.each(["paperclip.prp.event.v1", "paperclip.prp.event.v2"])("accepts the terminal and an authenticated %s idle goal-read suffix without changing its order", schema => { + const suffix = { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, schema } } }; + const control = state([terminal, suffix]); + expect(isSettledRemoteWarmControlState(control, identity)).toBe(true); + expect(control.committedEvents).toEqual([terminal, suffix]); + expect(isSettledRemoteWarmControlState(state([terminal]), identity)).toBe(true); + }); + + it.each([ + ["active goal", { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, payload: { ...goal, goal: { status: "active" } } } } }], + ["provider work", { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, payload: { ...goal, workingNow: true } } } }], + ["unknown event", { ...snapshot, eventType: "unknown" }], + ["unknown protocol version", { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, schema: "paperclip.prp.event.v3" } } }], + ["new turn", { ...snapshot, eventType: "turn.started" }], + ["foreign run", { ...snapshot, envelope: { ...snapshot.envelope, runId: "other" } }], + ["foreign session", { ...snapshot, envelope: { ...snapshot.envelope, normalizedSessionId: "other" } }], + ["foreign source", { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, sourceInstanceId: "other" } } }], + ["foreign lease", { ...snapshot, envelope: { ...snapshot.envelope, environmentLeaseId: "other" } }], + ["foreign notification", { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, runId: "other" } } }], + ["missing payload", { ...snapshot, envelope: { ...identity } }], + ])("holds a %s suffix", (_name, suffix) => { + expect(isSettledRemoteWarmControlState(state([terminal, suffix]), identity)).toBe(false); + }); + + it("holds missing terminals, pending commands and mixed core identities", () => { + expect(isSettledRemoteWarmControlState(state([snapshot]), identity)).toBe(false); + expect(isSettledRemoteWarmControlState({ ...state([terminal, snapshot]), commands: [{ status: "dispatched" }] }, identity)).toBe(false); + expect(isSettledRemoteWarmControlState({ ...state([terminal, snapshot]), identity: { ...identity, runnerInstanceId: "other" } }, identity)).toBe(false); + expect(isSettledRemoteWarmControlState(state([{ ...terminal, envelope: { ...identity, runId: "other" } }, snapshot]), identity)).toBe(false); + }); +}); diff --git a/server/src/services/native-runtime/remote-warm-session-recovery.ts b/server/src/services/native-runtime/remote-warm-session-recovery.ts index 2d91c49f6c7..893dfefaef6 100644 --- a/server/src/services/native-runtime/remote-warm-session-recovery.ts +++ b/server/src/services/native-runtime/remote-warm-session-recovery.ts @@ -3,6 +3,7 @@ import { agents, companies, environmentLeases, heartbeatRuns, issues, nativeRunF import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import { parseNativeExecutionInput, type NativeExecutionInput, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; import { currentNativeControllerIdentity, evaluateNativeControllerTakeover } from "./native-restart-recovery.js"; +import { readNativeControllerRecovery } from "./native-controller-recovery.js"; export type RemoteWarmSessionRecovery = { identity: Record; @@ -13,6 +14,36 @@ export type RemoteWarmSessionRecovery = { revalidate: (leaseOwner: string) => Promise; }; +/** A goal read can publish an idle snapshot after the terminal. It carries no + * new work; every other suffix still requires reconciliation before handoff. */ +export function isSettledRemoteWarmControlState(control: Record, identity: Record): boolean { + const record = (value: unknown): Record => + value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; + const commands = control.commands; + const events = control.committedEvents; + if (!Array.isArray(commands) || commands.some(command => record(command).status !== "completed") || + !Array.isArray(events) || Object.entries(identity).some(([key, value]) => record(control.identity)[key] !== value)) return false; + const terminalIndex = events.findLastIndex(event => record(event).eventType === "run.terminal"); + if (terminalIndex < 0) return false; + const matchesSession = (event: unknown) => { + const envelope = record(record(event).envelope); + return envelope.runId === identity.runId && envelope.normalizedSessionId === identity.normalizedSessionId; + }; + if (!matchesSession(events[terminalIndex])) return false; + return events.slice(terminalIndex + 1).every(event => { + const envelope = record(record(event).envelope); + const notification = record(envelope.payload); + const payload = record(notification.payload); + return record(event).eventType === "session.goal.snapshot" && matchesSession(event) && + Object.entries(identity).every(([key, value]) => envelope[key] === value) && + (notification.schema === "paperclip.prp.event.v1" || notification.schema === "paperclip.prp.event.v2") && + notification.eventType === "session.goal.snapshot" && + notification.sourceInstanceId === identity.runnerInstanceId && notification.sourceKind === "runner" && + notification.runId === identity.runId && notification.normalizedSessionId === identity.normalizedSessionId && + payload.schema === "paperclip.session_goal.snapshot.v1" && payload.goal === null && payload.workingNow === false; + }); +} + export async function inspectRemoteWarmSessionRecovery(input: { db: Db; execution: NativeExecutionInput; @@ -52,15 +83,14 @@ export async function inspectRemoteWarmSessionRecovery(input: { checkpoint.driverKind !== previous.session.driverKind || !checkpoint.providerSessionId || checkpoint.activeTurnId !== null || checkpoint.pendingRuntimeRequests?.length !== 0) return deny(); const control = input.readControlState(); - const commands = control.commands; - const events = control.committedEvents; - const terminal = Array.isArray(events) ? events.at(-1) : undefined; - if (!Array.isArray(commands) || commands.some(command => command.status !== "completed") || - terminal?.eventType !== "run.terminal" || terminal?.envelope?.runId !== identity.runId || - terminal?.envelope?.normalizedSessionId !== identity.normalizedSessionId || - Object.entries(identity).some(([key, value]) => (control.identity as Record)?.[key] !== value)) return deny(); + if (!isSettledRemoteWarmControlState(control, identity)) return deny(); const assertPriorStopped = async () => { - const stopped = await evaluateNativeControllerTakeover({ owner: prior.owner, now: new Date(), requireStoppedController: true }); + const persistedRecovery = readNativeControllerRecovery(prior.run.runnerProfileJson?.nativeControllerRecovery, + prior.run.runnerProfileJson?.sessionCheckpoint); + // A PID in another webserver container cannot prove life or death. New + // durable controllers are fenced by the database lease and current turn. + const stopped = await evaluateNativeControllerTakeover({ owner: prior.owner, now: new Date(), requireStoppedController: true, + remoteLeaseFenced: persistedRecovery !== null }); if (!stopped.allowed) return deny(); }; await assertPriorStopped(); diff --git a/server/src/services/run-secret-redaction.ts b/server/src/services/run-secret-redaction.ts index ad07a402920..597d12a9bd9 100644 --- a/server/src/services/run-secret-redaction.ts +++ b/server/src/services/run-secret-redaction.ts @@ -52,7 +52,7 @@ export function redactRegisteredSecretValues(input: T, values: string[], cred if (!record) return input; return Object.fromEntries( Object.entries(record) - .filter(([key]) => key !== REGISTRY_KEY) + .filter(([key]) => key !== REGISTRY_KEY && key !== "nativeControllerRecovery" && key !== "nativeTaskSessionPublication") .map(([key, value]) => [key, key === "credentialAccess" ? (credentialConnectionName ? { connectionName: redactText(credentialConnectionName, values) } : {}) : redactRegisteredSecretValues(value, values, credentialConnectionName)]), diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index 6e84dd0fbbd..63a918160ea 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -94,6 +94,9 @@ export const RUNNER_PRP_PROTOCOL_MIN = runner.RUNNER_PRP_PROTOCOL_MIN; export const RUNNER_PRP_PROTOCOL_MAX = runner.RUNNER_PRP_PROTOCOL_MAX; export const runnerCodexDynamicToolsFit = runner.runnerCodexDynamicToolsFit; export const inspectWarmRunTransition = runner.inspectWarmRunTransition; +export const inspectStagedRunAttachment = runner.inspectStagedRunAttachment; +export const inspectStagedRunAttachmentHeader = runner.inspectStagedRunAttachmentHeader; +export type StagedRunAttachment = import("@paperclipai/paperclip-runner").StagedRunAttachment; export const readRunnerdArtifactBinding = runner.readRunnerdArtifactBinding; export const readLinuxProcessStartedAt = runner.readLinuxProcessStartedAt; export const NativeSessionCloseUnrecoverableError = diff --git a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md index dbb11f6b884..1d7bf6f6fcb 100644 --- a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md +++ b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md @@ -114,6 +114,13 @@ effects with different tool IDs cannot repeat. `remote-restart-before.json`, `remote-restart-after.json` and `remote-restart-final.json` retain receipt identities and hashes. They exclude provider text, tool arguments, command output, and raw provider sandbox IDs. +The dedicated explicit-only `daytona-restart-continuity` definition 7 additionally +replaces the controller after completion, then posts its unchanged saved-project +follow-up through the normal authenticated task-comment API. Exactly one matching +persisted user comment, consumed wake, same native/provider identity, new completed +tool and useful output are required. Its initial browser journey remains; post-C +readback is API-only and does not qualify browser reconnection or rendered status. +The definition 6 browser timeout remains failed with an unknown browser cause. The suite definition version and harness digest change with this oracle. Its deterministic calibration is not a claim that a live Daytona run passed. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 01049d9e5a2..fbfbe27169c 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -10,13 +10,45 @@ qualification. The explicit-only `daytona-restart-continuity` suite runs the existing `recover-controller` journey with the recorded Codex and Claude staging models -on fresh Daytona fixtures. After a completed task tool in an active revision -turn, it keeps the owned development controller offline for 125 seconds, beyond -the controller's two-minute reconnect budget. The existing independent oracle +on fresh reusable warm Daytona fixtures. The sandbox, Runner and provider history +must survive both controller replacements. After a completed task tool in an active revision +turn, it keeps the owned development controller offline for 300 seconds. This +exceeds the 60-second ownership lease and remains within the Runner's 420-second +reconnect grace. The existing independent oracle requires the original process, native session, sandbox lease and receipt prefix, one completion and successful workspace finalization, plus verified downloaded -work. It permits one attempt per cell and retains failed evidence. This is +work. After completion, it replaces the controller again with its native cache +unavailable and asks a new turn to execute the saved project. The continuation +oracle requires the same retained provider session and a new useful result. +It permits one attempt per cell and retains failed evidence. This is development API-key acceptance, distinct from staging subscription recovery. +The five-minute outage is a resilience test, not an HTTP downtime allowance. +This suite gives its Daytona sandbox a 15-minute provider auto-stop interval, +leaving headroom for the outage and replacement startup. Native agent idle timing, +ownership/reconnect limits, archive/delete backstops and owned cleanup remain unchanged. +The retained continuation accepts whitespace between its exact marker and the +exact `queue_r` output token; readiness and final grading use the same matcher. + +Definition version 7 uses the existing warm reusable environment with a dedicated +configuration key. Only the post-completion replacement follow-up uses the normal +board-authenticated task-comment API, with one persisted user comment and the +unchanged follow-up request. Initial creation, active outage and queued continuation +retain their browser journey. Post-C settlement, task status and useful reply are +read through production APIs, without post-C browser navigation or rendered-status +assertions. The required final screenshot captures the actual retained Chromium +page, which may be stale or show an error; it is labeled as unqualified browser +evidence and does not establish fresh product state. The same strict native/provider +identity, new tool, +saved-project output and finalization checks remain required. Definition 6's +observed post-C browser navigation timeout remains failed; its cause is unknown. +This definition does not claim that browser reconnection was fixed. +The prior version 5 per-turn/no-reuse measurement **failed** +retained native/provider identity and remains failed; it does not qualify this +warm fixture. Cache retirement moves the entire controller Runner directory, +including provider failover-backup bytes. Once a per-turn sandbox is disposed, +its DB checkpoint alone cannot replace those bytes. Per-turn application +qualification under this disk loss awaits the provider-backup owner's durable +storage contract; native controller recovery does not waive that dependency. The explicit-only [GitHub review bot journey](GITHUB-REVIEW-JOURNEY.md) tests attended bot setup and an automatic PR review against an attached deployment. @@ -596,6 +628,15 @@ inputs, or a tracked env file. ## Local commands +When the host cannot start embedded Postgres, set +`PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL` to an explicitly supplied loopback +test Postgres administrator. The launcher creates a new `runner_e2e_` +database per attempt; the serving app applies its migrations. This does not +reuse the administrator URL's database. The administrator variable is not +forwarded to the server or browser. The launcher checks the private ownership +receipt and deletes only its database after confirmed process/resource cleanup. +Unknown remote cleanup preserves that database and the private fixture for recovery. + Install dependencies and Chromium once. Native local cells also need the local runner binaries: @@ -2507,7 +2548,13 @@ The same four installed-package pins also admit three existing local native Codex cells: `first-task.runner-codex.local.task-card-accept`, `everyday-workflows.runner-codex.local.build-revise`, and `agent-chat.runner-codex.local.continuity-restart`. Select exact IDs; other -Codex cells, legacy profiles and remote environments remain outside this lane. +Codex cells and legacy profiles remain outside this lane. The two explicit +Daytona controller-recovery cells are also admitted: +`everyday-workflows.runner-codex.daytona.recover-controller` and +`daytona-restart-continuity.runner-codex.daytona.recover-controller`. +The latter uses the 300-second outage and retained-session follow-up. Daytona +cells require the pinned installed plugin described below. Admission support +does not establish that either cell has passed against installed artifacts. These reuse the onboarding, independently checked file/revision, and restart graders. They require separately authorized provider access and are not executed by the provider-free release smoke. diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index a8ba74083b5..f7b1d783ee8 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -33,6 +33,47 @@ import { } from "./selectors.js"; describe("runner E2E catalog", () => { + it("keeps the Daytona controller outage on a reusable warm sandbox inside its idle lifetime", () => { + const suite = runnerSuites.find(candidate => candidate.id === "daytona-restart-continuity")!; + const secretRefs = { + OPENAI_API_KEY: { type: "secret_ref" as const, secretId: "22222222-2222-4222-8222-222222222222", version: "latest" as const }, + ANTHROPIC_API_KEY: { type: "secret_ref" as const, secretId: "33333333-3333-4333-8333-333333333333", version: "latest" as const }, + DAYTONA_API_KEY: { type: "secret_ref" as const, secretId: "44444444-4444-4444-8444-444444444444", version: "latest" as const }, + }; + const environmentInput = { executionId: "controller-outage", secretRefs, daytonaImage: `fixture@sha256:${"a".repeat(64)}` }; + const base = runnerEnvironments.find(environment => environment.id === "daytona")!; + const baseConfig = base.buildEnvironment(environmentInput).config as Record; + const warmConfig = daytonaWarmEnvironment.buildEnvironment(environmentInput).config as Record; + const cells = runnerMatrix.filter(entry => entry.suite.id === suite.id); + expect(cells).toHaveLength(2); + expect(cells.map(cell => cell.id)).toEqual([ + "daytona-restart-continuity.runner-codex.daytona.recover-controller", + "daytona-restart-continuity.runner-acpx-claude.daytona.recover-controller", + ]); + for (const cell of cells) { + const config = cell.environment.buildEnvironment(environmentInput).config as Record; + expect(config).toMatchObject({ reuseLease: true, runnerLifecycleMode: "warm" }); + expect(config.autoStopInterval).toBe(15); + expect(Number(config.autoStopInterval) * 60_000).toBeGreaterThan(420_000); + expect(config).toEqual({ ...warmConfig, autoStopInterval: 15 }); + expect(cell.environment.configurationKey).toBe("warm-controller-outage-v1"); + expect(cell.environment.label).toContain("warm reusable"); + const agentInput = { ...environmentInput, environmentId: "env-1", environmentFixtureId: "daytona" as const, workspacePath: "/workspace" }; + const adapterConfig = cell.profile.buildAgent(agentInput).adapterConfig as Record; + const original = runnerProfiles.find(profile => profile.id === cell.profile.id)!.buildAgent(agentInput).adapterConfig as Record; + expect(adapterConfig.idleTimeoutMs).toBe(300_000); + expect(adapterConfig.lifecycleMode).toBe(original.lifecycleMode); + expect(cell.suite.manualOnly).toBe(true); + expect(cell.task.automaticRetryPolicy).toBe("single_attempt"); + expect(selectRunnerExecutions(parseRunnerSelectors(["--id", cell.id])).map(entry => entry.id)).toEqual([cell.id]); + expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(entry => entry.id === cell.id)).toBe(false); + } + expect(baseConfig).toMatchObject({ autoStopInterval: 5, reuseLease: false, runnerLifecycleMode: "per_turn" }); + expect(warmConfig).toMatchObject({ autoStopInterval: 5, reuseLease: true, runnerLifecycleMode: "warm" }); + expect(suite.definitionMetadata).toMatchObject({ version: 7, controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, + retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", maximumAttemptsPerCell: 1 }); + }); + it("selects non-reused Kimi lifecycle leases without changing its warm or local cases", () => { const suite = runnerSuites.find(row => row.id === "kimi-qualification")!; const cells = runnerMatrix.filter(row => row.suite.id === suite.id); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 9ee243f3e8e..8b6fae6296c 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1160,13 +1160,21 @@ const repositoryModelProfiles = [ export const runnerSuites: readonly RunnerSuiteFixture[] = [ { - id: "daytona-restart-continuity", label: "Daytona controller outage continuity", manualOnly: true, - description: "Keep the original active native runner, task tools and saved files through more than two minutes without a controller, then commit the result without replay.", + id: "daytona-restart-continuity", label: "Daytona warm controller outage continuity", manualOnly: true, + description: "Keep the original active native runner, reusable warm sandbox, task tools and saved files through five minutes without a controller, then continue the completed session after another replacement.", groups: [], profiles: repositoryModelProfiles.map(({ native }) => productionStoryProfile(native)), - environments: runnerEnvironments.filter(environment => environment.id === "daytona"), + environments: [{ + ...daytonaWarmEnvironment, + configurationKey: "warm-controller-outage-v1", + buildEnvironment(input: EnvironmentFixtureBuildInput) { + const payload = daytonaWarmEnvironment.buildEnvironment(input); + // Keep provider idle stop beyond the outage and replacement budget. + return { ...payload, config: { ...(payload.config as Record), autoStopInterval: 15 } }; + }, + }], tasks: everydayTasks.filter(task => task.id === "recover-controller") - .map(task => ({ ...task, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2, - definitionMetadata: { version: 2, scheduling: "explicit-only", controllerDowntimeMs: 125_000, + .map(task => ({ ...task, expectedRunCount: task.expectedRunCount + 1, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2, + definitionMetadata: { version: 7, scheduling: "explicit-only", controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", maximumAttemptsPerCell: 1, budgetMonthlyCents: 1000, instructions: "production", grading: "same-active-process-lease-native-prefix-and-workspace-receipt", authentication: "api-key", stagingAuthentication: "subscription", stagingQualification: false }, }, diff --git a/tests/runner-e2e/controller-outage.test.ts b/tests/runner-e2e/controller-outage.test.ts index d8bb79b9743..edb1c9a5718 100644 --- a/tests/runner-e2e/controller-outage.test.ts +++ b/tests/runner-e2e/controller-outage.test.ts @@ -25,7 +25,7 @@ describe("controller outage qualification", () => { expect(executions).toHaveLength(2); for (const execution of executions) { const outage = controllerOutageForExecution(execution, {}); - expect(outage).toEqual({ signal: "SIGKILL", holdMs: 125_000 }); + expect(outage).toEqual({ signal: "SIGKILL", holdMs: 300_000 }); expect(controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "300000" })) .toEqual({ signal: "SIGKILL", holdMs: 300_000 }); expect(() => controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "1000" })) diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index 9f24264026f..cbba89e4db7 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -8,7 +8,7 @@ import { gradeConnectionGuidanceDecline, hasConnectionGuidanceDeclineReply, expl import { expect, type Page } from "@playwright/test"; import { runnerApiToolsEnabled } from "../../server/src/services/native-runtime/runner-api-rollout.js"; import { spawn } from "node:child_process"; -import { createHash } from "node:crypto"; +import { createHash, randomUUID } from "node:crypto"; import { mkdir, readFile, readdir } from "node:fs/promises"; import path from "node:path"; import { isDeepStrictEqual } from "node:util"; @@ -26,6 +26,7 @@ import { hasPersistedSource, isSavedSourceCheckpoint } from "./everyday-interrup import { collectRunEvents } from "./run-observations.js"; import { gradeRemoteRestart, observeRemoteRestartBoundary, projectRemoteRestartEvents, projectRemoteRestartRun, waitForRemoteRestartBoundary, type RemoteRestartEvidence } from "./everyday-remote-restart.js"; import { controllerOutageForExecution, type ControllerOutageReceipt } from "./controller-outage.js"; +import { gradeRetainedControllerContinuation, retainedControllerRun, retainedControllerFollowupTransport, submitRetainedControllerComment, onePersistedRetainedControllerComment } from "./retained-controller-continuation.js"; import { setupAggregatorFixture } from "./aggregator-fixture.js"; import { gradeProviderChoice, gradeProviderOutcome, requireProviderAccessCard } from "./connection-routing-evidence.js"; import { setupConnectionReview } from "./connection-reviews.js"; @@ -44,6 +45,8 @@ import { storyLifecycleChecks, storyRepliesConsumed, storyHasAgentReply, + storyAgentReplyMatches, + retainedControllerReplyMatcher, storyHasPendingHumanInteraction, storyReviewContinuationTimeoutDetail, storyHasStrandedBlockedLeaf, @@ -59,6 +62,7 @@ import { type StoryCheck, type StoryIssue, type StoryRun, + type StoryReplyExpectation, } from "./everyday-observations.js"; type Row = Record; @@ -187,6 +191,8 @@ export async function runEverydayFlow(input: Input) { let providerAccessDecision: { id: string; connectionId: string } | undefined; let stoppedWorkspace: Record | undefined; let settledAgentReply: Row | undefined; + let postControllerApiFollowup = false; + let retainedFollowupCommentId: string | undefined; let reviewHandoffBoundary: { childId: string; assigneeAgentId: string | null | undefined; @@ -280,7 +286,7 @@ export async function runEverydayFlow(input: Input) { ), ]; } - async function reply(message: string, target: StoryIssue = parent!) { + async function reply(message: string, target: StoryIssue = parent!, transport: "browser" | "task_comment_api" = "browser") { const priorFailures = ev.checks.filter((c) => !c.passed); if (priorFailures.length) throw new Error( @@ -288,7 +294,9 @@ export async function runEverydayFlow(input: Input) { ); const before = await api.get(`/api/issues/${target.id}/comments`); lastSubmissionAt = Date.now(); - await submitTaskReply(page, message); + const submitted = transport === "task_comment_api" + ? await submitRetainedControllerComment(api, target.id, message, randomUUID()) + : (await submitTaskReply(page, message), undefined); const after = await pollUntil({ label: "one persisted user reply", deadlineAt: Date.now() + 30_000, @@ -303,11 +311,12 @@ export async function runEverydayFlow(input: Input) { ); check( `reply-${ev.timeline.length}-stored-once`, - added.length === 1, - "A composer submission creates exactly one user message.", + transport === "task_comment_api" ? onePersistedRetainedControllerComment({ added, commentId: submitted?.id ?? "", body: message }) : added.length === 1, + transport === "task_comment_api" ? "The production comment API creates exactly one matching user message." : "A composer submission creates exactly one user message.", ); submittedCommentIds.push(...added.map((c) => c.id)); - note("composer-message-persisted", { + if (submitted) retainedFollowupCommentId = submitted.id; + note(transport === "task_comment_api" ? "task-api-message-persisted" : "composer-message-persisted", { issueId: target.id, commentIds: added.map((c) => c.id), }); @@ -324,7 +333,7 @@ export async function runEverydayFlow(input: Input) { sameConnections: false, }; } - async function settled(expectedAgentReply?: string) { + async function settled(expectedAgentReply?: StoryReplyExpectation, readback: "browser" | "api" = "browser") { const settledState = await pollUntil({ label: `everyday ${caseId} settled`, deadlineAt: input.deadlineAt, @@ -389,16 +398,14 @@ export async function runEverydayFlow(input: Input) { if (expectedAgentReply) { const issue = settledState.issues.find((candidate) => candidate.id === parent?.id); const reply = issue?.comments?.find( - (comment) => - comment.authorAgentId === fixtures.agent.id && - String(comment.body ?? "").includes(expectedAgentReply), + (comment) => storyAgentReplyMatches(comment, fixtures.agent.id, expectedAgentReply), ); if (reply) { // Keep the exact snapshot that satisfied the readiness predicate. A // later refresh may return a newer projection and must not change the // evidence used by the assertion below. settledAgentReply = { ...reply }; - note("expected-agent-reply-visible-at-settlement", { + note(readback === "api" ? "expected-agent-reply-persisted-at-settlement" : "expected-agent-reply-visible-at-settlement", { issueId: issue?.id, commentId: reply.id, authorAgentId: reply.authorAgentId, @@ -407,14 +414,16 @@ export async function runEverydayFlow(input: Input) { }); } } - await openParent(); - await expect( - page.getByTestId("issue-detail-header").getByRole("button", { - name: "Change status (current: Done)", - exact: true, - }), - ).toBeVisible(); - note("all-tasks-done"); + if (readback === "browser") { + await openParent(); + await expect( + page.getByTestId("issue-detail-header").getByRole("button", { + name: "Change status (current: Done)", + exact: true, + }), + ).toBeVisible(); + } + note("all-tasks-done", { readback }); } async function downloadDelegated( mode: "base" | "separator" | "max-length", @@ -574,6 +583,7 @@ export async function runEverydayFlow(input: Input) { "everyday-delivery.ts", "everyday-observations.ts", "everyday-remote-restart.ts", + "retained-controller-continuation.ts", "controller-outage.ts", "controller-recovery-observer.ts", "upgrade-artifacts.ts", @@ -1440,13 +1450,39 @@ export async function runEverydayFlow(input: Input) { artifactGradeModeForPhase("reviewed-delivery"), "reviewed-delivery", ); - } else if (caseId === "recover-controller") + } else if (caseId === "recover-controller") { await download( parent!.id, artifactGradeModeForPhase("recovered-delivery"), "recovered-delivery", ); - else if (caseId === "stop-redirect") + if (execution.suite.definitionMetadata?.retainedControllerContinuation === true) { + const prior = ev.runs.filter(run => run.status === "succeeded" && run.agentId === fixtures.agent.id) + .sort((a, b) => Date.parse(b.finishedAt ?? "") - Date.parse(a.finishedAt ?? ""))[0]; + if (!prior) throw new Error("Retained continuation requires a completed source run"); + const before = retainedControllerRun(prior), priorIds = new Set(ev.runs.map(run => run.id)); + const outage = await input.restart(); + const followupTransport = retainedControllerFollowupTransport(execution); + postControllerApiFollowup = followupTransport === "task_comment_api"; + if (!postControllerApiFollowup) await openParent(); + note("retained-controller-followup-entry", { transport: followupTransport, + browserNavigationVerified: !postControllerApiFollowup }); + const requestedAt = new Date().toISOString(), marker = `CONTINUED_${nonce}`; + const expectedReply = retainedControllerReplyMatcher(marker); + await reply(`Run the existing delivered slugify.py on " Queue--Ready!! " with --separator _ and --max-length 7. Report its output after the marker ${marker}. Use the saved project; do not rebuild it.`, parent!, followupTransport); + await settled(expectedReply, postControllerApiFollowup ? "api" : "browser"); + const after = ev.runs.filter(run => !priorIds.has(run.id) && run.status === "succeeded" && run.agentId === fixtures.agent.id) + .sort((a, b) => Date.parse(b.finishedAt ?? "") - Date.parse(a.finishedAt ?? ""))[0]; + if (!after) throw new Error("Retained continuation produced no new successful run"); + const events = await collectRunEvents((afterSeq, limit) => api.get(`/api/heartbeat-runs/${after.id}/events?afterSeq=${afterSeq}&limit=${limit}`)); + const evidence = { before, after: retainedControllerRun(after), outage: outage || undefined, requestedAt, + followupTransport, followupCommentId: retainedFollowupCommentId, + completedTool: events.some(row => row.payload?.prpEvent?.eventType === "tool.execution.completed" && row.payload.prpEvent.payload?.status === "completed"), + usefulReply: storyAgentReplyMatches(settledAgentReply, fixtures.agent.id, expectedReply) }; + ev.checks.push(...gradeRetainedControllerContinuation(evidence)); + await input.evidence("retained-controller-continuation.json", evidence); + } + } else if (caseId === "stop-redirect") check( "new-direction-delivered", settledAgentReply?.authorAgentId === fixtures.agent.id && @@ -1480,7 +1516,7 @@ export async function runEverydayFlow(input: Input) { ev.issues.find((i) => i.id === parent!.id)!.documents = bodies; } } - await page.reload(); + if (!postControllerApiFollowup) await page.reload(); await refresh(); check( "submitted-replies-consumed", @@ -1575,25 +1611,30 @@ export async function runEverydayFlow(input: Input) { ), "No completion confirmation or unanswered interaction remains.", ); - if (providerChoice || nativeProviderCase) await openParent(); - else { - await waitForTaskIdentity(page, taskUrl(parent!), parent!.identifier ?? parent!.id); - await waitForTaskChatRendered(page); + if (!postControllerApiFollowup) { + if (providerChoice || nativeProviderCase) await openParent(); + else { + await waitForTaskIdentity(page, taskUrl(parent!), parent!.identifier ?? parent!.id); + await waitForTaskChatRendered(page); + } + const latestAgentComment = ev.issues + .find((i) => i.id === parent!.id) + ?.comments?.filter((c: Row) => c.authorAgentId) + .at(-1); + if (latestAgentComment && !providerChoice && !nativeProviderCase) { + const response = page.locator(`[id="comment-${latestAgentComment.id}"]`); + await expect(response).toBeVisible(); + await response.scrollIntoViewIfNeeded(); + } + await input.capture( + "final-state", + "Finished everyday workflow", + "final-state.png", + ); + } else { + note("final-state-api-readback", { browserNavigationVerified: false, renderedStatusVerified: false }); + await input.capture("final-state", "Browser state retained after API-verified native continuation; browser reconnection unqualified", "final-state.png"); } - const latestAgentComment = ev.issues - .find((i) => i.id === parent!.id) - ?.comments?.filter((c: Row) => c.authorAgentId) - .at(-1); - if (latestAgentComment && !providerChoice && !nativeProviderCase) { - const response = page.locator(`[id="comment-${latestAgentComment.id}"]`); - await expect(response).toBeVisible(); - await response.scrollIntoViewIfNeeded(); - } - await input.capture( - "final-state", - "Finished everyday workflow", - "final-state.png", - ); const failed = ev.checks.filter((c) => !c.passed); if (failed.length) throw new Error( diff --git a/tests/runner-e2e/everyday-observations.ts b/tests/runner-e2e/everyday-observations.ts index 40ab35e4a44..798b85e5899 100644 --- a/tests/runner-e2e/everyday-observations.ts +++ b/tests/runner-e2e/everyday-observations.ts @@ -344,17 +344,34 @@ export function storyRepliesConsumed( ); } +export type StoryReplyExpectation = string | ((body: string) => boolean); + +export function storyAgentReplyMatches( + comment: StoryComment | undefined, + agentId: string, + expected: StoryReplyExpectation, +): boolean { + if (!comment || comment.authorAgentId !== agentId) return false; + const body = String(comment.body ?? ""); + return typeof expected === "string" ? body.includes(expected) : expected(body); +} + +/** The requested output is one exact token after this continuation's marker. */ +export function retainedControllerReplyMatcher(marker: string): (body: string) => boolean { + const escaped = marker.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + const pattern = new RegExp(`(?:^|\\s)${escaped}\\s+queue_r(?=\\s|$)`); + return body => pattern.test(body); +} + /** A terminal task is only ready for response assertions after its agent reply is visible. */ export function storyHasAgentReply( issue: StoryIssue | undefined, agentId: string, - expectedText: string, + expectedText: StoryReplyExpectation, ): boolean { return ( issue?.comments?.some( - (comment) => - comment.authorAgentId === agentId && - String(comment.body ?? "").includes(expectedText), + (comment) => storyAgentReplyMatches(comment, agentId, expectedText), ) ?? false ); } diff --git a/tests/runner-e2e/everyday.test.ts b/tests/runner-e2e/everyday.test.ts index 88cbecddd72..c9880edaa23 100644 --- a/tests/runner-e2e/everyday.test.ts +++ b/tests/runner-e2e/everyday.test.ts @@ -9,6 +9,8 @@ import { storyLifecycleChecks, storyRepliesConsumed, storyHasAgentReply, + storyAgentReplyMatches, + retainedControllerReplyMatcher, storyHasPendingAgentReview, storyHasPendingAgentWake, storyHasPendingAgentWork, @@ -385,6 +387,49 @@ describe("lifecycle oracle calibrated failures", () => { }); describe("reply completion boundary", () => { + it("accepts an explicit reply matcher for continuation output on a new line", () => { + const issue = { id: "parent", companyId: "company", title: "Continue", status: "done", + comments: [{ authorAgentId: "lead", body: "CONTINUED_nonce\nqueue_r" }] }; + const expected = (body: string) => body === "CONTINUED_nonce\nqueue_r"; + expect(storyHasAgentReply(issue, "lead", expected)).toBe(true); + }); + + it.each([" ", "\n", "\r\n", "\t", " \n"])("accepts continuation output separated by %j", separator => { + const expected = retainedControllerReplyMatcher("CONTINUED_nonce.1"); + const comment = { authorAgentId: "lead", body: `CONTINUED_nonce.1${separator}queue_r` }; + expect(storyHasAgentReply({ id: "task", companyId: "company", title: "Continue", status: "done", comments: [comment] }, "lead", expected)).toBe(true); + expect(storyAgentReplyMatches(comment, "lead", expected)).toBe(true); + }); + + it.each(["\nI used the saved project.", " I used the saved project.", "\r\nqueue_ready"])("accepts explanation after the exact continuation token %j", suffix => { + const expected = retainedControllerReplyMatcher("CONTINUED_nonce.1"); + const comment = { authorAgentId: "lead", body: `CONTINUED_nonce.1\nqueue_r${suffix}` }; + expect(storyHasAgentReply({ id: "task", companyId: "company", title: "Continue", status: "done", comments: [comment] }, "lead", expected)).toBe(true); + expect(storyAgentReplyMatches(comment, "lead", expected)).toBe(true); + }); + + it.each([ + "CONTINUED_nonce.1 queue_ready", "CONTINUED_nonce.1 queue_r_extra", "CONTINUED_nonce.1 wrong", + "CONTINUED_other queue_r", "CONTINUED_nonceX1 queue_r", "queue_r", "CONTINUED_nonce.1queue_r", + "OTHER_CONTINUED_nonce.1 queue_r", "CONTINUED_nonce.1 queue_r-extra", "CONTINUED_nonce.1 queue_r.", + ])("rejects an incorrect continuation reply %j", body => { + const expected = retainedControllerReplyMatcher("CONTINUED_nonce.1"); + const comment = { authorAgentId: "lead", body }; + expect(storyHasAgentReply({ id: "task", companyId: "company", title: "Continue", status: "done", comments: [comment] }, "lead", expected)).toBe(false); + expect(storyAgentReplyMatches(comment, "lead", expected)).toBe(false); + }); + + it("preserves literal string expectations and continuation author checks", () => { + const expected = retainedControllerReplyMatcher("CONTINUED_nonce"); + const comment = { authorAgentId: "other", body: "CONTINUED_nonce\nqueue_r" }; + expect(storyAgentReplyMatches(comment, "lead", expected)).toBe(false); + const issue = { id: "task", companyId: "company", title: "Continue", status: "done", comments: [comment] }; + expect(storyHasAgentReply(issue, "lead", expected)).toBe(false); + expect(storyAgentReplyMatches({ ...comment, authorAgentId: "lead" }, "lead", "CONTINUED_nonce queue_r")).toBe(false); + expect(storyAgentReplyMatches({ authorAgentId: "lead", body: "a.b" }, "lead", "a.b")).toBe(true); + expect(storyAgentReplyMatches({ authorAgentId: "lead", body: "axb" }, "lead", "a.b")).toBe(false); + }); + const run: StoryRun = { id: "old", companyId: "company", diff --git a/tests/runner-e2e/external-controller-database.test.ts b/tests/runner-e2e/external-controller-database.test.ts new file mode 100644 index 00000000000..5570b38d6fb --- /dev/null +++ b/tests/runner-e2e/external-controller-database.test.ts @@ -0,0 +1,52 @@ +import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { expect, it } from "vitest"; +import { createDb } from "../../packages/db/src/index.js"; +import { createExternalControllerDatabase } from "./external-controller-database.js"; +import { assertEmbeddedDatabaseIsolation } from "./instance-isolation.js"; +import { createRunnerE2ETemporaryRoot } from "./server-config.js"; +import { buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment } from "./harness-env.js"; + +it("keeps the database administrator out of both child environments", () => { + const source = { PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL: "postgres://test-secret@localhost/postgres" }; + expect(buildRunnerE2EProcessEnvironment(source, []).PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL).toBeUndefined(); + expect(buildPaperclipServerEnvironment(source).PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL).toBeUndefined(); +}); + +it("rejects a nonlocal administrator before creating a fixture", async () => { + await expect(createExternalControllerDatabase({ adminUrl: "postgres://example.invalid/postgres", + temporaryRoot: "/unused", configPath: "/unused/config.json", serverPort: 3100 })) + .rejects.toThrow("explicit loopback test server"); +}); + +it.skipIf(!process.env.PAPERCLIP_TEST_DATABASE_URL)("owns one empty database, rejects a changed binding, and deletes only that database", async () => { + const root = await createRunnerE2ETemporaryRoot(), configPath = path.join(root, "instance", "config.json"); + const admin = createDb(process.env.PAPERCLIP_TEST_DATABASE_URL!, { maxConnections: 1 }); + let fixture: Awaited> | undefined; + let client: ReturnType | undefined; + try { + fixture = await createExternalControllerDatabase({ adminUrl: process.env.PAPERCLIP_TEST_DATABASE_URL!, + temporaryRoot: root, configPath, serverPort: 3100 }); + const config = JSON.parse(await readFile(configPath, "utf8")); + const name = new URL(config.database.connectionString).pathname.slice(1); + expect(name).toMatch(/^runner_e2e_[a-f0-9]{32}$/); + client = createDb(config.database.connectionString); + expect(await client.$client`select tablename from pg_tables where schemaname = 'public'`).toEqual([]); + await mkdir(path.dirname(config.secrets.localEncrypted.keyFilePath), { recursive: true }); + await writeFile(config.secrets.localEncrypted.keyFilePath, "fixture-key"); + await expect(assertEmbeddedDatabaseIsolation(configPath, root)).resolves.toBeUndefined(); + config.database.connectionString = process.env.PAPERCLIP_TEST_DATABASE_URL; + await writeFile(configPath, JSON.stringify(config)); + await expect(assertEmbeddedDatabaseIsolation(configPath, root)).rejects.toThrow("fixture-owned database"); + await client.$client.end(); client = undefined; + await fixture.cleanup(); + await fixture.cleanup(); + expect(await admin.$client`select datname from pg_database where datname=${name}`).toEqual([]); + expect(await admin.$client`select current_database() as name`).toHaveLength(1); + } finally { + await client?.$client.end({ timeout: 1 }); + await fixture?.cleanup(); + await admin.$client.end({ timeout: 1 }); + await rm(root, { recursive: true, force: true }); + } +}); diff --git a/tests/runner-e2e/external-controller-database.ts b/tests/runner-e2e/external-controller-database.ts new file mode 100644 index 00000000000..5a538bd1f80 --- /dev/null +++ b/tests/runner-e2e/external-controller-database.ts @@ -0,0 +1,47 @@ +/** Launcher-owned disposable database. The admin credential never enters the + * server/browser environment. Unknown remote cleanup keeps this database alive. */ +import { randomUUID } from "node:crypto"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { createDb } from "../../packages/db/src/index.js"; +import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js"; +import { recordControllerDatabaseOwnership } from "./instance-isolation.js"; +import { prepareRunnerE2EServerConfig } from "./server-config.js"; + +export async function createExternalControllerDatabase(input: { + adminUrl: string; temporaryRoot: string; configPath: string; serverPort: number; +}) { + const url = new URL(input.adminUrl); + if (!["postgres:", "postgresql:"].includes(url.protocol) + || !["127.0.0.1", "localhost", "[::1]"].includes(url.hostname)) { + throw new Error("E2E Postgres administrator must use an explicit loopback test server"); + } + await (await prepareRunnerE2EServerConfig(input))?.close(); + const config = paperclipConfigSchema.parse(JSON.parse(await readFile(input.configPath, "utf8"))); + if (config.database.mode !== "embedded-postgres") throw new Error("E2E database fixture already configured"); + const databaseName = `runner_e2e_${randomUUID().replaceAll("-", "")}`; + const admin = createDb(input.adminUrl, { maxConnections: 1 }); + let created = false; + const cleanup = async () => { + if (!created) return; + const connection = createDb(input.adminUrl, { maxConnections: 1 }); + try { + // No FORCE: a remaining controller keeps its evidence and fails cleanup. + await connection.$client.unsafe(`drop database "${databaseName}"`); + created = false; + } finally { await connection.$client.end({ timeout: 1 }); } + }; + try { + await admin.$client.unsafe(`create database "${databaseName}"`); + created = true; + url.pathname = `/${databaseName}`; + config.database = { ...config.database, mode: "postgres", connectionString: url.toString() }; + await mkdir(config.database.embeddedPostgresDataDir, { recursive: true, mode: 0o700 }); + await recordControllerDatabaseOwnership({ ...input, dataDirectory: config.database.embeddedPostgresDataDir, + connectionString: url.toString(), externalDatabaseName: databaseName }); + await writeFile(input.configPath, `${JSON.stringify(config, null, 2)}\n`, { mode: 0o600 }); + return { cleanup }; + } catch (error) { + await cleanup(); + throw error; + } finally { await admin.$client.end({ timeout: 1 }); } +} diff --git a/tests/runner-e2e/harness-env.ts b/tests/runner-e2e/harness-env.ts index 0f9de2d0c3f..b33e3f2125b 100644 --- a/tests/runner-e2e/harness-env.ts +++ b/tests/runner-e2e/harness-env.ts @@ -136,6 +136,7 @@ export function buildRunnerE2EProcessEnvironment( const installed = usesInstalledCli(source); if (installed) assertInstalledCliSelection(source, executions); const result = { ...source }; + delete result.PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL; // The supervisor's read-only recovery observer must use the same disposable // instance key as the serving child, never a launching shell's vault key. delete result.PAPERCLIP_SECRETS_MASTER_KEY; @@ -209,6 +210,7 @@ export function buildPaperclipServerEnvironment( overrides: NodeJS.ProcessEnv = {}, ): NodeJS.ProcessEnv { const result = { ...source }; + delete result.PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL; for (const key of Object.keys(result)) { if (PROVIDER_SECRET_KEY.test(key)) delete result[key]; } diff --git a/tests/runner-e2e/installed-cli.test.ts b/tests/runner-e2e/installed-cli.test.ts index 7b917353ba3..7f3074e0fd8 100644 --- a/tests/runner-e2e/installed-cli.test.ts +++ b/tests/runner-e2e/installed-cli.test.ts @@ -32,7 +32,7 @@ it("admits all26 current Pi cases through pinned installed public CLI/server wit }); it("admits existing Codex onboarding, useful-file revision and restart journeys without broadening other cells", async () => { const { env, server } = await fixture(); - for (const id of ["first-task.runner-codex.local.task-card-accept", "everyday-workflows.runner-codex.local.build-revise", "agent-chat.runner-codex.local.continuity-restart", "everyday-workflows.runner-codex.daytona.recover-controller"]) { + for (const id of ["first-task.runner-codex.local.task-card-accept", "everyday-workflows.runner-codex.local.build-revise", "agent-chat.runner-codex.local.continuity-restart", "everyday-workflows.runner-codex.daytona.recover-controller", "daytona-restart-continuity.runner-codex.daytona.recover-controller"]) { const cell = runnerMatrix.find(e => e.id === id)!; expect(cell).toBeDefined(); expect(await verifyInstalledCli(env, [cell])).toMatchObject({ serverRoot: server, defaultRuntimeResolution: true, qualificationOverride: false }); diff --git a/tests/runner-e2e/installed-cli.ts b/tests/runner-e2e/installed-cli.ts index 0e5fe3c55c5..820577869a5 100644 --- a/tests/runner-e2e/installed-cli.ts +++ b/tests/runner-e2e/installed-cli.ts @@ -17,6 +17,7 @@ const installedCodexCells = new Set([ "everyday-workflows.runner-codex.local.build-revise", "agent-chat.runner-codex.local.continuity-restart", "everyday-workflows.runner-codex.daytona.recover-controller", + "daytona-restart-continuity.runner-codex.daytona.recover-controller", ]); const installedCopilotCells = new Set([ "copilot-protection.runner-acpx-copilot.daytona.attached-async-settlement", diff --git a/tests/runner-e2e/instance-isolation.ts b/tests/runner-e2e/instance-isolation.ts index 9bd98b60a3e..1c2117dbdcd 100644 --- a/tests/runner-e2e/instance-isolation.ts +++ b/tests/runner-e2e/instance-isolation.ts @@ -6,6 +6,7 @@ const CONTROLLER_DATABASE_OWNER_FILE = ".runner-e2e-controller-owner.json"; /** Written only after the supervisor has started this fixture's Postgres. */ export async function recordControllerDatabaseOwnership(input: { configPath: string; temporaryRoot: string; dataDirectory: string; connectionString: string; + externalDatabaseName?: string; }) { const temporaryRoot = await realpath(input.temporaryRoot); const dataDirectory = await realpath(input.dataDirectory); @@ -13,7 +14,8 @@ export async function recordControllerDatabaseOwnership(input: { throw new Error("Controller database path escaped its fixture-owned root"); } await writeFile(path.join(dataDirectory, CONTROLLER_DATABASE_OWNER_FILE), JSON.stringify({ - version: 1, configPath: await realpath(input.configPath), temporaryRoot, + version: input.externalDatabaseName ? 2 : 1, externalDatabaseName: input.externalDatabaseName, + configPath: await realpath(input.configPath), temporaryRoot, dataDirectory, connectionString: input.connectionString, }), { mode: 0o600 }); } @@ -82,12 +84,16 @@ export async function assertEmbeddedDatabaseIsolation( if (!directory.startsWith(`${root}${path.sep}`)) throw new Error("Database escaped fixture"); const owner = JSON.parse(await readFile(path.join(directory, CONTROLLER_DATABASE_OWNER_FILE), "utf8")); const port = config.database.embeddedPostgresPort; - const expectedConnection = `postgres://paperclip:paperclip@127.0.0.1:${port}/paperclip`; - if (!Number.isInteger(port) || Number(port) < 1 || Number(port) > 65535 - || config.database.connectionString !== expectedConnection - || owner.version !== 1 || owner.temporaryRoot !== root || owner.dataDirectory !== directory - || owner.configPath !== await realpath(configPath) || owner.connectionString !== expectedConnection - || !(await stat(path.join(directory, "PG_VERSION"))).isFile()) { + const connection = new URL(config.database.connectionString!); + const external = owner.version === 2 && /^runner_e2e_[a-f0-9]{32}$/.test(owner.externalDatabaseName) + && ["postgres:", "postgresql:"].includes(connection.protocol) + && ["127.0.0.1", "localhost", "[::1]"].includes(connection.hostname) + && connection.pathname === `/${owner.externalDatabaseName}`; + const embedded = owner.version === 1 && Number.isInteger(port) && Number(port) > 0 && Number(port) <= 65535 + && config.database.connectionString === `postgres://paperclip:paperclip@127.0.0.1:${port}/paperclip` + && (await stat(path.join(directory, "PG_VERSION"))).isFile(); + if ((!external && !embedded) || owner.temporaryRoot !== root || owner.dataDirectory !== directory + || owner.configPath !== await realpath(configPath) || owner.connectionString !== config.database.connectionString) { throw new Error("Supervisor database ownership mismatch"); } } catch (cause) { diff --git a/tests/runner-e2e/launch.ts b/tests/runner-e2e/launch.ts index 215cbd3a863..3a278d79dbe 100644 --- a/tests/runner-e2e/launch.ts +++ b/tests/runner-e2e/launch.ts @@ -10,6 +10,7 @@ import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js"; import { assertInstalledCliSelection, assertInstalledStartupOnly, verifyInstalledCli } from "./installed-cli.js"; import { createProcessTreeOwner, stopOwnedProcessTree } from "./process-tree-owner.js"; import { createRunnerE2ETemporaryRoot } from "./server-config.js"; +import { createExternalControllerDatabase } from "./external-controller-database.js"; import { createHash, randomBytes } from "node:crypto"; import { prepareCodexCiSandbox, requiresCodexCiSandbox } from "./codex-ci-sandbox.js"; import { spawn } from "node:child_process"; @@ -508,6 +509,7 @@ async function runAttempt(input: { let attemptSecrets: string[] = []; let processCleanupFailed = false; let startupReceiptPath: string | undefined; + let externalDatabase: Awaited> | undefined; let gateway: Awaited> | undefined; const rawCleanupResults: Array<{ cleanup: string; synthetic: boolean; status: string }> = []; try { @@ -521,6 +523,11 @@ async function runAttempt(input: { "config.json", ); const port = await reserveRunnerE2EServerPort(); + if (process.env.PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL) { + externalDatabase = await createExternalControllerDatabase({ + adminUrl: process.env.PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL, temporaryRoot, configPath, serverPort: port, + }); + } await Promise.all([ mkdir(paperclipHome, { recursive: true }), mkdir(workspace, { recursive: true }), @@ -891,7 +898,11 @@ async function runAttempt(input: { let cleanupError: unknown; const resourceAdmissionStarted = await access(path.join(privateDir, "resource-admission-started")) .then(() => true).catch((error: NodeJS.ErrnoException) => error.code !== "ENOENT"); - if (mustPreserveRecoveryState({ processCleanupFailed, resourceAdmissionStarted, results: rawCleanupResults })) { + const preserveRecovery = mustPreserveRecoveryState({ processCleanupFailed, resourceAdmissionStarted, results: rawCleanupResults }); + if (!preserveRecovery) { + try { await externalDatabase?.cleanup(); } catch (error) { cleanupError = error; } + } + if (preserveRecovery || cleanupError) { // Remote allocation cleanup is journaled in this instance's database. // Deleting it after the controller exits would strand uncertain creates. await chmod(temporaryRoot, 0o700); diff --git a/tests/runner-e2e/retained-controller-continuation.test.ts b/tests/runner-e2e/retained-controller-continuation.test.ts new file mode 100644 index 00000000000..a9636dfb06e --- /dev/null +++ b/tests/runner-e2e/retained-controller-continuation.test.ts @@ -0,0 +1,65 @@ +import { describe, expect, it, vi } from "vitest"; +import { gradeRetainedControllerContinuation, retainedControllerRun, retainedControllerFollowupTransport, submitRetainedControllerComment, onePersistedRetainedControllerComment } from "./retained-controller-continuation.js"; + +import { runnerMatrix } from "./catalog.js"; + +function fixture() { + const before = retainedControllerRun({ id: "a", companyId: "company", agentId: "agent", nativeIssueId: "task", + runtimeMode: "native", status: "succeeded", nativeSessionId: "private-native-session", sessionIdAfter: "private-provider-session", + finishedAt: "2026-10-30T00:00:00Z" }); + return { before, after: { ...before, runId: "b", finishedAt: "2026-10-30T00:01:00Z" }, + outage: { schema: "paperclip.e2e.controller-outage/v1" as const, requestId: "retained", signal: "SIGKILL" as const, + holdMs: 1000, startedAt: "2026-10-30T00:00:01Z", previousPid: 1, replacementPid: 2, + stoppedAfterMs: 0, replacementStartedAfterMs: 1000, readyAfterMs: 2000, cacheDiscarded: true }, + requestedAt: "2026-10-30T00:00:04Z", completedTool: true, usefulReply: true }; +} + +describe("retained controller continuation evidence", () => { + it("accepts a new useful turn after a completed source and cache retirement", () => { + expect(gradeRetainedControllerContinuation(fixture())[0]?.passed).toBe(true); + expect(JSON.stringify(fixture())).not.toContain("private-provider-session"); + }); + it.each(["provider", "native", "same-run", "before-completion", "no-cache-loss", "no-tool", "no-answer"])("rejects %s evidence", fault => { + const input = fixture(); + if (fault === "provider") input.after.providerSessionSha256 = "different"; + if (fault === "native") input.after.nativeSessionSha256 = null; + if (fault === "same-run") input.after.runId = input.before.runId; + if (fault === "before-completion") input.before.finishedAt = "2026-10-30T00:00:10Z"; + if (fault === "no-cache-loss") input.outage.cacheDiscarded = false; + if (fault === "no-tool") input.completedTool = false; + if (fault === "no-answer") input.usefulReply = false; + expect(gradeRetainedControllerContinuation(input)[0]?.passed).toBe(false); + }); +}); + +describe("post-controller API follow-up selection", () => { + const execution = runnerMatrix.find(cell => cell.id === "daytona-restart-continuity.runner-codex.daytona.recover-controller")!; + it("selects only the explicit dedicated definition and defaults to the browser", () => { + expect(retainedControllerFollowupTransport(execution)).toBe("task_comment_api"); + for (const changed of [ + { ...execution, suite: { ...execution.suite, id: "everyday-workflows" } }, + { ...execution, task: { ...execution.task, id: "another-case" } }, + { ...execution, suite: { ...execution.suite, definitionMetadata: { retainedControllerContinuation: true } } }, + { ...execution, suite: { ...execution.suite, definitionMetadata: { retainedControllerFollowupTransport: "task_comment_api" } } }, + ]) expect(retainedControllerFollowupTransport(changed)).toBe("browser"); + }); + it("posts one ordinary authenticated comment request with a stable client request ID", async () => { + const post = vi.fn(async () => ({ id: "saved-user-comment" })); + const api = { post } as unknown as Parameters[0]; + await expect(submitRetainedControllerComment(api, "task-id", "Exact original follow-up.", "request-id")) + .resolves.toEqual({ id: "saved-user-comment" }); + expect(post).toHaveBeenCalledExactlyOnceWith("/api/issues/task-id/comments", { + body: "Exact original follow-up.", reopen: true, clientRequestId: "request-id", + }); + }); + it("requires exactly the matching persisted user comment", () => { + const comment = { id: "comment", body: "Exact follow-up", authorType: "user" }; + const proof = { added: [comment], commentId: comment.id, body: comment.body }; + expect(onePersistedRetainedControllerComment(proof)).toBe(true); + expect(onePersistedRetainedControllerComment({ ...proof, commentId: "" })).toBe(false); + for (const added of [[], [comment, { ...comment, id: "duplicate" }], [{ ...comment, id: "other" }], + [{ ...comment, body: "changed" }], [{ ...comment, authorType: "system" }], [{ ...comment, authorAgentId: "agent" }]]) { + expect(onePersistedRetainedControllerComment({ ...proof, added })).toBe(false); + } + }); +}); diff --git a/tests/runner-e2e/retained-controller-continuation.ts b/tests/runner-e2e/retained-controller-continuation.ts new file mode 100644 index 00000000000..d0f5fa4bb31 --- /dev/null +++ b/tests/runner-e2e/retained-controller-continuation.ts @@ -0,0 +1,59 @@ +import type { RunnerApi } from "./api.js"; +import type { MatrixExecution } from "./types.js"; +import { createHash } from "node:crypto"; +import type { StoryCheck, StoryRun } from "./everyday-observations.js"; +import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js"; + +/** Only this explicit definition changes the post-replacement user entry point. */ +export function retainedControllerFollowupTransport(execution: Pick): "browser" | "task_comment_api" { + return execution.suite.id === "daytona-restart-continuity" && execution.task.id === "recover-controller" + && execution.suite.definitionMetadata?.retainedControllerContinuation === true + && execution.suite.definitionMetadata.retainedControllerFollowupTransport === "task_comment_api" + ? "task_comment_api" : "browser"; +} + +export async function submitRetainedControllerComment(api: Pick, issueId: string, body: string, clientRequestId: string) { + return api.post<{ id: string }>(`/api/issues/${issueId}/comments`, { body, reopen: true, clientRequestId }); +} + +export function onePersistedRetainedControllerComment(input: { + added: Array<{ id?: unknown; body?: unknown; authorAgentId?: unknown; authorType?: unknown }>; + commentId: string; body: string; +}) { + return typeof input.commentId === "string" && input.commentId.length > 0 + && input.added.length === 1 && input.added[0]?.id === input.commentId + && input.added[0].body === input.body && !input.added[0].authorAgentId && input.added[0].authorType === "user"; +} + +const hash = (value: string | null | undefined) => value ? createHash("sha256").update(value).digest("hex") : null; + +/** Public evidence keeps provider/session identifiers hashed. */ +export function retainedControllerRun(run: StoryRun) { + return { runId: run.id, companyId: run.companyId, agentId: run.agentId, issueId: run.nativeIssueId, + status: run.status, runtimeMode: run.runtimeMode, finishedAt: run.finishedAt, + nativeSessionSha256: hash(run.nativeSessionId), providerSessionSha256: hash(run.sessionIdAfter) }; +} + +export function gradeRetainedControllerContinuation(input: { + before: ReturnType; + after: ReturnType; + outage: ControllerOutageReceipt | undefined; + requestedAt: string; + completedTool: boolean; + usefulReply: boolean; +}): StoryCheck[] { + const { before, after, outage } = input; + const passed = validControllerOutageReceipt(outage) + && before.status === "succeeded" && after.status === "succeeded" + && before.runtimeMode === "native" && after.runtimeMode === "native" + && before.runId !== after.runId && before.companyId === after.companyId + && before.agentId === after.agentId && before.issueId === after.issueId + && Boolean(before.nativeSessionSha256) && before.nativeSessionSha256 === after.nativeSessionSha256 + && Boolean(before.providerSessionSha256) && before.providerSessionSha256 === after.providerSessionSha256 + && Date.parse(before.finishedAt ?? "") <= Date.parse(outage?.startedAt ?? "") + && Date.parse(input.requestedAt) >= Date.parse(outage?.startedAt ?? "") + (outage?.readyAfterMs ?? Infinity) + && Date.parse(after.finishedAt ?? "") >= Date.parse(input.requestedAt) + && input.completedTool && input.usefulReply; + return [{ id: "retained-controller-continuation", passed, + detail: "After completion and another empty-cache replacement, a new native turn must use the same retained provider session and execute the saved project for a useful reply." }]; +} diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 03739da45de..988ce11cb53 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1253,12 +1253,15 @@ for (const execution of executions) { const story = await runEverydayFlow({ page, api, fixtures, execution, nonce, workspacePath, privateDir, deadlineAt: startedAtMs + deadlineMs, - restartCount: upgradeManifest ? upgradeManifest.sequence.length - 1 : 1, + restartCount: upgradeManifest ? upgradeManifest.sequence.length - + (execution.suite.definitionMetadata?.retainedControllerContinuation === true ? 2 : 1) : 1, restart: async (recoveryBinding, controllerDowntimeMs) => { let receipt: ControllerOutageReceipt | undefined; restartIndex += 1; await restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}-${restartIndex}`, deadlineAt: startedAtMs + deadlineMs, - discardControllerCache: controllerOutage !== undefined, controllerOutage, controllerDowntimeMs, + discardControllerCache: controllerOutage !== undefined, + controllerOutage: controllerOutage && !recoveryBinding ? { ...controllerOutage, holdMs: 1000 } : controllerOutage, + controllerDowntimeMs, ...(controllerOutage ? { recoveryBinding } : {}), onControllerOutage: async value => { receipt = value;