DottaandPaperclip
492d537340
## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution runs in a remote Runner while the webserver owns its controller. > - Encrypted PRP checkpoints and session snapshots already survive a webserver restart in Postgres. > - Ordinary warm and per-turn admission still needs local markers and supervisor metadata to find that state. > - Replacing the webserver with an empty native controller directory can therefore lose a usable retained session. > - This change adds bounded recovery metadata to the existing snapshot transaction and reconstructs local projections from the selected durable source. > - The benefit is native execution continuity across webserver replacement, with the existing ownership and agent lifecycle policies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native remote controller recovery and retained-session continuation during a webserver upgrade. **Subsystem affected** server/ — native runtime, checkpoint persistence, recovery scheduling and internal upgrade reporting. Product E2E qualification also changes. **Current behavior** Without both the local PRP marker and supervisor file, normal continuation can return before discovering the saved Postgres checkpoint. The snapshot already exists in Postgres, but its local reuse compatibility envelope cannot be reconstructed. **Proposed behavior** Select the existing durable source, validate its identity and ownership, restore its PRP state and supervisor projection, then use normal authenticated admission. Missing or inconsistent evidence holds recovery. It does not authorize fresh provider work. **Reason and benefit** A replacement webserver must recover active execution and later continue its retained session without the old server's native files. This is one component of deployment continuity. HTTP traffic switching, workspace storage migration and Railway rollout remain separate. **Breaking changes** No new public endpoint, schema migration or database pool. The private run-profile envelope is additive. Disk-independent rollback also requires a reader for the staged controller format and compact checkpoint part; the first production release containing these readers is the floor. Older file-backed sessions need an owner checkpoint or the bounded internal importer. **Additional context** Related foundations: #17401, #17916, #17254, #14621, #14622. The implementation contract and open acceptance gates are in [the dated plan](doc/plans/2026-10-30-native-controller-checkpoints.md). ## What Changed - Persist a versioned recovery envelope, at most 4 KiB, beside the redacted session snapshot in its existing transaction. Keep it out of public run projections. - Compare frozen admission separately from prepared attachment instructions during staged recovery, preserving exact template/identity/authority checks. - Bound the existing lazy checkpoint pool at three connections after measured two-slot burst backlog; keep request/renewal capacities and SQL operation counts unchanged. The tuning is a separate commit. - Discover and restore the exact selected retained source without local native markers. Preserve the 32-candidate window, progress barriers and current authority checks. - Fence snapshot and result writes. Acquire coordinator, issue and run locks in order and check expiry after lock waits. Combine snapshot ownership/read work into one statement. - Reuse existing recovery deadlines and drain/resume scheduling. Retained-owner waits preserve the original unstarted run and actual lease deadline. Stop, ownership and provider-progress fences precede deferral. Add bounded native evidence reporting, retained-source import and retention protection. - Admit a settled warm source with an authenticated idle goal-read suffix while retaining active-work and mixed-authority holds. - Permit lifecycle `turn.stop` checkpoints after accepted results under existing ownership fences. Claim the existing successor coordinator before retiring the old warm session, and stage shared attachments only after verified retirement. Preserve omitted isolated-home defaults and existing fresh skill-selection order during recovery without hiding real configuration changes. - Carry the exact durable six-field PRP binding during active recovery after a healthy warm handoff. Keep the Runner mismatch guard and preserve opaque attachment IDs. - Preserve an already verified active or idle warm process during remote preparation; revalidate its proof/liveness and retain later authenticated PRP attachment. Ordinary resumed leases retain suspension checks. - Drain retained source events under their original authority, reconcile immutable source accounting and block unknown source tool dispatch before target work. - Persist compact staged handoff intent through authenticated adoption and recover direct/listener command-loss windows without another Runner or duplicate attachment. - Rebuild suspended lifecycle receipts for resumed prepared tasks, and retry missing lifecycle reads after partial PRP materialization without overwriting contradictory evidence. - Add normal warm/per-turn diskless continuation coverage, real Postgres races and SQL-cost checks. Add a reproducible paired load workload. - Give only the explicit outage fixture a 15-minute provider idle window, with a red/green catalog regression preserving all production timing and lifecycle policies. - Publish the goal-less task-session pointer in the existing post-acceptance terminal checkpoint transaction, with current ownership, accepted-result, reset and newer-progress fences. Keep a private bound receipt so late completion cannot duplicate or resurrect publication. - Use a retained warm sandbox in the dedicated continuity fixture, and accept an exact output token followed by an explanation. Keep the strict identity oracle. - Extend the explicit Daytona cell to a five-minute active outage and another replacement after completion. Require retained provider identity, a new tool receipt and a useful follow-up result. Add an owned loopback Postgres E2E fixture for hosts that cannot start embedded Postgres. ## Verification **Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`: ready for the qualified native-controller component.** Full CI, fresh review, local application paths, controlled performance qualification and the strict live Linux/Daytona case pass. Independent resource cleanup is complete. Broader rollout, browser reconnection and the unexecuted transport/platform combinations remain integration gates. Integrated 484 passed full local recursive typecheck/build and ordinary active→completion→cache-loss→C component qualification. Fresh review then found a valid attached-wake failure: preparation appends attachment instructions before the verifier compares execution to the frozen admission. 9f4 fixes that by separately checking the pre-preparation admission and exact prepared execution. Both hashes are captured privately per admission; the full template, identity, configuration and authority checks remain. Its regression fails before the fix; 36 focused cases, 42 staged source/target cases and server typecheck pass afterward. The review thread is resolved. Exact-head full CI passes ([run 41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822): 48 successful jobs, one intended skip); the complete `pnpm test:run` partitions are forced, with the five native suites in the Runner lane. This is not a claim that an unsharded local command passed. The fresh exact-head Greptile check succeeds with 5/5 and all six threads resolved. The original two-connection database comparison closes **steady PASS and burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all operations/ownership remain correct, but candidate arm 3 drains in 1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged 1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2 diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72 ms: three is the smallest tested passing cap, and the repeated two-cap failure survives run-order/cache effects. Separately reviewable `fc68584ce6` changes the existing lazy checkpoint pool default to three connections. Request capacity stays ten and liveness two; idle release, shutdown fencing, transaction counts and pool count are unchanged. Thirteen DB lifecycle/cap tests and DB typecheck pass. Full recursive typecheck/build pass at exactfc68584, with the matching server build stamp. The final paired native feature-cost comparison explicitly uses the **same cap-three policy and common current DB factory in both variants**; its baseline is controlled, not the unmodified historical cap-two configuration. Original gates and workload counts remain unchanged. The one strict Linux/Daytona case started at 04:40:56 UTC only after both new profiles passed; it passed on the first attempt with all 49 workflow checks. Independent cleanup confirms all three owned sandboxes absent, owned Postgres/controllers/observer gone, VM terminated, root volume absent and ephemeral SSH key/agent removed. **Final controlled cap-three qualification: PASS.** Baseline native implementation `59da15066a` and candidate `fc68584ce6` use the same current DB factory and checkpoint policy (request/liveness/checkpoint = 10/2/3). Eight 65-second ABBA arms cover 300 controllers and 10,000 historical tasks. All correctness, absolute and 10% relative gates pass; original cap-two failures remain unchanged. | Mode | DB CPU µs/op, baseline → candidate | WAL B/op, baseline → candidate | DB-request p99 ms, baseline → candidate | Maximum drain | | --- | --- | --- | --- | --- | | Steady | 542.40 → 554.83 (+2.29%) | 1509.01 → 1558.27 (+3.26%) | 107.11 → 111.90 (+4.48%) | 26.35 ms | | Burst | 1036.36 → 1063.11 (+2.58%) | 4153.12 → 4317.64 (+3.96%) | 103.736 → 103.841 (+0.10%) | 930.14 ms | Every matched arm preserves operation/statement/transaction counts, including 3,900 PRP and 3,900 snapshot transactions. All three actual pools are included; there are zero lost owners, missed lease deadlines, dropped operations or errors. Peak Postgres CPU is 18.31%/9.47% of two cores and memory 614.29/684.42 MiB (steady/burst). The benchmark uses one Node process simulating 300 controllers; it does not qualify two-webserver overlap, HTTP latency or hosted capacity. Its observer SQL contributes to measured database CPU. The original unmodified cap-two baseline is separately retained as failed; this cap-three comparison is explicitly controlled. The exact final-source 8/512/8,192-chunk test also passes: zero recovery SQL, transactions or changed rows in each text-only interval, zero idle SQL, seven lifecycle transactions and twelve changed rows per sample. Existing lease traffic remains separate. The final ordinary managed application replay passes active recovery, completion, native-cache removal and C continuation; its UNKNOWN local debug-daemon provenance and partial instruction-observer limits remain recorded. The live case uses the attested Linux producer instead. **Live qualification: PASS, one attempt, no retries.** Campaign `native-controller-retained-fix-fc68584ce6-ec2-20261031T044055Z` started 04:40:56 UTC on the exact final Linux webserver source; suite definition hash `12f9bb97823c1537e43f0398e7cddf39890f77bce7b413ca4dfd944f4a30e5d5`. The existing immutable image and attested Bookworm Runner artifact are unchanged. - A was SIGKILLed and its native cache discarded. B started after 300,149 ms, naturally acquired generation 2, authenticated at 316,624 ms and resumed at 317,682 ms within the original 420,000 ms grace. The same active run, Runner, provider PID/start fingerprint and one submitted/started/accepted turn were preserved; one final result/terminal and workspace finalization completed, with no retry/recovery action. - After active and queued completion, B was SIGKILLed and its native cache discarded again. C received exactly one ordinary task-comment API follow-up, completed a new tool and produced a useful reply while preserving native/provider/Runner identities. This closes the earlier failed post-C gate. A separate C OS-process PID field was not exported; the physical-PID proof is specifically A→B. - Four distinct results were accepted and published; all four finalizations committed at attempt 1, all four accounting-v2 projections settled, and the task-session pointer advanced to C. The coordinator independently checked these facts and all 49 canonical checks against the sanitized receipts. - Canonical cleanup passed and independent checks confirmed three exact sandbox 404s plus complete inventory/zero owned remnants, no owned PG or controller/observer processes, EC2 terminated, root volume absent, and the ephemeral SSH key/agent removed by 04:54:25 UTC. Live scope is classic remote Codex warm continuity with non-native storage dependencies available. Prepared-task completed B→C, full per-turn application disk loss, browser reconnection, two-server load and overall Railway/HTTP continuity were not live-qualified here. Earlier failed campaigns and the original cap-two burst failure remain failed historical evidence. A report-renderer cache setup failure was preserved; the normal sanitizer subsequently succeeded with an explicit cache path and a new output directory, without changing validators. The repair keeps retained events under their original six-field authority until persistence and accounting accept them. Unknown semantic effects stay unresolved. A compact staged target survives source bootstrap, cached-command delivery and authenticated handoff, including an uncertain commit response. Direct and same-guest-listener recovery require the verified original Runner and the exact endpoint. Target snapshot publication marks readiness in memory, with retirement coalesced into an existing command write. Headers contain only the staged descriptor/digest; a prequeue template is written once as a changed checkpoint part and then recovered from the existing command/receipt. No second session snapshot transaction, streaming SQL path, new pool or recurring idle scan is added. | Current repair gate | Evidence and scope | | --- | --- | | Source-tail regression | Real Runner protocol RED reproduced attachment rejection. Frozen core/transport pass five source-tail cases: source usage attribution, held persistence/no premature ACK, lost commit response plus cache loss, unknown semantic hold and accounting rejection hold. | | Handoff loss windows | Direct and listener modes pass queued/prepared/activated/completed boundaries (eight cells), preserving the actual Runner/provider process. Missing adoption proof and changed endpoints reject recovery. | | Compact storage | A 31 KiB attachment grant writes its separate part once, leaves it unchanged across source snapshots and removes it at queueing. Headers do not duplicate the grant. | | Accounting | 21 real-Postgres usage tests and 19 source-binding tests pass. Recovery hydrates immutable source receipts without resetting their producer or billing classification; conflicting/incomplete usage holds. | | Admission and malformed evidence | Driver 16, staged evidence 42, restart classifier 27 and source usage 19 focused cases pass. Four malformed-file regressions pass at d6a7, whose executor is byte-identical at 484, and server typecheck passes. Full executor file: 990 pass; 16 embedded-Postgres `initdb` setup failures before assertions, retained separately. | | Required checks | Exactfc68584recursive typecheck and build pass locally, with matching server stamp. Forced full CI succeeds (48 jobs pass, one intended skip); all PR checks are clean. Maps/token gates pass for their unchanged source surfaces. Fresh exact-head Greptile is 5/5, all six threads resolved. | | Ordinary application path | Exactfc68584managed component passes active recovery after natural expiry, completion, native-cache loss and ordinary C continuation. Production bytes match before/after; temporary fixtures restored and owned DB/services cleaned. UNKNOWN local debug-daemon/partial instruction-observer limits remain; the separate live case uses the attested Linux producer. | | Producer | Locked Bookworm build from exact 5b7 Rust treed03e494217, equal to finalfc68584Rust source. SHA-256 65b8f8cfdd6099e9ad24f1ca154e8a260777269cc82f6e23c6e6a180c531eaaf; 38,515,328 bytes. Actual Debian 12 execution and live admission pass. Controller/admission/DB changes do not relabel the producer source. | | Historical cap-two comparison | Exact 59da baseline vs 5b7 candidate, 300 controllers/10,000 history, ABBA 4×65 s per mode, all three actual pools and real snapshots. Steady CPU/op 541.36→554.64 µs (+2.45%); WAL/op 1512.13→1560.99 B (+3.23%); DB-request p99 102.78→105.99 ms (+3.12%): PASS. Burst CPU/op +4.71%, WAL/op +5.29%, p99 −0.10% pass relative gates, but absolute drains above fail. Identical operation/SQL counts, zero lost leases/drops/errors. One Node process simulates 300 controllers; this does not qualify two-webserver overlap or HTTP capacity. | | Live acceptance | Strict final-source Linux classic Codex case passes 1/1 on attempt 1, all 49 checks. Active recovery within grace and useful retained C continuation succeed; four accepted/published/committed/settled outcomes. Independent provider, PG/process, VM/root and SSH cleanup complete. Prepared/task and broader rollout limits are explicit above. | **Historical qualification snapshot through `9eb9ac6`.** The head at this snapshot was `9eb9ac6f9e312f3ebe8040f5a7565e88ae8824f3`. Server and Runner package sources are byte-identical to `283e05ffaa`; the two later commits change the explicitly scoped qualification and its documentation only. It includes master through the storage-owner fix `45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` (#18357). At that snapshot, CI/review passed but the strict post-C native recovery gate failed; the PR stayed draft and required another native repair. Prior `39a4343` CI and review passed, but its full live workflow failed; that grade remains unchanged. The first repaired `435ccd2` live attempt failed before any injected outage. It exposed a legitimate terminal snapshot before result acceptance; no accepted result ID or publication receipt was observed. Its canonical report remains 0/1 `candidate_failure`, with cleanup passed and independent provider absence confirmed. `0862062` defers publication until result acceptance and fixes the fresh review's coordinator/issue lock-order finding. Both failures have deterministic real-Postgres regressions. The `0862062` live attempt then proved accepted-result publication but failed active recovery: a resumed-lease preparation guard demanded suspension after the exact live Runner was already verified for adoption. `64c0b746` preserves that verified active/idle-warm process, revalidates its proof/liveness, and keeps ordinary suspension checks and later PRP authentication. Its live run passed preparation but exposed a second active-warm boundary: all durable binding fields matched except opaque turn/item attachment IDs, which the server had regenerated. `7a203274` carries the complete verified six-field binding and keeps the strict Runner mismatch guard. Its cloud case then passed active adoption, completion, publication and accounting. The queued continuation exposed accepted-result retirement writes and an implicit-home fingerprint drift; both are fixed with deterministic regressions in `39a4343` before this fresh full qualification. The prior live case exposed a real publication gap. A goal-less native turn could persist its accepted result and terminal snapshot before provider cleanup, while the task-session pointer was still empty. Normal later admission then selected a fresh session. The fix publishes that pointer and a private bound receipt in the **existing post-acceptance terminal checkpoint transaction**, before cleanup. Earlier terminal observations persist without a handled receipt or continuation pointer. No result-acceptance or snapshot transaction was added. The late completion path recognizes the receipt. This also prevents a late callback from restoring a pointer after a user reset. Goal/control publication keeps its existing owner. The dedicated Daytona fixture now uses the existing warm/reusable environment. Its sandbox, Runner and provider storage must survive. The former generic per-turn fixture disposed the sandbox after completion and kept provider-history backup bytes under the discarded directory. Whole per-turn disk-loss acceptance therefore has a separate provider-backup storage dependency. It remains distinct from the passing native per-turn component checks with backup storage available. The 60-second lease, 300-second outage and 420-second grace are unchanged. | Gate | Current repair evidence | | --- | --- | | Assembled regression | Baseline fails with an accepted result and terminal snapshot but a NULL task pointer while real provider cleanup is held. The fix passes normal continuation and reset cases, with no pointer seeding. Exact merged `92b7447` passed 2/2 twice, including after normal package TypeScript output refresh. Production files are byte-identical at frozen `435ccd2`. | | Persistence and concurrency | 19 helper/transaction cases and all 59 existing control-port cases pass on `0862062`. They cover bound result IDs, reset/newer epochs, Stop, lock waits and expiry, receipt replay, missing stubs, foreign bindings and atomic 4 KiB rollback. Existing snapshot transaction count stays one. The new tests reproduce the exact live missing-result-ID error and the inverse replay/status lock order against `435ccd2`, then pass after the correction. Replayed results still reject stale generations. | | Completion | All 32 completion-boundary tests pass on merged `92b7447`; the source is unchanged at `435ccd2`. Native result-shaped session metadata is preserved. | | Qualification fixture | A valid review report reproduced three rejected correct replies. Exact token-boundary matching fixes them. 140 focused E2E/catalog cases and Product E2E typecheck pass. Wrong tokens, marker prefixes/suffixes and wrong authors remain rejected; retained-identity checks are unchanged. | | Required checks | Full recursive typecheck and standard build passed at `435ccd2`. Both exact commands also passed on `0862062`, with its exact server build stamp; both also pass on `7a203274`, with its exact server build stamp. Fresh `0862062` Greptile passed 5/5, with zero unresolved threads. The `64c0b746` normal CI then passed all 49 jobs (48 success, one expected skip) and Greptile passed 5/5 with zero unresolved findings. The `7a203274` normal CI also passed 48 jobs with one expected skip, with fresh Greptile 5/5 and zero unresolved findings. Current `39a4343` recursive typecheck and standard build both pass with persisted exit codes and its exact build stamp; fresh Greptile is 5/5 with zero unresolved findings. Its normal CI completed successfully: 48 jobs passed and one expected job skipped. The fresh live result remains separately failed. The `0862062` normal CI had 46 passing jobs, one expected skip, one worker communication failure and its aggregate failure; GitHub states the self-hosted runner lost communication. No failed assertion or specific OOM/spot cause was established, and no blind retry was used. | | PostgreSQL supplement | One corrected all-pool pair at the preceding repair passed 300-controller/10,000-history correctness and absolute gates. SQL 76,590→75,990; transactions 9,013→8,713; UPDATE rows 11,613 each; 300 correct pointer writes each. PG CPU 31.99→30.46 s, WAL 70.36→67.87 MB, request p99 100.98→103.12 ms. This is one pair, not a repeated improvement claim or an exact `0862062` CPU measurement. Its two later changes defer premature publication and reorder existing locks; they add no transaction, stream/idle writer, pool or timer. The first collapsed-pool attempt remains invalid and retained. This is not an exact final-head CPU/WAL measurement; later fixes change ordering, identity selection and in-memory skill replacement without adding snapshot transactions, pools, stream writes or idle SQL. | | Live publication proof (`0862062`) | Baseline run `06fcc806-f186-4190-9b57-087b61d75533` accepted result `58f912dc-94d9-4b4d-bd3d-d5f689c83fd3`; the publication receipt names the same result, the task pointer has parameters and `lastRunId` matches, and the normalized native identity matches the following active run. Accounting v2 settled. | | Active recovery (`0862062`) | **FAIL.** SIGKILL/cache discard at 23:07:28.666 UTC; hold 300,000.859 ms; B ready at 308,896.486 ms; natural generation-2 ownership at 312,906 ms. At about 319 s, preparation failed `runner_harness_state_mismatch: resumed_not_suspended`. No authenticated after-receipt passed within 420 s. Exact original Runner PID 668/start/fingerprint was independently still alive; provider auto-stop was 15 minutes. No retained C follow-up qualified. Canonical 0/1 `cleanup_failure`/cleanup-failed report remains failed; subsequent independent cleanup deleted all three owned resources with exact 404s and confirmed full intended-account inventory with zero owned resources. | | Verified adoption preparation (`64c0b746`) | 91 focused cases pass, including actual preparation for four active providers and a distinct prior/current warm proof. Unproved resumed state, dead process and stale warm owner remain rejected. The earlier options-only fixture missed this gate. No spawn, backup restore or PRP-authentication bypass was added. Server typecheck passes. | | Active warm identity (`64c0b746`) | **FAIL.** B claimed generation 2 at 312,995 ms. Preparation passed, then strict PRP recovery rejected the generated attachment IDs at about 319 s. The saved and desired Runner, lease, normalized session and run IDs matched; only `turn_attachment_…`/`item_attachment_…` differed from generated `turn-<run>`/`item-<run>`. Canonical 0/1 cleanup-failed report remains failed; independent cleanup confirmed deletion/404s and zero owned provider resources. | | Six-field identity repair (`7a203274`) | Actual server-factory regression RED4→GREEN38. A real Rust Runner test passes healthy first run → random-ID warm handoff → held active second run → restored controller. It verifies a new live authenticated connection, exact identity, same Runner/provider OS identities, no duplicate launch/turn, and no redispatch or invented receipt for an uncertain tool. The synthetic wrong pair still fails the unchanged guard. This protocol case does not claim business completion. Server and Runner typechecks pass. | | Active recovery/completion (`7a203274`) | **PASS for the active component.** Full five-minute SIGKILL/cache-loss hold. B ready at 308,936.651 ms; natural generation 2 observed 312,979 ms; authenticated at 317,720 ms and resumed processing at 318,527 ms. Same run `1e1ccb00-0728-4935-97da-57562d81cb42`, Runner `246f73f7-a689-4177-a9c2-7ce711d63aed`, remote PID 669/start 23:48:27.470, provider lease/session and one submitted turn. Accepted result `49097529-5310-4ac3-af95-5bf9b91d19f4`, publication, finalization and accounting v2 all settled. | | Full workflow (`7a203274`) | **FAIL, preserved.** Queued late-requirement run `5e61f38d-c7d1-4fc1-bcbf-bce9d46ce4eb` failed `runner did not durably suspend before checkpoint` before outage 2. Canonical 0/1 candidate failure, cleanup passed, evidence valid. C retention was not exercised. Independent provider absence, controllers/PG cleanup, EC2 termination and root-volume absence all confirmed; closed receipt SHA-256 `676a3b15eb9937b6bf995c36fb8666c88964d509e5ceb9109fb02da0aa2bc909`. | | Retirement and stable configuration (`39a4343`) | Real Runner + Postgres RED reproduced the exact suspension failure and nine owner rejections. `turn.stop` is now lifecycle checkpoint maintenance under unchanged owner/result/generation fences. GREEN25/25 proves actual persisted suspension, exact identity and completed stop/drain/suspend; unaccepted, foreign, stale and new-work negatives remain rejected. Two default-home fingerprint REDs become GREEN9/9; explicit mode changes still invalidate. No global matching exemption or historical fingerprint rewrite. | | Fresh full Daytona warm cell (`39a4343`) | **FAIL, preserved.** Launched 00:15:29 UTC with automatic retries disabled. Active adoption passed: actual offline hold 300,000.848 ms; B ready 309,922.155 ms; natural generation 2 at 311,821 ms; authenticated 317,989 ms; resumed 330,440 ms. Same run `3e126819-e786-4b3b-b923-c975cf4f5b79`, Runner `4079138a-8d7c-44a4-8558-07a5411d1f55`, PID 668/start and provider/session identities. Accepted result `08e504af-4dec-48a8-bf05-bf0a32280559` and publication passed. Queued run `31d168c9-7ab9-4700-abb6-fafb90912237` failed the same strict suspension error before outage 2. Its row selected a new native identity, but no new third provider process or invocation was observed. Recovery-time configuration still reset only `adapterConfig`; native provider, context and workspace subtrees were unchanged. Canonical 0/1 candidate failure, evidence valid, normal cleanup passed; all campaign provider/PG/process resources independently absent. Closed campaign receipt SHA-256 `5e7d6281c13c812f3a946667f97f3c7d339b5ad86a3a819bd185b2fa73bc4a19`; EC2/SSH remain temporarily held for the concrete follow-up repair. Original 60/300/420-second limits and strict queued/second-replacement checks remain unchanged. Direct-process path; no new live prepared-task claim. | | Native retirement authority (`283e05ffaa`) | Exact `39a4343` facade regression RED3→GREEN9. The successor now uses its existing production coordinator claim and prompt renewal before strict retirement. A real Runner/Postgres case starts in the actual native-observed/unclaimed phase, claims through production code, then closes and durably suspends the predecessor in 1.657 s. All 25 real-Postgres cases pass; unclaimed/foreign writers remain rejected. Shared attachment staging stays after retirement; Stop and denied claims neither close nor stage. No new claim or SQL path was added. | | Configuration continuity (`283e05ffaa`) | Real shared/project-primary environment selection → configuration with four skill versions reproduces `reset: true`, changed category exactly `adapterConfig`, after a Postgres JSONB read. Pinned restoration had reordered the generated preference while actual runtime skills stayed identical. Replacing existing pinned rows in place preserves the original fresh name/key order and fingerprint. The assembled regression passes, as do three existing pinned-version/rename/alias cases and server typecheck. Genuine model/skill-selection changes still invalidate. No fingerprint-format change or broad compatibility exemption. | | Final production repair (`283e05ffaa`) | Full recursive typecheck/build and [normal CI](https://github.com/paperclipai/paperclip/actions/runs/41151145106) PASS: 48 jobs plus one expected skip. Fresh review 5/5, zero unresolved threads. Live baseline, recovered run and queued continuation all succeeded, each accepted/published/committed once with accounting v2 settled. Their native/provider identities and configuration fingerprint match; both recovery and continuation report no reset, no changed categories and task-session reuse. Active outage/authentication remains within original timing. | | Browser-dependent continuation (`283e05ffaa`, definition 6) | **Whole cell FAIL, preserved.** After B was killed at 00:46:38.928 and its cache discarded, C became ready in 9,151.410 ms. `openParent()` then hit a 30-second `page.goto`/DOMContentLoaded timeout before posting the final request. No fourth native run or C continuation was exercised. Cause of navigation timeout is unknown. C task had no execution lock, queued comments, handoff or recovery blocker; all three prior runs were committed/accounted. Normal cleanup passed; independent provider absence, controller/PG cleanup, EC2 termination and exact root-volume absence confirmed. Sanitized report bundle SHA-256 `2a3b841d62efb2ae30ef14f96a431ac4fa7fe3a782f1144831512f78edc65022`. | | Native API continuation qualification (`9eb9ac6f9e`, definition 7) | **FAIL, preserved.** The real post-C API request dispatched run `c71fbb18-f979-4fe4-93ce-9c4be46dbf21`, which failed `native_session_interrupted` / `native_session_recovery_failed` at 01:10:00.640 UTC. It selected the same normalized native identity, Runner and checkpoint-provider identity as the three succeeded turns. No C result was accepted; pending accounting remains explicit. Active recovery passed: actual offline hold 300,001.098 ms, B ready 308,335.019 ms, natural generation 2 observed 311,818 ms, authenticated 315,540 ms, resumed 316,612 ms. Queued continuation succeeded/accepted/published. B was then killed/cache discarded at 01:09:34.500 and C ready in 8,783.803 ms. Original classifier `transient_infrastructure`, whole-case failure and cleanup passed are unchanged; that classifier does not establish the root cause. All owned provider/PG/process resources, EC2, root volume and SSH agent independently absent. Closed receipt SHA-256 `7f21ac4a22efc07b849fa7e17866083b6127b3321af12671b4fc054ce57039a9`; sanitized bundle `71ffef14a30a553e008320a31840a6d6c0de85c1bfdb60f783fdb53a6aae812a`. Exact backend detail is unavailable in approved structured evidence. | | Current required checks (`9eb9ac6f9e`) | Recursive typecheck, standard build and exact build stamp PASS. [Normal CI](https://github.com/paperclipai/paperclip/actions/runs/41152908417) PASS: 48 jobs plus one expected skip. Fresh Greptile 5/5, latest edited summary checked, zero unresolved threads. Exact8/512/8,192 stream probes again report zero recovery SQL/transactions/rows and settled-idle SQL; seven control transactions/twelve changed rows each, with 36 separate legitimate renewals in the longest case. CI success does not supersede the failed live C gate. | The completion supplement observed secondary request/liveness/checkpoint transaction p99 increases of 23.0%/45.7%/15.6% in its single pair. Renewal p99 stayed below 0.865 s, with 900 successful renewals per arm and no lost leases. The named pool caps stayed 10+2+2. An aggregate activity sample peaked at 15 non-observer entries without backend-type classification, so no claim is made that all were client connections. No sampled lock episodes does not establish zero lock waits. Twelve pure reader/selector cases also pass with the new adjacent receipt. This is not another executed process rollback. Development reader `5cf7aead` can parse the recovery state, but its old run-detail redactor can expose receipt IDs/outcome to an authorized run reader. The first production release containing recovery readers, complete attachment-identity handling and both private-field redactions is the rollback floor. The implicit-home fix prevents new recovery-time fingerprint drift; it does not rewrite previously stored fingerprints from failed candidates. The development probe is not a published supported rollback target. Latest actual `0862062` Runner tests independently passed 8/512/8,192 text chunks with zero recovery SQL, transactions and changed rows, plus zero settled-idle SQL. All three observed seven setup/control transactions and twelve changed rows. The 8,192-chunk case separately recorded 31 legitimate lease-renewal statements, with no lost lease. These are component SQL observations, not a production CPU/WAL claim. The new adoption and identity branches affect recovery preparation only. The `64c0b746`, `7a203274` and exact `39a4343` CI independently passed all three stream sizes with the same zero-SQL/idle result; each latest 8,192-chunk case recorded 36 separate renewal statements. Exact `39a4343` Runner 2/2 passed 133 files / 2,317 tests plus two skips; its separate native group passed 150 tests plus eight skips. The first API fixture revision omitted the mandatory final screenshot; this was caught before paid launch and allocated no provider/model work. The correction captures the actual retained Chromium page with an explicit unqualified-browser label, without changing the standard report validator. Automatic approval review rejected exporting raw private API/workflow snapshots because of sensitive payloads and insufficient destination-specific authorization. Sanitized reports and bound/hash-only receipts are retained; raw snapshots were not copied through another channel. This does not change the failed grade. Repair characterization also preserves its failed attempts. An unconditional staging relocation broke three existing same-scope scripted cases; the final implementation defers staging only for an actual superseded warm scope, and all nine order/race cases pass. One baseline transform overlapped an edit and is explicitly invalid; the actual immutable `39a4343` loader then produced the three genuine REDs. The independent task-directory hypothesis produced a RED but was excluded by the live shared-workspace evidence; no directory policy change was landed. A shared-workspace fixture authorization omission was corrected separately from the four-skill semantic RED. Setup failures are retained separately from semantic outcomes. These include an old host port after the owned Postgres fixture restarted and a sandbox-denied TSX IPC pipe. The supported external-Postgres run passed, and the final exact recursive typecheck passes with normal process permissions. Seven stale declaration errors after the master merge cleared after compiling the actual merged SDK/Runner TypeScript sources. No declarations were edited by hand. <details> <summary>Earlier qualification measurements and failed attempts (historical, before this repair)</summary> **Historical blocked gate (before the repair below).** Final native implementation is `abed921e12d543b62f57a56435419c0dafb0bc33`; current branch head is `af9e2d157dcc349e55374ad186c9be2d6cb96bf9`. The two later commits change only qualification fixtures, their matcher and documentation; `server/` and `packages/` are unchanged. It includes master through `fd008979f5794fda3d5f54151d36acd31d0de140`. The live Daytona measurement proves five-minute active recovery, once-only completion and accounting, but **fails retained-session identity after the second replacement**. The saved checkpoint and recovery metadata exist. The task-session pointer remains empty, and ordinary admission creates new native/provider sessions. The exact skipped publication boundary remains unproven. A useful saved-file result alone does not satisfy retained-session continuation. This application-path gap blocks merge independently of CI. A separate new-master CI failure is owned by the storage workstream (fix in #18357). No storage guard, session-identity oracle, lease/grace limit or provider result was relaxed. | Gate | Observed result and exact scope | | --- | --- | | Distinct source recovery and compatible rollback | PASS at A `5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → B `5ea45f8d17e33c9a62bd79982dcf0375b07733d0` → fresh B → A. Four controller processes, natural 60-second lease expiry, absent native projections before replacements, three succeeded/committed unique results, one provider thread, unchanged Runner identity, and strict idle cleanup. 177.782 seconds. | | Immutable installed native component | PASS at installed A `5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → installed B `eaef0d56552c087aaa4fa68a8655282ffbd12667` → fresh B → A. Each controller imports its contained compiled server and DB. Three unique committed results, stable Runner/thread, no extra provider restart on same-contract rollback, strict cleanup. 106.672 seconds. This is compiled component proof, separate from full CLI-native application acceptance. | | Normal public CLI startup | PASS for B `eaef0d56552c087aaa4fa68a8655282ffbd12667`, Node 24.22.1, darwin-arm64: fresh external DB, 415 migrations, health ready, root HTML 200, company POST 201 plus GET readback. The installed tree remained unchanged after execution. | | Five-minute component outage | Actual controller absence 300,168 ms. Natural generation-2 claim at 407.196 seconds after crash, same provider attempt and Runner, original run succeeded/committed. The larger 540-second case timed out before its retained follow-ups, so it is an active-adoption pass, not a complete four-controller pass. The complete short source/installed cases above independently cover follow-up and rollback. | | Drain/resume review regression | Fixed in `226f03e736d1ee99fea1762ab32ea3a8a9241c20`. Five facade-level cases pass: overdue normal/restart waits, future timer cancellation, another facade resuming the original owner, and transient retry. Against exact prior scheduler `5ea45f8d17e33c9a62bd79982dcf0375b07733d0`, four fail and one passes. The later head changes documentation only. | | Final corrected-fixture Daytona | **PARTIAL: active recovery PASS; retained identity FAIL**, exact live source `0b6afc499f`. A was SIGKILLed and native cache discarded; actual offline hold 300,000.375 ms. B was ready at 308,326.961 ms, naturally claimed generation 2 at 312,059 ms, authenticated at 315,996 ms and resumed processing at 316,519 ms, within the original 420-second boundary. The same run, Runner PID/start, session, lease and event prefix survived; one turn submitted. Original execution succeeded and settled accounting. After completion, B was killed and its native cache discarded; C was ready in 9,077.779 ms. C executed the saved project and returned correct `queue_r`, with all four runs succeeded/accounted. However, native and true provider-session identities changed: the unchanged retained oracle fails. No whole-workflow pass is claimed. | | Prior matched-producer Daytona | FAILED at exact `abed921e12`. The matching current-source Runner passed normal admission, completed the first task and settled its accounting. During the active revision, A was SIGKILLed and its native cache discarded. Actual offline hold was 300,000.727 ms; B was ready at 308,304 ms and naturally claimed generation 2 at 309,651 ms. At about 318 seconds, the remote process check reported `process_not_running` for the original admitted Runner. No authenticated after-receipt or retained follow-up passed. The current failure is preserved; no automatic retry, lease/grace relaxation, debt edit or provider replay is used. Bounded forensic read found preserved exact-bound state with no recoverable failure, a transport-reset diagnostic and no terminal delivery; this supports external process loss without proving the exact cause. The sandbox had a five-minute auto-stop policy, conflicting with the required surviving-sandbox assumption. Owned cleanup precedes the separate corrected-fixture case. | **Live retained-session boundary.** Prior completed run `3806…` and C follow-up `8adf1df3-6744-40bf-b35d-100451f29d9f` both have database session snapshots and `nativeControllerRecovery`. Their tool fingerprint, provider configuration, workspace descriptor, runtime-context digest, driver and work mode match. There is no recorded force-fresh/workspace-selection change, and this is an ordinary task rather than a conversation-generation reset. The same issue/agent task-session row has `session_params_json: null` and `last_run_id: null` from baseline time. Both native normalized UUID and actual provider session/thread hashes differ after C. Full application admission therefore did not select the retained source. No fallback to file execution or relaxed identity comparison is accepted as proof of continuity. **Independent cleanup completed.** Full current-account provider inventory confirms no resources owned by either final fixture, with exact-ID absence checks. Owned controllers and campaign Postgres are stopped/removed. EC2 `i-01551c09af098dde5` is terminated, its delete-on-termination root volume returned `InvalidVolume.NotFound` at 22:14:08 UTC, and the temporary SSH key/agent were removed. The canonical failed report and immutable oracle input are retained. Closed qualification receipt SHA-256: `b4b5fefce287ebfdbf2cc2238efaad62d8ad485aed0e9c0628ae95dbc18a8575`. **Original measurement and grader repair remain separate.** The C reply contains the requested marker followed by a newline and the correct output. The old workflow waited for one literal space, so its normal receipt collection stalled; that original attempt was closed and its canonical `cleanup_failure`/interrupted result is preserved. Commit `af9e2d157d` repairs only this reply predicate, with 144 focused cases, E2E typecheck, discovery and diff checks passing. Ordinary string matching stays literal. The corrected predicate accepts the saved reply, but the unchanged retained-identity oracle still fails. Pure active-recovery grading of the captured measurement passes 7/7; this is not a rewritten passing campaign. The sandbox idle-window conflict was corrected only for this explicit fixture (15 minutes). Actual baseline and later sandbox metadata confirm that policy; the original active box was disposed before a post-gate policy read, so its 15-minute setting is established by the normal create/environment configuration rather than an after-read of that box. Production native lifecycle and the 60/300/420-second constants remain unchanged. The development compatibility reader A is a genuine backport commit based on `59ff89a18fe1312b5f0bb908439bbdb1237b0252`, with the required idle-goal reader fixes. It is not a previously published release. Its older executor differs from B. The rollback floor must include this reader capability. The source component cases admitted the Linux ARM64 Runner packaged at `59ff89a18fe1312b5f0bb908439bbdb1237b0252`; the installed component case admitted the `5cf7aead` package. Their Runner bytes are identical: SHA-256 `b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8`. Each admitted artifact remained unchanged throughout its case. [Final public package build](https://github.com/paperclipai/paperclip/actions/runs/41121460635) and [reader-floor build](https://github.com/paperclipai/paperclip/actions/runs/41122170506) both passed all four daemon builds, package assembly and both Darwin clean consumers. Both Linux ARM consumers failed before provider work with the same existing local Codex permission-profile error: bubblewrap attempts `activity.fifo/.git` and reports “Not a directory.” This matched baseline failure remains open; it is not a native remote continuity pass. Full post-run tree and build-stamp revalidation also passed after the compiled native case. The installed component trees are A `c29114c1bf71c5af00f761aeb9e3d84ed3d3075b9f5600dee3ef2543eb2b6ec5` and B `eda575505373a3bdc630fa4919862b359791fe3a7ef637961a25b73b38efe4b0`. **Controlled PostgreSQL qualification passed.** Baseline/candidate/candidate/baseline workloads used 300 controllers, 10,000 historical tasks, real snapshot/encrypted PRP writes and unchanged pools. PostgreSQL and client were each limited to two CPUs and 1536 MiB on an isolated eight-vCPU Linux host. | Workload | PG CPU per operation | WAL per operation | Database request p99 | | --- | ---: | ---: | ---: | | Steady | +1.18% | +1.81% | +1.91% | | Aligned burst | +3.40% | +2.56% | +3.81% | | Two logical servers | +3.11% | +2.56% | +9.68% | All arms passed absolute/correctness gates with unchanged control transaction counts, no dropped operations and no owner loss. Overlap p99 has little margin below the relative threshold. It uses two clients in one Node process, not two HTTP servers. [Closed measurements and limitations](doc/plans/2026-10-30-native-controller-qualification-evidence.json) and exact-source reproduction patches are retained. [Earlier contended-host failures](doc/plans/2026-10-30-native-controller-load-evidence.json) remain unchanged. The production DB-claim probe claimed 300 owners in 2.84 seconds and restored/committed controls and snapshots in 8.74 seconds, with zero owner loss or settled-idle SQL; its provider references were synthetic. The measured checkpoint/snapshot/renewal paths retain the same persistence behavior after integration; later accounting and business writes have separate owners. Actual Runner stream tests at `da8c7f6e1837ddd896dfe83a7524812fee813487` passed 8/512/8,192 chunks: zero recovery SQL, transactions and changed rows during text intervals; zero settled-idle SQL. Each retained seven control transactions and twelve changed records. Existing renewal SQL was measured separately. The guard/scheduler fixes add no stream path, pool or recurring idle scan. The live fixture used current-source producer SHA-256 `f0000c90cb83880191479da639201e659aa6ce0fc7d67c859faa172ec977849e`, a normal locked exact-head build with pinned Rust 1.97.1 in Bookworm. Actual network-isolated, read-only ABI execution passed (GLIBC requirement 2.34 against runtime 2.36). The immutable base image remained `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:36069788f53c51f228764fa36180575443dfb5d1f84a342d7c0daceb28dc61cc`, source `a0ef3ce7`. This was an unpublished development producer, explicitly distinct from the base image and admitted before the fresh fixture began. The earlier host build failed ABI preflight before admission. The preceding old-producer fixture completed its initial task but lacked the usage-basis marker required by current master; it remained pending/unpriced and correctly paused. It was independently cleaned without debt or budget changes. Two preceding local Linux setup attempts OOM-killed before provider allocation, and their cleanup passed. These attempts remain separate from the final recovery failure. **Failures retained and resolved separately.** The earlier `98f05fc` probe rejected a valid settled PRP v2 goal suffix; `eaef0d5` fixed that reader and the fresh follow-up passed. Later component cleanup failures were traced by a scoped exception observer to the fixture adapter rejecting directory downloads. A bounded owned-directory transfer with exact-byte/mode preflight corrected the fixture; source and installed strict-cleanup cases then passed. No production guard was relaxed. The initial local integrated control-port run hit its 30-second embedded Postgres startup hook before executing 59 cases; 18 other focused cases passed. A bounded rerun used a fresh owned external Postgres fixture, applied normal migrations before Vitest, and passed all 59 cases with the original timeouts unchanged. Its container and data were removed; the failed attempt remains retained. The additional opt-in full Product E2E Vitest run is not green: 5,313 passed, seven failed and three skipped. An actual `abed921e12` archive with the same dependencies, Node and IPC permissions reproduces all seven failures with the same reasons. Five are unchanged workflow-policy/profile-pin assertions, including Pi profile version 37 versus the test pin 35. Two database suites failed during embedded Postgres startup before their assertions; semaphore exhaustion is confirmed in the observer log. The observer passes its supported external-Postgres rerun with original timeouts and source unchanged. The controller-database client-kill/postmaster assertions remain unexecuted after `initdb` bootstrap failure; no external override is supported. The unit launcher stopped before its Node-test phase, which was then run independently and passed 181/181. The original failed full run is preserved. These results remain distinct from the passing focused checks and normal repository CI. The first local Product E2E typecheck also encountered stale compiled Runner declarations; a genuine package-local TypeScript compile refreshed them and the unchanged check passed, without a dependency install or producer rebuild. Earlier [Daytona campaign 41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900) failed at sandbox creation before any agent, with the underlying provider error unavailable. Its [sanitized report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-41108162900-2/index.html) remains a `cleanup_failure`. Both old creation names are absent in the current authenticated account, but the old CI account identity was not retained, so account equivalence is unproven. The subsequent exact `eaef0d5` source Product E2E attempt passed image/agent setup and the initial project, then held the controller offline for 300,003.7 ms. Its Mac supervisor failed owned-process inspection before B readiness; no adoption passed. Full current-company lease/label checks, provider deletion, exact-ID 404s and complete inventory confirmed cleanup of that attempt. Its local server and owned Postgres stopped. The prior AWS campaigns and database qualification VM were terminated with root volumes removed. The old admitted test Runner lacks the newer usage-basis marker introduced on master. The integrated component B turns retained unpriced, incomplete accounting obligations with `costAccountingPending: true` and no settled charge. The compatibility-reader A turn used its legacy receipt interpretation but likewise retained pending accounting and no settlement timestamp. This proves obligation preservation, not a whole-run accounting migration. Runner/receiver release ordering remains governed by the usage owner's contract. At native implementation `abed921e12d543b62f57a56435419c0dafb0bc33`, Greptile was 5/5 with zero unresolved threads and CI completed with 54 successful checks and five expected skips. At `0b6afc499f`, Greptile again passed 5/5 with zero unresolved threads, but CI ended with 51 successful checks, five skips and three failures. Server 9/12 fails a deterministic managed-skills maintenance assertion newly inherited from master `198218bf` in merge `c80e00ad`; that test is absent from the PR head, and the storage owner has the explicit-StorageService fix in #18357. Runner 2/2 passed 2,283 unit cases but its runner received a shutdown signal during the long stream probe; its cancelled step and aggregate failure are preserved. No blind retry was used. Current `af9e2d157d` checks are pending; the matcher-only change passes 144 focused cases and Product E2E typecheck. Earlier scheduling-sensitive CI failures and the interrupted unsharded local test run remain recorded in the dated plan. CI runs the repository's normal test partitions. Final verification does not claim Railway rollout, uninterrupted HTTP service or whole-webserver filesystem loss. </details> ## Risks - The task-pointer repair passes its assembled regression. The first live repair exposed pre-acceptance ordering and the fresh review exposed a replay lock-order issue; both are fixed and pass real Postgres tests at `0862062`. The following live attempt exposed the verified-running-versus-suspended preparation gate. Its narrow fix passes 91 focused cases at `64c0b746`. The next live case exposed synthetic attachment IDs; `7a203274` fixes that field selection and passes real Runner authentication. The subsequent queued continuation exposed strict retirement maintenance and a false default-home change; `39a4343` fixes both with real Runner/Postgres and fingerprint regressions. The `39a4343` live continuation failed again despite green CI. `283e05ffaa` repairs the actual native-observed admission ordering and reproduces/fixes the JSONB skill-order fingerprint drift. Both pass targeted real-Postgres regressions and the live active plus queued paths. The definition 6 post-C browser navigation timed out before admission. Explicit definition 7 now verifies native continuation through the ordinary comment API; its strict post-C native gate failed after a real dispatch and remains historical failed evidence. The finalfc68584repair now passes its fresh strict 49-check Linux/Daytona campaign; the prior failed grades remain unchanged. Browser reconnection is handed to its own workstream with the original failed evidence. No prior failed campaign is reclassified as passing. - Incorrect checkpoint/source binding or stale ownership could duplicate external work. Missing evidence remains a hold; existing receipt and finalization owners keep uncertain effects unresolved. - Recovery must retain maintenance writes after accepted results without allowing a stale business turn to write. Real lock-wait and generation tests cover that boundary. - The current two-connection paired burst profile fails its absolute drain gate in both baseline and candidate, despite passing relative load gates and all correctness checks. A causal capacity diagnostic selected three slots; the final controlled cap-three profiles pass unchanged thresholds. Full application replacement and two-server overlap capacity remain separate. There is no new streaming writer, pool or recurring idle scan. - The remote sandbox and Runner must survive the recovery window. Provider auto-stop is a separate lifetime bound from reconnect grace; the explicit qualification fixture uses a longer idle window. - Workspace and provider backup availability remain external dependencies. This PR does not certify whole-webserver filesystem loss, uninterrupted HTTP service or a Railway rollout. ## Model Used Implementation and initial verification used OpenAI GPT-6 through Codex; the initial hosted snapshot was not exposed. The follow-up qualification used three OpenAI `gpt-6.1-sol` workers coordinated through Codex, with repository tools, code execution, CI and scoped test infrastructure. Context-window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused and exact-source application/cost checks pass; full forced CI passes; earlier local embedded-PG setup failures remain recorded) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (exactfc68584full CI and auxiliary checks pass; forced complete test partitions) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (exactfc68584check and edited summary pass; all six threads resolved) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
Paperclip Native Runner
To add a harness/provider to native Runner, start with create-runner-adapter, the Runner integration playbook, and acceptance/evidence templates. They cover the new Runner integration path; legacy adapters are preserved as a compatibility boundary.
This package is the standalone development boundary for Paperclip's native
runner protocol, process supervision, durable transport, provider drivers, and
normalized session backends. Rust owns the production runner under runner/;
TypeScript provides the control-plane reference, browser SDK, scenario tools,
and conformance oracle.
The package includes one coherent set of capabilities: PRP v1 validation and replay, a supervised local runner with a scripted fake harness, durable WebSocket delivery and recovery, qualified Codex, OpenCode, ACPX, Claude Managed, and AWS AgentCore drivers, live session and issue-thread surfaces, a public browser/React SDK, a standalone adapter demo, and a deterministic mock control plane. None of these surfaces imports or starts Paperclip's server, UI, CLI, or production database.
Execution environment lifecycle
RunnerExecutionDriver defines the environment hooks shared by coding harnesses
and external bot harnesses. The server's createNativeRunnerExecutionDriver
supplies this contract for local, SSH, and sandbox execution. Provider-specific
session setup stays separate from environment selection.
Connection registration prepares remote resources and returns the process's connection configuration before launch. Local Runners connect to Core over loopback WS. SSH Runners and eligible sandboxes connect to the configured Core WSS endpoint. Sandboxes with authorized WebSocket ingress listen inside the sandbox; Core connects through the provider endpoint after process dispatch. Ingress selection fails closed and does not fall back to outbound WSS.
The returned activation, readiness, failure, checkpoint, and release hooks feed the existing shared PRP control plane. PRP owns authentication, commands, replay, tool authority, and run transitions. Process ownership hooks provide launch, verified adoption, and durable state access. A lost connection does not prove process termination, and a required checkpoint failure must prevent lease release.
The driver is an in-process interface. Its filesystem paths and process launch specifications are internal to built-in execution; they are not a plugin wire contract. Existing flat transport options remain compatible through this shared interface. This refactor does not add Launcher APIs or change harness capabilities.
An execution owner can supply prepareLaunch, a Runner artifact identity, and
process and connection hooks. Core supplies the launch requirements. The owner
returns a provider descriptor bound to those requirements. Core checks the policy
and assigned resources. See Runner launch preparation
for the contract and recovery record.
Connection continuations inspect the harness descriptor's optional
toolRefreshOnResume capability. Native Codex, Claude Managed Agents, AgentCore,
OpenCode, and qualified Claude/Codex/Grok ACPX profiles expose it; unqualified
harnesses leave it false or absent. Changing tools can replace a provider process while retaining
the provider conversation. Company, agent, task, workspace, model, instruction,
and skill compatibility still gate recovery. An MCP-only assignment change can
resume only when the selected harness explicitly supports refreshing tools.
Compact continuation messages include the active completion revision and criterion
IDs plus a reminder to obtain an accepted paperclip_finish or paperclip_block
result for this turn. Reports from earlier turns do not finish the new turn.
Provider final text alone remains insufficient; governed waits and strict native
completion validation keep their existing behavior. This reminder changes the
resumed model input, not the tool catalog or automatic retry policy.
When recovery needs a fresh conversation, the server supplies a deterministic
handoff through a lazy history loader at the fresh attempt boundary.
It includes the original request, recent messages, resolved interaction
summaries, agent replies, and document excerpts, with source identities and
retrieval instructions. Reads and excerpts are bounded; the handoff has a
24,000-byte ceiling and explicit truncation/omission markers. Conversation
reset boundaries, deleted messages, source quarantine, and secret redaction
apply before model submission. Successful recovery does not fetch or replay the
handoff.
Legacy adapters advertise supportsToolRefreshOnResume for their selected
harness: Claude and Codex CLI/ACP, Grok CLI, Gemini/Kimi CLI/ACP, and
Cursor/OpenCode/Pi CLI. CLI adapters using environment tool delivery start each
invocation with current endpoints and credentials, including resumed turns. ACP reloads
current MCP bindings, including run-scoped credentials, while preserving the
conversation; an unqualified/custom harness retains its restart fence.
Legacy fresh attempts receive the same bounded handoff, including resume-failure
fallbacks. Provider
authentication repairs retain their existing fresh-session recovery behavior.
ACPX release declarations
acpx-profiles.json owns the package versions, command/profile digests, and
required execution policies used by TypeScript and Rust. It contains no models.
After changing it, run pnpm --filter @paperclipai/paperclip-runner generate:acpx-profiles.
Normal build and typecheck reject stale generated declarations. Generation also
checks installed dependency pins and agreement with Cursor's distribution manifest
and immutable release attestation (cursor-contract.json). Cursor's per-platform
closure pins are generated from cursor-distributions.json.
Codex startup and browser login prefer the CLI from the installed dependency
graph. If that dependency is absent, they use an executable codex from the
selected execution host's PATH. An explicit execution command takes precedence,
and resumed sessions retain their recorded command. Older or newer CLI versions
are allowed. Relative PATH entries use the selected process working directory.
Compatibility is established by the actual protocol or login attempt.
Missing executables and real protocol failures still return actionable errors.
Package identity and executable containment remain checked. Browser login keeps
its provider-specific credential home and resolves the installed Codex CLI on
64-bit Linux and macOS independently of the daemon's libc requirements.
Exact dependency pins remain release and ACPX artifact-qualification checks. They make published builds reproducible and verify sandbox artifacts; they do not add a version-number gate to ordinary native Codex startup or browser login.
Release packages retain the pinned Codex JavaScript dependency graph, but do not bundle Codex native binaries. The published manifest declares the official, exact-version platform packages as optional dependencies. npm installs the package for the consumer's operating system and architecture. Those declarations belong to the published server manifest; the bundled JavaScript wrapper delegates platform installation there so npm can keep native and legacy versions separate. The wrapper code and patched ACP bridge stay unchanged. Codex sandbox read roots include the separately installed native vendor resources without granting access to enclosing npm directories or credential homes. This packaging does not change agent defaults or the selected runner of an existing agent.
Every ACPX harness accepts an explicit caller-selected model without a Paperclip
model allowlist. The adapter sends that ID unchanged and verifies the provider's
effective model before prompting. An incomplete discovery catalog does not block
selection; a provider rejection or mismatch fails without choosing a fallback.
Qualification model selections live in test catalogs, separately from optional
product defaults. The legacy resolved snapshot field qualificationModel contains
the caller's selected model; its serialized name preserves recovery identities.
Historical profile fixtures remain immutable evidence, not release declarations.
Pi credentials are selected from the explicit run environment and remain session-bound.
The native Rust launcher forwards the controller-bound credential names, including
custom provider references, without restricting Pi to an OpenRouter credential.
Built-in providers can use their usual API-key environment variables. Custom Pi
providers are supported through PAPERCLIP_PI_PROVIDERS: a JSON object containing
the native models.json provider entries (without the outer providers key).
The runner writes that configuration to the private Pi agent directory and binds
its digest to recovery. API-key and header environment references are forwarded
only from the explicit run environment; command-based credential resolution is
unsupported. Select the provider/model ID explicitly in the agent configuration.
The bundled ACPX package tests exercise unlisted model selection, rejection,
exact acknowledgement, and replay on a loaded connection. Catalog membership
and Cursor model-alias expansion do not determine the selected model.
Provider activity adapters own native tool identities, evidence, and diagnostic
usage projection; the shared driver and sidecar consume those hooks.
Public package surfaces
@paperclipai/paperclip-runner— production contracts, clients/backends, PRP validation/replay, canonical catalog/dispatcher, and compatibility check.@paperclipai/paperclip-runner/testing— deterministic mocks plus PRP and semantic conformance kits. Tests and external conformance consumers import this explicitly.@paperclipai/paperclip-runner/evals— versioned native-attempt metadata, fail-closed package/binary compatibility checks, and explicit runnerd artifact resolution for eval consumers.
The package root has no mock or scenario exports. Generic credential-free
matrix orchestration lives in the workspace-private
@paperclipai/paperclip-eval-kernel; scenario content and provider-backed eval
campaigns remain outside the runtime package.
See ADR 0001.
The two conformance surfaces intentionally prove different contracts. The
existing runControlPlanePortConformance suite checks narrow PRP run/event
persistence. CAPABILITY_HIGH_RISK_SEMANTIC_VECTORS and
runSemanticConformanceKit compare normalized tool authorization, state,
effects, audit, retries, conflicts, redaction, continuation, and terminal
decisions. The production adapter stays App-owned and invokes Paperclip's real
route/service authorities; it does not copy those rules into this package.
Quick start
Native provider debug-trace correlation uses an incremental index owned by its transport. Pending event lookups read only newly appended bytes, with a 1 MiB read budget per lookup; they retry until the observed suffix is indexed. Partial records remain pending, and trace replacement or truncation invalidates the index. Closing a transport clears its index. Other active transports cannot evict its progress. Records over 64 KiB are skipped by the correlation index without buffering or parsing their full contents; the original trace file retains them. Do not restore a full synchronous trace scan for each pending event: it blocks event delivery and can leave the board showing an active run after the provider turn has already ended.
The package also builds paperclip-runner-acpx-sidecar. This bounded v2
stdin/stdout bridge admits the pinned Claude, Codex, Grok and Cursor ACPX profiles.
It validates the exact model, session identity, tool catalog, structured input,
and terminal settlement at the process boundary. Copilot and Pi remain gated.
Verified distributions are build-owned; no provider accepts an arbitrary executable.
See the rich ACP capability report.
If the controller sends assigned MCP tools in the PRP catalog, the sidecar uses its authenticated tool bridge. Isolated provider admission accepts this route without a separate native MCP endpoint. Direct runtime-host callers must supply the native gateway binding unless they select PRP tool delivery and supply the semantic tool bridge. PRP delivery keeps the controller's exact operation names, including compact MCP catalog tools. The selected permission and read-only policies still apply.
Install Cursor explicitly with paperclipai runtime setup cursor; npm installation
does not download it. Run setup as the OS user that runs Paperclip (the service
account for a managed service). Setup writes to that account's
~/.paperclip/runtimes/cursor/<platform>-<arch>/<closure-sha256>, so a system-wide
npm installation can remain read-only. Isolated provider HOME/XDG settings do not
redirect this cache. Container images continue to use their packaged assets.
Configure a company secret binding for CURSOR_API_KEY or
CURSOR_AUTH_TOKEN, select Cursor in the Runner configuration, and select an exact
model ID. Agent is the default; Plan and Ask are explicit modes. Paperclip semantic
questions are supported. Native AskQuestion and authoritative per-run dollar usage
are unavailable. Accepting a native plan ends the planning run successfully while
the task waits for the next instruction. See the
Cursor release report.
Google Antigravity is an unqualified, pending native ACPX integration. The
explicit paperclipai runtime setup antigravity command installs the verified
Antigravity ACP 1.4.3 distribution and its owned admission patch on macOS ARM64.
It verifies both the official vendor closure and the separately pinned patched
closure; npm installation downloads no Antigravity runtime. This setup installs
assets only. Global release qualification remains pending and normal execution
admission stays closed. An isolated operator can admit one company, exact model,
local platform and pinned native closure/profile until a fixed expiry through
trusted server startup authority. The candidate supports a managed saved Google
subscription; the recorded acceptance reused the same owner's personal default
grant and gemini-3.8-flash-low on macOS ARM64. This does not qualify fresh Google
consent or the separate API-key route. Google Workspace OAuth does not substitute
for a runtime credential. See the
integration contract and evidence.
A release includes all four platform daemons and the Linux provider-pack identity
from its matching Daytona image. Run stage:release-binaries with a manifest that
binds each daemon path and SHA-256, plus remoteProviderPacks: {"linux-x64": {path, sha256}, "linux-arm64": {path, sha256}} for
the actual image's provider-pack.json. Assemble these assets after the normal
build and include them in the server's vendored Runner output before npm packing.
Assembly requires the provider pack's source revision to match sourceRevision
and its ACPX profiles and Cursor distribution to match the current source pins.
An independently rehashed older pack is rejected.
Provider-pack builds replace the installed Copilot platform wrappers with relative
launchers. These wrappers remain usable after the pack moves into an image. The
build still rejects any wrapper that retains its temporary deployment path.
Ordinary remote Cursor startup uses the packaged Linux daemon and verifies every
image asset against that manifest. A mismatched image fails before the provider
starts; install the matching package and image together.
Native binary platform baseline
Release packages contain Paperclip daemons for linux-x64, linux-arm64,
darwin-arm64, and darwin-x64. Codex native binaries are installed separately
through the official optional npm platform dependencies described above.
The shipped Linux daemons target GNU libc 2.39 or newer, on x86-64 or AArch64.
Their ELF version requirements include GLIBC_2.39; both depend on libc.so.6,
libm.so.6, libgcc_s.so.1, and the matching GNU dynamic loader. The ARM64
loader is /lib/ld-linux-aarch64.so.1; x64 uses
/lib64/ld-linux-x86-64.so.2. This is a binary compatibility baseline, not a
promise that every Linux distribution or kernel has been qualified.
Hosted native producers use Ubuntu 24.04 for both Linux architectures. Clean npm
install checks use the pinned Debian 13 Node image; the cloud Dockerfile uses
Debian 13 (node:24-trixie-slim). macOS binaries build on the existing macOS 15
Apple Silicon and Intel hosts. Actual task and follow-up acceptance is performed
on Apple Silicon; Intel qualification covers build, installation, and protocol
startup.
For sandbox environments, the existing explicit environment probe can measure
OS, architecture and libc from its allocated lease. It records those facts only
for an image pinned with @sha256, after confirmed lease cleanup. The record is
server-owned and bound to the company, effective provider/image configuration
and active custom-image identity. Creation and discovery read this record without
allocating a sandbox. Configuration or image changes invalidate it. Mutable tags,
provider defaults and unverified snapshot names do not establish compatibility;
they keep automatic Legacy selection until there is verifiable immutable image
or Daytona snapshot-ID evidence. See managed snapshot qualification below.
A connection-only success does not establish native readiness.
Managed Daytona snapshot qualification
Claude's automatic Connect flow can qualify a named Daytona snapshot through the existing administrator-only saved environment probe. The bundled Daytona plugin resolves the name to its provider snapshot ID, creates the probe sandbox by ID, and returns its account and creation identity. OS, architecture and libc facts are recorded only after confirmed cleanup. Native execution uses that same snapshot ID; discovery does not allocate or perform provider lookups. A deleted snapshot fails at the provider rather than silently selecting a replacement with the same name. Configuration, resolved credentials and active captured-template changes require another probe. Other named provider templates and mutable OCI tags remain unqualified. A provider plugin without snapshot identity support returns an actionable setup failure; it cannot certify a native default.
On Linux, Codex's sandboxed filesystem modes also require the host to permit its unprivileged namespaces. A no-privilege Docker container can prohibit nested namespace creation even when installation and binary compatibility pass. Qualification therefore retains the restricted Docker install/lifecycle checks and exercises the same installed graph's read-only native startup on the actual ARM host. This does not qualify arbitrary container or managed setup targets; their selected runtime must pass setup in that environment.
Ubuntu 24.04 also needs system bubblewrap and the stock
bwrap-userns-restrict AppArmor profile described in
Codex's Linux prerequisites.
The qualified ARM64 host uses that distribution helper/profile while retaining
AppArmor enforcement and global namespace restrictions. Hosted qualification
materializes authenticated stock package files without running maintainer
scripts; it verifies native startup and package continuity separately from the
restricted Docker installation check. This does not prove a normal OS package
installation journey or a paid provider task on ARM64.
Automatic creation selects legacy execution when a Linux host has musl, unknown
libc, or a measured GNU libc version below the baseline. Existing agents retain
their stored runner. Windows, 32-bit ARM, and Alpine/musl native qualification
remain deferred. The Runner Windows workflow builds and tests the Rust
workspace on Windows x64. It does not qualify Windows agent runs.
Native release jobs cache Cargo dependencies per platform and OS baseline, including compiler, lockfile, and external generated-contract inputs. Workspace crates and released binaries are rebuilt and verified every time. Only canonical exact-source master pushes may save trusted caches; preview qualification can restore them. JavaScript preparation and native compilation run concurrently, and both must succeed before artifact metadata and source-bound hashes are recorded. Run Docker and native qualification on hosted CI rather than a local developer machine.
Pi cold provider admission has an absolute 60-second budget. Warm run attachment checkpoints the old ACPX sidecar and starts a new one, so it uses that same budget while retaining the existing Runner authority. Live adoption and ordinary commands retain their 30-second bounds. Closing the transport cancels admission; an acknowledgement received after the admission deadline cannot revive it.
The board's transcript parser coalesces consecutive identical Pi runtime-failure display rows within one run, turn and session, including the exit handler and its late prompt rejection. Both original PRP facts remain in the run log. Distinct failure details, intervening retry activity and later turns remain visible. The profile-14 notice projection, wrapper bytes and terminal settlement remain unchanged.
Cursor candidate configuration accepts acpxSessionMode: "agent" | "plan" | "ask"
(default agent). This selects the native Cursor mode independently of
acpxPermissionMode and Paperclip task planning or company approvals. The mode
is validated at the API boundary and is bound to provider admission and recovery;
changing it cannot reuse an incompatible warm session. Other providers reject
this setting. Cursor configuration requires an explicit model. Pending providers retain
operator-controlled qualification admission.
Remote Codex sessions relay assigned app tools through the server's configured
gateway. Small catalogs are sent directly. When a catalog would exceed the
runner's 256-operation or 768 KiB contract limit, the server exposes
paperclip_search_assigned_tools and paperclip_call_assigned_tool instead.
Search accepts a limit from 1 to 20 and defaults to 5 results per page.
Use the returned nextOffset as offset to read the next page. The tool and
parameter descriptions include these bounds for providers that render numeric
schema constraints as plain number types.
Search returns bounded pages of names, descriptions, and input schemas. Each
page intersects the session's pinned assignments with current gateway grants.
An individual schema that exceeds a page returns an inputSchemaRef. The same
search tool retrieves that schema in chunks via schemaTool and
schemaOffset; discovery can continue past the large tool.
Calls retain task ownership, work-mode restrictions, gateway authorization,
approvals, and audit. Core task tools and the runner's completion tools keep
their reserved space; no assigned tools are silently removed to fit the limit.
Native Claude skill assignments travel in the runtime-context snapshot through
runnerd to the ACPX sidecar. After acquiring the provider lifetime lease, the
host materializes the assigned bundles under the isolated Claude home's
skills/ directory before launch. Reopening a provider refreshes that snapshot;
project and ambient host settings remain excluded. This path is separate from
the legacy claude_local adapter's remote skill staging.
The isolated Claude settings pin both model and availableModels to the
user's requested ID. This keeps ACP from replacing an exact ID with a picker
alias during selection and verification. Users can keep selecting models from
the normal Claude catalog or entering custom IDs; unavailable models still fail
at the provider rather than silently falling back.
ACPX Claude defaults to approve-all, shown as Full auto (approve all).
OpenCode defaults to allow; native Codex defaults to never (no approval
pauses). These defaults cover all assigned tools and connections, including
provider-native operations. Full auto is resolved consistently for agent
creation, adapter conversion, direct driver launches, and fresh/resumed turns.
Explicitly stored restrictive modes still apply.
Native OpenCode forwards incremental text and reasoning parts as they arrive; the final full snapshot selects the final response without repeating streamed text. Delta parts must match an observed message and part identity. Identical chunks without event IDs remain distinct tokens. Each isolated launch disables automatic updates, remote model-catalog refreshes, and default plugins; the runner supplies the pinned executable, selected model, and assigned MCP tools. Provider inference and reasoning still contribute to time to first text. These startup settings do not disable model reasoning.
On macOS, the verified OpenCode executable keeps its private launch pathname until the provider process exits. Removing that pathname immediately after spawn can kill or stall the signed binary before its health check succeeds. Ownership, permissions, and file-identity checks remain enforced; cleanup also runs on failed launches, and retries create a fresh launch snapshot.
Call session.startTurn({ message, reasoningMode: "disabled" }) to disable
reasoning for one OpenCode/OpenRouter turn. CapabilityLiveSession.sendMessage
accepts the same option in its second argument. The choice travels in the durable
turn.start command and selects an OpenCode model variant for that prompt only.
Omitting it (or passing "default") on the next turn restores provider defaults,
including on a warm or recovered session. It is not an environment variable or
agent-wide setting. No automatic decision policy selects it. The model must
support disabling reasoning; other runner providers reject an explicit selection
before starting work. Native sessions expose support through the
perTurnReasoning capability.
For the native openrouter/ route, the parent-held provider proxy retains the
upstream OpenRouter key. The child config keeps the selected openrouter provider
and exact model, using a random loopback token restricted to that model's chat
completions endpoint. The proxy and token expire when the session closes. The
local token remains a credential and is redacted from diagnostics; this does not
claim isolation for every child credential or persisted value.
OpenCode's proxy sends agent instructions once per turn through the system prompt. The task envelope retains task-specific constraints and the completion contract, without repeating the system instructions. This applies to all users of the proxy; already-recorded conversation history is not rewritten.
The runner's authenticated bridge and controller still enforce company access,
action claims, task modes, and governed approvals. Provider permission defaults
do not change workspace isolation or grant credentials or connection access.
approve-paperclip remains an optional narrower mode for assigned planning and
task tools; approve-reads allows assigned reads; deny-all rejects requests.
None of these restrictive modes is the default.
prepare_repository requires a provider permission decision in restrictive
modes, including approve-paperclip: repository acquisition can clone onto the
controller and is not automatically approved as a planning action. Full auto
still allows it; repository acquisition currently has no cumulative disk quota.
Restrictive profiles route supported permission decisions through durable runtime requests and the existing task interaction controls. Requests are persisted before presentation; answers are checked against the offered decisions and acknowledged by the sidecar before settlement. Unknown, stale and duplicate responses fail. Missing provider decision support remains a blocked disposition, not implicit approval. Company access checks still run for each Paperclip tool. Provider death expires pending promises; approvals are never replayed into a replacement process.
Automatic Paperclip/read allowances currently require the Claude SDK dispatch
boundary. Grok preserves these restricted settings, but its ACP requests lack
independently bound tool authority. Those operations require a supported operator
permission decision; a missing interactive responder stops with
approval_required. An explicitly selected approve-all policy permits unattended
Grok work in an assigned sandbox. Paperclip authorization and governed approvals
still apply.
Runnerd selects only qualified provider profiles. Claude Managed and AWS AgentCore receive immutable company-profile snapshots with explicit retention, spend, and invocation limits. No provider process receives a Paperclip API credential or unrestricted server environment.
Claude Managed resolves its API key from the company secret bound to the selected profile. AWS AgentCore uses workload identity only; long-lived static AWS access keys are intentionally removed from the runner environment.
The Rust core includes a bounded client for the sidecar protocol. It enforces request identity, event order, frame and queue limits, timeouts, redacted diagnostics, and process-group cleanup. Runnerd selects this package-local transport only through an exact qualified provider descriptor.
Before a later provider adapter consumes a valid sidecar event, the Rust core also requires its optional or mandatory run and turn scope to match the active execution. Process and diagnostic events can remain global. All operational, tool, input, permission, and terminal events require the exact active binding.
A package-local payload boundary decodes events only after that scope check. It validates control identities, terminal status, question sets, and the admitted runtime event types and bounded fields. It redacts diagnostic and retained event values again before they can enter provider state. Authoritative semantic tool arguments are validated for transport bounds and forwarded unchanged, including credential-bearing document and instruction content. The provider harness owns credential policy; redaction of logs and audit previews must not reject or rewrite execution arguments. Diagnostic detection requires explicit credential fields/assignments or recognizable key, Bearer, JWT, or PEM formats, not ordinary prose such as "credential handling" or dotted filenames.
Human question tools accept one complete payload.questionSet for text and
choice questions. The control plane generates legacy questions entries with
stable free-text option IDs. Legacy callers remain supported. Calls that supply
both forms must describe the same complete form; partial forms remain invalid.
The native recovery bridge uses the same projection for answer delivery.
The server validates canonical answer constraints before persistence. Regex matching runs in isolated workers with a one-second deadline and at most four active workers. A timeout or capacity error leaves the question pending. The ordinary and native answer paths both await this validation before persistence. Saved native answer delivery does not repeat regex matching.
Validated ACPX runtime events normalize into the same provider-neutral activity families as the direct Codex transport. Reasoning contents stay private. Tool targets are resolved within the workspace under the provider host's path semantics and receive a versioned sidecar boundary marker before becoming bounded, display-only PRP safe paths. Raw or unmarked provider locations fail closed. URI-scheme and Windows drive-shaped values require a separate sidecar attestation backed by an existing in-workspace entry or, for a not-yet-created edit target, an existing in-workspace parent. This preserves real POSIX colon filenames without treating arbitrary URI text as a path. Windows separators are canonicalized, and consumers must not reinterpret the display value as file-access authority. Operational semantic-result and terminal events remain reserved for the stateful adapter rather than being duplicated.
The ACPX provider reducer preserves that order while it tracks one active turn, bounded assistant text, semantic results, and pending tool or input correlations. Terminal events flush the final assistant message first and clear unresolved turn-scoped requests.
The package-local session bootstrap starts the bounded sidecar transport, verifies the qualified capability handshake and effective model, opens one identity-bound session, and confirms its run attachment. Any failed bootstrap terminates the process; session shutdown preserves persistent provider state. The session can then start one immutable-workspace turn, request interruption, and reduce polled events through the scope-first state boundary. A mismatched command acknowledgement or invalid event terminates the session fail closed. Polled semantic calls pass through the run-scoped authorized tool bridge before they can be returned to a caller. Before a follow-up turn releases settled tool receipts, runner-core suspends and reaps the idle sidecar/provider generation, then resumes the same verified persistent identity in a fresh generation. This prevents a late session-lifetime MCP callback from inheriting the next turn's event authority.
The Rust question-response validator checks the versioned response envelope against the exact persisted question IDs, answer modes, options, required answers, custom-answer policy, and text constraints before provider delivery. Tool results and structured question responses then use two-phase resolution: validate retained identity and schema, require the exact sidecar acknowledgement, and only then clear pending local state. Codex permission requests violate its pinned sidecar policy and terminate the session fail closed. Safe suspension is available only with no active turn or pending request. The sidecar must return the exact persistent session identity before runnerd terminates the local process. Already validated ACPX reducer events project into provider-neutral durable events only with an exact run, session, turn, and item binding. Raw sidecar envelopes and permission requests are not admitted at this boundary. A safely suspended session can be recorded as a bounded private checkpoint. The checkpoint binds the exact provider identity, run, catalog revision, and catalog digest and is replaced atomically before a later recovery attempt. Recovery releases the stored identity only after those bindings match the prospective session configuration exactly.
Run the complete contract gate with:
pnpm install --filter @paperclipai/paperclip-runner --lockfile=false --offline --ignore-scripts --dev
pnpm --filter @paperclipai/paperclip-runner verify
The verification command requires a stable Rust toolchain with cargo on
PATH, in addition to Node.js 24.11+ and pnpm 9+.
Rust development builds use debug = "line-tables-only" to reduce build storage
while retaining file names and line numbers in backtraces. The default test
profile inherits this setting. Incremental compilation remains enabled, and
release profiles keep their existing settings. For full debugger support,
including local variables and types, set CARGO_PROFILE_DEV_DEBUG=2 and
CARGO_PROFILE_TEST_DEBUG=2 for the build or test command.
Minimal Debian/Ubuntu hosts without root access can extract the required Playwright browser libraries into a user-owned cache and run the same acceptance sequence with:
pnpm --filter @paperclipai/paperclip-runner verify:rootless
The tracer's final line is stable:
{
"schemaVersion": "paperclip.runner.conformance.output.v1",
"runIdentity": {
"runId": "run_conformance_0001",
"sessionId": "session_conformance_0001"
},
"result": {
"status": "succeeded",
"summary": "Standalone Conformance fixture accepted."
}
}
Run only the tracer with:
pnpm --filter @paperclipai/paperclip-runner trace:conformance
Replay the Replay happy path, run a Local session, or open the browser devtool:
pnpm --filter @paperclipai/paperclip-runner replay:fixture
pnpm --filter @paperclipai/paperclip-runner trace:local-runner -- --scenario happy-path
pnpm --filter @paperclipai/paperclip-runner trace:codex
pnpm --filter @paperclipai/paperclip-runner demo:live-console -- --host 127.0.0.1 --port 4174
# Live console: chat with a live session in the browser.
pnpm --filter @paperclipai/paperclip-runner console:live-console
pnpm --filter @paperclipai/paperclip-runner browser:dev --host 127.0.0.1 --port 4179
# SDK: open the public-SDK reference console and mini consumer.
pnpm --filter @paperclipai/paperclip-runner console:sdk
# Standalone: run the standalone legacy/native/kill-switch tracer and page.
pnpm --filter @paperclipai/paperclip-runner trace:standalone
pnpm --filter @paperclipai/paperclip-runner trace:standalone -- --feature-flag enabled
pnpm --filter @paperclipai/paperclip-runner trace:standalone -- --feature-flag enabled --kill-switch enabled
pnpm --filter @paperclipai/paperclip-runner demo:standalone
Live console provider-backed routes are loopback-only and reject wildcard/LAN
binds. Browser mutations require same-origin Fetch Metadata, matching Origin,
and JSON content; see the protocol-server tutorial for direct curl examples.
The runtime activity publisher has an offline final-command regression against the qualified Codex executable. Run it on a host that supports Codex's OS sandbox, with the pinned executable and the matching freshly built Runner helper:
CODEX_NOTIFICATION_SANDBOX_COMMAND=/absolute/path/to/pinned/codex \
PAPERCLIP_RUNTIME_NOTIFICATION_HELPER=/absolute/path/to/paperclip-runnerd \
cargo test --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --lib \
publisher_crosses_real_codex_tool_boundary_without_runner_state_access -- --ignored
This uses app-server command/exec without credentials or a model call. It checks
actual shell discovery, successful durable activity recording, and denial of unrelated
Runner state, descriptor writes and ambient secrets. The normal suite covers
bounded framing, concurrent publishers, fixed-path cleanup and attachment fencing.
Direct live protocol qualification
The canonical direct live protocol suite lives in the separate
paperclip-evals repository under evals/paperclip-runner/. Its
live-mini.json roster is the complete 35-case Codex qualification lane. Build
this package's TypeScript output, release paperclip-runnerd, package tarball,
and dist-issue-thread viewer, then use the roster runner documented in that
repository. The package ships the required orchestration entry point as
paperclip-runner-eval-session (dist/cli/eval-session.js). Evalbook owns the
consistent HTML matrix and read-only attempt drill-down pages.
The hosted full-campaign workflow, parallel matrix, credential boundaries,
canonical report merge, and versioned S3 index are documented in
docs/runner-protocol-live-evals.md.
This direct protocol qualification is separate from the stress-derived Runner workflow schedule below and from the full-stack browser model E2E suite.
Stress-derived workflow, chaos, and AWS AgentCore operations
The deterministic workflow scorer and the chaos schedule do not require provider credentials:
pnpm --filter @paperclipai/paperclip-runner test:runner-workflow-evals
pnpm --filter @paperclipai/paperclip-runner report:runner-chaos-evals
report:runner-live-evals is a paid, provider-backed command. Native Codex
requires OPENAI_API_KEY; ACPX Claude requires
ANTHROPIC_API_KEY; OpenCode candidates require OPENROUTER_API_KEY. The live
matrix remains qualified-only and does not persist credential values. Candidate
qualification uses eval-session --candidate-profile <pi|cursor|copilot> with an
explicit model and a separately materialized pinned candidate pack. This option
is a constructor-bound diagnostic opt-in; session JSON cannot enable a candidate.
Missing credentials or unverifiable spend block paid qualification. Set
PAPERCLIP_EVAL_MAX_CAMPAIGN_COST_USD to a positive finite number to bound
additional scheduling after the observed campaign total reaches that value:
PAPERCLIP_EVAL_MAX_CAMPAIGN_COST_USD=12 \
PAPERCLIP_EVALS_ROOT=/path/to/paperclip-evals \
pnpm --filter @paperclipai/paperclip-runner report:runner-live-evals
# Run two scheduled native Codex executions only.
PAPERCLIP_EVALS_ROOT=/path/to/paperclip-evals \
pnpm --filter @paperclipai/paperclip-runner report:runner-live-evals -- \
--candidate codex-luna --limit 2
GitHub-hosted live campaigns additionally require the default branch, an
allowlisted numeric actor ID, the protected runner-e2e-paid environment, and
an explicit repository variable before scheduled runs are enabled. Manual
dispatches accept the same candidate, case, and execution-limit selectors. The
paid job uses the reviewed RunsOn Fleet label when RUNNER_E2E_AWS_ENABLED=true
and otherwise stays on ubuntu-latest. Uploaded reports contain redacted
observations and trace digests, not raw provider
frames, prompts, credentials, tool arguments, or hidden reasoning.
The AgentCore proof-of-concept uses an AWS CLI v2 profile to provision a
dedicated invocation role and scoped resources. Its local mode-0600 metadata
file contains no access keys; probes assume short-lived STS credentials and
clear them after use. Validate locally, provision or inspect the stack, run the
bounded lab/smoke, and tear it down explicitly with:
pnpm --filter @paperclipai/paperclip-runner test:aws-agentcore-provisioning
pnpm --filter @paperclipai/paperclip-runner aws-agentcore:provision -- --dry-run
pnpm --filter @paperclipai/paperclip-runner aws-agentcore:provision
pnpm --filter @paperclipai/paperclip-runner aws-agentcore:probe
pnpm --filter @paperclipai/paperclip-runner aws-agentcore:lab
pnpm --filter @paperclipai/paperclip-runner smoke:capability:aws-agentcore
pnpm --filter @paperclipai/paperclip-runner aws-agentcore:destroy -- --yes
To admit the hosted direct-eval workflow, provision with the account-local GitHub Actions OIDC provider and keep the default exact repository and protected environment binding:
pnpm --filter @paperclipai/paperclip-runner aws-agentcore:provision -- \
--aws-profile paperclip-dev \
--github-oidc-provider-arn arn:aws:iam::<account-id>:oidc-provider/token.actions.githubusercontent.com
This adds only repo:paperclipai/paperclip:environment:runner-e2e-paid as a
web-identity subject on the scoped invocation role. The generated nonsecret
profile records that role as both the local invocation role and the hosted
execution role.
Provisioning can incur Bedrock, AgentCore Runtime/Memory, storage, and private
networking charges. Provisioning refuses to modify a colliding stack unless its
Paperclip ownership tags and template description match. A verified
ROLLBACK_COMPLETE stack still requires --replace-failed-stack plus an
interactive confirmation (or --yes) before it can be deleted and recreated.
Destruction requires --yes and refuses to remove a stack with an active
recorded lab unless --force is also supplied.
Package-owned commands
| Command | Purpose |
|---|---|
build |
Compile the TypeScript public surface, Rust workspace, and browser devtool. |
typecheck |
Check TypeScript, Rust, generated schema sources, and browser types. |
test |
Run Rust/TypeScript fixture, supervisor, fake-driver, live/replay, and boundary tests. |
check:forbidden-imports |
Reject TypeScript imports and Cargo path dependencies that cross into Paperclip core. |
check:tracked-imports |
Reject tracked imports and package.json entry points that only resolve against untracked files, so a clean checkout of any commit builds. |
check:numbered-milestones |
Reject numbered construction-milestone names in tracked package paths and source. |
check:package-boundaries |
Enforce the acyclic runtime/testing/eval dependency and manifest boundary. |
check:clean-consumers |
Pack the runner and install its root, evals, and testing exports in a clean consumer. |
test:eval-slice |
Run the credential-free eval bundle, scoring, and behavior/fault slice. |
test:runner-workflow-evals |
Run the deterministic provider-neutral workflow matrix. |
report:runner-workflow-evals |
Validate deterministic fail-closed results and write JSON, Markdown, JUnit, and GitHub-safe reports. |
report:runner-live-evals |
Execute the paid provider schedule and render its immutable attempts with the canonical paperclip-evals HTML grid. |
report:runner-chaos-evals |
Write the credential-free eight-scenario chaos schedule. |
test:aws-agentcore-provisioning |
Validate the AgentCore template and wrapper safety contracts without provisioning. |
aws-agentcore:provision / probe / lab / destroy |
Manage the scoped AgentCore proof-of-concept lifecycle. |
smoke:capability:aws-agentcore |
Exercise the qualified AgentCore profile through the capability harness. |
check:conformance-parity |
Require byte-for-byte equivalent Rust and TypeScript tracer output. |
check:replay-goldens |
Require all reducer snapshots and cross-language summaries to match checked goldens. |
check:replay-parity |
Run TypeScript and Rust against the same Replay fixture summaries. |
check:browser-tokens |
Reject component-local visual literals and require the standalone token layer. |
docs:validate |
Validate local documentation links. |
trace:conformance |
Run the Rust mock-core tracer, print the stable result, and exit. |
trace:conformance:typescript |
Run the TypeScript reference tracer directly. |
replay:fixture |
Validate and reduce a fixture to a final snapshot. |
trace:local-runner |
Run one native local session through the Rust runner and fake harness. |
trace:codex |
Run the mock core with a real, local skillless Codex app-server session. |
demo:live-console |
Start the package-local HTTP/SSE server with server-only Codex authentication. |
console:live-console |
Start the standalone browser devtool with the Live console on 127.0.0.1:4180. |
console:sdk |
Start the public-SDK reference console and mini consumer on 127.0.0.1:4181. |
test:sdk |
Run targeted browser-client, reducer-projection, and React component contract tests. |
test:browser:sdk |
Exercise both consumers with the fake driver, keyboard/a11y checks, reconnect/replay, measurements, and screenshots. |
record:sdk:codex |
Run both public consumers against a safe real Codex session and capture live screenshots. |
check:capability-contract |
Verify the generated capability, legacy MCP, and eval traceability contract. |
check:semantic-contracts |
Verify the provider-neutral semantic tool contract is current. |
trace:live-runner |
Run the real runnerd/Codex semantic loop against the mock control plane. |
demo:scenarios |
Start the Capability scenario explorer over the mock control plane on 127.0.0.1:4183. |
console:issue-thread |
Start the Paperclip-style issue thread on 127.0.0.1:4184. |
test:scenarios |
Run the scenario index, run-artifact, parity, explorer component, and route tests. |
test:browser:scenarios |
Exercise both the scenario explorer and issue-thread browser contracts. |
browser:dev |
Start the standalone live/replay browser devtool. |
test:browser |
Exercise static replay and live scenarios, then capture temporary screenshots under ignored test output. |
verify |
Run the complete deterministic Conformance through SDK acceptance sequence. |
verify:rootless |
Extract Debian/Ubuntu browser libraries without root, then run verify. |
Navigate
- Architecture and dependency boundary
- ADR 0001: runner and testing package boundaries
- Tutorial index
- Conformance hand-run tutorial
- Replay hand-run tutorial
- Local runner hand-run tutorial
- Local protocol reference
- Durable transport reference
- Codex skillless Codex tutorial
- Codex skillless Codex driver reference
- Live console protocol/server tutorial
- Live console protocol/server reference
- Live console tutorial
- Live console reference
- SDK console tutorial
- SDK browser SDK reference
- SDK component decision record
- Capability semantic catalog and authorization
- Capability live runnerd/Codex loop
- Capability issue-thread UI
- PRP compatibility/versioning policy
- Adding a harness and permission-mode requirements
- PRP v1 expressiveness audit
- Cumulative end-to-end tutorial
Codex adds the package-local real-model reference driver, Live console adds the package-local browser console, and SDK extracts a reusable public SDK plus two standalone consumers. Runtime production Paperclip integration remains deferred; the App-owned production conformance adapter is test-only.
The SDK reference console opens in direct chat mode. Enter a normal prompt,
then open the protocol inspector to review events and reducer state. Expand a
Terminal row and its nested Debug details disclosure to inspect every
canonical event retained for that command. The header marker 🖇️ v0.1.2
identifies the current console iteration.
create_task accepts an optional initial status of backlog or todo. Use backlog when the user wants a saved task or plan without execution: assignment and the initial plan are committed without scheduling a wake, even when dependencies are already complete. Omitting status preserves immediate delegation (todo, or blocked for unresolved dependencies).
Linux provider-pack producers run on the corresponding native Linux target and
retain provider-pack.json with a relocatable provider-pack.tar.gz. Release
assembly binds both identities to the same source as its four daemons. The
historical remoteProviderPack field remains the Linux x64 identity. Linux
ARM64 packs do not qualify Pi, Cursor or Copilot on that target.
For an installed SSH host, install the same release producer's pack at
$HOME/.local/share/paperclip-runner/provider-pack or
/opt/paperclip-runner/provider-pack. A Mac controller selects its packaged
Linux target identity after probing the selected host. The runtime verifies the
installed pack before prompting. No development pack override is needed. A
missing or mismatched pack fails clearly. Existing explicit overrides remain
authoritative. ARM64 cloud-image production remains deferred.
Oversized OpenCode events
The OpenCode driver keeps SSE frames up to 1,048,576 characters exact. It compacts longer frames while retaining tool identity, status, and bounded output excerpts. This also applies when a complete large frame arrives inside one network chunk. Canonical tool rows mark shortened output, report its original byte count, and do not claim a digest for text that was not retained. A structure that cannot fit the compacted bound produces a diagnostic; the event stream continues.
Tool output and error fields have reserved retention space and size records, so large input values cannot consume their output tails or make excerpts look complete. A separator split between reads does not count against the frame limit.
OpenCode MCP tool results
The Runner-owned MCP bridge sends complete JSON tool results up to 1 MiB of
UTF-8 output. It does not slice JSON text. This includes the native server's
bounded 500 KB app-tool results and saved-result pages. OpenCode can put a long
result in its own output file; that file must still contain the complete JSON.
The existing read_tool_result artifact protocol handles larger app results.
Serialization has a bounded traversal budget and checks escaped UTF-8 output. If delivery exceeds the limit, the bridge reports an error without a partial success value. The action may already have completed. Reconcile its saved result or receipt instead of repeating it. Replaying the same MCP call identity reuses the original handler execution, including when delivery failed.
Prepared harness storage
An execution environment can supply an existing Claude configuration directory
through PAPERCLIP_RUNNER_PREPARED_HARNESS_HOME, using an absolute, non-symlinked
path. Runner derives CLAUDE_CONFIG_DIR from this value, overriding any separately
supplied value. This selects persistent harness storage without
changing the run's permission mode or enabling preconfigured-home behavior.
HOME remains the execution account's home. Each account must use its own
configuration directory. The environment must publish the selected skills before
admission; Runner does not copy or delete skills in this directory.
On Unix, the directory must belong to the Runner user and use mode 0700;
the provider lifetime lease rejects group/world access or another owner.
Runner binds the home into recovery identity and uses its existing provider lifetime lease. After acquiring that lease, it replaces Core-owned Claude settings with the current launch settings. History remains in the account directory. Task control files remain in the session runtime directory. Separate directories under one OS user are not a credential isolation boundary.
Preconfigured home
Set homeMode: "preconfigured" in a paperclip_runner agent's adapter config
(or the runnerd transport options) to use the execution account's existing home.
The default is "isolated". The initial supported providers are native Codex
and ACPX Claude. Preconfigured ACPX Codex fails before launch until its session
permissions can enforce denial of Runner state; use native Codex or isolated mode.
An operator must select this mode on the agent/Runner
configuration. Task-level overrides are rejected, and agent credentials cannot
set or inherit this option. Other providers reject this option. The internal ACPX task bridge uses a stable
session name so an existing user MCP server named paperclip remains available.
Prepare authentication, skills, and external MCP servers on the execution machine
before launching. Runner does not install skills, copy or remove credentials, or
rewrite user configuration in this mode. Providers can still refresh credentials
and update their own metadata. Runner still supplies the requested
model, task instructions, and its internal task/completion tools. Claude session
loads and resumes receive the same native session options as session creation,
including allowed tools and the selected settings sources. Local permission
rules and hooks apply. Native Codex tools can read and modify the execution home
(read-only for planning); credentials in that home are not isolated from tools.
Native Codex separately denies Runner-owned protocol state; ACPX Claude retains
its existing provider trust boundary. Execution is unattended: native Codex uses never, and
ACPX rejects requests for approval. When Claude starts in its home directory,
its native user settings retain user trust even if the SDK also reports that
same file as project settings. Other project settings keep the native trust filter.
The initial ACPX mode requires the Runner
setting acpxPermissionMode: "approve-all" (its default); other Runner modes are
unqualified here and fail before launch. Local permission rules still apply.
Preconfigured ACPX read-only/planning tasks also fail before provider launch
until a read-only launch is qualified; use isolated mode or native Codex for planning.
If local settings block a task, change them
and retry. Existing blocked results and failure diagnostics report the outcome;
a hook that also blocks the reporting tool can leave only a generic diagnostic.
Runner retains private protocol/session state and a fence for each session.
Different sessions may share the same home. Mode and home bindings fence recovery;
switching either requires a new session. Backups never include the external home.
Recovering a run uses its saved mode even if the agent configuration has changed.
Remote execution discovers the execution account's home on the remote machine
using getent; it does not copy the controller's HOME, XDG paths, or login state.
The external provider history must still exist to resume a session after a remote
machine is replaced. Missing history fails recovery rather than importing an
isolated home. OpenCode and the remaining ACPX profiles require separate
qualification before this mode can be enabled for them.
Assigned local MCP servers
An execution host can assign local stdio MCP servers alongside the existing
PAPERCLIP_NATIVE_MCP_NAME / URL / TOKEN HTTP gateway. This is general transport
support for local MCP implementations that satisfy the executable-verification
and private Unix socket requirements below. Runner uses each server's own tool
schemas and responses. Set
PAPERCLIP_NATIVE_MCP_STDIO to a JSON array of bindings with these fields:
[{"name":"local_tools","command":"/opt/local-tools/bin/mcp-server","args":[],"env":{"HOME":"/home/agent"},"sha256":"sha256:<64 lowercase hex digits>","serialResource":"/home/agent/.local/state/local-tools/runtime/service.sock"}]
serialResource identifies the shared resource for cross-process call locking;
the MCP transport itself uses stdio. It must name an existing private Unix socket
with the ownership and path requirements below.
Set PAPERCLIP_NATIVE_MCP_PROXY to the assigned Runner executable. These are
execution-host inputs, excluded from configurable task environment and provider
shell inheritance. Names use letters, digits, underscores or hyphens and cannot
replace the internal task server or gateway. Commands are verified against their
assigned digest before execution. The backend receives only its assigned environment.
Runner supplies these session-local bindings to native Codex and through the shared ACPX runtime. ACPX clients must support stdio MCP assignments. User or project MCP files are never rewritten. Read-only sessions omit these bindings. Host assignment authorizes native Codex to use these local tools without a new per-call prompt, matching execution-host utilities available through shell. ACPX adapters use the existing ACPX permission policy. Explicit native Codex tool deny/prompt rules remain effective; requested approvals use the existing PRP permission request, bound to the active thread and turn. Only one-time accept, decline or cancel decisions are accepted. Shell escalation stays disabled. Preconfigured Claude remains unattended and rejects approval requests.
The relay forwards tool schemas and native MCP content. For native Codex
0.160/0.162, image-bearing results put structured metadata in a text block instead of
structuredContent, because those clients otherwise discard the image blocks.
ACPX adapters receive the original response unchanged. Tool schemas and actions
are forwarded without provider-specific translation.
Provider-client smoke tests exercised native Codex 0.160.0 and Claude SDK 0.3.286 with local MCP servers and loopback model APIs. This records the clients used to verify image delivery and approvals; it is not an exhaustive adapter compatibility test.
All tools/call requests for the same private Unix socket are serialized across
sessions and processes, including reads. Metadata requests stay responsive.
Cancelling a queued request prevents dispatch. An active request retains its fence
until the backend acknowledges it; client cancellation cannot prove daemon work
stopped. If a relay exits with an unacknowledged call, later calls fail closed until
the host restarts that resource with a new socket. Do not remove its lock marker
while daemon work might still be running. The socket and its parent must be private,
canonical paths owned by the same account. Local stdio assignment requires Unix;
other platforms retain the HTTP path.
Kimi Code
Kimi Code uses the pinned @moonshot-ai/kimi-code@2.3.3 distribution through
ACPX through the qualified native profile. Ordinary selection and Runtime Test
require no candidate grant.
The installation targets are macOS ARM64 and Linux x64. Platform startup,
authenticated driver checks, and product acceptance are separate evidence levels.
Run paperclipai runtime setup kimi on the execution host to install its complete
verified runtime. Setup downloads public pinned dependencies without model calls
or provider credentials. An invalid existing installation fails without ambient
Kimi fallback. Bind a company secret as KIMI_API_KEY and select an exact model.
This release uses the global Code endpoint at https://api.kimi.ai/coding/v1.
Model suggestions are empty; a compatible typed model is decided by the provider.
The account needs Kimi Code entitlement. Runtime Test verifies installation and
queries the authenticated Code model catalog for the exact selected model. It
sends no model prompt; an actual task verifies inference and tool execution.
The qualified native profile uses ordinary setup, hiring, metadata Test and task startup on these two supported targets without operator admission flags. Protected candidate startup records remain reserved for future pending profiles; saved agent configuration and public requests cannot supply that authority.
Existing legacy Kimi agents retain their runtime and history. Managed Kimi connections, ambient login borrowing, other regions/platforms, steering, native child work, and per-run token/dollar accounting are outside this release. Read-only Kimi work uses scoped Paperclip tools; native filesystem and shell tools are denied. The provider-owned patch binds native instructions, preserves permission arguments, and removes provider auth from tool subprocesses. See the protocol dossier and release ledger for current evidence and limits.
ACPX controller replacement regression tests
src/control-plane/acpx-controller-recovery.test.ts runs the production Rust
Runner and authenticated PRP controller with a scripted ACPX peer. It covers
active-turn reconnection, checkpoint restoration after deleting controller
storage, native transport adoption and old-controller finalization, completion
while disconnected, a provider crash, and an explicitly closed provider.
It builds the debug Runner before execution and requires Cargo and a POSIX host.
No model credentials, provider requests, application server or database are used.
pnpm --filter @paperclipai/paperclip-runner exec vitest run \
src/control-plane/acpx-controller-recovery.test.ts --maxWorkers=1
Set PAPERCLIP_ACPX_REPRO_BINARY to an existing Runner executable to skip the
Cargo build. All five tests still run. Set PAPERCLIP_ACPX_REPRO_PROFILES to an
alternate acpx-profiles.json when that executable uses a different profile
manifest. Both overrides are optional.
The failure controls distinguish a lost provider from a controller outage; matching a diagnostic after explicit close does not identify the cause of a hosted failure. Real Claude/ACPX acceptance, server/Postgres recovery, and Daytona deployment continuity need their own evidence. See the remote continuity guide and the Daytona Product E2E suite.