diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index b4bec17b31c..13f3868869a 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -2649,6 +2649,26 @@ and `packages/adapter-utils/src/sandbox-callback-bridge.test.ts` for these contr ### Native runner restart recovery +For remote controller disk-loss work, use the existing control-plane snapshot +transaction and `native-controller-recovery.ts`; do not add a second snapshot +writer. The private versioned metadata is limited to 4 KiB. Its importer handles +one verified retained source while the source disk is available. Workspace and +provider-backup migrations remain with their storage owners. See +[the continuity verification guide](testing-native-remote-continuity.md) for +the direct/prepared transport and three-controller continuation fixtures. +An upgrade readiness report contains preliminary native evidence, not permission +to remove a deployment volume. Keep the checkpoint pool bounded and measure all +pools when qualifying database cost; the current size of two is a default. + +For retained Codex event tails, keep source drain/accounting and target admission +separate. `native-staged-attachment-recovery.ts` validates private source/target +header evidence; `native-recovery-usage-delivery.ts` binds the source's accounting +receiver. The transport persists its staged target before source bootstrap, then +uses the existing authenticated attachment protocol after durable drain. Do not +rotate the identity first, acknowledge before receipt persistence, or weaken the +Runner's settled-provider check. A staged target does not yet certify a resumable +session. Its eventual snapshot still uses the existing checkpoint transaction. + Custom source-mode HTTP launchers should create their listener with `createPaperclipHttpServer` from `server/src/http/server.ts`, passing the selected API origin as `apiUrl`. This registers the native Runner WebSocket transport @@ -2791,6 +2811,35 @@ and hard restarts with live and dead runners, the result-finalization race, incomplete bootstrap, repeated crashes with steering, and fail-closed process identity mismatches. +Native task-session publication is part of the terminal checkpoint transaction, +not resource cleanup. `native-task-session-publication.ts` owns its accepted-result, +owner, task-epoch and reset checks; heartbeat supplies prepared session metadata. +The goal-less regression in `native-session-resumption.test.ts` holds provider +cleanup open and then exercises a normal follow-up and an explicit session reset. +Control-port tests also cover terminal snapshots before accepted results and +result-replay contention with the status writer's coordinator-first lock order. +Remote adoption tests must invoke actual external-state preparation. Inspecting +transport options alone does not exercise the resumed-lease harness checks. +Include a healthy warm handoff before the crash so recovery exercises the +handoff's turn and item attachment IDs as well as first-run identities. +Retirement tests must exercise the preceding `turn.stop` as well as drain and +suspend after result acceptance. Use the actual native-observed, unclaimed +admission phase and the production claim before retiring the predecessor; +a synthetic legacy preparation does not cover that boundary. Verify failed +claims and Stop leave the predecessor and shared attachment staging untouched. +Configuration tests compare fresh and recovered +home-mode fingerprints while preserving rejection of real mode changes. +Include multiple skill versions read back from Postgres JSONB: version-map key +order must not change the generated preference or session fingerprint. Exercise +the real environment-selection and configuration services with a retained shared +workspace, and keep actual model or skill-version changes incompatible. +Run it against an isolated database with `PAPERCLIP_TEST_DATABASE_URL` when the +host cannot start embedded Postgres: + +```sh +pnpm exec vitest run server/src/__tests__/native-session-resumption.test.ts -t 'publishes the completed task session before provider cleanup' +``` + ## Version-based skill authoring and delivery Managed content uses `managed-skill-content.ts`: prepare and verify outside the diff --git a/doc/evals.md b/doc/evals.md index 353fe60e268..e396d676907 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -55,8 +55,9 @@ original bounds. Its source registration is not an AWS or TestOps pass. The broader YAML catalog remains planning inventory. The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/README.md) -extends the existing native restart oracle to a 125-second controller outage, -using the recorded Claude and Codex models in disposable development fixtures. +extends the existing native restart oracle to a 300-second controller outage +and another replacement after completion, using the recorded Claude and Codex +models in disposable development fixtures. It requires same-process adoption, saved files and finalization without replay; local regression checks and catalog discovery are separate from live acceptance. diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index c59bab4b3f6..9409d0d608e 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -2167,8 +2167,11 @@ digest. Earlier live qualification results keep their original profile identity. Healthy native warm turns retain their runner and provider processes. After a controller crash, continuation can recover a completed remote session from its -saved checkpoint and exact sandbox process identity. The prior controller must -be stopped, its result committed, and the successor must hold current task and +saved checkpoint and exact sandbox process identity. For an externally persisted +controller with validated recovery metadata, the prior lease must be released +and its result committed. A PID in the replacement container does not establish +whether the prior remote controller is alive. Legacy file-backed recovery retains +its stopped-process check. The successor must hold current task and coordinator authority. Recovery reconnects through the prior run's route before the authenticated run attachment changes authority. Unavailable identity probes have a shared 30-second retry budget; an identity conflict never permits replay. @@ -2647,3 +2650,159 @@ The existing prior-owner, suspended-state and backup checks still decide whether the next turn may start. Missing checkpoints and ambiguous identity remain blocked; restoration does not authorize a new provider session or change the configured lifecycle policy. + +Remote admission also handles the absence of both native controller projections. +`native-controller-recovery.ts` selects the existing bounded task/session source +and restores its encrypted PRP records before admission reads local identity. +It reconstructs the supervisor envelope from `runnerProfileJson.sessionCheckpoint` +and the private `nativeControllerRecovery` metadata. Missing, incompatible, or +mixed evidence holds the continuation; it does not authorize fresh execution. +The 32-candidate boundary and newer-progress barriers remain in force. Exhausted +history and current-run progress cannot establish fresh-bootstrap proof. + +A settled warm journal may contain an authenticated idle goal-read snapshot +after `run.terminal`. Recovery accepts only the supported PRP event v1/v2 +envelopes with the same complete attachment identity, a null goal and +`workingNow: false`. All commands must already be complete. Active goals, new +work, unknown versions and mixed bindings remain holds. This check adds no +streaming persistence or database work. + +If admission finds a live retained controller owner, it preserves the unstarted +business run and records the owner's actual deadline through the existing native +resumption path. This does not consume a provider attempt or stop the retained +sandbox. A locked run/coordinator check rejects Stop, changed ownership or any +provider progress. Startup restores even an overdue wait to the same scheduler; +ordinary resumption rechecks current permissions and task authority. No idle +scan is added. Per-turn continuation reconstructs a missing suspended-state +receipt from the admitted sandbox or prepared-task owner, including after a +previous read failed with only a PRP projection written. + +The version-one metadata contains a digest of the exact redacted stored snapshot, +configuration digests and the PRP checkpoint binding. It is bounded to 4 KiB, +excluded from public run projections, and saved by `checkpointSession` in its +existing snapshot transaction. No second snapshot writer, per-text-chunk query, +pool, or recurring scan is added. A null checkpoint binding records an incomplete +publication and prevents expiration. Snapshot and result writes recheck the +coordinator generation and lease after lock waits, including the existing limited +post-result maintenance exception. + +Goal-less native turns publish their task-session pointer with the first accepted, +successful terminal checkpoint, before provider or instruction cleanup can delay +the heartbeat completion callback. The pointer uses the native normalized UUID +and the checkpoint's provider display identity. Its prepared configuration metadata +comes from normal admission; neither streaming events nor ordinary checkpoints +add task-session writes. Goal checkpoints retain their existing publication path. + +The snapshot transaction validates the exact accepted result and current owner, +then updates only the authenticated run's existing task-session projection. Its +private `nativeTaskSessionPublication` receipt shares the 4 KiB recovery-metadata +budget. Coordinator, issue and run locks precede the ownership clock check; the +issue's `NO KEY UPDATE` lock remains compatible with result insertion's parent +`KEY SHARE`. Reset/deletion, a newer task-session epoch or a competing execution +prevents publication. Later maintenance and heartbeat completion honor the receipt, +so they cannot restore a cleared pointer or overwrite its successor. Unsupported +legacy bindings keep the existing completion path. The receipt is excluded from +public run projections and contains no session parameters or credentials. + +A terminal snapshot can arrive before result acceptance. That snapshot publishes +neither a continuation pointer nor a handled publication receipt. The existing +post-acceptance checkpoint completes publication in its normal transaction. +The first production release containing recovery readers, complete attachment +identity handling and redaction for both private recovery fields is the +disk-independent rollback floor. Earlier +development-reader probes do not establish a published rollback target. + +Active remote adoption and compatible idle warm reuse carry an existing verified +Runner proof. Preparation revalidates that proof and the exact process's liveness, +then preserves its in-place harness state. The later protocol attachment still +authenticates the durable PRP authority. A provider lease acquisition described as +`resumed` does not establish that this already-running harness was suspended. +Ordinary resumed leases retain their verified suspension and backup checks. + +Reattachment keeps the complete durable PRP binding, including its turn and item +attachment IDs. A healthy warm handoff can create new attachment IDs within the +retained Runner. Replacing that run's controller must preserve those IDs; deriving +them again from the business run ID would request an invalid authority rotation. + +Cold Codex continuation can restore a completed provider with a retained event +suffix. Native remote controllers first persist a private staged attachment under +the target run's existing checkpoint owner, then restore and drain the exact +source authority before attaching the target. The source's run, turn and item +identity stays on every old event. Receipt persistence precedes acknowledgement; +unknown semantic work holds the handoff and never reaches the target's tool +authority. The provider's settled-session attachment guard remains mandatory. + +The staged controller format is +`paperclip.runner.durable.control-plane-state.staged-attachment.v1`. It contains +the source and target attachment tuples and one bound target attachment template +inside the existing encrypted PRP authority, subject to its existing size limits. +The header stores a bounded descriptor and digest. Before command delivery, the +template is a separate changed-record part; after queueing, recovery uses the +existing command or receipt. Control checkpoints do not rewrite another copy of +instructions or tool context in the header. +It is separate from the 4 KiB run-profile recovery envelope. Startup discovers +this target obligation through bounded encrypted-header reads before a target +session snapshot exists. It verifies both frozen runs, the accepted and released +source, its original Runner artifact and the exact source header. Remote process +verification then chooses authenticated source adoption or verified suspended +source restoration. An in-memory rebound snapshot supplies normal admission; +an existing target snapshot always takes precedence over that source baseline. +The intent remains through command delivery and authenticated handoff, including +an uncertain commit response. Source and target receipt phases require their +exact physical authority; receipt replay cannot start a second Runner or attach +again under a different endpoint. After the ordinary target snapshot commits, +an in-memory notification permits intent retirement in the next existing command +write. There is no separate metadata/snapshot transaction for that retirement. + +Source accounting uses the source's immutable producer receipts and billing +classification. Opening this receiver does not reset accounting ownership or +reopen a finalized run. Equal finalized usage retains the existing audit policy; +changed, incomplete or conflicting usage remains unacknowledged for +reconciliation. The target's usage receiver never receives source events. Staging +adds recovery control work only: text streaming and settled idle operation add no +queries, snapshot transactions, pools or recurring scans. The first rollback +release supporting this continuation must also read the staged controller format; +older readers fail closed. Prepared-launch and other provider paths retain their +existing admission and recovery contracts. + +Accepted-result retirement still stops the old provider turn before draining and +suspending its Runner. The checkpoint codec classifies `turn.stop` as lifecycle +maintenance. Existing exact ownership, generation and accepted-result fences +govern this cleanup; starting provider work still requires an active owner. +A successor claims its existing native coordinator and starts renewal before +retiring a superseded warm session. An unclaimed preparation cannot authorize +the predecessor's checkpoint writes. Shared attachment staging follows verified +retirement; a failed claim or Stop cannot close that predecessor or stage files. +Recovery also preserves the implicit isolated-home default when binding admitted +configuration, so an unchanged home does not invalidate the retained session. +Explicit home-mode changes retain their normal configuration-change checks. +Restoring pinned skill versions also preserves the fresh selection's existing +name/key order. JSONB map ordering must not change the generated skill preference +and invalidate the session when the selected keys, versions and bytes are unchanged. + +For per-turn direct sandbox recovery, the executor reads the retained Runner's +suspended lifecycle through the recorded sandbox lease and validates its complete +identity and lack of pending delivery/cleanup. Only a lifecycle projection is +cached locally; provider files remain on the execution host. Prepared tasks use +their existing `read_state` contract. The normal snapshot rebinder, tool-contract +checks and current admission authority still govern the next turn. + +An active remote owner causes startup recovery to retain the obligation at the +actual lease deadline. The existing recovery dispatch timer is rearmed on drain +resume; successful reconciliation leaves no recurring recovery work. +Follow-up waits retain their original dispatch mode and owner across a drain, +including when another heartbeat service facade resumes the work. A failed +dispatch retains a bounded retry obligation rather than losing its deadline. +Upgrade diagnostics retain their existing `ready` meaning and add bounded retained-pointer +header evidence. That inventory is preliminary, reports partial coverage, and +does not certify workspace durability or volume removal. + +`importNativeControllerRecovery` is an internal one-source operation for the +migration owner while old controller files remain available. It requires the +admitted configuration digest and exact supervisor snapshot, locks the issue, +coordinator and run in order, rejects live or unsettled ownership, and publishes +only missing metadata. Repeating the same import is a no-op. Normal lifecycle +checkpoints publish the metadata without a separate migration write. Disk-free +rollback requires a server that reads this envelope and the supported settled +warm journal, including its PRP v2 idle goal suffix. Earlier local-only readers +and readers lacking that suffix handling remain outside that rollback contract. diff --git a/doc/plans/2026-10-30-native-controller-checkpoints.md b/doc/plans/2026-10-30-native-controller-checkpoints.md new file mode 100644 index 00000000000..8c4780f08aa --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-checkpoints.md @@ -0,0 +1,583 @@ +# Native-controller recovery during webserver upgrades + +Date: 2026-10-30. Status: implementation delivered; final qualification is tracked +in [PR #18343](https://github.com/paperclipai/paperclip/pull/18343). +Source inspected: `0c70cedce9a803bdc201324528abb35e423e7cd0` in the current task worktree. + +## Goal and finish line + +Make native controller state independent of the main Paperclip webserver's disk, +as one prerequisite for zero-downtime webserver deployments. A person starts a +task, the webserver is replaced, and the same remote execution continues. After +completion, another replacement can continue the retained session without the +previous webserver's files. Recovery must preserve Stop, current authorization, +accepted results, pending tool decisions, and single-owner execution. + +The implementation endpoint is a tested, reviewable PR for the native-controller +portion only. General HTTP readiness, traffic switching, request draining, +browser delivery, storage migration and Railway rollout belong to other chats. +Native shutdown integrates with the existing server lifecycle coordinator. + +Initial supported environment: native remote Runner execution, external Postgres, +a stable externally supplied instance encryption key, surviving remote sandbox +storage, and mutually compatible server/Runner releases. Qualify the existing +Daytona path, including its prepared-task transport. Preserve local/self-hosted +behavior. No new database, queue service, or shared filesystem is needed. + +Keep the existing outage contract: a 300-second controller outage, with recovery +completed within the Runner's existing 420-second reconnect grace. The remote +controller lease is currently 60 seconds. The 300-second fault-tolerance test is +not the permitted downtime of the webserver deployment. Do not shorten it or renew the Runner's +disconnected lifetime on each retry to make a test pass. This contract permits +server-dependent tools to wait; it does not promise uninterrupted HTTP service. + +## What already exists + +The earlier [Explore zero-downtime Railway deploy](codex://threads/01a1836e-8c8e-7b83-b09c-5db49a5e89de) +plan included remotely persisted native controller checkpoints. Much of that work +has since landed. Build on it instead of introducing another checkpoint store. + +| Existing capability | Owner / evidence inspected | +| --- | --- | +| Encrypted, company/task/run-bound PRP checkpoints in Postgres | [`native-runner-checkpoint.ts`](../../server/src/services/native-runtime/native-runner-checkpoint.ts), [`native_runner_checkpoints`](../../packages/db/src/schema/native_runner_checkpoints.ts); landed in `09ad9adce2` / PR #17439 | +| Opaque checkpoint codec, restore, authenticated replay cursor and persistence-before-effect barriers | [`durable-prp-control-plane.ts`](../../packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts) | +| Expiring ownership, controller generations, stale-writer fencing and isolated renewal/checkpoint pools | [`native-restart-recovery.ts`](../../server/src/services/native-runtime/native-restart-recovery.ts), [`native-execution-lease.ts`](../../server/src/services/native-runtime/native-execution-lease.ts) | +| Active remote reattachment from DB state and the admitted Runner artifact | [`native-session-executor.ts`](../../server/src/services/native-runtime/native-session-executor.ts), `executePaperclipNativeSessionWithinScope` | +| Bounded database discovery of prior task/session checkpoints | [`native-session-resume.ts`](../../server/src/services/native-runtime/native-session-resume.ts) | +| Empty-controller-directory, receipt-recovery, concurrency, retention and SQL-cost tests | [`native-runner-restart-recovery.integration.test.ts`](../../server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts), [continuity runbook](../testing-native-remote-continuity.md) | +| Read-only active-run upgrade eligibility | [`heartbeat/upgrade-readiness.ts`](../../server/src/services/heartbeat/upgrade-readiness.ts); landed in `567c043b17` / PR #18037 | + +The related [Enable uninterrupted agent upgrades](codex://threads/01a15e84-545f-7023-a854-9a642cbc76e1) +chat reports actual PAP staging replacements preserving active executions. It is +also working on broader default enablement. Those are reported results from that +chat, not a new qualification of this checkout or proof that the entire webserver +filesystem can be discarded. Reconcile that work before implementing overlapping +changes; do not duplicate its rollout or configuration work. + +## Remaining dependencies and evidence gaps + +1. **Ordinary continuation still discovers PRP state through local identity.** + `restoreWarmRunnerdController` and `restorePerTurnRunnerdController` first read + the local full state or external marker. They return when neither supplies a + matching identity. Active restart recovery instead starts from a DB run + binding. Extend that approach to ordinary continuation without skipping the + existing prior-owner, session, configuration, and suspended-state checks. +2. **Warm supervisor state has its own local persistence path.** + `persistWarmNativeCheckpoint` writes `paperclip.native-session-supervisor.v1` + with the snapshot, configuration digest, and process-configuration digest; + `loadWarmNativeCheckpoint` and warm process reuse read that file. The separate + Postgres PRP restore writes only `control-plane-state.json`. Existing saved + `runnerProfileJson.sessionCheckpoint` and admitted execution input provide + reusable data, but the complete warm reuse decision must become reconstructible. +3. **Current focused tests do not establish the whole application contract.** + The empty-directory transport fixture restores explicitly and sets the old + lease expired. It is valuable protocol evidence, but does not prove normal + admission with no markers, natural ownership expiry, HTTP routing during + overlap, or a second fresh-server continuation. Existing supervisor/Product + E2E coverage may prove some of these; reconcile it before adding new cells. +4. **Upgrade eligibility is narrower than disk independence.** + The current report checks active runs and bounded PRP headers. It does not + certify completed retained sessions or dereference workspace recovery files. + An empty active-run list must not imply that all controller history is safe + to discard. + +These are inspected dependencies and missing proof, not newly reproduced +production failures. Begin with characterization tests to determine which paths +already reconstruct sufficient state through their callers. + +## Design + +### One durable source for each kind of state + +| State | Durable owner | Local treatment | +| --- | --- | --- | +| PRP authentication, command receipts, control events and replay state | Existing encrypted `native_runner_checkpoints` | Reconstructed cache/marker | +| Run authority and finalization | Existing `native_run_finalizations`, run and issue locks | No local substitute | +| Provider/session snapshot and native reuse compatibility | Existing run profile plus a versioned native recovery envelope where fields are missing | Reconstructed supervisor projection | +| Session lookup | Existing task-session pointer and bounded prior-run selection | Never depends on directory enumeration or a marker | +| Provider process, provider history and unacknowledged outbox | Surviving remote Runner/sandbox | Never copied from an older controller cache over live state | +| Workspace descriptors, manifests, archives and provider backup blobs | Their existing workspace/backup owners, with remote durability supplied by that workstream | Logical references; independent eligibility requirement | + +Keep Runner's codec and opaque record contract in the Runner package. Native +runtime owns session selection, compatibility, and the server-side recovery +envelope. The database adapter owns encryption and transactional fencing. +Heartbeat only coordinates these contracts. Extract the affected persistence and +lookup responsibility from the large executor into a cohesive native-runtime +module rather than expanding the executor or creating a generic recovery system. + +### Reconstruct from durable identity + +Use the exact active run for restart adoption. For a subsequent turn, use the +existing task/session pointer and `native-session-resume.ts` selection policy to +locate the eligible prior run. Validate company, task, agent, normalized session, +workspace scope, admitted execution, provider configuration and current authority. +Retain the existing progress barriers: a newer incompatible or incomplete record +must not make recovery select an older convenient checkpoint. + +Load and validate the PRP checkpoint and native snapshot before materializing +local projections. The destination is computed from the replacement's local +root; absolute paths recorded by the old controller never identify durable data. +An existing local projection must agree with the selected committed state or be +replaced under the same authority. Missing, corrupt, undecodable or mismatched +state remains a specific recovery hold, without a fresh provider turn. + +Default persistence design: reuse `runnerProfileJson.sessionCheckpoint` and add +only a private, versioned native recovery envelope for missing compatibility and +checkpoint-binding metadata. Persist the snapshot and its metadata together +under the current run's fenced ownership. Do not duplicate the PRP journal or +store server-specific fields as arbitrary Runner checkpoint parts. If existing +profile metadata already supplies a field, reference it instead of copying it. +Use a schema migration only if characterization demonstrates that the existing +run/session records cannot express a required durable lookup or concurrency check. + +The private `nativeControllerRecovery` envelope is limited to 4 KiB serialized. +It contains a format version, digest of the exact redacted stored snapshot, +native configuration digest, process-configuration digest or explicit absence, +and PRP attachment/checkpoint binding. Run/company/task/agent/session identity is +validated against existing records and the snapshot, not duplicated execution +inputs. Keep credentials in the +existing encrypted authority; persist logical resource identities, not secret +values or host paths. Any newly sensitive recovery material uses the existing +encryption provider and remains excluded from public run projections. + +`PaperclipControlPlanePort.checkpointSession` already persists the snapshot before +the local supervisor callback. Add the envelope to that same transaction and +update; do not introduce a second snapshot writer. Compute stable compatibility +inputs once per admission/attachment, preserve unrelated profile fields, and +recheck durable ownership after lock waits with the established lock ordering. +Keep legitimate post-result suspension/cleanup checkpoints permitted. The generic +Runner persistence port must not learn Paperclip database fields. Remote local +files become projections; local-only consumers retain authoritative files. + +The recovery owner resolves, validates, materializes and attaches through existing +owners. Outcomes distinguish recoverable, waiting for an owner with its actual +deadline, missing/inconsistent evidence, unsupported format/capability, dependency +unavailable and proven no-prior-execution. Missing evidence is not fresh bootstrap. +Keep the 32-candidate selection window and newer-progress barriers; fetch the full +payload only for the selected source. PRP attachment identity and current business +run write authority differ during warm handoff and must remain explicit. + +Publish a resumable snapshot only after its referenced PRP boundary is durable. +Across separate commits, interruption may leave a pending obligation, never a +false ready state. Recovery reconciles that exact obligation without rerunning +the provider. Completion acceptance, workspace settlement, session reuse, and +cleanup remain separate facts. + +### Preserve ownership during replacement + +The new server may load recovery data while the old server serves, but it cannot +send commands, acknowledge events, or settle results until it owns the execution. +Reuse the existing boot identity, generation and lease-owner fences. Cross-host +takeover uses DB lease authority, never a local check of another container's PID. + +Graceful shutdown fences new local controller work, completes outstanding durable +barriers, detaches supported sessions, and releases only its own authority. A +crash uses natural expiry. The successor takes the existing fenced claim, +restores state, revalidates permissions and attaches to the exact remote process. +Lost authority detaches the stale controller without killing the surviving Runner. + +Retain wake/recovery obligations while the previous lease is live. Reconcile at +startup and the actual lease/retry deadline through the shared scheduler. Cover +a lost local release notification with durable state and the deadline. Add no +permanent checkpoint scans or independent polling timers. A future shared +cross-instance notification mechanism can accelerate this path without becoming +its source of truth. + +Exercise Stop and tool decisions arriving at either instance. Use existing durable +intents and receipts; fix a demonstrated missing native delivery path in its +owner. Never retransmit an uncertain external mutation just because its HTTP +response was lost. Current access, pause, cancellation and budget controls still +apply; restoring an admitted run is not a new grant of authority. + +### Compatibility, adoption and retention + +Use additive readers/writers. Before an ephemeral deployment is eligible, its +rollback target must understand the recovery envelope, sparse checkpoint policy +and authenticated cursor. A v2 decoder alone is insufficient. Preserve the +original Runner artifact identity; an old running binary does not gain new +capabilities merely because the webserver upgraded. + +For existing file-backed sessions, allow the current legitimate owner to publish +a validated durable snapshot at a normal checkpoint boundary. A bounded explicit +backfill may import retained supervisor state while the disk is still available, +after rechecking the same bindings and ownership. Never fabricate missing +credentials, checkpoints or remote process evidence. Unprovable sessions remain +ineligible until normal completion/retirement or explicit recovery. + +Keep local mode intact. Require the same externally supplied encryption key on +replacement; a recovery read must not generate a new key. Broad external-mode +startup validation belongs to the parent persistence workstream. + +Extend existing retention reference checks for any new envelope references. +Retain active owners, unresolved handoffs/finalization and selected continuation +sources. Cleanup must not race a new continuation claim or allow a late writer to +resurrect deleted authority. Preserve bounded startup cleanup and zero idle SQL. + +Expose an additive, versioned **native controller durability** diagnostic through +the existing internal upgrade report, with explicit unknown/ineligible reasons. +It must distinguish active adoption from retained-session continuation, and it +must never claim whole-instance volume removal readiness. Keep the existing +`ready` meaning compatible; the broader deploy coordinator composes this evidence +with workspace and other persistence checks. Derive eligibility from committed +records and supported readers, not a persisted boolean that can become stale. +Report active adoption, selected retained continuation, outstanding persistence or +handoff obligations and unsupported evidence separately. Include observation time, +bounded reason counts and coverage/truncation. Use bounded batched metadata/header +reads and pagination, and report incomplete inventory as unknown coverage. Add no +public endpoint. Establish the first envelope-capable release as the rollback +floor; an older local-only reader is not a valid empty-disk rollback target. + +## PostgreSQL load contract + +- Text streaming adds zero recovery SQL, transactions or checkpoint rows per chunk. +- Ordinary usage remains coalesced into control checkpoints. Preserve changed-part + PRP writes and exact removals; do not add a journal, transcript or archive copy. +- Snapshot metadata rides the existing snapshot transaction, including unchanged + metadata; no parallel writer, per-agent connection or new pool. +- Idle servers add no recurring SQL/network work once obligations settle. +- Scoped candidate selection fetches only the selected payload. Diagnostics use + bounded headers/metadata, never an automatic scan of historical payloads. + +Keep the existing dedicated checkpoint pool initially with its two-connection +default, lazy creation, idle release and shutdown fencing. Two is not an +architectural invariant. The invariant is bounded checkpoint traffic that meets +recovery deadlines without starving requests or renewals. Include both servers' +pools during overlap. Measure queue wait separately from SQL and lock wait. If +capacity fails acceptance, compare bounded sizes under the same workload and +choose the smallest passing size in a separately reviewable tuning change. Do not +increase the cap to conceal slow queries or lock contention. + +Run paired baseline/candidate workloads with identical fixtures and counts and +all database clients included. Record statement/transaction classes, changed rows, +persisted bytes, WAL, CPU, memory, connections, request latency, checkpoint queue/ +lock/execution latency, renewal latency/deadlines, throughput, retries and failures. +Use the existing 8/512-chunk test plus a stream beyond the retained-event window; +require exactly zero extra recovery SQL during text-only intervals. Repeat the +documented 300-controller/10,000-historical-task steady and aligned-burst workload, +alternating order and including actual session snapshot persistence. + +Gates: unchanged lifecycle sequence has no additional normal checkpoint +transactions; no new idle work, missed lease deadlines, lost owners, dropped +operations or premature ACKs; existing absolute latency/throughput gates pass. +Investigate any repeatable increase above 10% in steady CPU per completed operation, +WAL per completed operation or request p99 before handoff; report absolute values +too. Measure replacement bursts separately within the 420-second grace. Never +relax persistence-before-effect guarantees to meet a performance target. + +## Implementation slices + +| Slice | Owned surfaces and useful result | Dependency / review evidence | +| --- | --- | --- | +| 1. Complete native recovery state | Native checkpoint adapter, executor persistence/lookup, session-resume owner and focused tests. Normal active, warm and per-turn paths reconstruct with no local marker or supervisor file. | Reuse landed checkpoint work; first reproduce the exact missing path. Test current local mode and incompatible-session barriers. Add only demonstrated missing envelope fields. | +| 2. Replacement ownership and compatibility | Native lifecycle, shared recovery scheduling, shutdown wiring, retention and internal upgrade diagnostics. Competing servers hand off safely, and historical sessions have an honest eligibility path. | Slice 1; real Postgres ownership races, natural expiry, stale callbacks, Stop, migration/read compatibility and rollback tests. Keep heartbeat wiring small. | +| 3. Application and hosted qualification | Existing restart supervisor/Product E2E suites, continuity docs and acceptance evidence. Two replacements preserve execution and later continuation. | Slices 1–2; workspace durability dependency for full-home cases. Real immutable A/B releases and compatible rollback; no synthetic claims of Daytona qualification. | + +Tests and documentation travel with the behavior they verify. These are review +boundaries, not a fixed PR count. Combine tightly coupled implementation where +splitting would create an unusable intermediate state. Rebase onto the current +default branch and reconcile the active upgrades chat before writing source. + +## Acceptance matrix + +| Scenario | Required result and independently inspectable evidence | +| --- | --- | +| Fresh active remote run; graceful replacement | Same run, Runner process/start identity, provider session and active turn; one provider start, one final result | +| SIGKILL and absent controller storage | Normal startup claims after natural expiry; no manually edited lease, injected recovery marker or manual restore call | +| Output-only gap and pending tool/decision | Authenticated cursor recovers; pending input settles once; independent business receipt/effect count confirms no duplicate mutation | +| Completion while disconnected or during shutdown | Accepted result, accounting and finalization survive; incomplete persistence resumes without another provider turn | +| Completed warm and per-turn sessions | B can discover the correct source from DB; a later C continues it after B's controller storage is removed; no new provider session solely because a marker was missing | +| Warm config changes | Verified close/suspend and permitted saved-history resume; never incorrectly reuse changed credentials, tool contract or workspace | +| Old and new server overlap | B respects A's live lease; only one successor wins; delayed A cannot checkpoint, ACK, dispatch or finalize after losing ownership | +| Stop, revoked access, pause and cross-company attempts | Saved intent and current authorization prevail; no recovery bypass or delayed mutation | +| Corrupt/missing checkpoint, key or unsupported version | Precise ineligibility/hold; no silent fallback, replay, key generation or destruction of the last recoverable copy | +| Existing records and rollback | Valid local state can be imported under authority; unsupported state stays explicit; compatible prior release reads newly written records | +| Cleanup and idle | Selected checkpoints survive cleanup races; settled/idle states issue no new recurring SQL or network work; next legitimate producer reactivates recovery | +| Full application with empty homes | A → B → C uses the normal API/admission path and independently verifies the saved result; blocked until required workspace/backup dependencies are remotely durable | + +## Feedback path and verification + +First experiment: extend the existing real-Postgres remote continuation fixture +with a completed warm turn, remove both PRP marker and supervisor directories, +then submit the next turn through normal admission. Keep the remote Runner alive. +Assert the prior run/session chosen, process identity, one new turn and a useful +result. Also run the per-turn control. Do not prepopulate the desired recovery +state. This distinguishes a lookup/envelope gap from missing workspace material. + +Use the existing local Linux/container harness and scripted provider initially; +it avoids paid model calls while preserving database locks and process lifetime. +Measure setup separately from edit → run → inspected evidence. Start with a short +outage; run the real 300-second outage after the narrow behavior passes. Record +elapsed time and the exact revision in this plan when implementation begins. + +Then exercise separate compatible application revisions with distinct native roots +and shared Postgres, through normal startup/admission. Remove access to A's native +controller storage. Identify surviving non-native dependencies explicitly: this +is not evidence of whole-webserver filesystem loss. Test both SIGTERM and SIGKILL, B starting +while A still owns the run, decisions/Stop during handoff, and a second B → C +replacement. Retain an independent effects ledger and process identity evidence. +If this reaches a workspace/backup dependency, keep that failure as explicit +parent-workstream evidence; do not hide it with a copied controller directory. + +Qualify one disposable Daytona environment using the existing Product E2E path, +including completion and follow-up, plus a compatible rollback. Record server +revisions, Runner artifacts, run/session/sandbox identities, lease timestamps, +receipt counts, recovery latency and cleanup. Reuse an existing healthy campaign +only after checking its owner and exact revision. No fleet rollout is part of +this checkpoint feature. + +Required handoff checks for implementation: focused affected suites, then +`pnpm -r typecheck`, `pnpm test:run`, and `pnpm build`. If schema changes prove +necessary, use `pnpm db:generate` and the migration/snapshot checks. Update +`doc/execution-semantics.md`, `doc/testing-native-remote-continuity.md`, relevant +developer/database documentation and affected capability maps. Run +`node scripts/check-maps.mjs` after map edits. PRs use the complete repo template. +Report skipped or unavailable checks without promoting older evidence to a pass. + +## Parent-workstream dependencies and non-goals + +Native checkpoint success alone does not permit removing the webserver volume. +Workspace sync still resolves local descriptors/manifests in +[`native-workspace-sync.ts`](../../server/src/services/native-runtime/native-workspace-sync.ts). +That owner must provide durable logical recovery references and saved generations; +this plan consumes that contract rather than moving workspace archives into PRP. +Provider backup blobs and prepared resources needed after a process has suspended +must likewise resolve without the old server disk. + +Agent files/instructions, logs and accounting journals, imports, mutable packages, +global background-work claims, browser live-event delivery, HTTP readiness/drain, +and Cloud/Railway volume migration retain their own slices in the broader plan. +Do not turn native checkpoints into a second transcript: the existing sparse +checkpoint excludes display-only output, and this work does not strengthen the +run log's durability guarantee. Destroyed sandbox recovery, legacy local stdio +continuity and permanent horizontal scaling are outside this release. + +Railway still prevents overlapping deployments on a service with an attached +volume. Its separate overlap/drain controls become useful after all required +volume dependencies and concurrent-server behavior are qualified. Sources checked +2026-10-30: [volume caveats](https://docs.railway.com/volumes/reference#caveats), +[deployment teardown](https://docs.railway.com/deployments/deployment-teardown). + +## Current state and next action + +The observations below retain earlier qualification attempts and their limits. +The PR's Verification section is the current acceptance record. The final native +runtime candidate is `eaef0d56552c087aaa4fa68a8655282ffbd12667`; documentation-only +updates do not change those runtime bytes. Its CI and fresh 5/5 review passed. +The corrected source-process test has completed active adoption and a fresh +retained follow-up. Distinct compatible rollback, installed-application and live +Daytona gates remain under verification as of 2026-10-30 20:28 UTC. Preserve failed +attempts when adding later passes. The rollback floor must include the idle-goal +reader fix; an envelope-only reader is insufficient. + +Implementation is underway on `codex/native-controller-disk-independent-recovery`. +The related Cloud policy work merged as paperclip-cloud PR #1048; it retains +compatibility checks and drain fallback and does not implement this component. +Verified locally: metadata/source-selection/lock-wait checks (21 tests), readiness +and recovery scheduling (55 tests), durable discovery/import/inventory (7 tests), +and the complete native executor suite. The combined executor/selection/retention +run passed 1,063 tests and exposed two overbroad retention guards; those guards +were corrected and all 75 retention cases then passed. The snapshot fence test +also checks stale result acceptance and lease expiry during a run-row lock wait. + +Both warm and per-turn fixtures passed three separate controller processes, +deleting both native controller directories before B and again before C. They +check provider history, Runner identity, invocation counts, and the existing +suspension policy. Per-turn uses the production admission selector and snapshot +rebinder. These processes use the same candidate source, not distinct release +artifacts. Workspace and remote sandbox storage remain available. This is not +full-webserver filesystem-loss evidence. + +Both direct and prepared-task transport cases passed the 300-second outage with +natural lease expiry. The 8/512/8,192-chunk SQL test passed: each sample used seven +control transactions and twelve changed checkpoint records, with zero recovery +SQL/transactions/rows during the text interval and zero idle SQL. The long sample +included normal lease renewals. Its cluster-wide WAL values include simultaneous +test migrations and are diagnostic only, not a load acceptance measurement. + +Workspace typecheck, build, and map structural checks passed on the earlier +candidate; the later check dispositions are recorded below. The paired +300-controller/10,000-task workload now includes actual snapshot persistence. +Its initial host pilot missed throughput gates; the subsequent isolated Linux +measurements are recorded below. Database cost acceptance, distinct-release application startup, +compatible rollback, and native Daytona qualification remain open. The published Daytona qualification image was independently resolved from successful +workflow 40709784515 to `sha256:b38ed051d00fed409092d354eea9ae1b01fb232bcb7d5a187ed763382dda2d0b` +(source `8fe5d5defcdfb55a1b9049a801cfa5498629cebc`). The exact Linux Runner was +extracted from that image. The local attempt failed during the environment probe before agent creation. A read-only lookup independently returned `DaytonaAuthenticationError: Invalid credentials`. The private database and cleanup obligation remain preserved. The protected single-cell workflow [41093316372](https://github.com/paperclipai/paperclip/actions/runs/41093316372), targeting `512cc6d6dc`, was cancelled during image/artifact builds before provider allocation so acceptance can use the rebased candidate. Its protected secret is separate from the invalid local fixture. +No production deployment or complete zero-downtime acceptance is claimed. + +Failures retained in the implementation record: sandbox listener denial and host +embedded-Postgres startup limits; a corrected missing `phase` in a new fixture; +a corrected unexecuted clock query in result persistence; Linux fixture builds +first failed on shared build storage and then missing protocol source files, +resolved by a container-local build with the complete inputs. The 8,192-chunk +case exposed a test-provider cap of 4,096 (expanded in the test binary), then +exceeded the old short-stream wait limit. Its longer measurement now uses the +normal 60-second lease and renewal rather than an artificially long owner lease. +The long-stream renewal fixture initially omitted the controller PID/start identity; +renewal correctly rejected it. Complete fixture identity fixed the test. Per-turn +characterization then exposed the missing remote suspended-lifecycle projection; +that is now verified against the bound surviving sandbox. The fixture also now +uses the existing admission snapshot rebinder, which its warm-only predecessor +omitted. The first benchmark attempts lacked ES-module setup and the observer's +pg_stat_statements extension; the isolated Linux pilot needed its native Sharp +dependency. No database-load acceptance claim has been made from failed attempts. +There is no Paperclip issue bound to this local chat; this versioned source plan +is the handoff, rather than an upload to an unrelated company or issue. + +The remaining native qualification and compatibility cells require reproducible +measurements and a reviewed PR. Full-home application +acceptance remains dependent on workspace/backup durability; no controller-only +test can waive that gate. + +Qualification update: the legacy and changed-process-configuration warm cases +exposed a marker left by suspension before authority rotation. Restoring the +committed PRP checkpoint before constructing the replacement backend fixes both; +both real-process cases pass. The initial rerun used the wrong fixture executable +path and failed before provider startup; its log is retained. All 59 control-plane +port tests pass after narrowing the snapshot read to the binding and profile +columns it actually needs. The initial paired 300-controller ABBA comparison +completed every offered operation with no ownership losses, but candidate latency +and CPU increased above the investigation threshold. It is not performance +acceptance. A new paired run will measure the narrower snapshot projection. + + +Qualification update at `512cc6d6dc`: the snapshot port now combines coordinator +and run locks into one statement, with a database-clock check after both waits. +All 59 real-Postgres port tests pass. Snapshot and PRP transaction counts remain +unchanged; the first completed 300-controller baseline/candidate pair each issued +74,790 total statements and 7,800 control transactions across all pools. The +candidate adds the metadata bytes but no SQL per streamed chunk. + +The dedicated Daytona suite now fixes the active outage at 300 seconds and adds +normal UI follow-up after a second replacement following completion. Its oracle +requires a new successful native run, retained native/provider session identity, +a completed tool receipt and correct saved-project output. Focused E2E support +checks passed (56 tests, one optional external-database case skipped), plus all +137 Node support tests and E2E typecheck. The explicit external-database fixture +was separately exercised against owned Postgres earlier. The broad E2E unit +attempt lacked local listener permissions and was stopped; that failure remains +in `/tmp/native-controller-e2e-unit.log`, and broad qualification is still open. + +Database load remains **unqualified**. The narrowed-read and combined-query ABBA +runs became heavily contended as other local checks ran, including our E2E support +suite during the final pair. Both baseline and candidate lost absolute gates; +dropped operations invalidate relative comparisons. The combined-query first +pair completed all work, but its candidate p99 was 303 ms versus 131 ms and PG CPU +35.69 s versus 25.89 s. Unchanged business queries also slowed. These observations +require a quiet-host paired rerun; they are neither a performance pass nor proof +that the implementation caused all observed regression. The connection limit +remains unchanged. Summarized counters are retained in the adjacent +`2026-10-30-native-controller-load-evidence.json`; raw measurements remain under +`/tmp/native-controller-load/`. The reproducible script records failed absolute +and relative gates and marks unequal-operation comparisons ineligible. + +Remaining critical path: finish the protected Daytona attempt and cleanup; +complete full repository checks on the final rebased revision; finish the quiet +steady/burst/replacement load gates and explicit lock-wait latency measurement; +qualify distinct installed A/B/A artifacts; then resolve review on the draft +PR. Current server startup smoke and three-controller component tests are useful +layer-specific evidence, not substitutes for those remaining gates. + + +The full local `pnpm test:run` attempt was stopped after roughly 45 minutes while +still progressing through the general chat suite. Its output is retained at +`/tmp/native-controller-full-tests.log`; it is incomplete, not a pass. Broad +checks will run on hosted Linux for the rebased candidate. The broad E2E support +rerun with two workers was also stopped before rebasing; focused support checks +remain the completed evidence. A separate `server-config.test.ts` diagnostic +confirmed the original restricted run's `listen EPERM` cause. + + +### Rebased review and qualification follow-up + +Draft implementation: https://github.com/paperclipai/paperclip/pull/18343. +At rebased head `3aa90fbf0b5553ffa66a9cdc0fb153f348b56d75`, all 66 focused +persistence/discovery cases and full `pnpm -r typecheck` passed. The PR's general +server shard found 190 executor unit failures where synthetic IDs and a partial +Db double reached the new real discovery boundary. That fixture boundary is +being corrected; no production validation is relaxed. Greptile scored 3/5 and +identified missing prepared-task lifecycle reconstruction and retry after a +failed lifecycle read. Corrections and regression cases are under verification. +A separately identified retained-owner admission wait now uses the existing +same-run retry record/deadline, with unstarted-run and current-authority fences. + +Protected Daytona workflow `41094539946` failed before any test cell ran: the +image publisher exhausted disk (`No space left on device`, exit 102), including +its cleanup/log writes. It is not live recovery evidence, and repeating the +same build without a capacity fix is not justified. The default-branch workflow +has no per-run disk-capacity or upgrade-manifest override. CI infrastructure owns +that dependency. The invalid local credential attempt remains unresolved evidence. + +Local review-check attempts also encountered exhausted macOS Postgres semaphores; +only these owned test processes were stopped and moved to dedicated Docker +Postgres fixtures. One additional direct-read retry scenario timed out during +its source-controller fixture startup before a Runner appeared. Fixture import +readiness is now measured separately from the unchanged 60-second admission +limit. These failed/incomplete attempts remain distinct from subsequent passes. + +Review-fix verification: all 1,066 cases across native executor, durable +discovery, startup recovery and heartbeat runtime execution passed against owned +Docker Postgres. After the final database-clock and operator-Stop fences, all +nine durable discovery/import/wait tests passed and server typecheck passed. +The direct per-turn failed-read regression passed through normal admission with +three real controller processes: the failed read leaves the reconstructed PRP +projection, the next admission reconstructs the missing lifecycle receipt, and +the next replacement still continues the same provider history. Prepared-task +resumed and failed-read/retry cases also pass for Codex and Claude fixtures. +These are scripted-provider component results with remote workspace storage +available, not live Daytona or distinct-release acceptance. + + +### Bounded qualification campaign on October 30 + +The follow-up campaign began at 18:33 UTC with an approximately one-hour budget. +Three GPT-6.1 Sol workers own live Daytona, database load, and distinct-reader +replacement/rollback; the coordinator owns integration, artifact builds and review. +The exact native implementation under test is `a0ef3ce7493c8bb76c4766ff007e8a390118d145`. +Qualification commit `da8c7f6e1837ddd896dfe83a7524812fee813487` adds measurement and +installed-journey admission only; its native runtime, Runner, shared and database +production files are byte-identical to that implementation. + +The independent Linux database campaign uses an eight-vCPU, 32 GiB disposable +host, with PostgreSQL and its Node workload each limited to two CPUs and 1536 MiB. +It preserves the existing 10/2/2 request/renewal/checkpoint pool caps per logical +server. Steady and aligned-burst ABBA comparisons passed the existing absolute, +correctness and 10% relative regression gates with 300 controllers and 10,000 +historical tasks. Neither changes snapshot/PRP transaction counts. The separate +claim-capacity probe waits for natural lease expiry, invokes the production +claim path for 300 synthetic executions, restores real encrypted checkpoints +under empty controller directories, and measures current-owner checkpoint and +renewal work. Synthetic provider references do not establish sandbox existence +or authenticated provider reattachment. + +The closed, sanitized measurements and limits are recorded in +`2026-10-30-native-controller-qualification-evidence.json`. The earlier failed, +contended-host measurements remain in `2026-10-30-native-controller-load-evidence.json`. +Lock measurements use calibrated PostgreSQL wait-start timestamps and sampled +duration bounds. Observer traffic contributes to the measured database cost; +short unsampled waits and full HTTP latency remain outside that measurement. + +Live qualification uses protected campaign +[41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900). +Its initial catalog timeout occurred before provider allocation after Azure +reported a dependency-cache egress limit. A scoped exact-entry cache invalidation +and targeted retry forced a successful cold install. Only this campaign's +ephemeral image publisher received a larger, delete-on-termination disk; image +identity, source admission and provider authorization gates remain in force. + +Provider-free package builds pin first reader `59ff89a18fe1312b5f0bb908439bbdb1237b0252` +and candidate `a0ef3ce7493c8bb76c4766ff007e8a390118d145` in runs +[41108342354](https://github.com/paperclipai/paperclip/actions/runs/41108342354) and +[41108352138](https://github.com/paperclipai/paperclip/actions/runs/41108352138). +The source-process crash probe independently observed B waiting for A's live +lease and claiming the same execution after natural expiry, with no lease edits. +Its initial all-turns/single-provider-process assertion also crossed completion +contract revision 1 to revision 2. The existing Runner attachment policy can +retire a provider at a changed semantic contract while preserving its conversation; +phase-specific observations distinguish that boundary from same-contract rollback. +Earlier strict failures remain retained until that distinction is demonstrated. + +The final live, reader, installed-artifact, cleanup and CI outcomes are maintained +in [PR #18343](https://github.com/paperclipai/paperclip/pull/18343), with their exact +revisions and evidence limits. Campaign registration, successful artifact builds +and this plan entry do not themselves certify that a live journey passed. diff --git a/doc/plans/2026-10-30-native-controller-claim.patch b/doc/plans/2026-10-30-native-controller-claim.patch new file mode 100644 index 00000000000..0a3fe495bba --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-claim.patch @@ -0,0 +1,170 @@ +--- scripts/benchmark-native-controller-recovery.ts ++++ scripts/benchmark-native-controller-recovery.ts +@@ -14 +14 @@ +-const repoRoot = fileURLToPath(new URL("../", import.meta.url)); ++const repoRoot = path.resolve(required("BENCH_CANDIDATE_ROOT")); +@@ -241,0 +242,112 @@ ++async function claimCapacityPhase(db: any, u: string, items: any[], monitor: any) { ++ const { claimNativeRestartRecoveries } = await load(repoRoot, "server/src/services/native-runtime/native-restart-recovery.ts"); ++ const { loadNativeRunnerCheckpoint } = await load(repoRoot, "server/src/services/native-runtime/native-runner-checkpoint.ts"); ++ const replacement = createDb(u, { maxConnections: 10, applicationName: "benchmark-claim-replacement" }); ++ const successor = { bootId: randomUUID(), pid: process.pid, processStartedAt: new Date() }; ++ const result: any = { schema: "paperclip.native-controller-claim-capacity/v1", ...provenance, ++ scope: "Production DB claim function against synthetic prepared-workspace references and real persisted snapshots/PRP records. Natural lease expiry; two server database clients. No provider attachment or external sandbox existence claim.", ++ errors: [], claims: [], reads: [], samples: [], sqlByClient: {}, renewedOwnerLosses: 0, restoredCheckpoints: 0, ++ retainedOldControllerLeaseMs: 60000, remoteReconnectGraceMs: 420000, checkpointPoolPerServer: 2 }; ++ const clients = [ ++ ["old_request", db.$client], ["old_liveness", livenessDb(db).$client], ["old_checkpoint", checkpointDb(db).$client], ++ ["new_request", replacement.$client], ["new_liveness", livenessDb(replacement).$client], ["new_checkpoint", checkpointDb(replacement).$client], ++ ]; ++ for (const [name, client] of clients) { result.sqlByClient[name] = 0; client.options.debug = () => { result.sqlByClient[name]++; }; } ++ // Data is persisted by the ordinary snapshot port during the preconditioning phase. ++ const validity = await db.$client.unsafe("select count(*)::int as valid from heartbeat_runs r join native_run_finalizations f on f.run_id=r.id where r.status='running' and r.runner_profile_json->'nativeWorkspaceSync'->>'state'='prepared' and r.runner_profile_json->'sessionCheckpoint'->'identity'->>'runId'=r.id::text and exists(select 1 from native_runner_checkpoints c where c.run_id=r.id)"); ++ assert.equal(validity[0].valid, 300, "All claim inputs need real snapshots/checkpoints and prepared references"); ++ const before = pgResources(), started = performance.now(), requests = new Set>(), renewals: any[] = []; ++ const observed = observeLockWaits(monitor); ++ let tick = 0, offered = 0, completed = 0; ++ const traffic = setInterval(() => { ++ for (let j = 0; j < 20; j++) { ++ const index = tick++, item = items[index % items.length], client = index % 2 ? replacement : db; ++ const at = performance.now(); offered++; ++ const task = loadAgentRunAuthentication(client, item).then(() => { completed++; result.reads.push(performance.now() - at); }) ++ .catch((e: any) => result.errors.push({ source: "request", message: e.message })).finally(() => requests.delete(task)); ++ requests.add(task); ++ } ++ }, 100); // 200 database authentication requests/second, split evenly between servers. ++ let sampling: Promise | null = null; ++ const sample = setInterval(() => { ++ if (sampling) return; ++ sampling = monitor`select application_name,count(*)::int as connections,count(*) filter(where state='active')::int as active from pg_stat_activity where datname=current_database() and application_name!='benchmark-observer' group by application_name` ++ .then((rows: any) => result.samples.push({ elapsedMs: performance.now() - started, rows, postgresMemoryBytes: pgResources().memoryBytes })) ++ .catch((e: any) => result.errors.push({ source: "observer", message: e.message })).finally(() => { sampling = null; }); ++ }, 1000); ++ try { ++ const [lease] = await db.$client.unsafe("select greatest(0,extract(epoch from(max(lease_expires_at)-clock_timestamp()))*1000)::float8 as remaining from native_run_finalizations where lease_owner is not null"); ++ result.naturalExpiryWaitMs = Number(lease.remaining); ++ await sleep(result.naturalExpiryWaitMs + 100); ++ const [expired] = await db.$client.unsafe("select count(*)::int as expired from native_run_finalizations where lease_owner is not null and lease_expires_at<=clock_timestamp()"); ++ assert.equal(expired.expired, 300, "Never rewrite or shorten a controller lease for the probe"); ++ const claimStarted = performance.now(); ++ const dispositions = await claimNativeRestartRecoveries({ db: replacement, controller: successor, restartKind: "hard", ++ runIds: items.map(item => item.runId), limit: 300, ++ onClaim: (claim: any, scope: any) => { ++ result.claims.push({ runId: claim.runId, kind: claim.kind, elapsedMs: performance.now() - claimStarted, generation: claim.controllerGeneration }); ++ renewals.push(startNativeSessionExecutionLeaseRenewal({ db: replacement, ++ companyId: scope.companyId, issueId: scope.issueId, runId: claim.runId, leaseOwner: claim.leaseOwner, ++ attempt: claim.providerAttempt, controller: successor, remote: true, ++ onLost: async () => { result.renewedOwnerLosses++; } })); ++ }, ++ }); ++ result.claimDrainMs = performance.now() - claimStarted; ++ assert.equal(dispositions.length, 300); ++ assert(dispositions.every((claim: any) => claim.kind === "reattach_remote_runner"), "Every DB claim must independently validate its evidence"); ++ // Restore persisted records into empty replacement directories, then persist a real new control and snapshot. ++ const work = [...dispositions]; ++ await Promise.all(Array.from({ length: 4 }, async () => { ++ while (work.length) { ++ const claim: any = work.shift(), item = items.find(i => i.runId === claim.runId); ++ const checkpoint = await loadNativeRunnerCheckpoint(replacement, { companyId: item.companyId, issueId: item.issueId, runId: item.runId }); ++ assert(checkpoint); ++ const directory = root + "/replacement-states/" + item.runId; ++ checkpoint.restoreSessionDirectory(directory); ++ const header = JSON.parse(checkpoint.records.get("header")!); ++ const core = new CandidateCore({ stateDirectory: directory, identity: header.state.identity, ++ expectedRunnerVersion: checkpoint.runnerVersion, expectedRunnerDigest: checkpoint.runnerDigest }); ++ const binding = { companyId: item.companyId, issueId: item.issueId, runId: item.runId, ++ authorityRunId: item.runId, bootId: successor.bootId, generation: claim.controllerGeneration, leaseOwner: claim.leaseOwner }; ++ core.setCheckpointPersistence(CandidateWriter(replacement, binding)); ++ core.queueCommand("semantic_tool.result", { callId: "replacement-capacity", operationId: "fixture_read", result: { body: "x".repeat(4096) }, isError: false }, "replacement-control"); ++ await core.flushCheckpoint(); ++ const identity = item.snapshot.identity; ++ const port = new CandidatePort(replacement, { ...identity, sourceInstanceId: checkpoint.recoveryEvidence.runnerInstanceId, ++ controlPlaneSourceInstanceId: "replacement-controller", completionContractId: item.completionContractId, completionContractSha256: item.completionContractSha256 }, { nativeRecovery: { ++ compatibility: { configDigest: "sha256:" + "a".repeat(64), processConfigDigest: null }, owner: binding } }); ++ await port.openRun({ identity, backendKind: "codex", sourceInstanceId: checkpoint.recoveryEvidence.runnerInstanceId }); ++ await port.checkpointSession(item.snapshot); ++ result.restoredCheckpoints++; ++ await core.stop(); ++ } ++ })); ++ result.claimAndRestoreDrainMs = performance.now() - claimStarted; ++ // Observe one natural 20-second renewal from the last replacement claim. ++ await sleep(Math.max(0, 21000 - (performance.now() - claimStarted - result.claimDrainMs))); ++ const [saved] = await replacement.$client.unsafe("select count(*)::int as owners,min(controller_generation)::int as min_generation,max(controller_generation)::int as max_generation,min(extract(epoch from(lease_expires_at-clock_timestamp()))*1000)::float8 as minimum_remaining_ms from native_run_finalizations where controller_boot_id=$1 and lease_owner is not null", [successor.bootId]); ++ result.postRenewalOwners = saved; ++ assert.equal(saved.owners, 300); assert.equal(saved.min_generation, 2); assert.equal(saved.max_generation, 2); ++ assert(saved.minimum_remaining_ms > 40000, "Replacement leases should have naturally renewed from the database clock"); ++ } catch (e: any) { result.errors.push({ source: "claim-or-restore", message: e.message, stack: e.stack }); } ++ finally { ++ clearInterval(traffic); clearInterval(sample); ++ await Promise.allSettled([...requests]); await sampling; ++ await Promise.allSettled(renewals.map(r => r.stop())); ++ result.lockWait = await observed.stop(); ++ } ++ result.elapsedMs = performance.now() - started; ++ result.requestsOffered = offered; result.requestsCompleted = completed; result.requestLatency = summary(result.reads); delete result.reads; ++ result.postgresCpuUsec = pgResources().cpuUsec - before.cpuUsec; ++ const totalSql = () => Object.values(result.sqlByClient).reduce((a: number, b: any) => a + Number(b), 0); ++ const idleStart = totalSql(); await sleep(1000); result.settledIdleSql = totalSql() - idleStart; ++ result.passed = result.errors.length === 0 && result.lockWait.errors.length === 0 && result.claims.length === 300 ++ && result.restoredCheckpoints === 300 && completed === offered && result.renewedOwnerLosses === 0 ++ && result.claimAndRestoreDrainMs < 420000 && result.requestLatency.p99 < 250 && result.settledIdleSql === 0; ++ await writeFile(root + "/claim-capacity.json", JSON.stringify(result, null, 2) + "\n"); ++ console.log(JSON.stringify({ phase: "claim-capacity-complete", passed: result.passed, claims: result.claims.length, ++ claimDrainMs: result.claimDrainMs, claimAndRestoreDrainMs: result.claimAndRestoreDrainMs, ++ requests: result.requestLatency, errors: result.errors.slice(0, 3) })); ++ if (!result.passed) process.exitCode = 1; ++} ++ +@@ -429,0 +542 @@ ++ runnerProfileJson: { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: randomUUID(), remoteCwd: "/workspace", resourceDisposition: "destroy" } }, +@@ -544,0 +658,2 @@ ++ completionContractId: contractId, ++ completionContractSha256: "fixture", +@@ -905,0 +1021 @@ ++ await claimCapacityPhase(db, u, items, monitor); +@@ -917,6 +1033 @@ +- const profile = [ +- ["baseline", 300], +- ["candidate", 300], +- ["candidate", 300], +- ["baseline", 300], +- ] as const; ++ const profile = [["candidate", 300]] as const; +@@ -953,35 +1064 @@ +- const perOperation = (r: any, key: string) => +- r[key] / (r.normalCompleted + r.normalWritesCompleted + r.controls + r.admissionsCompleted); +- const average = (values: number[]) => values.reduce((a, b) => a + b, 0) / values.length; +- const comparable = +- result.every((r) => r.correctnessPassed) && +- result.every( +- (r) => +- r.controls === result[0].controls && +- r.normalCompleted === result[0].normalCompleted && +- r.normalWritesCompleted === result[0].normalWritesCompleted, +- ); +- const relative = {}; +- for (const key of ["postgresCpuUsec", "walBytes", "requestP99"]) { +- const value = (r: any) => (key === "requestP99" ? r.normalReadSummary.p99 : perOperation(r, key)); +- const baseline = average(result.filter((r) => r.variant === "baseline").map(value)); +- const candidate = average(result.filter((r) => r.variant === "candidate").map(value)); +- relative[key] = { +- baseline, +- candidate, +- increasePercent: comparable ? 100 * (candidate / baseline - 1) : null, +- passed: comparable ? candidate <= baseline * 1.1 : null, +- }; +- } +- Object.assign(report, { +- relative, +- comparable, +- passed: result.every((r) => r.passed) && Object.values(relative).every((r: any) => r.passed), +- measurementLimits: [ +- "Database request latency is not HTTP latency", +- "Pool queue includes client encoding; BEGIN response includes network time", +- "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed", +- "Peak memory is sampled once a second", +- "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide", +- ], +- }); ++ Object.assign(report, { passed: result.every(r => r.passed), scope: "Preconditioning only for the separate claim-capacity.json report" }); diff --git a/doc/plans/2026-10-30-native-controller-load-evidence.json b/doc/plans/2026-10-30-native-controller-load-evidence.json new file mode 100644 index 00000000000..1fadc46a050 --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-load-evidence.json @@ -0,0 +1,390 @@ +{ + "schema": "paperclip.native-controller-load-summary/v1", + "baselineRevision": "59e191c368", + "candidateRevision": "4f710fd7e7", + "qualification": "failed; host contention and unequal completed work invalidate the ABBA comparison", + "sourceReport": "/tmp/native-controller-load/combined-steady/report.json", + "fixture": { + "mode": "steady", + "normalReadsPerSecond": 500, + "normalWritesPerSecond": 100, + "databaseAdmissionEverySeconds": 5, + "checkpointControlEverySeconds": 5, + "controlPayloadBytes": 4096, + "phaseSeconds": 65, + "snapshotTransactionsPerControl": 1, + "baselineCheckpointPool": 2, + "baselineRenewalIntervalSeconds": 20, + "candidateRenewalIntervalSeconds": 20, + "historicalIssues": 10000 + }, + "limits": [ + "Database request latency is not HTTP latency", + "Pool queue includes client encoding; BEGIN response includes network time", + "pg_stat_statements execution includes lock waits; samples report concurrent lock waiters, not a lock latency distribution", + "Peak memory is sampled once a second", + "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide" + ], + "results": [ + { + "label": "01_baseline_300", + "controls": 3900, + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesCompleted": 6500, + "dropped": 0, + "renewalLosses": 0, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "changedRows": 7800, + "changedBytes": 17371500, + "walBytes": 63819704, + "postgresCpuUsec": 25887716, + "postgresPeakSampledMemoryBytes": 697917440, + "normalReadSummary": { + "n": 32500, + "p50": 53.82264099999884, + "p95": 83.97040299999935, + "p99": 131.4543599999888, + "max": 249.75419500000135 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 3.5057919999962905, + "p95": 14.36212499999965, + "p99": 79.32949999999255, + "max": 220.70087599998806 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.04241599999659229, + "p95": 4.89183399999456, + "p99": 25.90625, + "max": 210.7756250000093 + }, + "liveness": { + "n": 900, + "p50": 401.07175099999586, + "p95": 1987.666584000006, + "p99": 2269.4225429999933, + "max": 2296.889626000004 + }, + "checkpoint": { + "n": 3900, + "p50": 0.05258299999695737, + "p95": 0.14449999999487773, + "p99": 41.99183400000038, + "max": 188.36633400000574 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 3.6501250000001164, + "p95": 14.756584000002476, + "p99": 62.41062499998952, + "max": 240.88450000000012 + }, + "liveness": { + "n": 900, + "p50": 406.2205419999955, + "p95": 2044.2198760000028, + "p99": 2277.324250999998, + "max": 2308.3716669999994 + }, + "checkpoint": { + "n": 3900, + "p50": 3.297791999997571, + "p95": 14.160208000001148, + "p99": 79.12295900000026, + "max": 220.50066700000025 + } + }, + "drainMs": 26.716459000002942, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "startedAt": "2026-10-30T16:40:56.503Z", + "endedAt": "2026-10-30T16:42:01.534Z" + }, + { + "label": "02_candidate_300", + "controls": 3900, + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesCompleted": 6500, + "dropped": 0, + "renewalLosses": 0, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "changedRows": 7800, + "changedBytes": 17371500, + "walBytes": 66746424, + "postgresCpuUsec": 35685605, + "postgresPeakSampledMemoryBytes": 666578944, + "normalReadSummary": { + "n": 32500, + "p50": 68.02364699999453, + "p95": 127.37661100001424, + "p99": 303.326441000012, + "max": 474.2829830000119 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 5.670542000007117, + "p95": 241.97500100001344, + "p99": 479.3440830000036, + "max": 553.2187080000003 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.05162500002188608, + "p95": 37.2935839999991, + "p99": 136.02920900000026, + "max": 420.89487499999814 + }, + "liveness": { + "n": 900, + "p50": 536.7470420000027, + "p95": 3620.454543, + "p99": 4146.325211000018, + "max": 4238.34758500001 + }, + "checkpoint": { + "n": 3900, + "p50": 0.057792000006884336, + "p95": 209.87512499999139, + "p99": 443.83324999999604, + "max": 528.2805419999931 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 6.640833000012208, + "p95": 66.06445900001563, + "p99": 184.52537499999744, + "max": 449.3999579999945 + }, + "liveness": { + "n": 900, + "p50": 545.1022919999959, + "p95": 3665.246376999974, + "p99": 4172.158336000022, + "max": 4248.078752000001 + }, + "checkpoint": { + "n": 3900, + "p50": 5.484875000009197, + "p95": 241.83212600002298, + "p99": 478.8787080000038, + "max": 553.1449999999895 + } + }, + "drainMs": 76.8760000000184, + "absoluteGates": { + "requestP99": false, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": false, + "startedAt": "2026-10-30T16:42:16.915Z", + "endedAt": "2026-10-30T16:43:21.995Z" + }, + { + "label": "03_candidate_300", + "controls": 3887, + "normalOffered": 32500, + "normalCompleted": 32379, + "normalWritesCompleted": 6470, + "dropped": 164, + "renewalLosses": 0, + "checkpointTransactions": 3887, + "snapshotTransactions": 3887, + "sqlCount": 54484, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15549, + "changedRows": 7774, + "changedBytes": 17313559, + "walBytes": 66470080, + "postgresCpuUsec": 53715752, + "postgresPeakSampledMemoryBytes": 670437376, + "normalReadSummary": { + "n": 32379, + "p50": 87.29533699998865, + "p95": 407.7868620000081, + "p99": 715.8863589999964, + "max": 1413.3376139999891 + }, + "checkpointLatencySummary": { + "n": 3887, + "p50": 35.722666999994544, + "p95": 2254.7568759999995, + "p99": 4195.890377000003, + "max": 4784.590544000006 + }, + "poolQueueByPool": { + "request": { + "n": 3900, + "p50": 0.12583400000585243, + "p95": 218.51912499999162, + "p99": 498.72595799999544, + "max": 1368.8183760000102 + }, + "liveness": { + "n": 900, + "p50": 2733.2611259999976, + "p95": 7990.602337999997, + "p99": 8707.207421, + "max": 8915.786796 + }, + "checkpoint": { + "n": 3887, + "p50": 3.155832999997074, + "p95": 2207.667333999998, + "p99": 4154.626294000016, + "max": 4726.676418999996 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3900, + "p50": 27.709541999996873, + "p95": 276.61000000001513, + "p99": 625.9165419999918, + "max": 1627.7409179999959 + }, + "liveness": { + "n": 900, + "p50": 2782.7440009999846, + "p95": 8019.347628999996, + "p99": 8734.34838000001, + "max": 8964.291171000019 + }, + "checkpoint": { + "n": 3887, + "p50": 35.49954200000502, + "p95": 2254.554459999985, + "p99": 4195.5352520000015, + "max": 4784.371294000011 + } + }, + "drainMs": 4928.940960000007, + "absoluteGates": { + "requestP99": false, + "drain": false, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": false, + "startedAt": "2026-10-30T16:43:41.479Z", + "endedAt": "2026-10-30T16:44:51.414Z" + }, + { + "label": "04_baseline_300", + "controls": 2390, + "normalOffered": 32500, + "normalCompleted": 10713, + "normalWritesCompleted": 1503, + "dropped": 28305, + "renewalLosses": 0, + "checkpointTransactions": 2390, + "snapshotTransactions": 2390, + "sqlCount": 21797, + "livenessSqlCount": 4502, + "checkpointSqlCount": 9561, + "changedRows": 4780, + "changedBytes": 10643952, + "walBytes": 31871472, + "postgresCpuUsec": 55047197, + "postgresPeakSampledMemoryBytes": 643657728, + "normalReadSummary": { + "n": 10713, + "p50": 291.001007999992, + "p95": 841.957837999973, + "p99": 2601.1680469999555, + "max": 3106.3534709999803 + }, + "checkpointLatencySummary": { + "n": 2390, + "p50": 6933.93029400002, + "p95": 16553.939049999986, + "p99": 17176.082591000013, + "max": 17434.499467000016 + }, + "poolQueueByPool": { + "request": { + "n": 2392, + "p50": 0.10450000001583248, + "p95": 520.4013340000529, + "p99": 798.97454200004, + "max": 2614.81958499999 + }, + "liveness": { + "n": 900, + "p50": 5497.6995860000025, + "p95": 16836.80413299997, + "p99": 17748.473174999992, + "max": 18895.275049999997 + }, + "checkpoint": { + "n": 2390, + "p50": 6882.121711000043, + "p95": 16492.48646600003, + "p99": 17097.382091999985, + "max": 17352.715134 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 2392, + "p50": 55.64137500000652, + "p95": 638.8517920000013, + "p99": 2574.2484169999952, + "max": 2879.7436260000104 + }, + "liveness": { + "n": 900, + "p50": 5623.322503000032, + "p95": 16872.938590999984, + "p99": 18552.84471799998, + "max": 18921.25067499996 + }, + "checkpoint": { + "n": 2390, + "p50": 6933.636502000038, + "p95": 16553.763133, + "p99": 17175.909424000012, + "max": 17433.771509000042 + } + }, + "drainMs": 12508.142965000006, + "absoluteGates": { + "requestP99": false, + "drain": false, + "throughput": false, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": false, + "startedAt": "2026-10-30T16:45:39.710Z", + "endedAt": "2026-10-30T16:46:57.219Z" + } + ] +} diff --git a/doc/plans/2026-10-30-native-controller-overlap.patch b/doc/plans/2026-10-30-native-controller-overlap.patch new file mode 100644 index 00000000000..4963f5045e2 --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-overlap.patch @@ -0,0 +1,76 @@ +--- scripts/benchmark-native-controller-recovery.ts ++++ scripts/benchmark-native-controller-recovery.ts +@@ -0,0 +1 @@ ++// Temporary qualification probe derived from frozen benchmark SHA256 abaa29d50eb95013836b54b8dacc94c72e080edf24744cc5d291cc5eb1ff2818. +@@ -14 +15 @@ +-const repoRoot = fileURLToPath(new URL("../", import.meta.url)); ++const repoRoot = path.resolve(required("BENCH_CANDIDATE_ROOT")); +@@ -187 +188 @@ +- const byPool = Object.fromEntries(["benchmark-application", "paperclip-controller-lease", "paperclip-runner-checkpoint"].map(pool => { ++ const byPool = Object.fromEntries(["benchmark-application", "benchmark-replacement-application", "paperclip-controller-lease", "paperclip-runner-checkpoint"].map(pool => { +@@ -250,0 +252 @@ ++ const replacementDb = createDb(u, { maxConnections: 10, applicationName: "benchmark-replacement-application" }); +@@ -256,0 +259 @@ ++ const replacementOwner = { ...owner, bootId: randomUUID() }; +@@ -299,0 +303,3 @@ ++ ["replacement_request", replacementDb.$client], ++ ["replacement_liveness", livenessDb(replacementDb).$client], ++ ["replacement_checkpoint", checkpointDb(replacementDb).$client], +@@ -369,0 +376 @@ ++ replacementDb.$client.options.debug = db.$client.options.debug; +@@ -386,0 +394,2 @@ ++ livenessDb(replacementDb).$client.options.debug = livenessDb(db).$client.options.debug; ++ checkpointDb(replacementDb).$client.options.debug = checkpointDb(db).$client.options.debug; +@@ -389,0 +399,2 @@ ++ const controllerDb = i % 2 ? replacementDb : db; ++ const controllerOwner = i % 2 ? replacementOwner : owner; +@@ -443,3 +454,3 @@ +- controllerBootId: owner.bootId, +- controllerPid: owner.pid, +- controllerProcessStartedAt: owner.processStartedAt, ++ controllerBootId: controllerOwner.bootId, ++ controllerPid: controllerOwner.pid, ++ controllerProcessStartedAt: controllerOwner.processStartedAt, +@@ -467 +478 @@ +- const writer = (variant === "candidate" ? CandidateWriter : BaselineWriter)(db, { ++ const writer = (variant === "candidate" ? CandidateWriter : BaselineWriter)(controllerDb, { +@@ -472 +483 @@ +- bootId: owner.bootId, ++ bootId: controllerOwner.bootId, +@@ -511 +522 @@ +- db, ++ controllerDb, +@@ -531 +542 @@ +- bootId: owner.bootId, ++ bootId: controllerOwner.bootId, +@@ -544,0 +556,2 @@ ++ db: controllerDb, ++ owner: controllerOwner, +@@ -592 +605 @@ +- for (let i = 0; i < 100; i++) await loadAgentRunAuthentication(db, items[i % n]); ++ for (let i = 0; i < 100; i++) await loadAgentRunAuthentication(items[i % n].db, items[i % n]); +@@ -639 +652 @@ +- const input = { db, ...item, attempt: 1, controller: owner }; ++ const input = { ...item, attempt: 1, controller: item.owner }; +@@ -664 +677 @@ +- const rows = await loadAgentRunAuthentication(db, item); ++ const rows = await loadAgentRunAuthentication(item.db, item); +@@ -700 +713 @@ +- await db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, item.issueId)); ++ await item.db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, item.issueId)); +@@ -713 +726 @@ +- await db.transaction(async (tx: any) => { ++ await (metrics.admissionsOffered % 2 ? replacementDb : db).transaction(async (tx: any) => { +@@ -917,6 +930 @@ +- const profile = [ +- ["baseline", 300], +- ["candidate", 300], +- ["candidate", 300], +- ["baseline", 300], +- ] as const; ++ const profile = [["baseline", 300], ["candidate", 300], ["candidate", 300], ["baseline", 300]] as const; +@@ -929 +937 @@ +- "Paired baseline/candidate, including real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", ++ "Two logical servers in one client process, 150 controllers each, separate request/liveness/checkpoint pools at unchanged caps; same aggregate aligned-burst workload. Matched baseline/candidate ABBA pool-overlap component qualification; no takeover latency or provider reattachment claim. Includes real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", +@@ -930,0 +939 @@ ++ logicalServers: 2, diff --git a/doc/plans/2026-10-30-native-controller-qualification-evidence.json b/doc/plans/2026-10-30-native-controller-qualification-evidence.json new file mode 100644 index 00000000000..1b8e6f50444 --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-qualification-evidence.json @@ -0,0 +1,3628 @@ +{ + "schema": "paperclip.native-controller-qualification-evidence/v1", + "recordedAtUtc": "2026-10-30T19:08:26.343965+00:00", + "qualification": "Passed isolated PostgreSQL steady, aligned-burst and matched two-logical-server ABBA correctness, absolute and relative gates, plus production database recovery-claim/restore capacity. These are component measurements.", + "baselineRevision": "59e191c368cb3119ff3206621fefa9c5d4ad3110", + "candidateRevision": "a0ef3ce7493c8bb76c4766ff007e8a390118d145", + "measurementHarnessRevision": "da8c7f6e1837ddd896dfe83a7524812fee813487", + "measurementHarnessSha256": "abaa29d50eb95013836b54b8dacc94c72e080edf24744cc5d291cc5eb1ff2818", + "overlapProbeSha256": "fae5f15053eab0eeac15f4689035c7225488e2b8fd9cf1920e3543b71187f25e", + "sourceInputs": [ + { + "path": "server/src/services/native-runtime/native-runner-checkpoint.ts", + "candidateSha256": "d79f0e3ab44c2cea158571d9f0235aba1f9985e1e3c30ae4b347eb3e15772acf", + "baselineSha256": "d798cd0859ca363ae3be33c607f10b4755420d77d796824d1ee9570d0f7a253c" + }, + { + "path": "server/src/services/native-runtime/paperclip-control-plane-port.ts", + "candidateSha256": "b9143eea07733b67eab80e7e000cf1018202b047c755c47533a6679b67d57c51", + "baselineSha256": "a1f807c5d8fbdcb30e966ebf8a4789974f6a8a7086a0056d9351da6e9817e5f2" + }, + { + "path": "packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts", + "candidateSha256": "a8a8bed373cdd18397a762b9f67b04c789d9e5d63208f06c5ebe000faff090ac", + "baselineSha256": "a8a8bed373cdd18397a762b9f67b04c789d9e5d63208f06c5ebe000faff090ac" + }, + { + "path": "server/src/services/native-runtime/native-execution-lease.ts", + "candidateSha256": "03e70c33342ce505011ff883f8f67892e3ac5f53c58eb758316b67b4b0b2b386", + "baselineSha256": "03e70c33342ce505011ff883f8f67892e3ac5f53c58eb758316b67b4b0b2b386" + } + ], + "environment": { + "host": "One isolated disposable AWS m7i.2xlarge Linux x86_64 VM", + "hostVcpus": 8, + "hostMemoryGiB": 32, + "hostSwapMiB": 0, + "nodeVersion": "v24.22.2", + "postgresMajor": 17, + "pnpmVersion": "9.15.4", + "postgresCpuLimit": 2, + "postgresMemoryLimitMiB": 1536, + "nodeCpuLimit": 2, + "nodeMemoryLimitMiB": 1536, + "perServerPoolLimits": { + "request": 10, + "liveness": 2, + "checkpoint": 2 + }, + "observerConnectionLimit": 1, + "leaseDurationMs": 60000, + "remoteReconnectGraceMs": 420000 + }, + "fixture": { + "historicalTasks": 10000, + "controllers": 300, + "phaseDurationSeconds": 65, + "controlIntervalSeconds": 5, + "controlPayloadBytes": 4096, + "snapshotsPerControl": 1, + "mainProfileOrder": [ + "baseline", + "candidate", + "candidate", + "baseline" + ], + "baselineAndCandidateUseIdenticalDependencyBytes": true, + "normalRequestAndLeaseRenewalImplementationsAreCommonControls": true + }, + "profiles": { + "steady": { + "schema": "paperclip.normal-recovery-load/v1", + "scope": "Paired baseline/candidate, including real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", + "hardware": { + "postgresCpuLimit": 2, + "postgresMemoryLimitMiB": 1536, + "mainPool": 10, + "livenessPool": 2, + "candidateCheckpointPool": 2 + }, + "workload": { + "mode": "steady", + "normalReadsPerSecond": 500, + "normalWritesPerSecond": 100, + "databaseAdmissionEverySeconds": 5, + "checkpointControlEverySeconds": 5, + "controlPayloadBytes": 4096, + "phaseSeconds": 65, + "snapshotTransactionsPerControl": 1, + "baselineCheckpointPool": 2, + "baselineRenewalIntervalSeconds": 20, + "candidateRenewalIntervalSeconds": 20, + "historicalIssues": 10000 + }, + "passed": true, + "comparable": true, + "measurementLimits": [ + "Database request latency is not HTTP latency", + "Pool queue includes client encoding; BEGIN response includes network time", + "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed", + "Peak memory is sampled once a second", + "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide" + ], + "sourceReport": "/tmp/native-controller-load-qualification-20261030/remote/evidence/steady/report.json", + "relative": { + "postgresCpuUsec": { + "baseline": 656.5912310954723, + "candidate": 664.3645165800573, + "increasePercent": 1.183885059143397, + "passed": true + }, + "walBytes": { + "baseline": 1532.2154125789389, + "candidate": 1559.9179735744415, + "increasePercent": 1.8080069400212562, + "passed": true + }, + "requestP99": { + "baseline": 91.71616899999935, + "candidate": 93.46717100001115, + "increasePercent": 1.9091530087915087, + "passed": true + } + }, + "results": [ + { + "label": "01_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:43:31.121Z", + "endedAt": "2026-10-30T18:44:36.144Z", + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesOffered": 6500, + "normalWritesCompleted": 6500, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 67048632, + "postgresCpuUsec": 28159533, + "postgresPeakSampledMemoryBytes": 917352448, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 19.461555999994744, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 32500, + "p50": 23.630726999996114, + "p95": 104.30216500000097, + "p99": 107.72847300001013, + "max": 111.46659400001226 + }, + "normalWriteSummary": { + "n": 6500, + "p50": 28.412395999999717, + "p95": 109.15322800001013, + "p99": 112.54311900000903, + "max": 117.06165799999872 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 14.350610333334771, + "p95": 21.62447166666243, + "p99": 42.645023666656925, + "max": 92.58285566666018 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 4.567198000004282, + "p95": 6.6246809999938705, + "p99": 11.499447000001965, + "max": 34.80159999999887 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 4.406659000000218, + "p95": 8.00715200000559, + "p99": 12.106628999998065, + "max": 81.07146300000022 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.031578000001900364, + "p95": 3.2165359999999055, + "p99": 6.9602370000066, + "max": 65.06650499999887 + }, + "liveness": { + "n": 900, + "p50": 222.9600139999966, + "p95": 408.67597899999964, + "p99": 436.3592290000015, + "max": 446.41857600000367 + }, + "checkpoint": { + "n": 3900, + "p50": 0.03176600000006147, + "p95": 0.044882999995024875, + "p99": 0.18860399999903166, + "max": 17.319057000000612 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 4.37169300000096, + "p95": 8.198765000001004, + "p99": 12.514188999994076, + "max": 81.02461400000175 + }, + "liveness": { + "n": 900, + "p50": 226.25227900000027, + "p95": 412.00253099999827, + "p99": 440.3486919999996, + "max": 448.6205390000032 + }, + "checkpoint": { + "n": 3900, + "p50": 4.4467920000024606, + "p95": 6.499621000002662, + "p99": 11.388421999999991, + "max": 34.62909299999956 + } + }, + "sqlAllApplicationPools": 74790, + "sampledPeakDatabaseActivityRows": 15, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2563, + "gapsMs": { + "n": 2563, + "p50": 25.121846000001824, + "p95": 26.47124900000199, + "p99": 28.41609900000185, + "max": 86.58039899999858 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2563, + "total_exec_time": 222.99437100000054, + "rows": 0 + } + ], + "nodeCpu": { + "user": 20498304, + "system": 1761522 + }, + "nodeCpuAverageOneCorePercent": 34.23372797056988, + "eventLoopDelay": { + "p95Ms": 11.337727, + "p99Ms": 12.173311, + "maxMs": 70.975487 + }, + "peakSampledNodeRssBytes": 829227008, + "peakSampledNodeHeapBytes": 579761248, + "postgresStatementExecutionMs": 7781.774002, + "lockObserverExecutionMs": 222.99437100000054 + }, + { + "label": "02_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:44:42.986Z", + "endedAt": "2026-10-30T18:45:48.008Z", + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesOffered": 6500, + "normalWritesCompleted": 6500, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 67046184, + "postgresCpuUsec": 28544913, + "postgresPeakSampledMemoryBytes": 901398528, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 18.60484000001452, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 32500, + "p50": 52.137395999990986, + "p95": 73.59120199998142, + "p99": 80.18543700000737, + "max": 160.65012499998556 + }, + "normalWriteSummary": { + "n": 6500, + "p50": 56.79107799999474, + "p95": 79.06722500000615, + "p99": 124.89146799998707, + "max": 164.84574299998349 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 14.875081999998656, + "p95": 22.424880666658282, + "p99": 59.884479666667175, + "max": 174.85218533332227 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 4.532260000007227, + "p95": 6.7100529999879654, + "p99": 14.624643999995897, + "max": 70.33155800000532 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 4.792860999994446, + "p95": 8.380057000002125, + "p99": 14.55555699999968, + "max": 96.5399219999963 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.03197399998316541, + "p95": 3.4958060000208206, + "p99": 8.42655800000648, + "max": 86.68744899999001 + }, + "liveness": { + "n": 900, + "p50": 226.07839499998954, + "p95": 631.9207260000112, + "p99": 687.6313799999916, + "max": 700.4564780000073 + }, + "checkpoint": { + "n": 3900, + "p50": 0.03252500000235159, + "p95": 0.04554400000779424, + "p99": 1.5021660000056727, + "max": 30.859597000002395 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 4.757719000001089, + "p95": 8.515767000004416, + "p99": 14.811911999990116, + "max": 96.50402399999439 + }, + "liveness": { + "n": 900, + "p50": 228.28790900000604, + "p95": 634.5756129999936, + "p99": 689.5229259999905, + "max": 703.2544249999919 + }, + "checkpoint": { + "n": 3900, + "p50": 4.425221999990754, + "p95": 6.599863999988884, + "p99": 14.496323999992455, + "max": 70.18957200000295 + } + }, + "sqlAllApplicationPools": 74790, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2564, + "gapsMs": { + "n": 2564, + "p50": 25.082806999998866, + "p95": 26.16317899999558, + "p99": 28.29036499999347, + "max": 96.75551200000336 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2564, + "total_exec_time": 222.22588199999979, + "rows": 0 + } + ], + "nodeCpu": { + "user": 21447274, + "system": 1827953 + }, + "nodeCpuAverageOneCorePercent": 35.79597645613477, + "eventLoopDelay": { + "p95Ms": 10.903551, + "p99Ms": 12.877823, + "maxMs": 70.778879 + }, + "peakSampledNodeRssBytes": 861179904, + "peakSampledNodeHeapBytes": 612108352, + "postgresStatementExecutionMs": 7869.057230000005, + "lockObserverExecutionMs": 222.22588199999979 + }, + { + "label": "03_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:45:55.253Z", + "endedAt": "2026-10-30T18:47:00.271Z", + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesOffered": 6500, + "normalWritesCompleted": 6500, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 66835336, + "postgresCpuUsec": 28474836, + "postgresPeakSampledMemoryBytes": 890142720, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 14.864391999988584, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 32500, + "p50": 70.75364599999739, + "p95": 101.21604299999308, + "p99": 106.74890500001493, + "max": 184.49763100000564 + }, + "normalWriteSummary": { + "n": 6500, + "p50": 75.31148000000394, + "p95": 106.2007309999899, + "p99": 112.02943600001163, + "max": 189.8124850000022 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 14.700123333343072, + "p95": 22.93642300000647, + "p99": 74.96976966666989, + "max": 167.43050899999798 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 4.546646000002511, + "p95": 6.599335000006249, + "p99": 16.56861200000276, + "max": 66.67637499998091 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 4.787251000001561, + "p95": 9.346571000001859, + "p99": 17.659020999999484, + "max": 89.22574900000473 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.032805000024382025, + "p95": 3.5023479999799747, + "p99": 7.330482000019401, + "max": 22.141313000000082 + }, + "liveness": { + "n": 900, + "p50": 229.3470420000085, + "p95": 658.2537230000016, + "p99": 692.6467640000046, + "max": 699.1289490000054 + }, + "checkpoint": { + "n": 3900, + "p50": 0.034085999999661, + "p95": 0.048119000013684854, + "p99": 4.801466999982949, + "max": 22.820584999979474 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 4.7497089999960735, + "p95": 9.576310999982525, + "p99": 17.61802799999714, + "max": 89.19793199998094 + }, + "liveness": { + "n": 900, + "p50": 232.34173700000974, + "p95": 665.4881870000099, + "p99": 694.4492489999975, + "max": 701.2261050000088 + }, + "checkpoint": { + "n": 3900, + "p50": 4.438666999980342, + "p95": 6.487386000022525, + "p99": 16.5198870000022, + "max": 66.55283199998667 + } + }, + "sqlAllApplicationPools": 74790, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2560, + "gapsMs": { + "n": 2560, + "p50": 25.087980000011157, + "p95": 26.18886699998984, + "p99": 29.31080700000166, + "max": 87.73278699998627 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2560, + "total_exec_time": 228.186316, + "rows": 0 + } + ], + "nodeCpu": { + "user": 21036535, + "system": 1821778 + }, + "nodeCpuAverageOneCorePercent": 35.15690455164375, + "eventLoopDelay": { + "p95Ms": 11.026431, + "p99Ms": 13.180927, + "maxMs": 74.579967 + }, + "peakSampledNodeRssBytes": 867966976, + "peakSampledNodeHeapBytes": 609559232, + "postgresStatementExecutionMs": 7841.367742000012, + "lockObserverExecutionMs": 228.186316 + }, + { + "label": "04_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:47:06.695Z", + "endedAt": "2026-10-30T18:48:11.712Z", + "normalOffered": 32500, + "normalCompleted": 32500, + "normalWritesOffered": 6500, + "normalWritesCompleted": 6500, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 54687, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 64455288, + "postgresCpuUsec": 28193066, + "postgresPeakSampledMemoryBytes": 876048384, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 14.899754000012763, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 32500, + "p50": 44.329154999984894, + "p95": 69.53714800003218, + "p99": 75.70386499998858, + "max": 144.0518920000177 + }, + "normalWriteSummary": { + "n": 6500, + "p50": 49.25532100000419, + "p95": 75.08631300000707, + "p99": 122.02264599999762, + "max": 148.6692589999875 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 14.625869666662766, + "p95": 21.669017333333613, + "p99": 61.56695066665998, + "max": 185.71976100001484 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 4.66447999997763, + "p95": 6.80966199998511, + "p99": 16.252097999997204, + "max": 80.11221200000728 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 4.480548000021372, + "p95": 8.228377999999793, + "p99": 14.50026900001103, + "max": 103.96219200000633 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.032313000003341585, + "p95": 3.0398300000233576, + "p99": 7.722563000017544, + "max": 90.98564000002807 + }, + "liveness": { + "n": 900, + "p50": 225.77165700000478, + "p95": 577.020279999997, + "p99": 636.1102529999916, + "max": 644.8716770000174 + }, + "checkpoint": { + "n": 3900, + "p50": 0.033404999994672835, + "p95": 0.04764499998418614, + "p99": 3.443251999997301, + "max": 24.120947999996133 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 4.446676999999909, + "p95": 8.55906799994409, + "p99": 14.462981000018772, + "max": 103.91490999999223 + }, + "liveness": { + "n": 900, + "p50": 228.31017700000666, + "p95": 580.3901179999812, + "p99": 639.0160339999711, + "max": 646.8926890000002 + }, + "checkpoint": { + "n": 3900, + "p50": 4.548077000014018, + "p95": 6.678080999990925, + "p99": 15.536747000005562, + "max": 79.98216100002173 + } + }, + "sqlAllApplicationPools": 74790, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2555, + "gapsMs": { + "n": 2555, + "p50": 25.13867300003767, + "p95": 26.488039000018034, + "p99": 28.65206400002353, + "max": 90.49584500002675 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2555, + "total_exec_time": 226.8500740000005, + "rows": 0 + } + ], + "nodeCpu": { + "user": 21150924, + "system": 1781032 + }, + "nodeCpuAverageOneCorePercent": 35.27067846892734, + "eventLoopDelay": { + "p95Ms": 11.354111, + "p99Ms": 12.230655, + "maxMs": 70.909951 + }, + "peakSampledNodeRssBytes": 869142528, + "peakSampledNodeHeapBytes": 618486624, + "postgresStatementExecutionMs": 7824.61821599999, + "lockObserverExecutionMs": 226.8500740000005 + } + ] + }, + "burst": { + "schema": "paperclip.normal-recovery-load/v1", + "scope": "Paired baseline/candidate, including real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", + "hardware": { + "postgresCpuLimit": 2, + "postgresMemoryLimitMiB": 1536, + "mainPool": 10, + "livenessPool": 2, + "candidateCheckpointPool": 2 + }, + "workload": { + "mode": "burst", + "normalReadsPerSecond": 100, + "normalWritesPerSecond": 20, + "databaseAdmissionEverySeconds": 5, + "checkpointControlEverySeconds": 5, + "controlPayloadBytes": 4096, + "phaseSeconds": 65, + "snapshotTransactionsPerControl": 1, + "baselineCheckpointPool": 2, + "baselineRenewalIntervalSeconds": 20, + "candidateRenewalIntervalSeconds": 20, + "historicalIssues": 10000 + }, + "passed": true, + "comparable": true, + "measurementLimits": [ + "Database request latency is not HTTP latency", + "Pool queue includes client encoding; BEGIN response includes network time", + "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed", + "Peak memory is sampled once a second", + "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide" + ], + "sourceReport": "/tmp/native-controller-load-qualification-20261030/remote/evidence/burst/report.json", + "relative": { + "postgresCpuUsec": { + "baseline": 1212.0604029710578, + "candidate": 1253.290446512422, + "increasePercent": 3.4016492445672952, + "passed": true + }, + "walBytes": { + "baseline": 4214.911636643046, + "candidate": 4322.979253820542, + "increasePercent": 2.563935533973982, + "passed": true + }, + "requestP99": { + "baseline": 99.50419749999128, + "candidate": 103.2992319999903, + "increasePercent": 3.813944130346214, + "passed": true + } + }, + "results": [ + { + "label": "01_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:48:31.444Z", + "endedAt": "2026-10-30T18:49:37.203Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23487, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 50551328, + "postgresCpuUsec": 14274208, + "postgresPeakSampledMemoryBytes": 987574272, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 754.7449020000058, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 74.4481569999989, + "p95": 98.20989800000098, + "p99": 101.2616299999936, + "max": 169.48861500000203 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 76.8531340000045, + "p95": 100.53923999999824, + "p99": 103.82765900000231, + "max": 171.57443800000328 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 504.43135499999335, + "p95": 904.9732890000014, + "p99": 987.5739109999995, + "max": 1058.7055260000052 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 464.0462219999972, + "p95": 860.9761490000092, + "p99": 942.788827999997, + "max": 998.9880149999954 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 5.402625000002445, + "p95": 9.567354999999225, + "p99": 32.887894999999844, + "max": 80.82659199999762 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.01833500000066124, + "p95": 0.03541399999812711, + "p99": 1.1884140000038315, + "max": 35.858127999992575 + }, + "liveness": { + "n": 900, + "p50": 577.4616719999976, + "p95": 935.9824230000013, + "p99": 967.3116440000013, + "max": 986.1326500000141 + }, + "checkpoint": { + "n": 3900, + "p50": 458.1738780000014, + "p95": 856.6399959999981, + "p99": 936.7588849999993, + "max": 995.3760920000059 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 5.375102999998489, + "p95": 9.639744000000064, + "p99": 39.09893100000045, + "max": 90.62846400000126 + }, + "liveness": { + "n": 900, + "p50": 582.0743040000016, + "p95": 940.3565129999988, + "p99": 971.0303759999952, + "max": 991.3499540000048 + }, + "checkpoint": { + "n": 3900, + "p50": 463.99203199999465, + "p95": 860.9391810000088, + "p99": 942.7536680000012, + "max": 998.9510200000077 + } + }, + "sqlAllApplicationPools": 43590, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2555, + "gapsMs": { + "n": 2555, + "p50": 25.1262850000021, + "p95": 26.189194999999017, + "p99": 62.625012000000424, + "max": 86.79922200000146 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2555, + "total_exec_time": 216.60417100000035, + "rows": 0 + } + ], + "nodeCpu": { + "user": 18625390, + "system": 871929 + }, + "nodeCpuAverageOneCorePercent": 29.649551657605677, + "eventLoopDelay": { + "p95Ms": 10.977279, + "p99Ms": 12.943359, + "maxMs": 70.647807 + }, + "peakSampledNodeRssBytes": 809156608, + "peakSampledNodeHeapBytes": 509942048, + "postgresStatementExecutionMs": 3523.919157, + "lockObserverExecutionMs": 216.60417100000035 + }, + { + "label": "02_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:49:43.659Z", + "endedAt": "2026-10-30T18:50:49.402Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23487, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 50596552, + "postgresCpuUsec": 14685071, + "postgresPeakSampledMemoryBytes": 990670848, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 739.15390200002, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 66.31109799999103, + "p95": 99.20572600000014, + "p99": 102.6184629999916, + "max": 152.73279199999524 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 69.19254099999671, + "p95": 101.30469299998367, + "p99": 151.40168099998846, + "max": 156.07804199999373 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 545.065419999999, + "p95": 943.5618349999859, + "p99": 1053.6890769999882, + "max": 1114.0595559999929 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 503.06268100001034, + "p95": 899.0765079999983, + "p99": 1003.1476519999997, + "max": 1050.1976159999904 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 5.609040000010282, + "p95": 10.175828000006732, + "p99": 66.40549299999839, + "max": 77.22545900000841 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.018979999993462116, + "p95": 0.03818100001080893, + "p99": 1.3919429999950808, + "max": 60.06078600001638 + }, + "liveness": { + "n": 900, + "p50": 618.575686000011, + "p95": 1052.3832659999898, + "p99": 1074.6600569999719, + "max": 1082.593469000014 + }, + "checkpoint": { + "n": 3900, + "p50": 498.0827250000002, + "p95": 893.5146509999904, + "p99": 996.6987400000071, + "max": 1045.0579469999939 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 5.584780999997747, + "p95": 10.301430999999866, + "p99": 66.71016699999745, + "max": 79.74655799999891 + }, + "liveness": { + "n": 900, + "p50": 625.6686590000027, + "p95": 1054.6769740000018, + "p99": 1076.872750999988, + "max": 1084.9572240000125 + }, + "checkpoint": { + "n": 3900, + "p50": 503.0248439999996, + "p95": 899.0387380000029, + "p99": 1003.1150250000064, + "max": 1050.135020000016 + } + }, + "sqlAllApplicationPools": 43590, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2542, + "gapsMs": { + "n": 2542, + "p50": 25.161112000001594, + "p95": 26.290963999999803, + "p99": 69.67558000001009, + "max": 85.49304400000256 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2542, + "total_exec_time": 216.69468099999958, + "rows": 0 + } + ], + "nodeCpu": { + "user": 17900804, + "system": 947711 + }, + "nodeCpuAverageOneCorePercent": 28.670043958722392, + "eventLoopDelay": { + "p95Ms": 11.100159, + "p99Ms": 12.918783, + "maxMs": 71.368703 + }, + "peakSampledNodeRssBytes": 870387712, + "peakSampledNodeHeapBytes": 576073168, + "postgresStatementExecutionMs": 3486.1393980000007, + "lockObserverExecutionMs": 216.69468099999958 + }, + { + "label": "03_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:50:56.635Z", + "endedAt": "2026-10-30T18:52:02.382Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23487, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 50673560, + "postgresCpuUsec": 14674511, + "postgresPeakSampledMemoryBytes": 973410304, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 744.6864390000119, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 65.08340400000452, + "p95": 101.02233700000215, + "p99": 103.98000099998899, + "max": 108.83414200000698 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 75.7316179999907, + "p95": 103.83602499999688, + "p99": 107.78453299999819, + "max": 111.01445499999681 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 496.9523610000033, + "p95": 920.7745010000071, + "p99": 1049.187115000008, + "max": 1091.9349159999983 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 457.1240420000104, + "p95": 873.8922820000153, + "p99": 995.4707909999997, + "max": 1029.6669490000058 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 5.59725800002343, + "p95": 9.960343999991892, + "p99": 63.98731100000441, + "max": 81.85869700001786 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.019010999996680766, + "p95": 0.034908000001451, + "p99": 1.545111999992514, + "max": 38.025777999981074 + }, + "liveness": { + "n": 900, + "p50": 624.3187389999803, + "p95": 1033.8756909999938, + "p99": 1092.2892289999872, + "max": 1099.7822960000194 + }, + "checkpoint": { + "n": 3900, + "p50": 451.83978900001966, + "p95": 868.8492180000176, + "p99": 990.5199409999768, + "max": 1025.1646750000073 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 5.570027000008849, + "p95": 10.061665000015637, + "p99": 64.13720699999249, + "max": 81.82990699997754 + }, + "liveness": { + "n": 900, + "p50": 630.1176869999908, + "p95": 1036.091023999994, + "p99": 1094.1707950000127, + "max": 1101.9002290000208 + }, + "checkpoint": { + "n": 3900, + "p50": 457.09078199998476, + "p95": 873.8558170000033, + "p99": 995.4385109999857, + "max": 1029.6382080000185 + } + }, + "sqlAllApplicationPools": 43590, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2544, + "gapsMs": { + "n": 2544, + "p50": 25.161667999986093, + "p95": 26.483623999985866, + "p99": 65.51908900000853, + "max": 86.48127200000454 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 1, + "observedLowerMs": { + "n": 1, + "p50": 0.694, + "p95": 0.694, + "p99": 0.694, + "max": 0.694 + }, + "estimatedUpperMs": { + "n": 1, + "p50": 26.656120999978157, + "p95": 26.656120999978157, + "p99": 26.656120999978157, + "max": 26.656120999978157 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 2, + "observedLowerMs": { + "n": 2, + "p50": 3.427, + "p95": 3.427, + "p99": 3.427, + "max": 3.427 + }, + "estimatedUpperMs": { + "n": 2, + "p50": 81.2373620000144, + "p95": 81.2373620000144, + "p99": 81.2373620000144, + "max": 81.2373620000144 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2544, + "total_exec_time": 218.81579299999981, + "rows": 3 + } + ], + "nodeCpu": { + "user": 18483524, + "system": 895970 + }, + "nodeCpuAverageOneCorePercent": 29.475632426669986, + "eventLoopDelay": { + "p95Ms": 11.198463, + "p99Ms": 12.804095, + "maxMs": 74.317823 + }, + "peakSampledNodeRssBytes": 871841792, + "peakSampledNodeHeapBytes": 566295128, + "postgresStatementExecutionMs": 3610.8802799999994, + "lockObserverExecutionMs": 218.81579299999981 + }, + { + "label": "04_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:52:08.252Z", + "endedAt": "2026-10-30T18:53:14.009Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23487, + "livenessSqlCount": 4502, + "checkpointSqlCount": 15601, + "walBytes": 48187192, + "postgresCpuUsec": 14119519, + "postgresPeakSampledMemoryBytes": 961527808, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 753.365232000011, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 70.38637200000812, + "p95": 94.09619499999098, + "p99": 97.74676499998895, + "max": 185.41928999999072 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 72.84908000001451, + "p95": 96.762837999966, + "p99": 105.79805099999066, + "max": 190.5021940000006 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 472.87740499997744, + "p95": 851.2413069999893, + "p99": 927.7114349999465, + "max": 1017.7665159999742 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 425.16228799999226, + "p95": 785.2056810000213, + "p99": 846.040737000003, + "max": 879.5945269999793 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 5.294340000022203, + "p95": 9.09644999995362, + "p99": 13.077560000005178, + "max": 79.4648959999904 + }, + "poolQueueByPool": { + "request": { + "n": 3913, + "p50": 0.01867399999173358, + "p95": 0.03246000001672655, + "p99": 1.0189679999893997, + "max": 38.69638399995165 + }, + "liveness": { + "n": 900, + "p50": 450.48504699999467, + "p95": 818.5160719999694, + "p99": 846.6014129999967, + "max": 852.2024820000515 + }, + "checkpoint": { + "n": 3900, + "p50": 420.15820399997756, + "p95": 780.1903570000432, + "p99": 842.1725940000033, + "max": 876.1292810000014 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 3913, + "p50": 5.267995000001974, + "p95": 9.177220999990823, + "p99": 13.902025000017602, + "max": 80.02995799999917 + }, + "liveness": { + "n": 900, + "p50": 455.3887499999837, + "p95": 824.6480000000156, + "p99": 848.961249999993, + "max": 854.1895970000187 + }, + "checkpoint": { + "n": 3900, + "p50": 425.0832320000045, + "p95": 785.1541730000172, + "p99": 845.993524999998, + "max": 879.5596040000091 + } + }, + "sqlAllApplicationPools": 43590, + "sampledPeakDatabaseActivityRows": 14, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2567, + "gapsMs": { + "n": 2567, + "p50": 25.124033000000054, + "p95": 26.08253999997396, + "p99": 50.32741100003477, + "max": 183.68320700002369 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 1, + "observedLowerMs": { + "n": 1, + "p50": 0.264, + "p95": 0.264, + "p99": 0.264, + "max": 0.264 + }, + "estimatedUpperMs": { + "n": 1, + "p50": 25.689116000023205, + "p95": 25.689116000023205, + "p99": 25.689116000023205, + "max": 25.689116000023205 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2567, + "total_exec_time": 212.36981699999987, + "rows": 1 + } + ], + "nodeCpu": { + "user": 17769478, + "system": 879767 + }, + "nodeCpuAverageOneCorePercent": 28.36105586844359, + "eventLoopDelay": { + "p95Ms": 10.936319, + "p99Ms": 12.083199, + "maxMs": 177.602559 + }, + "peakSampledNodeRssBytes": 872235008, + "peakSampledNodeHeapBytes": 588085240, + "postgresStatementExecutionMs": 3513.288256999997, + "lockObserverExecutionMs": 212.36981699999987 + } + ] + }, + "overlap-burst": { + "schema": "paperclip.normal-recovery-load/v1", + "scope": "Two logical servers in one client process, 150 controllers each, separate request/liveness/checkpoint pools at unchanged caps; same aggregate aligned-burst workload. Component overlap capacity only; no takeover latency or provider reattachment claim. Includes real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", + "hardware": { + "logicalServers": 2, + "postgresCpuLimit": 2, + "postgresMemoryLimitMiB": 1536, + "mainPool": 10, + "livenessPool": 2, + "candidateCheckpointPool": 2 + }, + "workload": { + "mode": "burst", + "normalReadsPerSecond": 100, + "normalWritesPerSecond": 20, + "databaseAdmissionEverySeconds": 5, + "checkpointControlEverySeconds": 5, + "controlPayloadBytes": 4096, + "phaseSeconds": 65, + "snapshotTransactionsPerControl": 1, + "baselineCheckpointPool": 2, + "baselineRenewalIntervalSeconds": 20, + "candidateRenewalIntervalSeconds": 20, + "historicalIssues": 10000 + }, + "passed": true, + "comparable": false, + "measurementLimits": [ + "Two logical server clients share one two-CPU Node process; this is a bounded component pool-overlap probe", + "Each logical server retains its own 10 request, 2 liveness, and 2 checkpoint connection limits", + "Same aggregate workload as the one-server aligned burst; no actual takeover or provider reattachment latency", + "Lock wait uses sampled PostgreSQL waitstart ages; waits shorter than polling gaps may be missed", + "Observer nominally adds 40 queries/second, included in database CPU", + "Database request latency is not HTTP latency; PostgreSQL CPU/WAL are cluster-wide" + ], + "sourceReport": "/tmp/native-controller-load-qualification-20261030/remote/evidence/overlap-burst/report.json", + "relative": null, + "results": [ + { + "label": "01_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:54:12.653Z", + "endedAt": "2026-10-30T18:55:18.280Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23494, + "livenessSqlCount": 4504, + "checkpointSqlCount": 15602, + "walBytes": 58953856, + "postgresCpuUsec": 16207490, + "postgresPeakSampledMemoryBytes": 1184854016, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 622.7349329999997, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 71.72571099999186, + "p95": 99.64835199999652, + "p99": 103.41825999999128, + "max": 167.0082839999959 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 73.87743999999657, + "p95": 101.96136399998795, + "p99": 106.03463099998771, + "max": 167.02244699999574 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 510.7785539999895, + "p95": 847.7751439999993, + "p99": 955.7120819999982, + "max": 1019.8632959999959 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 464.80313800000295, + "p95": 799.8753710000019, + "p99": 903.230091999998, + "max": 973.0980429999981 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 8.885602000002109, + "p95": 18.187949999999546, + "p99": 75.0605080000023, + "max": 103.59090300000025 + }, + "poolQueueByPool": { + "request": { + "n": 1956, + "p50": 0.01652999999350868, + "p95": 0.032654000009642914, + "p99": 0.5533529999956954, + "max": 32.65080599999783 + }, + "liveness": { + "n": 450, + "p50": 679.932338999999, + "p95": 983.9729049999951, + "p99": 1005.0673800000004, + "max": 1008.1818359999961 + }, + "checkpoint": { + "n": 1950, + "p50": 455.7207139999955, + "p95": 789.6575360000061, + "p99": 879.0804459999927, + "max": 927.088631999999 + }, + "replacement_request": { + "n": 1957, + "p50": 0.016338000001269393, + "p95": 0.030588000001444016, + "p99": 0.04346399999485584, + "max": 34.07667899999069 + }, + "replacement_liveness": { + "n": 450, + "p50": 680.664604000005, + "p95": 985.0139530000015, + "p99": 1007.0503020000033, + "max": 1013.1946360000002 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 451.09980499999074, + "p95": 795.8284449999992, + "p99": 905.3483259999994, + "max": 968.1059260000038 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 1956, + "p50": 8.849019000001135, + "p95": 18.71429999999964, + "p99": 79.22411199999988, + "max": 103.53132499999992 + }, + "liveness": { + "n": 450, + "p50": 690.4811160000027, + "p95": 989.9459690000003, + "p99": 1008.2045910000015, + "max": 1011.9318149999963 + }, + "checkpoint": { + "n": 1950, + "p50": 465.5724979999941, + "p95": 798.1944409999996, + "p99": 891.0923289999992, + "max": 935.9049690000029 + }, + "replacement_request": { + "n": 1957, + "p50": 8.899836000011419, + "p95": 20.206529000002774, + "p99": 72.87767399999575, + "max": 96.72780800001055 + }, + "replacement_liveness": { + "n": 450, + "p50": 690.242895000003, + "p95": 991.1967210000003, + "p99": 1010.9223639999982, + "max": 1015.3685380000024 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 461.74945899999875, + "p95": 805.4827579999983, + "p99": 914.226066999996, + "max": 973.0625819999987 + } + }, + "sqlAllApplicationPools": 43600, + "sampledPeakDatabaseActivityRows": 26, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2532, + "gapsMs": { + "n": 2532, + "p50": 25.151506999998674, + "p95": 26.541017999999895, + "p99": 66.61091300000408, + "max": 87.0732219999918 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "benchmark-replacement-application": { + "observedEpisodes": 2, + "observedLowerMs": { + "n": 2, + "p50": 1.992, + "p95": 1.992, + "p99": 1.992, + "max": 1.992 + }, + "estimatedUpperMs": { + "n": 2, + "p50": 77.9257440000074, + "p95": 77.9257440000074, + "p99": 77.9257440000074, + "max": 77.9257440000074 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 4, + "observedLowerMs": { + "n": 4, + "p50": 2.681, + "p95": 3.012, + "p99": 3.012, + "max": 3.012 + }, + "estimatedUpperMs": { + "n": 4, + "p50": 89.75422199998866, + "p95": 90.08522199999425, + "p99": 90.08522199999425, + "max": 90.08522199999425 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 6, + "observedLowerMs": { + "n": 6, + "p50": 1.226, + "p95": 3.516, + "p99": 3.516, + "max": 3.516 + }, + "estimatedUpperMs": { + "n": 6, + "p50": 28.208660000003874, + "p95": 64.30090599999676, + "p99": 64.30090599999676, + "max": 64.30090599999676 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2532, + "total_exec_time": 228.71010199999986, + "rows": 12 + } + ], + "nodeCpu": { + "user": 18403801, + "system": 860810 + }, + "nodeCpuAverageOneCorePercent": 29.35486475249997, + "eventLoopDelay": { + "p95Ms": 11.182079, + "p99Ms": 13.328383, + "maxMs": 72.482815 + }, + "peakSampledNodeRssBytes": 835289088, + "peakSampledNodeHeapBytes": 527752512, + "postgresStatementExecutionMs": 4084.128306999997, + "lockObserverExecutionMs": 228.71010199999986 + } + ] + }, + "overlap-abba": { + "schema": "paperclip.normal-recovery-load/v1", + "scope": "Two logical servers in one client process, 150 controllers each, separate request/liveness/checkpoint pools at unchanged caps; same aggregate aligned-burst workload. Matched baseline/candidate ABBA pool-overlap component qualification; no takeover latency or provider reattachment claim. Includes real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", + "hardware": { + "logicalServers": 2, + "postgresCpuLimit": 2, + "postgresMemoryLimitMiB": 1536, + "mainPool": 10, + "livenessPool": 2, + "candidateCheckpointPool": 2 + }, + "workload": { + "mode": "burst", + "normalReadsPerSecond": 100, + "normalWritesPerSecond": 20, + "databaseAdmissionEverySeconds": 5, + "checkpointControlEverySeconds": 5, + "controlPayloadBytes": 4096, + "phaseSeconds": 65, + "snapshotTransactionsPerControl": 1, + "baselineCheckpointPool": 2, + "baselineRenewalIntervalSeconds": 20, + "candidateRenewalIntervalSeconds": 20, + "historicalIssues": 10000 + }, + "passed": true, + "comparable": true, + "relative": { + "postgresCpuUsec": { + "baseline": 1334.8748826090668, + "candidate": 1376.3816699393838, + "increasePercent": 3.109414063525584, + "passed": true + }, + "walBytes": { + "baseline": 4209.21437718774, + "candidate": 4317.105779902672, + "increasePercent": 2.5632194762913674, + "passed": true + }, + "requestP99": { + "baseline": 106.66748450000887, + "candidate": 116.98804850000306, + "increasePercent": 9.675454566469433, + "passed": true + } + }, + "measurementLimits": [ + "Database request latency is not HTTP latency", + "Pool queue includes client encoding; BEGIN response includes network time", + "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed", + "Peak memory is sampled once a second", + "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide" + ], + "results": [ + { + "label": "01_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T18:59:52.122Z", + "endedAt": "2026-10-30T19:00:57.905Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23496, + "livenessSqlCount": 4504, + "checkpointSqlCount": 15602, + "walBytes": 50462184, + "postgresCpuUsec": 15645449, + "postgresPeakSampledMemoryBytes": 1343967232, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 721.3357259999902, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 18.673452000002726, + "p95": 101.14544200000091, + "p99": 105.69386800000211, + "max": 219.4722890000121 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 22.73070000000007, + "p95": 103.43886099999872, + "p99": 116.90596599999844, + "max": 224.30324400000973 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 487.1431490000032, + "p95": 804.2522070000123, + "p99": 901.6204700000017, + "max": 941.495895 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 436.6414150000055, + "p95": 746.4600769999961, + "p99": 826.2692409999945, + "max": 891.541333000001 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 8.546736999996938, + "p95": 15.356733000000531, + "p99": 75.99347599998873, + "max": 98.56840399999783 + }, + "poolQueueByPool": { + "request": { + "n": 1956, + "p50": 0.016741999999794643, + "p95": 0.03154400000494206, + "p99": 0.6992969999992056, + "max": 5.09585099999822 + }, + "liveness": { + "n": 450, + "p50": 501.6249909999897, + "p95": 876.1304410000012, + "p99": 913.067286999998, + "max": 919.2936289999998 + }, + "checkpoint": { + "n": 1950, + "p50": 427.4311180000004, + "p95": 738.933901999997, + "p99": 813.4696679999979, + "max": 886.4332880000002 + }, + "replacement_request": { + "n": 1957, + "p50": 0.0168159999884665, + "p95": 0.02985600000829436, + "p99": 0.04265299999678973, + "max": 10.915440000000672 + }, + "replacement_liveness": { + "n": 450, + "p50": 503.20186999999714, + "p95": 886.2206920000026, + "p99": 913.0074840000016, + "max": 918.3099900000016 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 426.9944880000112, + "p95": 739.3993829999963, + "p99": 821.3317409999981, + "max": 878.8103770000016 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 1956, + "p50": 8.524627999999211, + "p95": 15.395046999998158, + "p99": 76.11285200000202, + "max": 97.21485499999835 + }, + "liveness": { + "n": 450, + "p50": 510.662817000004, + "p95": 886.2594030000037, + "p99": 916.8632429999998, + "max": 921.3872209999972 + }, + "checkpoint": { + "n": 1950, + "p50": 436.56327500000043, + "p95": 745.776893000002, + "p99": 822.4703049999989, + "max": 891.5069800000056 + }, + "replacement_request": { + "n": 1957, + "p50": 8.534450999999535, + "p95": 15.9994200000001, + "p99": 75.55317300000024, + "max": 98.5201530000013 + }, + "replacement_liveness": { + "n": 450, + "p50": 513.2629840000009, + "p95": 890.6694930000012, + "p99": 915.7612549999976, + "max": 920.3067900000024 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 436.6130089999933, + "p95": 746.4048050000129, + "p99": 830.9584389999982, + "max": 887.7181599999967 + } + }, + "nodeCpu": { + "user": 17806507, + "system": 879516 + }, + "nodeCpuAverageOneCorePercent": 28.40549223176189, + "eventLoopDelay": { + "p95Ms": 11.010047, + "p99Ms": 12.697599, + "maxMs": 183.107583 + }, + "sqlAllApplicationPools": 43602, + "sampledPeakDatabaseActivityRows": 29, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2551, + "gapsMs": { + "n": 2551, + "p50": 25.145184999997582, + "p95": 26.409488999997848, + "p99": 59.03993599999376, + "max": 229.8140570000105 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "benchmark-replacement-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 10, + "observedLowerMs": { + "n": 10, + "p50": 2.481, + "p95": 11.224, + "p99": 11.224, + "max": 11.224 + }, + "estimatedUpperMs": { + "n": 10, + "p50": 27.13945600000443, + "p95": 36.74406199999794, + "p99": 36.74406199999794, + "max": 36.74406199999794 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2551, + "total_exec_time": 230.25003400000065, + "rows": 10 + } + ], + "peakSampledNodeRssBytes": 815190016, + "peakSampledNodeHeapBytes": 503697472, + "postgresStatementExecutionMs": 4126.207232999998, + "lockObserverExecutionMs": 230.25003400000065 + }, + { + "label": "02_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T19:01:04.413Z", + "endedAt": "2026-10-30T19:02:10.241Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23496, + "livenessSqlCount": 4504, + "checkpointSqlCount": 15602, + "walBytes": 50562024, + "postgresCpuUsec": 16161232, + "postgresPeakSampledMemoryBytes": 1324052480, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 825.5109480000101, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 14.214422999997623, + "p95": 103.00589499998023, + "p99": 109.18438099999912, + "max": 124.26887600000191 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 16.232166000001598, + "p95": 104.91053900000406, + "p99": 111.67462099998374, + "max": 124.31452200000058 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 497.3481239999819, + "p95": 844.0066550000047, + "p99": 960.1377219999995, + "max": 998.2879429999884 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 452.44012500000827, + "p95": 791.6675080000132, + "p99": 904.2310109999962, + "max": 948.914775000012 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 8.975842999992892, + "p95": 16.999474000011105, + "p99": 80.54993900000409, + "max": 92.77781900000991 + }, + "poolQueueByPool": { + "request": { + "n": 1956, + "p50": 0.01796499999181833, + "p95": 0.03485300000465941, + "p99": 0.5488950000144541, + "max": 16.1574099999998 + }, + "liveness": { + "n": 450, + "p50": 634.4087819999986, + "p95": 971.200656999994, + "p99": 989.703067999988, + "max": 996.0718149999884 + }, + "checkpoint": { + "n": 1950, + "p50": 443.02296800000477, + "p95": 778.6681580000004, + "p99": 884.4173180000071, + "max": 918.9185089999955 + }, + "replacement_request": { + "n": 1957, + "p50": 0.017980999997234903, + "p95": 0.03247500001452863, + "p99": 0.04606999998213723, + "max": 39.24437300002319 + }, + "replacement_liveness": { + "n": 450, + "p50": 634.364161000005, + "p95": 969.9561950000061, + "p99": 989.6641479999962, + "max": 995.097293999992 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 440.39501599999494, + "p95": 786.2747190000082, + "p99": 904.2403229999909, + "max": 943.5718220000126 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 1956, + "p50": 8.969991000019945, + "p95": 17.73811299999943, + "p99": 80.50918799999636, + "max": 92.72936200001277 + }, + "liveness": { + "n": 450, + "p50": 645.302377, + "p95": 976.2641570000123, + "p99": 993.3791540000093, + "max": 1001.4772759999905 + }, + "checkpoint": { + "n": 1950, + "p50": 453.5094059999974, + "p95": 786.6211449999828, + "p99": 893.5732319999952, + "max": 926.4708950000058 + }, + "replacement_request": { + "n": 1957, + "p50": 8.951632000011159, + "p95": 17.27179700000852, + "p99": 79.54755400000431, + "max": 92.341597000006 + }, + "replacement_liveness": { + "n": 450, + "p50": 645.1912089999969, + "p95": 975.069325000004, + "p99": 992.4281219999975, + "max": 997.0403169999918 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 451.6021180000098, + "p95": 793.450920000003, + "p99": 914.5407790000027, + "max": 948.8798110000062 + } + }, + "nodeCpu": { + "user": 18799633, + "system": 889825 + }, + "nodeCpuAverageOneCorePercent": 29.910693792544862, + "eventLoopDelay": { + "p95Ms": 11.124735, + "p99Ms": 12.779519, + "maxMs": 80.478207 + }, + "sqlAllApplicationPools": 43602, + "sampledPeakDatabaseActivityRows": 28, + "postgresThrottledUsec": 117766, + "lockWait": { + "intervalMs": 25, + "samples": 2548, + "gapsMs": { + "n": 2548, + "p50": 25.149598000003607, + "p95": 26.418762000001152, + "p99": 73.46682900001178, + "max": 90.0133790000109 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 2, + "observedLowerMs": { + "n": 2, + "p50": 57.415, + "p95": 57.415, + "p99": 57.415, + "max": 57.415 + }, + "estimatedUpperMs": { + "n": 2, + "p50": 78.82680399998208, + "p95": 78.82680399998208, + "p99": 78.82680399998208, + "max": 78.82680399998208 + } + }, + "benchmark-replacement-application": { + "observedEpisodes": 1, + "observedLowerMs": { + "n": 1, + "p50": 0.817, + "p95": 0.817, + "p99": 0.817, + "max": 0.817 + }, + "estimatedUpperMs": { + "n": 1, + "p50": 25.092009999993024, + "p95": 25.092009999993024, + "p99": 25.092009999993024, + "max": 25.092009999993024 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 11, + "observedLowerMs": { + "n": 11, + "p50": 3.97, + "p95": 58.82, + "p99": 58.82, + "max": 58.82 + }, + "estimatedUpperMs": { + "n": 11, + "p50": 30.69183900000644, + "p95": 83.27365099999588, + "p99": 83.27365099999588, + "max": 83.27365099999588 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2548, + "total_exec_time": 234.671226, + "rows": 14 + } + ], + "peakSampledNodeRssBytes": 870621184, + "peakSampledNodeHeapBytes": 593590232, + "postgresStatementExecutionMs": 4403.789213000001, + "lockObserverExecutionMs": 234.671226 + }, + { + "label": "03_candidate_300", + "variant": "candidate", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T19:02:16.491Z", + "endedAt": "2026-10-30T19:03:22.313Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23496, + "livenessSqlCount": 4504, + "checkpointSqlCount": 15602, + "walBytes": 50570496, + "postgresCpuUsec": 16081885, + "postgresPeakSampledMemoryBytes": 1332752384, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 818.6222430000198, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 39.5862119999947, + "p95": 74.14522999999463, + "p99": 124.791716000007, + "max": 129.85977700000512 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 41.993144000007305, + "p95": 77.44222500000615, + "p99": 125.49473299999954, + "max": 132.81708999999682 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 487.02540600000066, + "p95": 836.5916340000113, + "p99": 955.1296190000139, + "max": 1000.8564950000145 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 444.6453029999975, + "p95": 790.974338, + "p99": 903.6869510000106, + "max": 944.6195900000166 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 8.99197100001038, + "p95": 17.247062000009464, + "p99": 77.24802900000941, + "max": 89.03845799999544 + }, + "poolQueueByPool": { + "request": { + "n": 1956, + "p50": 0.01789499999722466, + "p95": 0.03461500001139939, + "p99": 1.2706449999823235, + "max": 7.25288700000965 + }, + "liveness": { + "n": 450, + "p50": 668.1329050000058, + "p95": 943.7231139999931, + "p99": 964.8324849999917, + "max": 971.0895420000015 + }, + "checkpoint": { + "n": 1950, + "p50": 436.23111700001755, + "p95": 776.24320300002, + "p99": 884.2146380000049, + "max": 935.5753669999831 + }, + "replacement_request": { + "n": 1957, + "p50": 0.017906000022776425, + "p95": 0.03174499998567626, + "p99": 0.04958200000692159, + "max": 37.47016100000474 + }, + "replacement_liveness": { + "n": 450, + "p50": 660.5174439999973, + "p95": 939.0108659999969, + "p99": 963.677406999981, + "max": 968.687052000023 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 436.32736499997554, + "p95": 788.8090800000064, + "p99": 902.0589020000189, + "max": 938.6039999999921 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 1956, + "p50": 8.946074000006774, + "p95": 17.429038000002038, + "p99": 76.37251200000173, + "max": 89.003899000003 + }, + "liveness": { + "n": 450, + "p50": 678.8731129999796, + "p95": 945.945057999983, + "p99": 967.5097349999996, + "max": 973.1065839999937 + }, + "checkpoint": { + "n": 1950, + "p50": 444.8119090000109, + "p95": 784.2202169999946, + "p99": 893.1187290000089, + "max": 943.4892260000051 + }, + "replacement_request": { + "n": 1957, + "p50": 9.007467999996152, + "p95": 18.345773000008194, + "p99": 78.94358199997805, + "max": 89.00447099999292 + }, + "replacement_liveness": { + "n": 450, + "p50": 668.120407999988, + "p95": 941.9229499999783, + "p99": 966.5471529999922, + "max": 971.1461609999824 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 444.2886780000117, + "p95": 798.4288989999914, + "p99": 912.1802920000046, + "max": 944.5919719999947 + } + }, + "nodeCpu": { + "user": 18563401, + "system": 892958 + }, + "nodeCpuAverageOneCorePercent": 29.559203827490894, + "eventLoopDelay": { + "p95Ms": 11.214847, + "p99Ms": 13.647871, + "maxMs": 80.478207 + }, + "sqlAllApplicationPools": 43602, + "sampledPeakDatabaseActivityRows": 28, + "postgresThrottledUsec": 0, + "lockWait": { + "intervalMs": 25, + "samples": 2541, + "gapsMs": { + "n": 2541, + "p50": 25.158021000010194, + "p95": 26.61356500000693, + "p99": 70.55964399999357, + "max": 90.20055800001137 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 3, + "observedLowerMs": { + "n": 3, + "p50": 2.624, + "p95": 42.948, + "p99": 42.948, + "max": 42.948 + }, + "estimatedUpperMs": { + "n": 3, + "p50": 68.97033699997701, + "p95": 72.40636000002269, + "p99": 72.40636000002269, + "max": 72.40636000002269 + } + }, + "benchmark-replacement-application": { + "observedEpisodes": 2, + "observedLowerMs": { + "n": 2, + "p50": 2.623, + "p95": 2.623, + "p99": 2.623, + "max": 2.623 + }, + "estimatedUpperMs": { + "n": 2, + "p50": 59.07080300000962, + "p95": 59.07080300000962, + "p99": 59.07080300000962, + "max": 59.07080300000962 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 4, + "observedLowerMs": { + "n": 4, + "p50": 1.948, + "p95": 4.972, + "p99": 4.972, + "max": 4.972 + }, + "estimatedUpperMs": { + "n": 4, + "p50": 56.89580300002126, + "p95": 76.26736000002711, + "p99": 76.26736000002711, + "max": 76.26736000002711 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 8, + "observedLowerMs": { + "n": 8, + "p50": 2.616, + "p95": 62.06, + "p99": 62.06, + "max": 62.06 + }, + "estimatedUpperMs": { + "n": 8, + "p50": 28.899388999998337, + "p95": 87.05610200000228, + "p99": 87.05610200000228, + "max": 87.05610200000228 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2541, + "total_exec_time": 230.91749600000003, + "rows": 17 + } + ], + "peakSampledNodeRssBytes": 887791616, + "peakSampledNodeHeapBytes": 633808520, + "postgresStatementExecutionMs": 4428.595989999998, + "lockObserverExecutionMs": 230.91749600000003 + }, + { + "label": "04_baseline_300", + "variant": "baseline", + "n": 300, + "durationMs": 65000, + "startedAt": "2026-10-30T19:03:29.319Z", + "endedAt": "2026-10-30T19:04:35.119Z", + "normalOffered": 6500, + "normalCompleted": 6500, + "normalWritesOffered": 1300, + "normalWritesCompleted": 1300, + "admissionsOffered": 13, + "admissionsCompleted": 13, + "controls": 3900, + "checkpointTransactions": 3900, + "snapshotTransactions": 3900, + "changedRows": 7800, + "changedBytes": 17371500, + "sqlCount": 23496, + "livenessSqlCount": 4504, + "checkpointSqlCount": 15602, + "walBytes": 48142872, + "postgresCpuUsec": 15625330, + "postgresPeakSampledMemoryBytes": 1307955200, + "renewalLosses": 0, + "dropped": 0, + "errors": [], + "drainMs": 796.7394680000143, + "correctnessPassed": true, + "absoluteGates": { + "requestP99": true, + "drain": true, + "throughput": true, + "postgresCpu": true, + "postgresMemory": true + }, + "passed": true, + "normalReadSummary": { + "n": 6500, + "p50": 14.284052000002703, + "p95": 102.27245000004768, + "p99": 107.64110100001562, + "max": 141.18229500000598 + }, + "normalWriteSummary": { + "n": 1300, + "p50": 16.601036000007298, + "p95": 104.65098200005013, + "p99": 110.24340700003086, + "max": 141.02557800000068 + }, + "controlLatencySummary": { + "n": 3900, + "p50": 511.11750200000824, + "p95": 835.1623750000144, + "p99": 932.1108870000462, + "max": 966.9618110000156 + }, + "checkpointLatencySummary": { + "n": 3900, + "p50": 466.6611930000363, + "p95": 785.0629789999803, + "p99": 882.4489049999975, + "max": 911.663005999988 + }, + "snapshotLatencySummary": { + "n": 3900, + "p50": 8.741485000005923, + "p95": 21.27701399999205, + "p99": 79.73378300003242, + "max": 98.61222199999611 + }, + "poolQueueByPool": { + "request": { + "n": 1956, + "p50": 0.017884999979287386, + "p95": 0.035715999954845756, + "p99": 0.7316219999920577, + "max": 6.9568530000397 + }, + "liveness": { + "n": 450, + "p50": 612.479364000028, + "p95": 931.2895360000257, + "p99": 968.4246090000088, + "max": 974.2110239999893 + }, + "checkpoint": { + "n": 1950, + "p50": 457.2952529999893, + "p95": 777.7845369999995, + "p99": 872.6880790000141, + "max": 904.4663130000117 + }, + "replacement_request": { + "n": 1957, + "p50": 0.017825999995693564, + "p95": 0.032011000032071024, + "p99": 0.05147999996552244, + "max": 58.91463199999998 + }, + "replacement_liveness": { + "n": 450, + "p50": 611.0748399999866, + "p95": 924.6075030000065, + "p99": 966.4776800000109, + "max": 972.2507760000008 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 458.15926700001, + "p95": 773.1846900000237, + "p99": 871.5540899999905, + "max": 906.4964309999486 + } + }, + "transactionLatencyByPool": { + "request": { + "n": 1956, + "p50": 8.748108999978285, + "p95": 21.241075000027195, + "p99": 79.69776200002525, + "max": 96.44556800002465 + }, + "liveness": { + "n": 450, + "p50": 623.5618810000014, + "p95": 938.7270220000064, + "p99": 971.3052120000066, + "max": 976.3884749999852 + }, + "checkpoint": { + "n": 1950, + "p50": 466.64239300001645, + "p95": 786.580037000007, + "p99": 882.414477000013, + "max": 910.3289480000385 + }, + "replacement_request": { + "n": 1957, + "p50": 8.698626999976113, + "p95": 22.4105860000127, + "p99": 80.60295999998925, + "max": 98.5777620000008 + }, + "replacement_liveness": { + "n": 450, + "p50": 620.2892739999807, + "p95": 935.5929790000082, + "p99": 969.3374020000047, + "max": 975.1367830000236 + }, + "replacement_checkpoint": { + "n": 1950, + "p50": 466.62874600000214, + "p95": 782.9779160000035, + "p99": 880.8073159999913, + "max": 911.626213999989 + } + }, + "nodeCpu": { + "user": 19222778, + "system": 829326 + }, + "nodeCpuAverageOneCorePercent": 30.474186242549273, + "eventLoopDelay": { + "p95Ms": 10.977279, + "p99Ms": 14.262271, + "maxMs": 82.444287 + }, + "sqlAllApplicationPools": 43602, + "sampledPeakDatabaseActivityRows": 28, + "postgresThrottledUsec": 86175, + "lockWait": { + "intervalMs": 25, + "samples": 2544, + "gapsMs": { + "n": 2544, + "p50": 25.112813999992795, + "p95": 26.291275000025053, + "p99": 71.21967900000163, + "max": 94.01448299997719 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "benchmark-replacement-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 7, + "observedLowerMs": { + "n": 7, + "p50": 3.134, + "p95": 62.973, + "p99": 62.973, + "max": 62.973 + }, + "estimatedUpperMs": { + "n": 7, + "p50": 29.69778799993219, + "p95": 91.62701000005472, + "p99": 91.62701000005472, + "max": 91.62701000005472 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + }, + "lockObserverSql": [ + { + "calls": 2544, + "total_exec_time": 225.15488799999974, + "rows": 7 + } + ], + "peakSampledNodeRssBytes": 900505600, + "peakSampledNodeHeapBytes": 621405448, + "postgresStatementExecutionMs": 4218.09364, + "lockObserverExecutionMs": 225.15488799999974 + } + ], + "sourceReport": "/tmp/native-controller-load-qualification-20261030/remote/evidence/overlap-abba/report.json" + } + }, + "lockObserverCalibration": { + "intentionalHoldMs": 250, + "releasedAtMs": 249.4750299999996, + "transactionElapsedMs": 304.0121600000002, + "observed": { + "intervalMs": 25, + "samples": 12, + "gapsMs": { + "n": 12, + "p50": 25.226912000000084, + "p95": 35.78078900000037, + "p99": 35.78078900000037, + "max": 35.78078900000037 + }, + "errors": [], + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 1, + "observedLowerMs": { + "n": 1, + "p50": 229.153, + "p95": 229.153, + "p99": 229.153, + "max": 229.153 + }, + "estimatedUpperMs": { + "n": 1, + "p50": 254.2638969999989, + "p95": 254.2638969999989, + "p99": 254.2638969999989, + "max": 254.2638969999989 + } + } + }, + "episodes": [ + { + "pid": 88, + "application_name": "paperclip-runner-checkpoint", + "query_start": "2026-10-30 18:42:31.6856+00", + "wait_event": "transactionid", + "wait_start": "2026-10-30 18:42:31.685612+00", + "wait_age_ms": 229.153, + "firstMs": 12704.136085, + "lastMs": 12899.644581, + "precedingGapMs": 35.78078900000037, + "observedLowerMs": 229.153, + "estimatedUpperMs": 254.2638969999989, + "rightCensored": false + } + ], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + } + }, + "overlapInterpretation": { + "scope": "Two logical server clients share one two-CPU Node process; each has its own bounded request/liveness/checkpoint pools. Same aggregate workload as one-server aligned burst.", + "postgresCpuSeconds": 16.20749, + "oneServerCandidateBurstMeanPostgresCpuSeconds": 14.679791, + "increasePercent": 10.406817099780241, + "interpretation": "Capacity passed. This additional CPU cost comes from a different topology and one sample; it is not an improvement claim or a repeatable baseline/candidate regression finding." + }, + "retainedFailures": { + "earlierReport": "doc/plans/2026-10-30-native-controller-load-evidence.json", + "earlierReportUnchanged": true, + "freshAttemptLedger": "/tmp/native-controller-load-qualification-20261030/failure-ledger.json", + "localFailedPilot": { + "requestP99Ms": 1211.193594999997, + "droppedOperations": 33, + "postgresThrottledUsec": 12294711, + "outcome": "Ineligible for acceptance; local scheduling pressure and unmatched work. A subsequent identical local pilot passed, demonstrating temporal host variability." + }, + "remoteSetupFailures": [ + "Abbreviated Git revision fetch rejected; exact full revision fetched.", + "Missing built plugin SDK export; compiled from locked installed dependencies.", + "Observer database missing pg_stat_statements; harness now creates the extension in the owned observer database." + ], + "allFailedAttemptsAccountedFor": true + }, + "limits": [ + "Database authentication/read latency is not HTTP latency or full application latency.", + "The baseline supplies writer, snapshot port and PRP controller; normal request and lease-renewal paths are common controls.", + "PostgreSQL CPU and WAL are cluster-wide and include observer overhead and PostgreSQL background work.", + "Lock observer nominally adds 40 queries/second; actual calls and SQL execution time are included in each result.", + "Lock episodes use PostgreSQL waitstart identities and server-reported ages. Short waits between polls can be missed, and client scheduling/network delay widens estimated upper bounds.", + "Pool admission includes client encoding; SQL execution statistics include lock waits.", + "Memory and database activity are sampled once per second. Activity-row counts include non-client database backends, so they are not exact client-connection counts.", + "The two-server overlap probe uses two logical clients in one Node process and does not measure real process replacement or authenticated provider reattachment.", + "The separate claim-capacity probe uses synthetic prepared-workspace references. Provider authentication and external sandbox existence are outside its scope." + ], + "rawEvidence": { + "root": "/tmp/native-controller-load-qualification-20261030", + "initialArchive": "/tmp/native-controller-load-qualification-20261030/remote-evidence-initial.tar.gz", + "initialArchiveSha256": "d1f2da9429e2ae1944df35d40da77c0333c506336fed4f78dcfd1d13447403f8", + "cloudInstanceCleanup": "Owned benchmark containers stopped and owned disposable VM confirmed terminated with no attached root volumes.", + "finalDeltaArchive": "/tmp/native-controller-load-qualification-20261030/remote-evidence-final-delta.tar.gz", + "finalDeltaArchiveSha256": "1e77edbfd3ee9e05bd5de6a9ecefb82fdb12c89e52e575f9e8b48ea0c94d26ed", + "cleanupReceipt": "/tmp/native-controller-load-qualification-20261030/ec2-final-state.json" + }, + "claimCapacity": { + "schema": "paperclip.native-controller-claim-capacity/v1", + "scope": "Production DB claim function against synthetic prepared-workspace references and real persisted snapshots/PRP records. Natural lease expiry; two server database clients. No provider attachment or external sandbox existence claim.", + "passed": true, + "harnessSha256": "b083e2e34eccf21a1921e3b32a434ac7d5b1dec24c903b0b3b5052f9f48adf2e", + "candidateRevision": "a0ef3ce7493c8bb76c4766ff007e8a390118d145", + "baselineRevision": "59e191c368cb3119ff3206621fefa9c5d4ad3110", + "errors": [], + "naturalExpiryWaitMs": 54026.14, + "claimDrainMs": 2838.7143720000167, + "claimAndRestoreDrainMs": 8742.97213000001, + "restoredCheckpoints": 300, + "renewedOwnerLosses": 0, + "postRenewalOwners": { + "owners": 300, + "min_generation": 2, + "max_generation": 2, + "minimum_remaining_ms": 56198.788 + }, + "elapsedMs": 77971.21296299998, + "requestsOffered": 15400, + "requestsCompleted": 15400, + "requestLatency": { + "n": 15400, + "p50": 2.684827999997651, + "p95": 5.376435999991372, + "p99": 9.759766000002855, + "max": 64.86076799999864 + }, + "postgresCpuUsec": 11280308, + "sqlByClient": { + "old_request": 7703, + "old_liveness": 0, + "old_checkpoint": 0, + "new_request": 12512, + "new_liveness": 1502, + "new_checkpoint": 1502 + }, + "settledIdleSql": 0, + "retainedOldControllerLeaseMs": 60000, + "remoteReconnectGraceMs": 420000, + "checkpointPoolPerServer": 2, + "claimsByKind": { + "reattach_remote_runner": 300 + }, + "distinctClaimedRuns": 300, + "claimGenerations": [ + 2 + ], + "fixtureDatabaseRequestsPerSecond": 200, + "sourceCode": "/tmp/native-controller-load-qualification-20261030/benchmark-native-controller-claim-capacity.ts", + "sourceReport": "/tmp/native-controller-load-qualification-20261030/remote/evidence/claim-capacity/claim-capacity.json", + "sourceReportSha256": "cd9fc360f1cc1df0b6e82ca50694f6b2ad4c3ff3f638381637dcc99dd3385bc2", + "peakSampledDatabaseActivityRows": 25, + "lockWait": { + "samples": 2987, + "gapsMs": { + "n": 2987, + "p50": 25.607273999994504, + "p95": 29.01517500000773, + "p99": 43.6128069999977, + "max": 85.42577500001062 + }, + "byPool": { + "benchmark-application": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-controller-lease": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + }, + "paperclip-runner-checkpoint": { + "observedEpisodes": 0, + "observedLowerMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + }, + "estimatedUpperMs": { + "n": 0, + "p50": 0, + "p95": 0, + "p99": 0, + "max": 0 + } + } + }, + "errors": [], + "limits": "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." + } + }, + "matchedOverlapAbba": { + "status": "passed", + "sourceCode": "/tmp/native-controller-load-qualification-20261030/benchmark-native-controller-overlap-abba.ts", + "sourceSha256": "2cc52afee21c34efab7f65af88bbd911e6efdd424094f7020bfccd4a16b4d441", + "requestP99IncreasePercent": 9.675454566469433, + "interpretation": "All relative gates passed; request p99 increase is close to the 10 percent threshold." + }, + "reproduction": { + "baseHarnessSha256": "abaa29d50eb95013836b54b8dacc94c72e080edf24744cc5d291cc5eb1ff2818", + "patches": { + "overlap": { + "path": "doc/plans/2026-10-30-native-controller-overlap.patch", + "patchSha256": "e8e14ddbdbed289f47e57fa0fe8ae2be337e77f2919c755d7003afd73150faf4", + "reconstructedSourceSha256": "2cc52afee21c34efab7f65af88bbd911e6efdd424094f7020bfccd4a16b4d441", + "verifiedPatchReconstruction": true + }, + "claim": { + "path": "doc/plans/2026-10-30-native-controller-claim.patch", + "patchSha256": "1fcbf028654499562c5f6b789f36e7c4681ec5d28c3dec6d13e2a51650496e2e", + "reconstructedSourceSha256": "b083e2e34eccf21a1921e3b32a434ac7d5b1dec24c903b0b3b5052f9f48adf2e", + "verifiedPatchReconstruction": true + } + }, + "instructions": [ + "Copy scripts/benchmark-native-controller-recovery.ts to a new probe source file.", + "Apply the corresponding dated patch with: patch /absolute/path/to/probe.ts /absolute/path/to/patch.", + "Set existing benchmark inputs plus BENCH_CANDIDATE_ROOT to the repository root; use the pinned baseline and identical installed dependencies.", + "Run Node 24 with the repository tsx loader and the patched probe file followed by burst. Each report directory must be fresh." + ], + "requiredInputs": [ + "BENCH_DISPOSABLE_POSTGRES=1", + "BENCH_DATABASE_URL", + "BENCH_BASELINE_ROOT", + "BENCH_REPORT_ROOT", + "BENCH_POSTGRES_CGROUP", + "BENCH_CANDIDATE_ROOT" + ] + }, + "streamingSqlIndependence": { + "status": "passed", + "sourceRevision": "da8c7f6e1837ddd896dfe83a7524812fee813487", + "sourceReport": "/tmp/native-controller-release-qualification-20261030/stream-cost-current.json", + "sourceLog": "/tmp/native-controller-release-qualification-20261030/stream-cost-current.log", + "samples": [ + { + "chunks": 8, + "streamSql": 0, + "streamTransactions": 0, + "streamChangedRows": 0, + "idleSql": 0, + "checkpointWrites": 8, + "checkpointTransactions": 7, + "changedRows": 12 + }, + { + "chunks": 512, + "streamSql": 0, + "streamTransactions": 0, + "streamChangedRows": 0, + "idleSql": 0, + "checkpointWrites": 8, + "checkpointTransactions": 7, + "changedRows": 12 + }, + { + "chunks": 8192, + "streamSql": 0, + "streamTransactions": 0, + "streamChangedRows": 0, + "idleSql": 0, + "checkpointWrites": 8, + "checkpointTransactions": 7, + "changedRows": 12 + } + ], + "positiveCalibrationPassed": true, + "existingRenewalSqlFor8192ChunkCase": 66, + "limits": [ + "Separate real Runner fixture executed by the release qualification agent.", + "Concurrent owned fixtures may affect its cluster-wide WAL and elapsed diagnostics; those values are excluded from performance claims." + ] + } +} diff --git a/doc/plans/2026-10-30-native-controller-source-probe-receipt.json b/doc/plans/2026-10-30-native-controller-source-probe-receipt.json new file mode 100644 index 00000000000..73d05dc06a6 --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-source-probe-receipt.json @@ -0,0 +1,70 @@ +{ + "schema": "paperclip.native-controller.release-qualification-receipt/v1", + "finishedAt": "2026-10-30T19:33:20.587023+00:00", + "sourceIntegrity": { + "A": { + "revision": "59ff89a18fe1312b5f0bb908439bbdb1237b0252", + "nativeRuntimeSourceBytesUnchanged": true, + "mismatches": [] + }, + "B": { + "revision": "98f05fcd5b452704131a35ff09486f5c04c6959a", + "nativeRuntimeSourceBytesUnchanged": true, + "mismatches": [] + } + }, + "sourceAndArtifact": { + "A": "59ff89a18fe1312b5f0bb908439bbdb1237b0252", + "B": "98f05fcd5b452704131a35ff09486f5c04c6959a", + "admittedRunnerSha256": "b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8", + "runnerArtifact": { + "role": "admitted Runner artifact A, unchanged through every controller", + "sourceRevision": "59ff89a18fe1312b5f0bb908439bbdb1237b0252", + "binary": "/private/tmp/native-controller-qualification-20261030-1833/installed-A/consumer/node_modules/@paperclipai/server/dist/vendor/paperclip-runner/bin/linux-arm64/paperclip-runnerd", + "sha256": "b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8", + "bytes": 35345144, + "installationReceipt": "/private/tmp/native-controller-qualification-20261030-1833/installed-A/receipt.json" + } + }, + "passed": [ + "Distinct source-controller active SIGKILL followed by natural 60-second lease takeover; exact same run/provider attempt/Runner retained", + "Current real-PostgreSQL stream oracle at 8,512,8192 chunks: zero incremental checkpoint SQL/transactions/changed rows; idle SQL zero; positive control passed", + "B1 recovered run completed and committed its result with ready native snapshot and valid v1 recovery envelope" + ], + "failed": [ + "Fresh B2 retained continuation rejected after B1 completed because durable control journal has a null-goal snapshot after run.terminal", + "Final 98f05fc live probe still rejects actual paperclip.prp.event.v2 because proposed guard accepts only v1" + ], + "notQualified": [ + "Complete A/B/A native installed-application startup/rollback gate", + "Uninterrupted warm provider process through changed completion-contract boundary", + "First compatible published empty-disk rollback reader" + ], + "limits": [ + "Source controller processes and scripted Codex-compatible provider in Docker; no live-provider or live-Daytona claim", + "Qualified immutable A release Runner used; source controller native modules hash-verified, not full installed server execution", + "Only native controller projections deleted; non-native configuration, encryption key, workspace/sandbox dependencies preserved", + "Local stream WAL/elapsed diagnostics excluded from quantitative performance due concurrent isolated fixture" + ], + "evidence": { + "finalProbe": "/tmp/native-controller-release-qualification-qualified-A-20261030/evidence.json", + "finalProbeLog": "/tmp/native-controller-release-qualification-qualified-A-20261030/run-final-fix.log", + "controlLog": "/tmp/native-controller-release-qualification-qualified-A-20261030/controller.log", + "streamOracle": "/tmp/native-controller-release-qualification-20261030/stream-cost-current.json", + "streamLog": "/tmp/native-controller-release-qualification-20261030/stream-cost-current.log", + "earlierThreeTurnBusinessContinuation": "/tmp/native-controller-release-qualification-20261030/evidence-run-06.json" + }, + "finalProbeFailure": { + "message": "Error: runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate\n at deny (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/remote-warm-session-recovery.ts:56:37)\n at inspectRemoteWarmSessionRecovery (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/remote-warm-session-recovery.ts:85:67)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async executePaperclipNativeSessionWithinScope (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/native-session-executor.ts:8237:28)\n at async executePaperclipNativeSession (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/native-session-executor.ts:8057:12)\n at async process. (/private/tmp/native-controller-release-qualification-qualified-A-20261030/controller-worker.mjs:36:16)", + "cause": null + }, + "candidateCorrection": { + "change": "Accept supported PRP event v1 and v2 idle goal suffixes; reject unknown versions", + "fullSourceReplay": "not rerun after the final v2 correction within the one-hour qualification window", + "focusedTests": { + "result": "pass", + "cases": 14, + "test": "server/src/services/native-runtime/remote-warm-session-recovery.test.ts" + } + } +} diff --git a/doc/testing-native-remote-continuity.md b/doc/testing-native-remote-continuity.md index 91887a1b71b..65671f91edf 100644 --- a/doc/testing-native-remote-continuity.md +++ b/doc/testing-native-remote-continuity.md @@ -7,7 +7,7 @@ and backup validation against a deterministic fake Codex process in Linux. It checks the same runner process identity and one provider process across three warm turns, provider history reuse, and verified suspension at idle close. Separate cases kill the actual controller process and recover the next turn. They reject -a still-live controller and a replacement sandbox before reconnecting. Legacy +a live execution lease and a replacement sandbox before reconnecting. Legacy checkpoints and changed process configuration require verified suspension and replacement, while preserving the conversation and starting each turn once. Recovery can restart a provider to refresh its tool contract; process reuse is @@ -24,6 +24,39 @@ subscription. The controller must register the connection before subscribing so completed message processing can drain and the suspended session can retire. It does not qualify a live model or a sandbox provider API. +The `diskless` and `diskless-per-turn` cases delete both native controller +directories after A completes, continue in a separate B process, delete those +directories after B completes, and continue in C. They use the production source +selector and snapshot rebinder for per-turn admission, followed by the real +native executor. They do not inject PRP files or edit leases to produce recovery. +Warm mode verifies compatible process reuse; per-turn mode verifies the existing +suspension and history-resume policy. These component tests preserve workspace +and remote sandbox storage, and use one source revision in three processes. +They do not qualify distinct application releases or complete webserver disk loss. + +`native-controller-recovery.integration.test.ts` separately verifies encrypted +database discovery, one-source import, idempotency, live-owner waiting, corrupt +bindings and paginated inventory. `native-session-resume-projection.test.ts` +retains the 32-candidate and newer-progress barrier checks. Snapshot transaction +counts and stale-generation rejection are exercised by the control-plane port +tests; incomplete-publication retention is exercised by the expiration tests. + +The cold Codex transport regressions additionally retain a real provider event +after completion, hold its persistence callback and require the original source +tuple and Runner outbox to survive until acceptance. A separate case deletes the +controller cache after its staged target is durably recorded and resumes through +the same source before target attachment. Unknown semantic tails must stay held +without target tool dispatch or a fabricated response. The staged restart helper +and classifier tests cover target discovery before its first session snapshot, +exact encrypted source/header/artifact bindings and the existing compatibility +rebinder. Source accounting tests use real PostgreSQL to distinguish immutable +receipt replay, equal finalized usage and late usage requiring reconciliation. +Handoff crash tests must restore the exact queued command or authenticated +receipt, reuse the verified physical Runner and preserve endpoint binding. Test +both direct callbacks and the same guest listener when that ingress is enabled; +changing an endpoint or losing physical proof must hold recovery. +These component cases do not replace the normal three-controller Daytona gate. + Per-turn plugin continuation uses a stopped checkpoint on a replacement lease. The executor tests cover Claude and Codex, including reopening blocked work. The lease-authority integration test uses PostgreSQL to check that checkpoint @@ -151,14 +184,21 @@ local file becomes a marker that requires restoration from the external checkpoint before restart. Ordinary control changes do not rewrite a second local journal. The sandbox runner keeps its own journal. -Checkpoint transactions share a separate pool capped at two connections per +Checkpoint transactions share a separate pool capped at three connections per server database client. They queue without occupying the application pool or the existing lease-renewal pool. This can delay a control acknowledgement during a burst; it does not release that acknowledgement before its commit. The cost is -up to two additional active database connections per instance, not per agent. +up to three additional active database connections per instance, not per agent. There is no new SQL per text chunk or while idle, and the number of control transactions is unchanged. SQL cost collection must include the checkpoint pool. +The cap follows a bounded 2 → 3 → 4 → 2 diagnostic: both cap-2 trials failed +the existing burst-drain gate; 3 was the smallest tested passing cap. Retain +those failures and the diagnostic separately. The subsequent native feature-cost +comparison uses the common current DB factory and equal cap-3 policy for both +variants, so its baseline is a controlled baseline rather than the unmodified +historical cap-2 configuration. All existing benchmark gates remain unchanged. + A sparse checkpoint is marked `checkpointEventPolicy: "control-v1"`. Its peer must advertise `recoveryCursorVersion: 1`. During reconnection the controller echoes the runner's durable `resume.ackedSourceSeq` as `recoveryAckedSourceSeq` in the signed @@ -270,8 +310,10 @@ adapters, database loss and destroyed sandboxes are outside this guarantee. The empty-controller-cache test uses a real Rust runner and scripted provider in separate directories, removes the controller directory, then restores solely -from Postgres. It checks unchanged runner/provider process identities and one -provider turn start. Build the host runner and fake provider first, then run: +from Postgres. It waits for natural lease expiry and checks unchanged +runner/provider process identities and one provider turn start. It explicitly +restores the transport checkpoint, so it does not prove application startup +discovery by itself. Build the host runner and fake provider first, then run: ```sh PAPERCLIP_TEST_CONTROLLER_OUTAGE_MS=300000 pnpm exec vitest run \ @@ -284,6 +326,85 @@ This is protocol and persistence evidence. The separate qualifies the actual Daytona/provider path, including completion and workspace finalization. Neither test substitutes for the other's failure boundary. +The explicit `daytona-restart-continuity.runner-codex.daytona.recover-controller` +cell fixes the outage at 300 seconds and adds a second replacement after +completion. Its new turn executes the saved project through normal admission. +The oracle compares hashed native/provider session identities and requires a new +completed tool receipt and correct answer. An installed A/B/A artifact manifest +uses the last replacement for this retained-session continuation/rollback check. +The general everyday suite retains its 125-second default. These intervals are +fault tests, not permitted HTTP deployment downtime. Both fixtures retain the +workspace and remote sandbox; they do not prove whole-webserver disk loss. + +The dedicated suite's version 6 uses the existing reusable warm environment, +with a 15-minute provider idle window and unchanged 60/300/420-second ownership, +outage and reconnect limits. Its prior per-turn/no-reuse measurement failed +retained native/provider identity and remains failed. The test retires the whole +`runtime/paperclip-runner` directory, including provider failover-backup bytes. +Per-turn application qualification after sandbox disposal and this disk loss +therefore awaits the provider-backup owner's durable storage contract; logical +native checkpoints do not replace provider history or weaken the identity oracle. + +The rollback reader must support both the recovery envelope and the settled PRP +v2 idle goal suffix. Record its actual source commit and installed tree hash. +An older reader rejecting that journal is an unsupported rollback, not a pass. +A backported compatibility build is a development fixture until it is released; +do not describe it as a previously published version. Distinct source-controller +tests, compiled native-module tests, installed CLI startup and the full Product +E2E journey provide different evidence and must retain those scope labels. + +## Database cost qualification + +`scripts/benchmark-native-controller-recovery.ts` compares a pinned source +baseline with the candidate's real PRP writer, control-plane snapshot port and +lease renewal. It uses identical fixtures with 300 controllers and 10,000 prior +tasks, in baseline/candidate/candidate/baseline order. `steady` distributes control +checkpoints; `burst` aligns them every five seconds. `pilot` checks instrumentation +with ten controllers. No provider is called. + +Run on an otherwise idle disposable Linux Postgres cluster with +`pg_stat_statements` preloaded and its extension available. Give Postgres and the +Node client two CPUs and 1536 MiB each. Set these explicit inputs: + +```sh +BENCH_DISPOSABLE_POSTGRES=1 \ +BENCH_DATABASE_URL=postgres://postgres:fixture-password@127.0.0.1:5432/postgres \ +BENCH_BASELINE_ROOT=/absolute/path/to/pinned-baseline \ +BENCH_REPORT_ROOT=/absolute/path/to/new-report-directory \ +BENCH_POSTGRES_CGROUP=/absolute/path/to/postgres-cgroup \ +node --import ./cli/node_modules/tsx/dist/loader.mjs \ + scripts/benchmark-native-controller-recovery.ts steady +``` + +The baseline needs the same installed dependency versions and built workspace +packages. It supplies the baseline writer, snapshot port and PRP controller; +the request fixtures, renewal path and pool policy are common controls. When +running an exported tree without Git metadata, supply its exact +`BENCH_BASELINE_REVISION`, `BENCH_CANDIDATE_REVISION` and +`BENCH_CANDIDATE_DIFF`. Record the export source and verify its bytes separately. +Use a fresh report directory for every attempt. Only new random databases are +created; successful trial databases are dropped, and the owned template remains +for inspection until the disposable cluster is removed. + +The report counts SQL across all three pools, actual snapshot/PRP transactions, +changed records/bytes, WAL, database CPU, sampled memory, pool admission, +transaction latency and database request latency. Pool admission ends at +connection assignment/BEGIN encoding; BEGIN response time is separate. +Postgres execution time includes lock waits. A 25 ms observer uses PostgreSQL's +`pg_locks.waitstart` to report duration bounds for observed lock episodes by pool. +The pilot calibrates it against an intentional 250 ms row-lock wait. Short waits +can be missed, and delayed samples widen the estimated upper bound. The nominal +40 observer queries per second contribute to measured database CPU and statement +statistics. Reports include actual sample counts, sampling gaps, CPU pressure, +quota throttling and the measurement harness hash. They do not provide exact +lock-exclusive SQL execution time or HTTP latency; this is component load +qualification. + +Correctness and absolute gates must pass for acceptance. +Dropped operations invalidate paired comparisons. The report retains failures +and refuses an overall pass if p99, drain, throughput, memory, CPU or the 10% +relative regression gate fails. Do not change pool limits to conceal a failure. + ## Required native usage acceptance Built-in native remote Runner execution accepts structured accounting before diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index 071404c8b83..2a2041d2e1a 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -272,6 +272,15 @@ provider restart. For that journey, interrupt a disposable deployment after workspace preparation and before its first executor claim, then confirm one provider turn starts on the original run through the normal admission gates. +The same suite's `diskless` and `diskless-per-turn` cases exercise three native +controller processes with both native projections removed between completed +turns. They check history and invocation counts through the real executor and +admission snapshot rebinder. The workspace and sandbox remain available, and +the processes use the same source revision. Separate Postgres tests cover +metadata import, bounded discovery, ownership holds and incomplete-publication +retention. These cases do not qualify full webserver disk loss, distinct releases, +HTTP traffic continuity, or the live Daytona provider path. + The [Runner transport regressions](../../packages/paperclip-runner/runner/crates/runner-core/src/durable/transport.rs) exercise a real WebSocket welcome delayed beyond two seconds. Authentication uses the ten-second connection setup budget, clipped by the current lifecycle diff --git a/packages/db/src/checkpoint-pool.test.ts b/packages/db/src/checkpoint-pool.test.ts index 05dd02a8208..e75842053b6 100644 --- a/packages/db/src/checkpoint-pool.test.ts +++ b/packages/db/src/checkpoint-pool.test.ts @@ -1,5 +1,5 @@ import { afterEach, expect, it } from "vitest"; -import { checkpointDb, closeCheckpointDb, closeRegisteredClients, createDb, type Db } from "./client.js"; +import { checkpointDb, closeCheckpointDb, closeRegisteredClients, createDb, livenessDb, type Db } from "./client.js"; // postgres.js pools are lazy: these tests exercise actual client lifetimes but // never connect to a database or depend on embedded Postgres availability. @@ -17,6 +17,7 @@ it("fences checkpoints when shutdown precedes the first pool request", async () it("keeps an already-created checkpoint pool ended after shutdown", async () => { const db = createDb(url); const pool = checkpointDb(db); + expect(pool.$client.options.max).toBe(3); await closeCheckpointDb(db, 1); expect(checkpointDb(db)).toBe(pool); await expect(pool.$client`select 1`).rejects.toMatchObject({ code: "CONNECTION_ENDED" }); @@ -30,3 +31,16 @@ it("preserves other owner lifetimes and non-owning test-double behavior", async await closeCheckpointDb(nonOwner, 1); expect(checkpointDb(nonOwner)).toBe(nonOwner); }); + +it("bounds one cached checkpoint pool per owner without changing request or renewal capacity", () => { + const first = createDb(url, { maxConnections: 10 }); + const second = createDb(url, { maxConnections: 10 }); + const pool = checkpointDb(first); + expect(pool).not.toBe(first); + expect(checkpointDb(first)).toBe(pool); + expect(checkpointDb(second)).not.toBe(pool); + expect(pool.$client.options.max).toBe(3); + expect(checkpointDb(second).$client.options.max).toBe(3); + expect(first.$client.options.max).toBe(10); + expect(livenessDb(first).$client.options.max).toBe(2); +}); diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index b43d472d19e..66ff29b3aa6 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -295,9 +295,10 @@ export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise Db>(); const checkpointDbs = new WeakMap(); +const CHECKPOINT_POOL_MAX_CONNECTIONS = 3; const closedCheckpointOwners = new WeakSet(); /** The bounded controller checkpoint pool; transactions and test doubles keep @@ -342,7 +343,7 @@ function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: ob ...resolved, maxConnections: LIVENESS_POOL_MAX_CONNECTIONS, applicationName: "paperclip-controller-lease", }, db)); checkpointDbFactories.set(db, () => createDbClient(url, { - ...resolved, maxConnections: 2, applicationName: "paperclip-runner-checkpoint", + ...resolved, maxConnections: CHECKPOINT_POOL_MAX_CONNECTIONS, applicationName: "paperclip-runner-checkpoint", }, db)); return db; } diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs index 88f6c18c27e..cf710278233 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs @@ -769,8 +769,8 @@ fn run() -> Result<(), Box> { .map(|value| value.parse::()) .transpose()? .unwrap_or(64); - if !(1..=4096).contains(&held_output_count) { - return Err("--held-output-count must be between 1 and 4096".into()); + if !(1..=8192).contains(&held_output_count) { + return Err("--held-output-count must be between 1 and 8192".into()); } let exit_after_turn_start = args.iter().any(|value| value == "--exit-after-turn-start"); let exit_after_turn_completion = args diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts index b9f982401d2..c4f29bdbb6a 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts @@ -32,6 +32,8 @@ import { DurablePrpControlPlane, decodeDurablePrpCheckpoint, inspectDurablePrpCheckpointHeader, + inspectStagedRunAttachment, + inspectStagedRunAttachmentHeader, type DurablePrpCheckpoint, durableRecoveryInternals, SemanticToolNotDispatchedError, @@ -53,6 +55,66 @@ const identity: DurableRecoveryIdentity = { const expectedRunnerVersion = "0.3.0"; const expectedRunnerDigest = `sha256:${"a".repeat(64)}`; +it("persists an undispatched source-to-target attachment before bootstrap and preserves it after cache loss", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-staged-attachment-")); + const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" }; + const template = { provider: { provider: "codex", command: "fixture", instructions: "assigned instructions ".repeat(1_500) }, authorizedTools: {} }; + const core = new DurablePrpControlPlane({ stateDirectory: resolve(root, "first"), identity, expectedRunnerVersion, expectedRunnerDigest }); + let checkpoint!: DurablePrpCheckpoint, templateWrites = 0; + core.setCheckpointPersistence(async value => { checkpoint = value; if (value.changedRecords.has("staged-attachment-template")) templateWrites++; }); + let restored: DurablePrpControlPlane | undefined; + let client: AuthenticatedClient | null = null; + try { + core.stageRunAttachment(target, template); + await core.flushCheckpoint(); + expect(checkpoint.runId).toBe(target.runId); + expect(checkpoint.predecessor).toEqual({ runId: identity.runId, normalizedSessionId: identity.normalizedSessionId }); + expect(checkpoint.workCommandIds).toEqual([]); + const proof = inspectStagedRunAttachmentHeader(checkpoint.records.get("header")!); + expect(proof).toMatchObject({ attachment: { sourceIdentity: identity, targetIdentity: target, commandId: null } }); + expect(proof?.attachment).not.toHaveProperty("runAttachTemplate"); + expect(checkpoint.records.get("header")!.length).toBeLessThan(JSON.stringify(template).length / 2); + expect(JSON.parse(checkpoint.records.get("staged-attachment-template")!)).toEqual(template); + expect(proof?.runnerVersion).toBe(expectedRunnerVersion); + expect(proof?.runnerDigest).toBe(expectedRunnerDigest); + checkpoint.restoreSessionDirectory(resolve(root, "replacement")); + restored = new DurablePrpControlPlane({ stateDirectory: resolve(root, "replacement", "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest }); + expect(inspectStagedRunAttachment(restored.store.state, target)).toMatchObject({ ...proof?.attachment, runAttachTemplate: template }); + expect(restored.store.state.commands).toEqual([]); + expect(() => restored!.stageRunAttachment({ ...target, itemId: "foreign" }, template)).toThrow("conflicts"); + expect(() => restored!.queueCommand("turn.start", {})).toThrow("maintenance only"); + expect(() => restored!.queueCommand("run.attach", { ...template, paperclipNextAuthority: { identity: { ...target, itemId: "foreign" } } })).toThrow("conflicts"); + const corruptedRecords = new Map(checkpoint.records); + corruptedRecords.set("staged-attachment-template", JSON.stringify({ provider: { command: "changed" } })); + expect(() => decodeDurablePrpCheckpoint(corruptedRecords)).toThrow("native_runner_checkpoint_invalid"); + expect(inspectStagedRunAttachment(restored.store.state, { ...target, turnId: "foreign" })).toBeNull(); + for (const field of ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId"] as const) { + const changed = JSON.parse(checkpoint.records.get("header")!); + changed.state.stagedRunAttachment.targetIdentity[field] = "foreign"; + expect(inspectStagedRunAttachmentHeader(JSON.stringify(changed))).toBeNull(); + } + const downgraded = JSON.parse(checkpoint.records.get("header")!); + downgraded.state.schema = "paperclip.runner.durable.control-plane-state.v1"; + expect(inspectStagedRunAttachmentHeader(JSON.stringify(downgraded))).toBeNull(); + const beforeRetry = checkpoint; + core.stageRunAttachment(target, template); await core.flushCheckpoint(); expect(checkpoint).toBe(beforeRetry); + core.queueCommand("session.snapshot", {}); await core.flushCheckpoint(); + expect(checkpoint.changedRecords.has("staged-attachment-template")).toBe(false); + await core.start(); client = await authenticate(core, core.issueBootstrapTicket()); + expect(client).not.toBeNull(); + core.queueCommand("run.attach", { ...template, paperclipNextAuthority: { identity: target, + connection: { mode: "connect", connectUrl: core.connectUrl } } }, "staged-next-attach"); + await core.flushCheckpoint(); + expect(templateWrites).toBe(1); + expect(checkpoint.records.has("staged-attachment-template")).toBe(false); + expect(checkpoint.removedRecords).toContain("staged-attachment-template"); + expect(() => decodeDurablePrpCheckpoint(checkpoint.records)).not.toThrow(); + } finally { + client?.socket.destroy(); + await restored?.stop(); await core.stop(); rmSync(root, { recursive: true, force: true }); + } +}); + it("continues externally persisted commands after losing the local controller directory", async () => { const root = mkdtempSync(resolve(tmpdir(), "runner-external-store-")); const options = { stateDirectory: resolve(root, "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest }; @@ -106,6 +168,7 @@ it("restores authentication and commands from opaque checkpoint records", async try { const ticket = first.issueBootstrapTicket(); first.queueCommand("session.snapshot", {}, "maintenance"); + first.queueCommand("turn.stop", {}, "retirement-maintenance"); first.queueCommand("turn.start", {}, "execution"); first.persistPreparedLaunchConfiguration(identity, "retained-configuration"); await first.flushCheckpoint(); diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index eb67e6d9849..ea82cd674e0 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -54,6 +54,7 @@ const protocolVersion = RUNNER_PRP_PROTOCOL_MAX; const secureFrameSchema = "paperclip.runner.secure-frame.v1"; const websocketGuid = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; const coreStateSchema = "paperclip.runner.durable.control-plane-state.v1"; +const stagedCoreStateSchema = "paperclip.runner.durable.control-plane-state.staged-attachment.v1"; const externalCoreStateSchema = "paperclip.runner.external-control-plane-state.v1"; const transitionCoreStateSchema = "paperclip.runner.durable.control-plane-state.warm-transition.v1"; @@ -140,8 +141,17 @@ interface ConnectionLeaseRecord { revokedAt: string | null; } +export interface StagedRunAttachment { + readonly sourceIdentity: DurableRecoveryIdentity; + readonly targetIdentity: DurableRecoveryIdentity; + readonly runAttachTemplate: Record; + readonly templateDigest: string; + readonly commandId: string | null; +} + interface StoredCoreState { - schema: typeof coreStateSchema | typeof transitionCoreStateSchema; + schema: typeof coreStateSchema | typeof transitionCoreStateSchema | typeof stagedCoreStateSchema; + stagedRunAttachment?: Omit & { runAttachTemplate?: Record }; identity: DurableRecoveryIdentity; /** External checkpoints contain control state, not a duplicate output log. */ checkpointEventPolicy?: "control-v1"; @@ -311,10 +321,12 @@ type CheckpointHeader = { schema: "paperclip.runner.controller-checkpoint.v2"; runnerVersion: string; runnerDigest: string; - state: Omit; + state: Omit + & { stagedRunAttachment?: Omit }; }; function checkpointRunId(state: StoredCoreState): string { + if (state.stagedRunAttachment) return state.stagedRunAttachment.targetIdentity.runId; if (state.warmTransition) return state.warmTransition.receipt.newIdentity.runId; const attachment = state.commands.findLast(command => command.type === "run.attach"); const next = attachment?.payload.paperclipNextAuthority as { identity?: { runId?: string } } | undefined; @@ -347,17 +359,19 @@ function checkpointChanges(records: ReadonlyMap, runId: string, function captureCheckpoint(state: StoredCoreState, runnerVersion: string, runnerDigest: string): DurablePrpCheckpoint { const runId = checkpointRunId(state); const records = new Map(); - const { commands, committedEvents, ackedSourceSeq: _cursor, replayDeliveries: _replays, ...metadata } = state; + const { commands, committedEvents, ackedSourceSeq: _cursor, replayDeliveries: _replays, stagedRunAttachment, ...metadata } = state; + const stagedDescriptor = stagedRunAttachment ? (({ runAttachTemplate: _template, ...descriptor }) => descriptor)(stagedRunAttachment) : undefined; + if (stagedRunAttachment?.commandId === null) records.set("staged-attachment-template", JSON.stringify(stagedRunAttachment.runAttachTemplate)); const latestUsage = committedEvents.findLast(event => event.eventType === "usage.reported"); const recoveryEvents = committedEvents.filter(event => !isStreamingEvent(event) || event === latestUsage); for (const command of commands) records.set(`command:${command.controllerSeq}`, JSON.stringify(command)); for (const event of recoveryEvents) records.set(`event:${event.sourceSeq}`, JSON.stringify(event)); const header: CheckpointHeader = { schema: "paperclip.runner.controller-checkpoint.v2", - runnerVersion, runnerDigest, state: { ...metadata, checkpointEventPolicy: "control-v1" } }; + runnerVersion, runnerDigest, state: { ...metadata, ...(stagedDescriptor ? { stagedRunAttachment: stagedDescriptor } : {}), checkpointEventPolicy: "control-v1" } }; records.set("header", JSON.stringify(header)); records.set("cursor", JSON.stringify({ ackedSourceSeq: state.ackedSourceSeq, replayDeliveries: state.replayDeliveries, commandCount: commands.length, eventCount: recoveryEvents.length })); - const maintenance = new Set(["runner.drain", "runner.suspend", "session.snapshot", "session.close"]); + const maintenance = new Set(["turn.stop", "runner.drain", "runner.suspend", "session.snapshot", "session.close"]); return { runnerVersion, runnerDigest, runId, records, ...checkpointChanges(records, runId), recoveryEvidence: { runnerInstanceId: state.identity.runnerInstanceId, normalizedSessionId: state.identity.normalizedSessionId, @@ -397,11 +411,43 @@ export function inspectDurablePrpCheckpointHeader(contents: string): DurableReco const header = parseCheckpointHeader(contents); if (header.schema !== "paperclip.runner.controller-checkpoint.v2" || header.state.checkpointEventPolicy !== "control-v1" - || ![coreStateSchema, transitionCoreStateSchema].includes(String(header.state.schema))) return null; + || ![coreStateSchema, transitionCoreStateSchema, stagedCoreStateSchema].includes(String(header.state.schema)) + || (header.state.schema === stagedCoreStateSchema && !validStagedRunAttachment(header.state as unknown as StoredCoreState, true))) return null; return { ...header.state.identity }; } catch { return null; } } +/** Bounded checkpoint metadata inspection; never itself grants recovery authority. */ +export function inspectStagedRunAttachmentHeader(contents: string): { + attachment: Omit; runnerVersion: string; runnerDigest: string; + protocolIdentity: DurableRecoveryIdentity; + phase: "source_bootstrap" | "attachment_queued" | "awaiting_result" | "prepared" | "activated" | "completed"; + receipt: DurableWarmRunTransition | null; +} | null { + try { + if (Buffer.byteLength(contents) > maxStateBytes) return null; + const header = parseCheckpointHeader(contents), state = header.state as unknown as StoredCoreState; + if (header.schema !== "paperclip.runner.controller-checkpoint.v2" + || state.checkpointEventPolicy !== "control-v1" || !validStagedRunAttachment(state, true) + || !state.stagedRunAttachment) return null; + const transition = state.warmTransition; + const completed = stagedTargetActivated(state) ? state.completedWarmTransition : undefined; + const bound = transition ?? completed; + if (bound) { + const receipt = bound.receipt; + const result = transition?.phase === "awaiting_result" ? transition.expectedResult : bound.command.result; + const expected = warmTransitionReceipt(receipt.oldIdentity, bound.command, result!, receipt.oldAckedSourceSeq, + { leaseId: receipt.leaseId, expiresAtUnixMs: receipt.leaseExpiresAtUnixMs, revocationEpoch: receipt.leaseRevocationEpoch }, + header.runnerVersion, header.runnerDigest); + if (canonicalJson(expected) !== canonicalJson(receipt)) return null; + } + return { attachment: structuredClone(state.stagedRunAttachment), protocolIdentity: structuredClone(state.identity), + phase: transition?.phase ?? (completed ? "completed" : state.stagedRunAttachment.commandId ? "attachment_queued" : "source_bootstrap"), + receipt: bound ? structuredClone(bound.receipt) : null, + runnerVersion: header.runnerVersion, runnerDigest: header.runnerDigest }; + } catch { return null; } +} + function checkpointState(records: ReadonlyMap) { const header = parseCheckpointHeader(records.get("header") ?? "null"); const read = (key: string): unknown => { @@ -423,9 +469,14 @@ function checkpointState(records: ReadonlyMap) { } const cursor = v2 ? read("cursor") : header.state; if (!isRecord(cursor) || (v2 && (cursor.commandCount !== commandKeys.length || cursor.eventCount !== eventKeys.length - || records.size !== commandKeys.length + eventKeys.length + 2))) throw new Error("native_runner_checkpoint_incomplete"); + || records.size !== commandKeys.length + eventKeys.length + 2 + (isRecord(header.state.stagedRunAttachment) && header.state.stagedRunAttachment.commandId === null ? 1 : 0)))) throw new Error("native_runner_checkpoint_incomplete"); const state = { ...header.state, ackedSourceSeq: cursor.ackedSourceSeq, replayDeliveries: cursor.replayDeliveries, - commands: commandKeys.map(read), committedEvents: eventKeys.map(read) }; + commands: commandKeys.map(read), committedEvents: eventKeys.map(read) } as unknown as StoredCoreState; + if (state.stagedRunAttachment?.commandId === null) { + const template = read("staged-attachment-template"); + if (!isRecord(template)) throw new Error("native_runner_checkpoint_invalid"); + state.stagedRunAttachment = { ...state.stagedRunAttachment, runAttachTemplate: template }; + } if (!isStoredCoreState(state, header.state.identity)) throw new Error("native_runner_checkpoint_invalid"); if (state.commands.some((command, index) => commandKeys[index] !== `command:${command.controllerSeq}`) || state.committedEvents.some((event, index) => eventKeys[index] !== `event:${event.sourceSeq}`)) { @@ -683,9 +734,9 @@ function warmTransitionReceipt( function validStoredWarmTransition(state: StoredCoreState): boolean { const transition = state.warmTransition; - if (!transition) return state.schema === coreStateSchema; + if (!transition) return state.schema === coreStateSchema || (state.schema === stagedCoreStateSchema && validStagedRunAttachment(state)); if ( - state.schema !== transitionCoreStateSchema || + (state.schema !== transitionCoreStateSchema && state.schema !== stagedCoreStateSchema) || !["awaiting_result", "prepared", "activated"].includes(transition.phase) || !isRecord(transition.receipt) || !isRecord(transition.command) || @@ -802,6 +853,76 @@ function unsettledSemanticInput( } } +function stagedCommandMatches(staged: Pick, command: DurableRecoveryCoreCommand): boolean { + if (command.type !== "run.attach" || command.commandId !== staged.commandId) return false; + const { paperclipNextAuthority, ...template } = command.payload; + return isRecord(paperclipNextAuthority) + && canonicalJson(paperclipNextAuthority.identity) === canonicalJson(staged.targetIdentity) + && `sha256:${canonicalDigest(template)}` === staged.templateDigest; +} + +function stagedAttachmentTemplate(state: StoredCoreState): Record | null { + const staged = state.stagedRunAttachment; + if (!staged) return null; + if (staged.commandId === null) return staged.runAttachTemplate ?? null; + const command = state.commands?.find(command => command.commandId === staged.commandId) + ?? state.warmTransition?.command ?? state.completedWarmTransition?.command; + if (!command || !stagedCommandMatches(staged, command)) return null; + const { paperclipNextAuthority: _authority, ...template } = command.payload; + return template; +} + +function stagedTargetActivated(state: StoredCoreState): boolean { + const staged = state.stagedRunAttachment, completed = state.completedWarmTransition; + return !!staged && !state.warmTransition && !!completed + && canonicalJson(state.identity) === canonicalJson(staged.targetIdentity) + && completed.receipt.commandId === staged.commandId + && canonicalJson(completed.receipt.oldIdentity) === canonicalJson(staged.sourceIdentity) + && canonicalJson(completed.receipt.newIdentity) === canonicalJson(staged.targetIdentity) + && stagedCommandMatches(staged, completed.command); +} + +function validStagedRunAttachment(state: StoredCoreState, headerOnly = false): boolean { + const staged = state.stagedRunAttachment; + if (staged === undefined) return state.schema !== stagedCoreStateSchema; + if (!(state.schema === stagedCoreStateSchema && isRecord(staged) + && Object.keys(staged).sort().join(",") === (headerOnly || staged.commandId !== null ? "commandId,sourceIdentity,targetIdentity,templateDigest" : "commandId,runAttachTemplate,sourceIdentity,targetIdentity,templateDigest") + && exactIdentity(staged.sourceIdentity) && exactIdentity(staged.targetIdentity) + && staged.sourceIdentity.runId !== staged.targetIdentity.runId + && ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId"].every(key => + staged.sourceIdentity[key as keyof DurableRecoveryIdentity] === staged.targetIdentity[key as keyof DurableRecoveryIdentity]) + && /^sha256:[0-9a-f]{64}$/.test(staged.templateDigest) + && (headerOnly || staged.commandId !== null || (isRecord(staged.runAttachTemplate) && isRecord(staged.runAttachTemplate.provider) + && staged.runAttachTemplate.paperclipNextAuthority === undefined + && `sha256:${canonicalDigest(staged.runAttachTemplate)}` === staged.templateDigest + && Buffer.byteLength(JSON.stringify(staged.runAttachTemplate)) <= maxCommandBytes)) + && (staged.commandId === null || (typeof staged.commandId === "string" && stableIdPattern.test(staged.commandId))))) return false; + const sourceCurrent = canonicalJson(state.identity) === canonicalJson(staged.sourceIdentity); + if (!sourceCurrent && canonicalJson(state.identity) !== canonicalJson(staged.targetIdentity)) return false; + if (state.warmTransition) { + const transition = state.warmTransition; + return staged.commandId !== null && stagedCommandMatches(staged, transition.command) + && canonicalJson(transition.receipt.oldIdentity) === canonicalJson(staged.sourceIdentity) + && canonicalJson(transition.receipt.newIdentity) === canonicalJson(staged.targetIdentity) + && (sourceCurrent ? transition.phase !== "activated" : transition.phase === "activated"); + } + if (!sourceCurrent) return stagedTargetActivated(state); + if (staged.commandId === null) return true; + // Header-only inspection cannot certify the separate command record. Full + // restoration below requires its exact tuple, payload and command ID. + return !Array.isArray(state.commands) || state.commands.some(command => stagedCommandMatches(staged, command)); +} + +/** Inspection is evidence only; the caller still owns recovery authorization. */ +export function inspectStagedRunAttachment(state: unknown, expectedTarget?: DurableRecoveryIdentity): StagedRunAttachment | null { + try { + if (!isRecord(state) || !exactIdentity(state.identity) || !isStoredCoreState(state, state.identity) + || !state.stagedRunAttachment || (expectedTarget && canonicalJson(state.stagedRunAttachment.targetIdentity) !== canonicalJson(expectedTarget))) return null; + const template = stagedAttachmentTemplate(state); + return template ? { ...structuredClone(state.stagedRunAttachment), runAttachTemplate: structuredClone(template) } : null; + } catch { return null; } +} + function isStoredCoreState( value: unknown, identity: DurableRecoveryIdentity, @@ -811,7 +932,7 @@ function isStoredCoreState( const events = value.committedEvents; if ( (value.schema !== coreStateSchema && - value.schema !== transitionCoreStateSchema) || + value.schema !== transitionCoreStateSchema && value.schema !== stagedCoreStateSchema) || canonicalJson(value.identity) !== canonicalJson(identity) || (value.checkpointEventPolicy !== undefined && value.checkpointEventPolicy !== "control-v1") || !isRecord(value.tickets) || @@ -828,6 +949,7 @@ function isStoredCoreState( ) { return false; } + if (!validStagedRunAttachment(value as unknown as StoredCoreState)) return false; if (!validStoredWarmTransition(value as unknown as StoredCoreState)) return false; if (value.completedWarmTransition !== undefined) { @@ -1758,6 +1880,7 @@ class AuthorityConnection { /** Authenticated, replay-safe PRP transport authority. Business operations are caller supplied. */ export class DurablePrpControlPlane { #identity: DurableRecoveryIdentity; + #stagedTargetCheckpointPublished: string | null = null; readonly #store: DurableCoreStore; #expectedRunnerVersion: string; #expectedRunnerDigest: string; @@ -1854,7 +1977,12 @@ export class DurablePrpControlPlane { cached.set(part, prior?.record === record ? prior : { record, command: JSON.parse(record) }); } this.#publicationCommands = cached; - return { ...header.state, ...cursor, commands: [...cached.values()].map(value => value.command), + const descriptor = header.state.stagedRunAttachment; + const workingStage = this.#store.state.stagedRunAttachment; + const staged = descriptor?.commandId === null ? { ...descriptor, + runAttachTemplate: workingStage?.templateDigest === descriptor.templateDigest && workingStage.runAttachTemplate + ? workingStage.runAttachTemplate : JSON.parse(snapshot.records.get("staged-attachment-template")!) as Record } : descriptor; + return { ...header.state, ...(staged ? { stagedRunAttachment: staged } : {}), ...cursor, commands: [...cached.values()].map(value => value.command), // Event records are immutable after insertion. Retain their array, not // another deep copy of the streaming payload window. committedEvents: this.#store.state.committedEvents }; @@ -2323,6 +2451,23 @@ export class DurablePrpControlPlane { return ticket; } + /** Import exact durable handoff evidence for a caller-verified surviving participant; issues no credential. */ + prepareWarmTransitionAdoption(input: { transitionId: string; runnerState: Record }): void { + this.#store.assertWritable(); + const pending = input.runnerState.warmTransition; + const proof = warmTransitionRecoveryProof({ controlPlaneState: this.#store.state, runnerState: input.runnerState, + expectedNewIdentity: (isRecord(pending) && isRecord(pending.receipt) ? pending.receipt.newIdentity : null) as DurableRecoveryIdentity, + expectedRunnerVersion: this.#expectedRunnerVersion, expectedRunnerDigest: this.#expectedRunnerDigest }); + if (!proof || proof.transition.receipt.transitionId !== input.transitionId || !this.#store.state.stagedRunAttachment) { + throw new Error("Warm transition adoption snapshot proof is not authorized."); + } + if (this.#store.state.warmTransition && canonicalJson(this.#store.state.warmTransition) === canonicalJson(proof.transition)) return; + const candidate = structuredClone(this.#store.state); + candidate.schema = stagedCoreStateSchema; + candidate.warmTransition = structuredClone(proof.transition); + this.#store.commit(candidate); + } + /** Caller-owned recovery admission is required; a receipt is not a credential. */ issueWarmTransitionBootstrapTicket( input: { @@ -2361,7 +2506,7 @@ export class DurablePrpControlPlane { original.expiresAtUnixMs, ); const candidate = structuredClone(this.#store.state); - candidate.schema = transitionCoreStateSchema; + candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : transitionCoreStateSchema; candidate.warmTransition = structuredClone(transition); candidate.tickets[material.credentialId] = { recordId: `bootstrap_ticket_${randomUUID()}`, @@ -2380,6 +2525,36 @@ export class DurablePrpControlPlane { return ticket; } + /** Caller confirms its existing session-checkpoint transaction committed. No journal write. */ + markStagedTargetCheckpointPublished(identity: DurableRecoveryIdentity): boolean { + if (!stagedTargetActivated(this.#store.state) || canonicalJson(identity) !== canonicalJson(this.#store.state.identity)) return false; + this.#stagedTargetCheckpointPublished = canonicalJson(identity); + return true; + } + + /** Durable intent precedes source restoration; it is never a dispatched command. */ + stageRunAttachment(targetIdentity: DurableRecoveryIdentity, runAttachTemplate: Record): void { + this.#store.assertWritable(); + const existing = this.#store.state.stagedRunAttachment; + if (existing && !(stagedTargetActivated(this.#store.state) + && this.#stagedTargetCheckpointPublished === canonicalJson(this.#store.state.identity) + && targetIdentity.runId !== existing.targetIdentity.runId)) { + if (!validStagedRunAttachment(this.#store.state) + || canonicalJson(existing.targetIdentity) !== canonicalJson(targetIdentity) + || canonicalJson(stagedAttachmentTemplate(this.#store.state)) !== canonicalJson(runAttachTemplate)) { + throw new Error("Staged run attachment conflicts with its durable intent."); + } + return; + } + const candidate = structuredClone(this.#store.state); + candidate.schema = stagedCoreStateSchema; + candidate.stagedRunAttachment = { sourceIdentity: structuredClone(candidate.identity), + targetIdentity: structuredClone(targetIdentity), runAttachTemplate: structuredClone(runAttachTemplate), + templateDigest: `sha256:${canonicalDigest(runAttachTemplate)}`, commandId: null }; + if (!validStagedRunAttachment(candidate)) throw new Error("Staged run attachment binding is invalid."); + this.#store.commit(candidate); + } + queueCommand( type: string, payload: Record = {}, @@ -2387,6 +2562,21 @@ export class DurablePrpControlPlane { deliverImmediately = false, ): DurableRecoveryCoreCommand { this.#store.assertWritable(); + const staged = this.#store.state.stagedRunAttachment; + const retireStaged = stagedTargetActivated(this.#store.state) && (type === "turn.start" + || (this.#stagedTargetCheckpointPublished === canonicalJson(this.#store.state.identity) + && !["runner.drain", "session.snapshot", "turn.stop", "runner.suspend", "session.close"].includes(type))); + if (staged && !retireStaged) { + if (type === "run.attach") { + const { paperclipNextAuthority, ...template } = payload; + if (!isRecord(paperclipNextAuthority) + || canonicalJson(paperclipNextAuthority.identity) !== canonicalJson(staged.targetIdentity) + || `sha256:${canonicalDigest(template)}` !== staged.templateDigest + || (staged.commandId !== null && staged.commandId !== commandId)) throw new Error("Staged run attachment command conflicts."); + } else if (!stagedTargetActivated(this.#store.state) && !["runner.drain", "session.open", "session.snapshot", "turn.stop", "runner.suspend", "session.close", "semantic_tool.result"].includes(type)) { + throw new Error("Staged run attachment permits recovery maintenance only."); + } + } const transition = this.#store.state.warmTransition; if (transition && transition.phase !== "activated") { if ( @@ -2461,6 +2651,16 @@ export class DurablePrpControlPlane { if (Buffer.byteLength(JSON.stringify(command)) > maxCommandBytes) throw new CommandJournalLimitError("command_payload_too_large"); this.#store.state.commands.push(command); + if (staged && !retireStaged && type === "run.attach") { + const { runAttachTemplate: _template, ...descriptor } = staged; + this.#store.state.stagedRunAttachment = { ...descriptor, commandId: command.commandId }; + } else if (retireStaged) { + // Native runtime persists the recovered session before its first turn. + // Read/goal controls during opening must not retire startup evidence. + delete this.#store.state.stagedRunAttachment; + this.#store.state.schema = coreStateSchema; + this.#stagedTargetCheckpointPublished = null; + } this.#store.save(); void this.flushCheckpoint().catch(() => undefined); if (deliverImmediately) { @@ -2677,7 +2877,7 @@ export class DurablePrpControlPlane { } if (transition) { const candidate = structuredClone(this.#store.state); - candidate.schema = coreStateSchema; + candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : coreStateSchema; candidate.leases[transition.credentialId]!.identity = structuredClone( receipt.newIdentity, ); @@ -3205,8 +3405,9 @@ export class DurablePrpControlPlane { } if (transition.phase === "prepared") { const candidate = initialCoreState(transition.receipt.newIdentity); + candidate.stagedRunAttachment = structuredClone(this.#store.state.stagedRunAttachment); candidate.preparedLaunchConfigurations = structuredClone(this.#store.state.preparedLaunchConfigurations); - candidate.schema = transitionCoreStateSchema; + candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : transitionCoreStateSchema; candidate.warmTransition = { ...structuredClone(transition), phase: "activated", @@ -3507,7 +3708,7 @@ export class DurablePrpControlPlane { )!; completed.status = "completed"; completed.result = structuredClone(result); - candidate.schema = transitionCoreStateSchema; + candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : transitionCoreStateSchema; candidate.warmTransition = { receipt, phase: "prepared", diff --git a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts index f94fa464082..526d628d9f7 100644 --- a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts +++ b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts @@ -36,6 +36,10 @@ export interface RunnerExecutionDriver { checkpoint?: (settlement: "settled" | "unsettled") => Promise | void; release: (reason?: "detach") => Promise | void; }>; + /** Register source and target checkpoint ownership before staging changes the stored run. */ + onStagedRunAttachment?: (authority: DurablePrpControlPlane, proof: { + sourceIdentity: DurableRecoveryIdentity; targetIdentity: DurableRecoveryIdentity; templateDigest: string; + }) => Promise; /** Override process ownership; omission uses the SDK's local process owner. */ runnerProcessLauncher?: ( spec: RunnerProcessLaunchSpec, diff --git a/packages/paperclip-runner/src/index.ts b/packages/paperclip-runner/src/index.ts index 264fb7ed4ed..9f628c39f0a 100644 --- a/packages/paperclip-runner/src/index.ts +++ b/packages/paperclip-runner/src/index.ts @@ -38,6 +38,9 @@ export { RUNNER_PRP_PROTOCOL_MIN, RUNNER_PRP_PROTOCOL_MAX, inspectWarmRunTransition, + inspectStagedRunAttachment, + inspectStagedRunAttachmentHeader, + type StagedRunAttachment, type DurablePrpControlPlaneOptions, type DurablePrpCheckpoint, decodeDurablePrpCheckpoint, diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index 68fc16f53f7..b1749f4653a 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -7646,6 +7646,122 @@ it("rotates PRP authority in place for a warm cross-run attachment", async () => } }, 30_000); +it("recovers an active second warm business run with its opaque attachment identity", async () => { + const root = await mkdtemp(join(tmpdir(), "runnerd-warm-second-active-")); + const firstRoot = join(root, "first-controller"), nextRoot = join(root, "next-controller"); + const server = createServer(); + let authority: DurablePrpControlPlane | null = null, generation = 0; + let checkpoint: durableControlPlane.DurablePrpCheckpoint | undefined; + const registered = new WeakSet(); + server.on("upgrade", (request, socket, head) => authority + ? authority.handleUpgrade(request, socket, "/runner", head) : socket.destroy()); + await new Promise(done => server.listen(0, "127.0.0.1", done)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Missing owned Runner route"); + const registration = async (core: DurablePrpControlPlane) => { + if (!registered.has(core)) { + core.setCheckpointPersistence(async value => { checkpoint = value; }); + registered.add(core); + } + authority = core; const current = ++generation; + return { connectUrl: `ws://127.0.0.1:${address.port}/runner`, release: async () => { if (generation === current) authority = null; } }; + }; + const initialIdentity = { runnerInstanceId: "runner-warm-second-active", environmentLeaseId: "lease-warm-second-active", + runId: "run-warm-first-healthy", normalizedSessionId: "session-warm-second-active", turnId: "turn-first", itemId: "item-first" }; + const options = { runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex, + codexArgs: fakeCodexArgs(root, "--emit-tool-call", "--durable-tool-ids"), stateDirectory: firstRoot, + prpIdentity: initialIdentity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 }, controlPlaneRegistration: registration }; + const first = createCapabilityRunnerdCodexTransport(options); + const within = async (label: string, promise: Promise) => { + let timer: ReturnType | undefined; + try { return await Promise.race([promise, new Promise((_, reject) => { timer = setTimeout(() => reject(new Error(`warm active recovery stage timed out: ${label}`)), 5_000); })]); } + finally { clearTimeout(timer); } + }; + const waitComplete = async (transport: typeof first.transport) => { + for await (const notification of transport.notifications()) if (notification.method === "turn/completed") return; + }; + let calls = 0, runnerPid: number | null = null, providerPid: number | null = null; + let recovered: ReturnType | undefined; + const result = { ok: true, task: { id: "task-1" } }; + first.transport.setServerRequestHandler(async () => ++calls === 1 + ? { success: true, contentItems: [{ type: "inputText", text: JSON.stringify(result) }] } + : new Promise>(() => undefined)); + try { + const opened = await within("first thread/start", first.transport.request("thread/start", { cwd: tmpdir(), dynamicTools: [{ name: "get_task_context", description: "Read current task", + inputSchema: { type: "object", properties: {}, additionalProperties: false } }] })); + runnerPid = first.evidence().runnerPid; + providerPid = first.evidence().codexPid; + expect(providerPid).toEqual(expect.any(Number)); + const providerStart = execFileSync("ps", ["-p", String(providerPid), "-o", "lstart="], { encoding: "utf8" }).trim(); + const startedAt = first.transport.processInfo().startedAt; + const firstCompleted = waitComplete(first.transport); + await within("first turn/start", first.transport.request("turn/start", { input: [{ type: "text", text: "first healthy business run" }] })); + await within("healthy first turn completion", firstCompleted); + const attachment = { runId: "run-warm-second-active", turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` }; + await within("warm second business attachment", first.transport.attachRun!(attachment)); + const active = await within("second active turn/start", first.transport.request("turn/start", { input: [{ type: "text", text: "second business run held at tool" }] })); + await vi.waitFor(() => expect(calls).toBe(2), { timeout: 5_000 }); + const activeCore = authority!; + const savedIdentity = { ...activeCore.store.state.identity }; + expect(savedIdentity).toEqual({ ...initialIdentity, ...attachment }); + await activeCore.flushCheckpoint(); + await within("controller detach", first.detachControllerForRestart()); + expect(() => process.kill(runnerPid!, 0)).not.toThrow(); + checkpoint!.restoreSessionDirectory(nextRoot); + const noSpawn = vi.fn(() => { throw new Error("Duplicate Runner launch"); }); + const openedThread = opened.thread as Record; + const activeTurn = active.turn as Record; + expect(openedThread.id).toEqual(expect.any(String)); + expect(activeTurn.id).toEqual(expect.any(String)); + const shared = { ...options, stateDirectory: nextRoot, + resumeProviderSession: { driverSessionId: String(openedThread.id), providerSessionId: String(openedThread.sessionId) }, + resumeActiveTurnId: String(activeTurn.id), runnerArtifact: readRunnerdArtifactBinding(options.runnerBinary), + runnerStateDirectory: join(firstRoot, "runner"), readRunnerState: async () => JSON.parse(await readFile(join(firstRoot, "runner/runner-state.json"), "utf8")), + prepareExternalRunnerState: async () => {}, archiveExternalRunnerState: async () => { throw new Error("Active authority must not rotate"); }, + runnerProcessLauncher: noSpawn, adoptExistingRunner: { pid: runnerPid!, processGroupId: runnerPid, startedAt, + isAlive: () => { try { process.kill(runnerPid!, 0); return true; } catch { return false; } }, + signal: (signal: NodeJS.Signals) => { try { process.kill(runnerPid!, signal); return true; } catch { return false; } } } }; + // The old server's synthetic pair must continue failing the strict Runner guard. + const wrong = createCapabilityRunnerdCodexTransport({ ...shared, prpIdentity: { ...savedIdentity, + turnId: `turn-${attachment.runId}`, itemId: `item-${attachment.runId}` } }); + await expect(wrong.transport.request("thread/read", {})).rejects.toThrow("PRP recovery identity does not match the durable session binding"); + await wrong.detachControllerForRestart(); + // Leave the pending tool outcome unknown. Reattachment must authenticate + // and preserve it without fabricating a receipt or redispatching its handler. + recovered = createCapabilityRunnerdCodexTransport({ ...shared, prpIdentity: savedIdentity }); + await within("replacement thread/read", recovered.transport.request("thread/read", {})); + expect(recovered.evidence()).toMatchObject({ runnerPid, runnerExited: false }); + expect(authority!.activeRunnerConnectionCount()).toBe(1); + expect(() => process.kill(providerPid!, 0)).not.toThrow(); + expect(execFileSync("ps", ["-p", String(providerPid), "-o", "lstart="], { encoding: "utf8" }).trim()).toBe(providerStart); + expect(calls).toBe(2); + expect(authority!.store.state.commands.filter(command => command.type === "semantic_tool.result")).toHaveLength(0); + expect(recovered.transport.processInfo().startedAt).toBe(startedAt); + expect(authority!.store.state.identity).toEqual(savedIdentity); + expect(authority!.store.state.commands.filter(command => command.type === "turn.start")).toHaveLength(1); + expect(noSpawn).not.toHaveBeenCalled(); + } catch (error) { + const saved = JSON.parse(await readFile(join(firstRoot, "runner/runner-state.json"), "utf8").catch(() => "{}")); + const stderr = await readFile(join(firstRoot, "diagnostics/runnerd.stderr.log"), "utf8").catch(() => ""); + throw new Error(`${String(error)}; owned protocol evidence ${JSON.stringify({ + transport: recovered?.evidence(), connections: authority?.activeRunnerConnectionCount(), + identity: authority?.store.state.identity, commands: authority?.store.state.commands.map(c => ({type:c.type,status:c.status})), + events: authority?.store.state.committedEvents.map(e => e.eventType).slice(-8), + runner: { lifecycle:saved.lifecycle, runId:saved.runId, turnId:saved.turnId, itemId:saved.itemId, + outbox:saved.outbox?.length }, stderr:stderr.slice(-1500), + })}`); + } finally { + await (recovered ? within("recovered cleanup", recovered.transport.close()).catch(() => undefined) : undefined); + await within("first cleanup", first.transport.close()).catch(() => undefined); + if (providerPid) { try { process.kill(providerPid, "SIGKILL"); } catch {} } + await (authority as DurablePrpControlPlane | null)?.stop(); + if (runnerPid) { try { process.kill(-runnerPid, "SIGKILL"); } catch {} } + server.closeAllConnections(); + if (server.listening) await new Promise(done => server.close(() => done())); + await rm(root, { recursive: true, force: true }); + } +}, 30_000); + it("waits for a warm runner to re-authenticate before probing attachment readiness", async () => { const stateDirectory = await mkdtemp( join(tmpdir(), "runnerd-warm-reattach-before-probe-"), @@ -8123,7 +8239,7 @@ it("still fails closed when a real close grace period cannot fit a durable suspe } }, 30_000); -it("cold-restores a suspended provider session under its durable run binding", async () => { +it.each([false, true, "restart-before-bootstrap", "unknown-semantic", "accounting-rejected"] as const)("cold-restores a suspended provider session under its durable run binding (pending tail: %s)", async (pendingTail) => { const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-cold-attach-")); const tracePath = join(stateDirectory, "provider-trace.ndjson"); const skillRoot = join(stateDirectory, "runtime-skill"); @@ -8216,14 +8332,57 @@ it("cold-restores a suspended provider session under its durable run binding", a throw new Error("cold attach fixture did not return a provider thread"); } + // Model a retained provider-owned suffix: rotation must deliver and durably + // accept it before attempting run.attach, rather than dropping or ACKing it. + const retainedDelta = "cold-recovery-retained-tail"; + if (pendingTail) { + const providerFile = join(stateDirectory, "runner", "codex-provider-state.json"); + const provider = JSON.parse(await readFile(providerFile, "utf8")); + expect(provider.pendingEvents).toEqual([]); + provider.pendingEvents.push({ + executorEventId: `codex_provider_${String(provider.nextProviderEventSeq++).padStart(16, "0")}`, + eventType: pendingTail === "unknown-semantic" ? "semantic_tool.input" : "usage.reported", + priority: "p0", + payload: pendingTail === "unknown-semantic" ? { semantic_tool: { + schema: "paperclip.prp.semantic_tool.v1", schemaVersion: 1, phase: "input", + callId: "unknown-source-call", operationId: "get_task_context", input: {}, idempotencyKey: null, + correlation: { runId: baseIdentity.runId, normalizedSessionId: baseIdentity.normalizedSessionId, + turnId: baseIdentity.turnId, itemId: baseIdentity.itemId }, + content: { digest: `sha256:${createHash("sha256").update("{}").digest("hex")}`, redactionDisposition: "digest_only", references: [] }, + } } : { provider: "codex", kind: "usage", delta: retainedDelta, + providerTurnId: provider.completed_provider_turn_id, runDelta: { inputTokens: 20, outputTokens: 2 }, + runDeltaAvailable: true, runDeltaBasis: "run" }, + }); + await writeFile(providerFile, JSON.stringify(provider)); + } + const retainedEvents: PrpEvent[] = []; + let releaseRetained!: () => void, retainedEntered!: () => void; + const retainedAccepted = new Promise(done => { releaseRetained = done; }); + const retainedAdmission = new Promise(done => { retainedEntered = done; }); const secondIdentity = { ...baseIdentity, runId: "run-cold-second", turnId: "turn-cold-second", itemId: "item-cold-second", }; - const rotated = createCapabilityRunnerdCodexTransport({ + let stagedCheckpoint: durableControlPlane.DurablePrpCheckpoint | undefined; + let rejectStagedCommit = pendingTail === "restart-before-bootstrap"; + const rotatedOptions = { ...options, + onStagedRunAttachment: pendingTail === "restart-before-bootstrap" ? async (authority: DurablePrpControlPlane) => { + authority.setCheckpointPersistence(async checkpoint => { + stagedCheckpoint = checkpoint; + if (rejectStagedCommit) throw new Error("fixture lost controller after durable intent"); + }); + } : undefined, + onRecoveryAccountingEvent: pendingTail ? async (event: PrpEvent) => { + if (event.payload.delta === retainedDelta) { + retainedEvents.push(structuredClone(event)); + retainedEntered(); + if (pendingTail === "accounting-rejected") throw new Error("source accounting receipt unavailable"); + await retainedAccepted; + } + } : undefined, environment: { ...options.environment, PAPERCLIP_GITHUB_BROKER_TOKEN: "test-run-B-capability" }, codexArgs: options.codexArgs.map((arg) => arg.replace('/run/A', '/run/B')), resumeDynamicTools: dynamicTools, @@ -8232,14 +8391,81 @@ it("cold-restores a suspended provider session under its durable run binding", a criterionIds: ["criterion-second"], }, prpIdentity: secondIdentity, - }); - rotated.transport.setServerRequestHandler(async () => ({ - success: true, - contentItems: [], - })); + }; + let rotated = createCapabilityRunnerdCodexTransport(rotatedOptions); + if (pendingTail === "restart-before-bootstrap") { + await expect(rotated.transport.request("thread/read", {})).rejects.toThrow("fixture lost controller after durable intent"); + expect(rotated.evidence().runnerPid).toBeNull(); + expect(stagedCheckpoint?.runId).toBe(secondIdentity.runId); + expect([...stagedCheckpoint!.records].filter(([key]) => key.startsWith("command:")).map(([, value]) => JSON.parse(value).type)).not.toContain("run.attach"); + await rotated.detachControllerForRestart().catch(() => undefined); + rejectStagedCommit = false; + await rm(join(stateDirectory, "control-plane"), { recursive: true, force: true }); + stagedCheckpoint!.restoreSessionDirectory(stateDirectory); + rotated = createCapabilityRunnerdCodexTransport(rotatedOptions); + } + const currentHandler = vi.fn(async () => ({ success: true, contentItems: [] })); + rotated.transport.setServerRequestHandler(currentHandler); + if (pendingTail === "accounting-rejected") { + const reading = rotated.transport.request("thread/read", {}); + void reading.catch(() => undefined); + try { + await Promise.race([retainedAdmission, reading]); + const source = JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8")); + expect(source.identity).toEqual(baseIdentity); + expect(source.stagedRunAttachment.targetIdentity).toEqual(secondIdentity); + expect(source.commands.some((command: { type: string }) => command.type === "run.attach")).toBe(false); + const retainedRunner = JSON.parse(await readFile(join(stateDirectory, "runner", "runner-state.json"), "utf8")); + expect(retainedRunner.outbox.some((entry: { envelope: { payload: PrpEvent } }) => + entry.envelope.payload.sourceEventId === retainedEvents[0]!.sourceEventId)).toBe(true); + expect(currentHandler).not.toHaveBeenCalled(); + } finally { + await expect(rotated.transport.close()).rejects.toMatchObject({ code: "native_session_close_unrecoverable" }); + await expect(reading).rejects.toThrow(); + await releaseMaterializedNativeRuntimeSkills(join(stateDirectory, "codex-home", "skills")); + await rm(stateDirectory, { recursive: true, force: true }); + } + return; + } + if (pendingTail === "unknown-semantic") { + try { + await expect(rotated.transport.request("thread/read", {})).rejects.toThrow("native_runner_source_semantic_reconciliation_required"); + expect(currentHandler).not.toHaveBeenCalled(); + const retained = JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8")); + expect(retained.identity).toEqual(baseIdentity); + expect(retained.stagedRunAttachment.targetIdentity).toEqual(secondIdentity); + expect(retained.commands.some((command: { type: string }) => ["run.attach", "semantic_tool.result"].includes(command.type))).toBe(false); + expect(retained.committedEvents.some((event: PrpEvent) => event.eventType === "semantic_tool.input")).toBe(true); + } finally { + await expect(rotated.transport.close()).rejects.toMatchObject({ code: "native_session_close_unrecoverable" }); + await releaseMaterializedNativeRuntimeSkills(join(stateDirectory, "codex-home", "skills")); + await rm(stateDirectory, { recursive: true, force: true }); + } + return; + } try { - const read = await rotated.transport.request("thread/read", {}); + const reading = rotated.transport.request("thread/read", {}); + void reading.catch(() => undefined); + if (pendingTail) { + await Promise.race([retainedAdmission, reading]); + const retainedRunner = JSON.parse(await readFile(join(stateDirectory, "runner", "runner-state.json"), "utf8")); + expect(retainedRunner.outbox.some((entry: { envelope: { payload: PrpEvent } }) => + entry.envelope.payload.sourceEventId === retainedEvents[0]!.sourceEventId)).toBe(true); + const retainedCore = stagedCheckpoint ? { + identity: JSON.parse(stagedCheckpoint.records.get("header")!).state.identity, + commands: [...stagedCheckpoint.records].filter(([key]) => key.startsWith("command:")).map(([, value]) => JSON.parse(value)), + } : JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8")); + expect(retainedCore.identity).toEqual(baseIdentity); + expect(retainedCore.commands.some((command: { type: string }) => command.type === "run.attach")).toBe(false); + } + releaseRetained(); + const read = await reading; + expect(retainedEvents).toHaveLength(pendingTail ? 1 : 0); + if (pendingTail) expect(retainedEvents[0]).toMatchObject({ + runId: baseIdentity.runId, turnId: baseIdentity.turnId, itemId: baseIdentity.itemId, + }); const persistedProvider = JSON.parse(await readFile(join(stateDirectory, "runner", "codex-provider-state.json"), "utf8")); + expect(persistedProvider.pendingEvents).toEqual([]); expect(persistedProvider.config.args.join("\n")).toContain('/run/B'); expect(JSON.stringify(persistedProvider)).not.toContain("test-run-B-capability"); expect(read.thread).toMatchObject({ @@ -8256,8 +8482,14 @@ it("cold-restores a suspended provider session under its durable run binding", a expect(rotated.evidence().diagnostics).toContain( "runnerd attached the durable provider session to a fresh PRP run authority", ); - expect(await stat(join(stateDirectory, "authority-epochs"))).toBeDefined(); + if (pendingTail) { + const target = stagedCheckpoint ? JSON.parse(stagedCheckpoint.records.get("header")!).state + : JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8")); + expect(target.stagedRunAttachment).toBeUndefined(); + expect(target.completedWarmTransition.receipt).toMatchObject({ oldIdentity: baseIdentity, newIdentity: secondIdentity }); + } else expect(await stat(join(stateDirectory, "authority-epochs"))).toBeDefined(); } finally { + releaseRetained(); await rotated.transport.close(); } const requestFrames = (await readFile(tracePath, "utf8")) @@ -8294,6 +8526,12 @@ it("cold-restores a suspended provider session under its durable run binding", a ); } + if (pendingTail) { + await releaseMaterializedNativeRuntimeSkills(join(stateDirectory, "codex-home", "skills")); + await rm(stateDirectory, { recursive: true, force: true }); + return; + } + // A remote process owner keeps runner-state outside the controller's local // session root. Resume must defer to its explicit state reader instead of // rejecting recovery before the remote checkpoint can be made available. diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index c7df8fe8bc9..811133f5b0a 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -69,6 +69,7 @@ import type { } from "../contracts/harness-driver.js"; import { DurablePrpControlPlane, + inspectStagedRunAttachment, SemanticToolNotDispatchedError, durableRecoveryInternals, inspectWarmRunTransition, @@ -1241,6 +1242,7 @@ export interface CapabilityRunnerdCodexTransportOptions extends RunnerExecutionD observeCredentials?: boolean; onCredentialsUpdated?: (event: PrpEvent) => Promise; onAccountingEvent?: (event: PrpEvent) => Promise; + onRecoveryAccountingEvent?: (event: PrpEvent, sourceIdentity: DurableRecoveryIdentity) => Promise; recoverSemanticToolInput?: DurablePrpControlPlaneOptions["recoverSemanticToolInput"]; opencodePermissionMode?: NativeOpenCodePermissionMode; acpxAgent?: QualifiedAcpxAgent; @@ -1379,7 +1381,7 @@ export interface CapabilityRunnerdCodexTransportOptions extends RunnerExecutionD itemId: string; }; /** Existing server-owned routes only; never provision two ingress owners. */ - warmTransitionRegistrationMode?: "routed_connect"; + warmTransitionRegistrationMode?: "routed_connect" | "routed_same_endpoint"; /** Rechecks the server-owned recovery claim after each asynchronous boundary. */ authorizeWarmTransitionRecovery?: ( stage: "before_bootstrap" | "before_spawn" | "before_authentication", @@ -3563,6 +3565,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { #pump: NodeJS.Timeout | null = null; #eventSourceSeq = 0; #eventIdentity: DurableRecoveryIdentity | null = null; + #sourceRecoveryIdentity: DurableRecoveryIdentity | null = null; + #sourceAttachmentDeadline: number | null = null; #pendingWarmRecoveryCompletion: { transitionId: string; identity: DurableRecoveryIdentity; @@ -3794,6 +3798,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ): Promise> { if (this.#closed) throw new Error("PRP Codex transport is closed"); this.#throwIfFailed(); + if (this.#sourceRecoveryIdentity && !["initialize", "collaborationMode/list"].includes(method)) { + throw new Error("native_runner_source_attachment_pending"); + } if ( this.#pendingWarmRecoveryCompletion !== null && !["thread/read", "initialize", "collaborationMode/list"].includes(method) @@ -4103,7 +4110,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } #recoverPendingSemanticInputsWhenReady(): void { - if (this.#startupComplete && this.#handlerInstalled) { + if (this.#startupComplete && this.#handlerInstalled && this.#sourceRecoveryIdentity === null && this.#pendingWarmRecoveryCompletion === null) { void Promise.resolve().then(() => this.#core?.recoverPendingSemanticInputs()) .catch(error => this.#failTransport(error instanceof Error ? error : new Error(String(error)))); } @@ -4113,7 +4120,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { // Remote runner ingress already has a bounded reconnect budget. Reuse the // same budget here so a transient tunnel reconnect cannot trip the shorter // generic command timeout and replace an otherwise healthy warm runner. - const reconnectGraceMs = this.options.runnerReconnectGraceMs ?? 5_000; + const reconnectGraceMs = Math.min(this.options.runnerReconnectGraceMs ?? 5_000, + this.#sourceAttachmentDeadline === null ? Infinity : Math.max(0, this.#sourceAttachmentDeadline - Date.now())); await waitForWarmAttachmentReadiness({ graceMs: reconnectGraceMs, waitForConnection: (deadline) => this.#awaitWarmRunnerConnection(deadline), @@ -4243,7 +4251,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { .update(`${prior.runId}:${desired.runId}:${desired.turnId}`) .digest("hex") .slice(0, 32)}`; - const runAttachTemplate = this.#runAttachTemplate + const stagedCandidate = inspectStagedRunAttachment(core.store.state); + const staged = stagedCandidate && recoveryIdentityMatches(stagedCandidate.targetIdentity, desired) ? stagedCandidate : undefined; + const runAttachTemplate = staged ? structuredClone(staged.runAttachTemplate) : this.#runAttachTemplate ? retargetRunAttachPayload( this.#runAttachTemplate, desired, @@ -4260,7 +4270,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { currentRunGrant?.runtimeContext, currentRunGrant ? { text: currentRunGrant.instructions, context: currentRunGrant.runtimeContext } : undefined, ); - if (this.options.prepareLaunch) { + if (this.options.prepareLaunch && !staged) { const priorProvider = record(runAttachTemplate.provider); const priorRequirements = new RunnerLaunchPreparationStore(this.#root).load(prior)?.requirements; await this.#prepareOwnedProvider(runAttachTemplate, desired, @@ -4270,6 +4280,11 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } this.#assertStartupOpen(); this.#runAttachTemplate = structuredClone(runAttachTemplate); + if (!staged && (this.options.provider ?? "codex") === "codex" && !this.options.prepareLaunch && this.options.onStagedRunAttachment) { + // Registration already installed the target writer. Stage and queue + // share the existing attachment checkpoint; no duplicate owner reads. + core.stageRunAttachment(desired, runAttachTemplate); + } const payload = { ...runAttachTemplate, paperclipNextAuthority: { identity: desired, connection, @@ -4285,7 +4300,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await this.#waitCommand( "run.attach", commandId, - this.#coldAdmissionDeadline(), + this.#sourceAttachmentDeadline === null ? this.#coldAdmissionDeadline() + : Math.min(Date.now() + 30_000, this.#sourceAttachmentDeadline), true, ); const attached = core.getCommand(commandId); @@ -4295,6 +4311,21 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { activationStarted = true; core.rotateRunIdentity(desired, runAttachTemplate); + if (core.store.state.stagedRunAttachment && registration?.connection?.mode === "listen" && this.options.readRunnerState) { + const deadline = Math.min(Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000), this.#sourceAttachmentDeadline ?? Infinity); + const artifact = this.options.runnerArtifact ?? approvedRunnerArtifact(this.options.runnerBinary ?? defaultCapabilityRunnerdBinary()); + for (;;) { + this.#assertStartupOpen(); + const physical = await this.options.readRunnerState(); + const proof = inspectWarmRunTransition({ controlPlaneState: core.store.state, runnerState: physical, + expectedNewIdentity: desired, expectedRunnerVersion: artifact.version, expectedRunnerDigest: artifact.digest }); + if (proof && recoveryIdentityMatches(proof.runnerIdentity, desired)) { + await previousRelease?.(); previousReleased = true; break; + } + if (Date.now() >= deadline || await this.#runnerHasExited()) throw new Error("native_runner_warm_transition_activation_pending"); + await new Promise(done => setTimeout(done, 100)); + } + } await registration?.activate?.(); if (registration?.failure) { void registration.failure.catch((error: unknown) => { @@ -4304,13 +4335,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { }); } await this.#awaitRegistrationReady(registration?.ready); - const activationDeadline = - Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000); + const activationDeadline = Math.min( + Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000), this.#sourceAttachmentDeadline ?? Infinity); while ( !recoveryIdentityMatches(core.store.state.identity, desired) || core.store.state.warmTransition !== undefined || core.activeRunnerConnectionCount() === 0 ) { + this.#assertStartupOpen(); if ( Date.now() >= activationDeadline || (await this.#runnerHasExited()) @@ -4319,11 +4351,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } await new Promise((resolveWait) => setTimeout(resolveWait, 10)); } + this.#assertStartupOpen(); this.#eventIdentity = structuredClone(desired); this.#eventSourceSeq = 0; this.#deferredTurnStartEvents = []; this.#durableTurnId = desired.turnId; - await previousRelease?.(); + if (!previousReleased) await previousRelease?.(); previousReleased = true; this.#controlPlaneRelease = registration?.release ?? null; } catch (error) { @@ -5251,7 +5284,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { onSemanticToolInput: (call) => this.#handleSemanticToolInput(call), recoverSemanticToolInput: call => this.#recoverSemanticToolInput(call), onCommittedEvent: async (event) => { - await this.options.onAccountingEvent?.(event); + if (this.#sourceRecoveryIdentity && event.runId === this.#sourceRecoveryIdentity.runId) { + if (!this.options.onRecoveryAccountingEvent && this.options.onAccountingEvent) throw new Error("native_runner_source_accounting_unavailable"); + await this.options.onRecoveryAccountingEvent?.(event, this.#sourceRecoveryIdentity); + } else await this.options.onAccountingEvent?.(event); if (event.eventType === "auth.credentials.updated") { await this.options.onCredentialsUpdated?.(event); } @@ -5393,7 +5429,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { async #resumeWork(): Promise { if (this.options.acpxCandidateAdmission?.agent === "antigravity") throw new Error("ACPX candidate durable runner recovery is not admitted"); - const desiredIdentity = this.options.prpIdentity; + let desiredIdentity = this.options.prpIdentity; if (desiredIdentity === undefined) { throw new Error("PRP provider resume state is unavailable"); } @@ -5457,6 +5493,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ) { candidateRunnerState = await this.options.readRunnerState(); } + const staged = inspectStagedRunAttachment(controlPlaneState, desiredIdentity); + if (controlPlaneState?.stagedRunAttachment !== undefined && staged === null) { + throw new Error("native_runner_staged_attachment_binding_mismatch"); + } let warmRecovery: { runnerState: Record; runnerIdentity: DurableRecoveryIdentity; @@ -5482,8 +5522,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { !controlPlaneState || !candidateRunnerState || (this.options.controlPlaneRegistration !== undefined && - this.options.warmTransitionRegistrationMode !== "routed_connect") || - this.options.adoptExistingRunner !== undefined + this.options.warmTransitionRegistrationMode !== "routed_connect" + && !(staged && this.options.adoptExistingRunner && this.options.warmTransitionRegistrationMode === "routed_same_endpoint")) || + (this.options.adoptExistingRunner !== undefined && !staged) ) { throw new Error( "native_runner_warm_transition_requires_exact_owned_endpoint", @@ -5508,14 +5549,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ? new URL(connection.connectUrl) : null; if ( - connection.mode !== "connect" || - endpoint === null || - endpoint.search || - endpoint.hash || - endpoint.username || - endpoint.password || + (connection.mode !== "connect" && !(connection.mode === "listen" && staged + && this.options.adoptExistingRunner && this.options.controlPlaneRegistration + && this.options.warmTransitionRegistrationMode === "routed_same_endpoint")) || + (connection.mode === "connect" && (endpoint === null || endpoint.search || endpoint.hash || endpoint.username || endpoint.password)) || (!this.options.controlPlaneRegistration && - (endpoint.protocol !== "ws:" || + (endpoint === null || endpoint.protocol !== "ws:" || endpoint.hostname !== "127.0.0.1" || endpoint.pathname !== "/durableRecovery/connect" || !endpoint.port)) @@ -5529,7 +5568,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { transitionId: receipt.transitionId, commandId: receipt.commandId, ...(!this.options.controlPlaneRegistration - ? { port: Number(endpoint.port) } + ? { port: Number(endpoint!.port) } : {}), }; // Fence concurrent public work before exposing the core or awaiting @@ -5539,6 +5578,40 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { identity: structuredClone(desiredIdentity), }; } + const completedStaged = staged && recoveryIdentityMatches(identity, staged.targetIdentity) + && record(record(controlPlaneState?.completedWarmTransition).receipt).commandId === staged.commandId; + const queuedSourceAttachment = staged && staged.commandId !== null && warmRecovery === null + && recoveryIdentityMatches(identity, staged.sourceIdentity) ? staged : null; + if (queuedSourceAttachment && !this.options.adoptExistingRunner) { + throw new Error("native_runner_queued_attachment_requires_verified_live_source"); + } + let sourceAttachmentTarget: DurableRecoveryIdentity | null = null; + if (controlPlaneState && warmRecovery === null && (this.options.provider ?? "codex") === "codex" + && !this.options.prepareLaunch && (staged || this.options.onStagedRunAttachment || this.options.onRecoveryAccountingEvent) + && identity.runId !== desiredIdentity.runId) { + // Restore and acknowledge the source under its own authority. The target + // remains a durable, undispatched intent until every old obligation settles. + if (!this.options.adoptExistingRunner) { + if (localStateOwner) { + if (!candidateRunnerState) throw new Error("native_runner_authority_rotation_state_unavailable"); + assertSuspendedRunnerState(candidateRunnerState, identity); + } else { + if (!this.options.readRunnerState || !this.options.prepareExternalRunnerState) throw new Error("native_runner_prp_run_rotation_unavailable"); + await this.options.prepareExternalRunnerState(); + assertSuspendedRunnerState(await this.options.readRunnerState(), identity); + } + } + sourceAttachmentTarget = queuedSourceAttachment ? null : desiredIdentity; + desiredIdentity = identity; + this.#sourceRecoveryIdentity = structuredClone(identity); + this.#sourceAttachmentDeadline = Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000); + } else if (staged && !completedStaged && !warmRecovery) { + throw new Error("native_runner_staged_attachment_profile_unsupported"); + } + if (warmRecovery && staged) { + this.#sourceRecoveryIdentity = structuredClone(staged.sourceIdentity); + this.#sourceAttachmentDeadline = Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000); + } const exactAuthority = controlPlaneState !== null && identity.runnerInstanceId === desiredIdentity.runnerInstanceId && @@ -5704,7 +5777,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { onSemanticToolInput: (call) => this.#handleSemanticToolInput(call), recoverSemanticToolInput: call => this.#recoverSemanticToolInput(call), onCommittedEvent: async (event) => { - await this.options.onAccountingEvent?.(event); + if (this.#sourceRecoveryIdentity && event.runId === this.#sourceRecoveryIdentity.runId) { + if (!this.options.onRecoveryAccountingEvent && this.options.onAccountingEvent) throw new Error("native_runner_source_accounting_unavailable"); + await this.options.onRecoveryAccountingEvent?.(event, this.#sourceRecoveryIdentity); + } else await this.options.onAccountingEvent?.(event); if (event.eventType === "auth.credentials.updated") { await this.options.onCredentialsUpdated?.(event); } @@ -5775,14 +5851,15 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { preparedContext, preparedInstructions, saved?.prepared, saved?.requirements); this.#runAttachTemplate = structuredClone(template); } - if (rotatedAuthority) { + if (rotatedAuthority || sourceAttachmentTarget) { if (provider === "acpx" && this.options.acpxAgent === "copilot" && typeof this.options.baseInstructions !== "string") { throw new Error("native_runner_current_instruction_grant_unavailable"); } + const attachmentIdentity = sourceAttachmentTarget ?? desiredIdentity; const runAttachTemplate = rotatedRunAttachPayload( controlPlaneState, - desiredIdentity, + attachmentIdentity, this.#authorizedTools, this.options.resumeCompletionContract, runtimeContext, @@ -5823,8 +5900,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { runAttachTemplate.provider = environmentPrepared.provider; } this.#runAttachTemplate = structuredClone(runAttachTemplate); - core.persistRunAttachTemplate(runAttachTemplate); - core.queueCommand("run.attach", runAttachTemplate); + if (sourceAttachmentTarget) { + await this.options.onStagedRunAttachment?.(core, { sourceIdentity: desiredIdentity, + targetIdentity: sourceAttachmentTarget, templateDigest: commandDigest(runAttachTemplate) }); + core.stageRunAttachment(sourceAttachmentTarget, runAttachTemplate); + } else { + core.persistRunAttachTemplate(runAttachTemplate); + core.queueCommand("run.attach", runAttachTemplate); + } } const committedEvents = core.store.state.committedEvents; const runAttachment = recoveredRunAttachment(core.store.state); @@ -5836,7 +5919,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ? `command_resume_probe_${randomUUID().replaceAll("-", "")}` : null; if (recoveryProbeCommandId !== null) { - core.queueCommand("runner.drain", {}, recoveryProbeCommandId); + core.queueCommand(sourceAttachmentTarget ? "session.open" : "runner.drain", {}, recoveryProbeCommandId); } // A controller retry can open the exact authority after run.attach has // already reached a durable outcome. Re-observe that command instead of @@ -5885,6 +5968,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { : "restored provider identity from the exact durable checkpoint; awaiting live confirmation", ); } + if (staged && !sourceAttachmentTarget) { + await this.options.onStagedRunAttachment?.(core, { sourceIdentity: staged.sourceIdentity, + targetIdentity: staged.targetIdentity, templateDigest: staged.templateDigest }); + if (warmRecovery && this.options.adoptExistingRunner) core.prepareWarmTransitionAdoption({ + transitionId: warmRecovery.transitionId, runnerState: warmRecovery.runnerState }); + } type RecoveryRegistration = Awaited< ReturnType< NonNullable< @@ -5895,67 +5984,48 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { let oldTransitionRegistration: RecoveryRegistration | null = null; let newTransitionRegistration: RecoveryRegistration | null = null; this.#assertStartupOpen(); - if (warmRecovery && this.options.controlPlaneRegistration) { + const recoveringAttachment = warmRecovery || queuedSourceAttachment; + const queuedCommand = queuedSourceAttachment ? core.getCommand(queuedSourceAttachment.commandId!) : undefined; + const expectedRecoveryConnection = (warmRecovery?.proof.receipt.connection + ?? record(queuedCommand?.payload.paperclipNextAuthority).connection) as RunnerProcessConnection | undefined; + const recoverySource = warmRecovery?.proof.receipt.oldIdentity ?? queuedSourceAttachment?.sourceIdentity; + const recoveryTarget = warmRecovery?.proof.receipt.newIdentity ?? queuedSourceAttachment?.targetIdentity; + if (recoveringAttachment && this.options.controlPlaneRegistration) { try { - oldTransitionRegistration = await this.options.controlPlaneRegistration( - core, - warmRecovery.proof.receipt.oldIdentity, - ); - const oldConnection = oldTransitionRegistration.connection ?? { - mode: "connect", - connectUrl: oldTransitionRegistration.connectUrl, - }; - if ( - oldConnection.mode !== "connect" || - typeof oldConnection.connectUrl !== "string" - ) { - throw new Error( - "native_runner_warm_transition_registered_endpoint_mismatch", - ); + if (!recoverySource || !recoveryTarget || !expectedRecoveryConnection + || (expectedRecoveryConnection.mode !== "connect" && !(staged && this.options.adoptExistingRunner + && this.options.warmTransitionRegistrationMode === "routed_same_endpoint"))) { + throw new Error("native_runner_warm_transition_requires_exact_owned_endpoint"); } + oldTransitionRegistration = await this.options.controlPlaneRegistration(core, recoverySource); this.#assertStartupOpen(); - newTransitionRegistration = await this.options.controlPlaneRegistration( - core, - warmRecovery.proof.receipt.newIdentity, - ); - const newConnection = newTransitionRegistration.connection ?? { - mode: "connect", - connectUrl: newTransitionRegistration.connectUrl, - }; - if ( - newConnection.mode !== "connect" || - durableRecoveryInternals.canonicalJson(newConnection) !== - durableRecoveryInternals.canonicalJson( - warmRecovery.proof.receipt.connection, - ) - ) { - throw new Error( - "native_runner_warm_transition_registered_endpoint_mismatch", - ); + newTransitionRegistration = await this.options.controlPlaneRegistration(core, recoveryTarget); + const connection = newTransitionRegistration.connection ?? { mode: "connect", connectUrl: newTransitionRegistration.connectUrl }; + if (durableRecoveryInternals.canonicalJson(connection) !== durableRecoveryInternals.canonicalJson(expectedRecoveryConnection)) { + throw new Error("native_runner_warm_transition_registered_endpoint_mismatch"); } } catch (error) { - await Promise.allSettled( - [oldTransitionRegistration, newTransitionRegistration].map((entry) => - Promise.resolve().then(() => entry?.release()), - ), - ); + await Promise.allSettled([oldTransitionRegistration, newTransitionRegistration].map(entry => Promise.resolve().then(() => entry?.release()))); throw error; } } - const registration = - warmRecovery && oldTransitionRegistration && newTransitionRegistration - ? recoveryIdentityMatches( - warmRecovery.runnerIdentity, - warmRecovery.proof.receipt.oldIdentity, - ) - ? oldTransitionRegistration - : newTransitionRegistration - : this.options.controlPlaneRegistration - ? await this.options.controlPlaneRegistration(core) - : null; + const registration = oldTransitionRegistration && newTransitionRegistration + ? recoveryIdentityMatches(warmRecovery?.runnerIdentity ?? queuedSourceAttachment!.sourceIdentity, recoverySource!) + ? oldTransitionRegistration : newTransitionRegistration + : this.options.controlPlaneRegistration + ? await this.options.controlPlaneRegistration(core, this.#sourceRecoveryIdentity ?? undefined) : null; this.#startupFailureCode = registration?.startupFailureCode ?? "runner_local_connect_failed"; - if (registration === null) await core.start(warmRecovery?.port); + if (registration === null) { + let queuedPort: number | undefined; + if (queuedSourceAttachment) { + const endpoint = expectedRecoveryConnection?.mode === "connect" ? new URL(expectedRecoveryConnection.connectUrl) : null; + if (!endpoint || endpoint.protocol !== "ws:" || endpoint.hostname !== "127.0.0.1" || endpoint.pathname !== "/durableRecovery/connect" + || !endpoint.port || endpoint.search || endpoint.hash || endpoint.username || endpoint.password) throw new Error("native_runner_warm_transition_snapshot_mismatch"); + queuedPort = Number(endpoint.port); + } + await core.start(warmRecovery?.port ?? queuedPort); + } else { this.#controlPlaneCheckpoint = registration.checkpoint ?? null; this.#controlPlaneRelease = @@ -5976,7 +6046,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#assertStartupOpen(); // Route and immutable endpoint admission precedes every launch-material // write. A refused transition leaves its retained home/context untouched. - if (runtimeContext !== null && !this.options.prepareLaunch) { + if (runtimeContext !== null && !this.options.prepareLaunch && this.#sourceRecoveryIdentity === null) { writeFileSync( localRuntimeContextPath, `${JSON.stringify(runtimeContext)}\n`, @@ -5985,7 +6055,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } if (provider === "aws_agentcore" && !this.options.prepareLaunch) mkdirSync(codexHome, { recursive: true, mode: 0o700 }); - if (!this.options.prepareLaunch && provider === "codex" && this.options.homeMode !== "preconfigured") { + if (!this.options.prepareLaunch && provider === "codex" && this.options.homeMode !== "preconfigured" && this.#sourceRecoveryIdentity === null) { await releaseMaterializedNativeRuntimeSkills( resolve(localCodexHome, "skills"), ); @@ -6025,7 +6095,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } // A replacement executor restores its cold provider before acknowledging recovery. this.#assertStartupOpen(); - const admissionDeadline = this.#coldAdmissionDeadline(adoptedRunner === undefined); + const admissionDeadline = this.#sourceAttachmentDeadline === null + ? this.#coldAdmissionDeadline(adoptedRunner === undefined) + : Math.min(Date.now() + 30_000, this.#sourceAttachmentDeadline); const handle = adoptedRunner ? null : spawnRunner({ @@ -6071,18 +6143,30 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await this.#publishSpawnedProcess(handle); } this.#assertStartupOpen(); + let targetRecoveryActivated = registration === newTransitionRegistration; if (oldTransitionRegistration && newTransitionRegistration) { - await oldTransitionRegistration.activate?.(); + // A guest listener still serves only its actual physical run. Open that + // route first; probing the future/retired path cannot prove readiness. + await registration?.activate?.(); this.#assertStartupOpen(); - await newTransitionRegistration.activate?.(); + if (registration === oldTransitionRegistration) { + // Source authentication can consume its receipt and disappear before + // a sampled count sees it. The durable/physical proof loop below owns + // that progress; only target authentication completes recovery. + if (expectedRecoveryConnection?.mode !== "listen") { + await newTransitionRegistration.activate?.(); targetRecoveryActivated = true; + } + } } else await registration?.activate?.(); this.#assertStartupOpen(); - if (registration?.failure) { - void registration.failure.catch((error: unknown) => { - this.#failTransport( - error instanceof Error ? error : new Error(String(error)), - ); - }); + if (registration?.failure && !recoveringAttachment) { + void registration.failure.catch((error: unknown) => this.#failTransport(error instanceof Error ? error : new Error(String(error)))); + } else if (recoveringAttachment) { + // The old listener can retire normally during recovery. Authentication + // and the shared deadline, rather than its stale readiness promise, own + // the transition. Only the target remains live after activation. + void oldTransitionRegistration?.failure?.catch(() => undefined); + void newTransitionRegistration?.failure?.catch(() => undefined); } if (!adoptedRunner) await this.#awaitRegistrationReady(registration?.ready); this.#assertStartupOpen(); @@ -6095,22 +6179,40 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#evidence.runnerProcessGroupId = handle?.processGroupId ?? null; } this.#publish(); - if (warmRecovery) { - const deadline = - Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000); + if (recoveringAttachment) { + const deadline = Math.min(Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000), this.#sourceAttachmentDeadline ?? Infinity); + let sourceRouteRetired = false, nextPhysicalInspection = 0, previousConnections = core.activeRunnerConnectionCount(); while ( - !recoveryIdentityMatches(core.store.state.identity, desiredIdentity) || - core.store.state.warmTransition !== undefined + !recoveryIdentityMatches(core.store.state.identity, recoveryTarget!) || + core.store.state.warmTransition !== undefined || core.activeRunnerConnectionCount() !== 1 ) { this.#assertStartupOpen(); + const connections = core.activeRunnerConnectionCount(); + if (connections === 0 && previousConnections > 0) nextPhysicalInspection = 0; + previousConnections = connections; + if (!sourceRouteRetired && expectedRecoveryConnection?.mode === "listen" && oldTransitionRegistration + && newTransitionRegistration && this.options.readRunnerState && Date.now() >= nextPhysicalInspection) { + nextPhysicalInspection = Date.now() + 250; + const physical = await this.options.readRunnerState(); + const proof = inspectWarmRunTransition({ controlPlaneState: core.store.state, runnerState: physical, + expectedNewIdentity: recoveryTarget!, expectedRunnerVersion: runnerArtifact.version, expectedRunnerDigest: runnerArtifact.digest }); + if (proof && recoveryIdentityMatches(proof.runnerIdentity, recoveryTarget!)) { + // The same participant has consumed the old ACK and now serves + // the target path. Stop obsolete ingress retries before they can + // compete with the target's authenticated connection. + await oldTransitionRegistration.release(); sourceRouteRetired = true; + if (!targetRecoveryActivated) { await newTransitionRegistration.activate?.(); targetRecoveryActivated = true; } + } + } if (Date.now() >= deadline || (await this.#runnerHasExited())) { throw new Error("native_runner_warm_transition_recovery_pending"); } await new Promise((resolveWait) => setTimeout(resolveWait, 10)); } - if (core.getCommand(warmRecovery.commandId)?.status !== "completed") { + if (core.getCommand(warmRecovery?.commandId ?? queuedSourceAttachment!.commandId!)?.status !== "completed") { throw new Error("native_runner_warm_transition_result_unproven"); } + desiredIdentity = recoveryTarget!; this.#durableTurnId = desiredIdentity.turnId; this.#eventIdentity = structuredClone(desiredIdentity); this.#eventSourceSeq = 0; @@ -6119,8 +6221,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await oldTransitionRegistration.release(); this.#assertStartupOpen(); this.#controlPlaneRelease = newTransitionRegistration.release; - this.#controlPlaneCheckpoint = - newTransitionRegistration.checkpoint ?? null; + this.#controlPlaneCheckpoint = newTransitionRegistration.checkpoint ?? null; + void newTransitionRegistration.failure?.catch(error => this.#failTransport(error instanceof Error ? error : new Error(String(error)))); } } this.#assertStartupOpen(); @@ -6128,7 +6230,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { if (adoptedRunner) { await this.#awaitAdoptedRunnerConnection( adoptedRunner, - registration?.ready, + newTransitionRegistration?.ready ?? registration?.ready, ); } // Only a replacement executor can be restoring a cold Pi process. Share @@ -6137,7 +6239,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await this.#waitCommand("run.attach", runAttachment.commandId, admissionDeadline, true); } if (recoveryProbeCommandId !== null) { - await this.#waitCommand("runner.drain", recoveryProbeCommandId, admissionDeadline, true); + await this.#waitCommand(sourceAttachmentTarget ? "session.open" : "runner.drain", recoveryProbeCommandId, admissionDeadline, true); if (this.#checkpointProviderIdentityExpectation !== null) { // A replacement runner can restore the exact provider while its fresh // session.resumed event is compacted or delayed behind the completed @@ -6166,8 +6268,20 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { admissionDeadline, ); this.#startupComplete = true; + if (recoveringAttachment && staged) { + this.#sourceRecoveryIdentity = null; + this.#sourceAttachmentDeadline = null; + } + if (sourceAttachmentTarget) { + await this.#awaitWarmRunAttachmentReady(); + if (!core.semanticToolResultsSettled()) throw new Error("native_runner_source_semantic_reconciliation_required"); + if (runtimeContext !== null) writeFileSync(localRuntimeContextPath, `${JSON.stringify(runtimeContext)}\n`, { mode: 0o600 }); + await this.#attachRunWork(sourceAttachmentTarget); + this.#sourceRecoveryIdentity = null; + this.#sourceAttachmentDeadline = null; + } this.#diagnostic( - rotatedAuthority + rotatedAuthority || sourceAttachmentTarget ? "runnerd attached the durable provider session to a fresh PRP run authority" : "runnerd restored its durable PRP session and provider thread", ); @@ -6176,6 +6290,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } async #recoverSemanticToolInput(call: Parameters>[0]) { + if (this.#sourceRecoveryIdentity) throw new Error("native_runner_source_semantic_reconciliation_required"); if (call.operationId === "paperclip_finish" || call.operationId === "paperclip_block") { return this.#handleSemanticToolInput(call); } @@ -6188,6 +6303,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { NonNullable >[0], ) { + if (this.#sourceRecoveryIdentity) throw new Error("native_runner_source_semantic_reconciliation_required"); try { this.#throwIfFailed(); } catch { @@ -6529,6 +6645,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await completing; if (this.#pendingWarmRecoveryCompletion === completion) { this.#pendingWarmRecoveryCompletion = null; + this.#recoverPendingSemanticInputsWhenReady(); } } finally { if (this.#warmRecoveryCompletionInFlight === completing) { diff --git a/packages/paperclip-runner/src/live/runnerd-staged-handoff.test.ts b/packages/paperclip-runner/src/live/runnerd-staged-handoff.test.ts new file mode 100644 index 00000000000..23a465d4b56 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-staged-handoff.test.ts @@ -0,0 +1,241 @@ +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { createServer, request } from "node:http"; +import { createHash, randomBytes } from "node:crypto"; +import { execFileSync } from "node:child_process"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { expect, it, vi } from "vitest"; +import { + DurablePrpControlPlane, inspectStagedRunAttachmentHeader, + type DurablePrpCheckpoint, type RunnerProcessConnection, type PrpWireConnection, +} from "../control-plane/durable-prp-control-plane.js"; +import { + createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary, + readRunnerdArtifactBinding, +} from "./runnerd-codex-transport.js"; + +// The guest rejects WebSocket extensions. Use the same uncompressed wire +// contract as production, without importing the server connector into Runner. +async function openOwnedWire(url: string): Promise<{ wire: PrpWireConnection; closed: Promise; close(): void }> { + return new Promise((resolvePeer, reject) => { + const key = randomBytes(16).toString("base64"); + const req = request(url.replace("ws:", "http:"), { headers: { + Connection: "Upgrade", Upgrade: "websocket", "Sec-WebSocket-Version": "13", "Sec-WebSocket-Key": key, + } }); + req.on("error", reject); req.on("response", response => { response.resume(); reject(new Error(`Guest upgrade refused: ${response.statusCode}`)); }); + req.setTimeout(1_000, () => req.destroy(new Error("Guest upgrade timeout"))); + req.on("upgrade", (response, socket, head) => { + socket.setTimeout(0); + const expected = createHash("sha1").update(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").digest("base64"); + if (response.headers["sec-websocket-accept"] !== expected) { socket.destroy(); reject(new Error("Invalid guest upgrade")); return; } + const listeners: Array<(value: unknown) => void> = [], closes: Array<() => void> = []; + let buffer: Buffer = head, ended = false; + let closeResolve!: () => void; + const closed = new Promise(done => { closeResolve = done; }); + const finish = () => { if (ended) return; ended = true; for (const close of closes) close(); closeResolve(); }; + const send = (payload: Buffer, opcode: number) => { + const mask = randomBytes(4), prefix = payload.length < 126 ? Buffer.from([0x80 | opcode, 0x80 | payload.length]) : Buffer.alloc(payload.length <= 65535 ? 4 : 10); + if (payload.length >= 126) { prefix[0] = 0x80 | opcode; prefix[1] = 0x80 | (payload.length <= 65535 ? 126 : 127); + if (payload.length <= 65535) prefix.writeUInt16BE(payload.length, 2); else prefix.writeBigUInt64BE(BigInt(payload.length), 2); } + const masked = Buffer.from(payload); for (let index = 0; index < masked.length; index++) masked[index] ^= mask[index % 4]!; + socket.write(Buffer.concat([prefix, mask, masked])); + }; + const parse = () => { + while (buffer.length >= 2) { + const opcode = buffer[0]! & 15, initial = buffer[1]! & 127; + if (buffer.length < (initial === 126 ? 4 : initial === 127 ? 10 : 2)) return; + const size = initial === 126 ? buffer.readUInt16BE(2) : initial === 127 ? Number(buffer.readBigUInt64BE(2)) : initial; + const offset = initial === 126 ? 4 : initial === 127 ? 10 : 2; + if (buffer.length < offset + size) return; + const payload = buffer.subarray(offset, offset + size); buffer = buffer.subarray(offset + size); + if (opcode === 8) { socket.destroy(); finish(); return; } + if (opcode === 9) send(payload, 10); + if (opcode === 1 || opcode === 2) for (const listener of listeners) listener(JSON.parse(payload.toString("utf8"))); + } + }; + socket.on("data", bytes => { buffer = Buffer.concat([buffer, bytes]); parse(); }); socket.on("close", finish); socket.on("error", finish); + const close = () => { socket.destroy(); finish(); }; + resolvePeer({ wire: { sendJson: value => send(Buffer.from(JSON.stringify(value)), 1), close, + onJson: listener => { listeners.push(listener); }, onClose: listener => { closes.push(() => listener({})); } }, closed, close }); + queueMicrotask(parse); + }); req.end(); + }); +} + +// Real encrypted PRP and real Runner/provider processes; no model or server imports. +it.each(["connect", "listen"] as const)("recovers a staged %s handoff at every durable boundary without another process", async mode => { + const root = await mkdtemp(join(tmpdir(), "runner-staged-handoff-")); + const shared = join(root, "shared"), calls = join(root, "calls.log"); + const source = { runnerInstanceId: "runner-staged", environmentLeaseId: "lease-staged", runId: "source-run", + normalizedSessionId: "session-staged", turnId: "source-turn", itemId: "source-item" }; + const target = { ...source, runId: "target-run", turnId: "target-turn", itemId: "target-item" }; + const http = createServer(); + const routes = new Map(); + http.on("upgrade", (request, socket, head) => { + const route = routes.get(request.url ?? ""); + if (route) route.core.handleUpgrade(request, socket, request.url!, head); else socket.destroy(); + }); + await new Promise(done => http.listen(0, "127.0.0.1", done)); + const address = http.address(); + if (!address || typeof address === "string") throw new Error("Missing owned endpoint"); + const guest = createServer(); + await new Promise(done => guest.listen(0, "127.0.0.1", done)); + const guestAddress = guest.address(); + if (!guestAddress || typeof guestAddress === "string") throw new Error("Missing owned guest endpoint"); + const guestPort = guestAddress.port; + await new Promise(done => guest.close(() => done())); + const within = async (promise: Promise, label: string, ms = 8_000): Promise => { + let timer: ReturnType | undefined; + try { return await Promise.race([promise, new Promise((_, reject) => { timer = setTimeout(() => reject(new Error(`staged fixture timeout: ${label}`)), ms); })]); } + finally { clearTimeout(timer); } + }; + const dead = (pid: number) => { try { process.kill(pid, 0); return false; } catch (error) { return (error as NodeJS.ErrnoException).code === "ESRCH"; } }; + // Each boundary uses a fresh company-free fixture so no previous command or + // activation can satisfy a later assertion. + try { + for (const boundary of ["queued", "prepared", "activated", "completed"] as const) { + const directory = join(shared, boundary), callLog = `${calls}-${boundary}`; + let checkpoint!: DurablePrpCheckpoint, currentCore!: DurablePrpControlPlane; + let lost = false, release!: () => void, reached!: () => void; + const held = new Promise(done => { release = done; }); + const atBoundary = new Promise(done => { reached = done; }); + const cores = new Set(); + const registered = new WeakSet(); + let generation = 0; + const registration = async (core: DurablePrpControlPlane, identity = core.store.state.identity) => { + currentCore = core; + if (!registered.has(core)) { + registered.add(core); cores.add(core); + const epoch = generation; + core.setCheckpointPersistence(async saved => { + if (epoch !== generation) throw new Error("retired fixture controller"); + checkpoint = saved; + const proof = inspectStagedRunAttachmentHeader(saved.records.get("header")!); + if (!lost && proof?.phase === (boundary === "queued" ? "attachment_queued" : boundary)) { + lost = true; reached(); await held; + if (epoch !== generation) throw new Error("retired fixture controller"); + } + }); + } + const path = `/api/runner/v1/connect/${identity.runId}`, token = Symbol(path); + const connection: RunnerProcessConnection = mode === "connect" + ? { mode: "connect", connectUrl: `ws://127.0.0.1:${address.port}${path}` } + : { mode: "listen", listenAddress: "0.0.0.0", listenPort: guestPort, listenPath: path }; + if (mode === "connect") { + routes.set(path, { core, token }); + return { connection, release: () => { if (routes.get(path)?.token === token) routes.delete(path); } }; + } + let stopped = false, socket: { close(): void } | undefined; + let readyResolve!: () => void, readyReject!: (error: Error) => void; + const ready = new Promise((done, reject) => { readyResolve = done; readyReject = reject; }); + void ready.catch(() => undefined); + const connect = async () => { + const deadline = Date.now() + 8_000; + while (!stopped && Date.now() < deadline) { + try { + const peer = await openOwnedWire(`ws://127.0.0.1:${guestPort}${path}`); socket = peer; + if (stopped) { peer.close(); return; } + core.attachWireConnection(peer.wire); readyResolve(); await peer.closed; + } catch { /* The target listener appears only after the old receipt ACK. */ } + // Match the production ingress connector's first retry delay (250ms). + if (!stopped) await new Promise(done => setTimeout(done, 250)); + } + if (!stopped) readyReject(new Error("Owned listener did not become available")); + }; + return { connection, activate: () => { void connect(); }, ready: () => ready, + release: () => { stopped = true; socket?.close(); } }; + }; + const options = { runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"), + codexArgs: ["--call-log", callLog, "--record-process-start", "--state-file", join(directory, "fake.json")], + stateDirectory: directory, readRunnerState: async () => JSON.parse(await readFile(join(directory, "runner/runner-state.json"), "utf8")), lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 }, + runnerReconnectGraceMs: 5_000, prpIdentity: source, controlPlaneRegistration: registration, + onStagedRunAttachment: async () => undefined, onRecoveryAccountingEvent: async () => undefined }; + const first = createCapabilityRunnerdCodexTransport(options); + let replacement: ReturnType | undefined; + let runnerPid: number | null = null, providerPid: number | null = null; + try { + const opened = await within(first.transport.request("thread/start", { cwd: root, dynamicTools: [] }), "source open"); + runnerPid = first.evidence().runnerPid; providerPid = first.evidence().providerPid; + const startedAt = first.transport.processInfo().startedAt; + const providerStart = execFileSync("ps", ["-p", String(providerPid), "-o", "lstart="], { encoding: "utf8" }).trim(); + expect(runnerPid).toBeGreaterThan(0); expect(providerPid).toBeGreaterThan(0); + const attach = first.transport.attachRun!({ runId: target.runId, turnId: target.turnId, itemId: target.itemId }); + void attach.catch(() => undefined); + await within(atBoundary, `reach ${boundary}`); + expect(inspectStagedRunAttachmentHeader(checkpoint.records.get("header")!)?.attachment.targetIdentity).toEqual(target); + generation += 1; + // Disconnect the controller only. The bounded failed commit is released + // after its route is retired; a stale callback cannot publish generation B. + const detached = first.detachControllerForRestart(); void detached.catch(() => undefined); + release(); await within(detached, "detach").catch(() => undefined); + const next = join(root, `replacement-${boundary}`); + checkpoint.restoreSessionDirectory(next); + const noSpawn = vi.fn(() => { throw new Error("Duplicate Runner launch"); }); + const recoveryOptions: Parameters[0] = { ...options, stateDirectory: next, + prpIdentity: target, runnerArtifact: readRunnerdArtifactBinding(options.runnerBinary), + runnerStateDirectory: join(directory, "runner"), readRunnerState: async () => JSON.parse(await readFile(join(directory, "runner/runner-state.json"), "utf8")), + prepareExternalRunnerState: async () => { throw new Error("Verified live handoff must not restore a backup"); }, + runnerProcessLauncher: noSpawn, warmTransitionRegistrationMode: mode === "listen" ? "routed_same_endpoint" : "routed_connect", + resumeProviderSession: { driverSessionId: String((opened.thread as { id: string }).id), providerSessionId: String((opened.thread as { sessionId: string }).sessionId) }, + adoptExistingRunner: { pid: runnerPid, processGroupId: runnerPid, startedAt, + isAlive: () => !dead(runnerPid!), signal: signal => { if (dead(runnerPid!)) return true; process.kill(-runnerPid!, signal); return true; } } }; + if (boundary === "queued") { + const noProof = createCapabilityRunnerdCodexTransport({ ...recoveryOptions, adoptExistingRunner: undefined }); + await expect(noProof.transport.request("thread/read", {})).rejects.toThrow("queued_attachment_requires_verified_live_source"); + await noProof.detachControllerForRestart().catch(() => undefined); + checkpoint.restoreSessionDirectory(next); + const wrongEndpoint = createCapabilityRunnerdCodexTransport({ ...recoveryOptions, + controlPlaneRegistration: async (core, identity) => { + const registered = await registration(core, identity); + if (identity?.runId !== target.runId) return registered; + return { ...registered, connection: mode === "listen" + ? { mode: "listen", listenAddress: "0.0.0.0", listenPort: guestPort + 1, listenPath: `/api/runner/v1/connect/${target.runId}` } + : { mode: "connect", connectUrl: `ws://127.0.0.1:${address.port}/changed` } }; + } }); + await expect(wrongEndpoint.transport.request("thread/read", {})).rejects.toThrow("registered_endpoint_mismatch"); + await wrongEndpoint.detachControllerForRestart().catch(() => undefined); + expect(noSpawn).not.toHaveBeenCalled(); + checkpoint.restoreSessionDirectory(next); + } + replacement = createCapabilityRunnerdCodexTransport(recoveryOptions); + const read = await within(replacement.transport.request("thread/read", {}), `recover ${boundary}`); + expect(read.thread).toMatchObject({ id: (opened.thread as { id: string }).id, + sessionId: (opened.thread as { sessionId: string }).sessionId }); + expect(dead(providerPid!)).toBe(false); + expect(execFileSync("ps", ["-p", String(providerPid), "-o", "lstart="], { encoding: "utf8" }).trim()).toBe(providerStart); + expect(replacement.evidence().runnerPid).toBe(runnerPid); + expect(currentCore.activeRunnerConnectionCount()).toBe(1); + expect(currentCore.store.state.identity).toEqual(target); + expect(currentCore.store.state.stagedRunAttachment).toBeDefined(); + expect(currentCore.markStagedTargetCheckpointPublished({ ...target, itemId: "wrong" })).toBe(false); + await currentCore.flushCheckpoint(); + const beforeMark = checkpoint; + expect(currentCore.markStagedTargetCheckpointPublished(target)).toBe(true); + await currentCore.flushCheckpoint(); expect(checkpoint).toBe(beforeMark); + await within(replacement.transport.request("thread/goal/get", {}), "post-checkpoint read"); + expect(currentCore.store.state.stagedRunAttachment).toBeUndefined(); + expect(noSpawn).not.toHaveBeenCalled(); + const effects = (await readFile(callLog, "utf8")).trim().split(/\r?\n/); + expect(effects.filter(line => line === "process-start")).toHaveLength(1); + expect(effects.filter(line => line === "thread/start")).toHaveLength(1); + expect(effects.filter(line => line === "turn/start")).toHaveLength(0); + } catch (error) { + const physical = JSON.parse(await readFile(join(directory, "runner/runner-state.json"), "utf8").catch(() => "{}")); + throw new Error(`${mode}/${boundary}: ${String(error)}; ${JSON.stringify({ + active: currentCore?.activeRunnerConnectionCount(), coreRun: currentCore?.store.state.identity.runId, + coreWarm: currentCore?.store.state.warmTransition?.phase, header: checkpoint && inspectStagedRunAttachmentHeader(checkpoint.records.get("header")!)?.phase, + runnerRun: physical.runId, runnerWarm: physical.warmTransition?.phase, lifecycle: physical.lifecycle, + failureCode: physical.recoverableFailure?.code, failureKeys: Object.keys(physical.recoverableFailure ?? {}), + })}`, { cause: error }); + } finally { + release(); + if (!runnerPid && !replacement) { await first.transport.close().catch(() => undefined); await first.retireProbeResources?.(1_000).catch(() => undefined); } + else await first.detachControllerForRestart().catch(() => undefined); + await replacement?.transport.close().catch(() => undefined); + for (const pid of [providerPid, runnerPid]) if (pid && !dead(-pid)) { try { process.kill(-pid, "SIGKILL"); } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; } } + for (const core of cores) await core.stop().catch(() => undefined); + for (const pid of [providerPid, runnerPid]) if (pid) await vi.waitFor(() => expect(dead(pid)).toBe(true)); + } + } + } finally { http.closeAllConnections(); await new Promise(done => http.close(() => done())); await rm(root, { recursive: true, force: true }); } +}, 90_000); diff --git a/scripts/benchmark-native-controller-recovery.ts b/scripts/benchmark-native-controller-recovery.ts new file mode 100644 index 00000000000..8681954b34c --- /dev/null +++ b/scripts/benchmark-native-controller-recovery.ts @@ -0,0 +1,996 @@ +import fs from "node:fs"; +import { syncBuiltinESMExports } from "node:module"; +import assert from "node:assert/strict"; +import { randomUUID, randomBytes, createHash } from "node:crypto"; +import { createRequire } from "node:module"; +import { mkdir, writeFile, readFile } from "node:fs/promises"; +import { monitorEventLoopDelay } from "node:perf_hooks"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { execFileSync } from "node:child_process"; + +// Run only against an explicitly disposable, otherwise idle Postgres cluster. +// pg_stat_statements must be preloaded; CPU/memory measurements use its cgroup. +const repoRoot = fileURLToPath(new URL("../", import.meta.url)); +function required(name: string) { + const value = process.env[name]; + assert(value, `Set ${name}`); + return value; +} +const baselineRoot = path.resolve(required("BENCH_BASELINE_ROOT")); +const root = path.resolve(required("BENCH_REPORT_ROOT")); +const url = required("BENCH_DATABASE_URL"); +const parsedUrl = new URL(url); +assert( + ["postgres:", "postgresql:"].includes(parsedUrl.protocol) && + ["127.0.0.1", "localhost", "[::1]"].includes(parsedUrl.hostname), + "Use an explicit loopback Postgres URL", +); +assert.equal(process.env.BENCH_DISPOSABLE_POSTGRES, "1", "The whole cluster must be owned by this benchmark"); +const mode = process.argv[2] ?? "steady"; +assert(["pilot", "steady", "burst"].includes(mode)); +const pgCgroup = required("BENCH_POSTGRES_CGROUP"); +const [cpuQuota, cpuPeriod] = fs.readFileSync(path.join(pgCgroup, "cpu.max"), "utf8").trim().split(/\s+/).map(Number); +assert.equal(cpuQuota / cpuPeriod, 2, "The reference workload requires two Postgres CPUs"); +assert.equal( + Number(fs.readFileSync(path.join(pgCgroup, "memory.max"), "utf8")), + 1536 * 1024 * 1024, + "The reference workload requires a 1536 MiB Postgres memory limit", +); +const load = (base: string, file: string) => import(pathToFileURL(path.join(base, file)).href); +const { + completionContracts, + createDb, + livenessDb, + checkpointDb, + closeRegisteredClients, + migratePostgresIfEmpty, + companies, + agents, + issues, + heartbeatRuns, + nativeRunFinalizations, +} = await load(repoRoot, "packages/db/src/index.ts"); +const { nativeRunnerCheckpointWriter: CandidateWriter } = await load( + repoRoot, + "server/src/services/native-runtime/native-runner-checkpoint.ts", +); +const { nativeRunnerCheckpointWriter: BaselineWriter } = await load( + baselineRoot, + "server/src/services/native-runtime/native-runner-checkpoint.ts", +); +const { startNativeSessionExecutionLeaseRenewal } = await load( + repoRoot, + "server/src/services/native-runtime/native-execution-lease.ts", +); +const { loadAgentRunAuthentication } = await load(repoRoot, "server/src/middleware/agent-run-authentication.ts"); +const { DurablePrpControlPlane: CandidateCore } = await load( + repoRoot, + "packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts", +); +const { DurablePrpControlPlane: BaselineCore } = await load( + baselineRoot, + "packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts", +); +const { PaperclipControlPlanePort: CandidatePort } = await load( + repoRoot, + "server/src/services/native-runtime/paperclip-control-plane-port.ts", +); +const { PaperclipControlPlanePort: BaselinePort } = await load( + baselineRoot, + "server/src/services/native-runtime/paperclip-control-plane-port.ts", +); +const require = createRequire(path.join(repoRoot, "server/package.json")); +const postgres = createRequire(path.join(repoRoot, "packages/db/package.json"))("postgres"); +const { and, eq, desc } = require("drizzle-orm"); +const pressure = (base: string, name: string) => { + const file = path.join(base, name + ".pressure"); + return fs.existsSync(file) ? fs.readFileSync(file, "utf8").trim() : null; +}; +const pgResources = () => { + const raw = fs.readFileSync(path.join(pgCgroup, "cpu.stat"), "utf8"); + return { + cpuUsec: Number(raw.match(/usage_usec (\d+)/)![1]), + throttledUsec: Number(raw.match(/throttled_usec (\d+)/)?.[1] ?? 0), + throttledPeriods: Number(raw.match(/nr_throttled (\d+)/)?.[1] ?? 0), + cpuPressure: pressure(pgCgroup, "cpu"), + memoryPressure: pressure(pgCgroup, "memory"), + memoryBytes: Number(fs.readFileSync(path.join(pgCgroup, "memory.current"), "utf8")), + }; +}; +const revision = (directory: string) => + execFileSync("git", ["rev-parse", "HEAD"], { + cwd: directory, + encoding: "utf8", + }).trim(); +const provenance = { + harnessSha256: createHash("sha256").update(fs.readFileSync(fileURLToPath(import.meta.url))).digest("hex"), + lockObserver: { intervalMs: 25, nominalQueriesPerSecond: 40, includedInPostgresCpu: true }, + baselineRevision: process.env.BENCH_BASELINE_REVISION ?? revision(baselineRoot), + candidateRevision: process.env.BENCH_CANDIDATE_REVISION ?? revision(repoRoot), + candidateDiff: + process.env.BENCH_CANDIDATE_DIFF ?? + execFileSync("git", ["diff", "HEAD", "--", "server/src/services/native-runtime", "packages/db/src"], { + cwd: repoRoot, + encoding: "utf8", + }), +}; +await mkdir(root); // Refuse to overwrite earlier measurements. +await writeFile(path.join(root, "provenance.json"), JSON.stringify(provenance, null, 2)); +const admin = postgres(url, { + max: 1, + onnotice: () => {}, + connection: { application_name: "benchmark-observer" }, +}); +const ownedPrefix = "native_recovery_bench_" + randomUUID().replaceAll("-", ""); +const templateName = ownedPrefix + "_template"; +const databaseUrl = (name: string) => { + const value = new URL(url); + value.pathname = "/" + name; + return value.toString(); +}; +process.env.PAPERCLIP_HOME = root + "/home"; +process.env.PAPERCLIP_SECRETS_MASTER_KEY = randomBytes(32).toString("hex"); +const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); +const summary = (a: number[]) => { + const b = [...a].sort((x, y) => x - y); + return { + n: b.length, + p50: b[Math.floor(b.length * 0.5)] ?? 0, + p95: b[Math.floor(b.length * 0.95)] ?? 0, + p99: b[Math.floor(b.length * 0.99)] ?? 0, + max: b.at(-1) ?? 0, + }; +}; +// Sampling reports bounded estimates for observed episodes, never exact lock time. +// A wait shorter than a polling gap can be missed entirely. Query text is excluded. +function observeLockWaits(monitor: any, intervalMs = 25) { + const active = new Map(); + const episodes: any[] = []; + const gaps: number[] = []; + const errors: string[] = []; + let previous = performance.now(), stopped = false; + let poll: Promise | undefined; + let samples = 0; + const collect = async () => { + const rows = await monitor`select a.pid,a.application_name,a.query_start::text as query_start,a.wait_event,l.waitstart::text as wait_start,extract(epoch from clock_timestamp()-l.waitstart)*1000 as wait_age_ms from pg_stat_activity a join pg_locks l on l.pid=a.pid and not l.granted and l.waitstart is not null where a.datname=current_database() and a.application_name!='benchmark-observer' and a.wait_event_type='Lock'`; + const now = performance.now(), gap = now - previous; + gaps.push(gap); + samples++; + const seen = new Set(); + for (const row of rows) { + const key = `${row.pid}/${row.query_start}/${row.wait_start}/${row.wait_event}`; + seen.add(key); + const old = active.get(key); + if (old) { old.lastMs = now; old.wait_age_ms = Number(row.wait_age_ms); } + else active.set(key, { ...row, firstMs: now, lastMs: now, precedingGapMs: gap }); + } + for (const [key, episode] of active) { + if (seen.has(key)) continue; + episodes.push({ ...episode, observedLowerMs: Number(episode.wait_age_ms), + estimatedUpperMs: Number(episode.wait_age_ms) + now - episode.lastMs, rightCensored: false }); + active.delete(key); + } + previous = now; + }; + const timer = setInterval(() => { + if (stopped || poll) return; + poll = collect().catch((e: any) => errors.push(e.message)).finally(() => { poll = undefined; }); + }, intervalMs); + return { async stop() { + stopped = true; + clearInterval(timer); + await poll; + await collect().catch((e: any) => errors.push(e.message)); + for (const episode of active.values()) episodes.push({ ...episode, + observedLowerMs: Number(episode.wait_age_ms), estimatedUpperMs: null, rightCensored: true }); + const byPool = Object.fromEntries(["benchmark-application", "paperclip-controller-lease", "paperclip-runner-checkpoint"].map(pool => { + const selected = episodes.filter(e => e.application_name === pool); + return [pool, { observedEpisodes: selected.length, + observedLowerMs: summary(selected.map(e => e.observedLowerMs)), + estimatedUpperMs: summary(selected.filter(e => e.estimatedUpperMs !== null).map(e => e.estimatedUpperMs)) }]; + })); + return { intervalMs, samples, gapsMs: summary(gaps), errors, byPool, episodes, + limits: "Episodes use PostgreSQL pg_locks.waitstart identity. Lower bounds are server-reported wait age at the last observation; estimated upper bounds add the next client polling gap. Waits shorter than polling gaps can be missed; network/event-loop delay widens upper bounds. Observer nominally adds 40 queries/second, included in database CPU. SQL execution statistics include lock time and are not lock-exclusive." }; + } }; +} +async function setup() { + await admin.unsafe(`create database ${templateName}`); + const u = databaseUrl(templateName); + await migratePostgresIfEmpty(u); + const db = createDb(u); + await db.$client.unsafe("create extension if not exists pg_stat_statements"); + const companyId = randomUUID(), + agentId = randomUUID(); + await db.insert(companies).values({ + id: companyId, + name: "Synthetic normal-operation benchmark", + issuePrefix: "BENCH", + requireBoardApprovalForNewAgents: false, + }); + await db.insert(agents).values({ + id: agentId, + companyId, + name: "Benchmark fixture", + adapterType: "paperclip_runner", + adapterConfig: { provider: "codex" }, + status: "active", + }); + for (let batch = 0; batch < 20; batch++) + await db.insert(issues).values( + Array.from({ length: 500 }, (_, j) => ({ + companyId, + title: `Historical task ${batch * 500 + j}`, + status: j % 3 ? "done" : "backlog", + identifier: `BENCH-${batch * 500 + j + 1}`, + issueNumber: batch * 500 + j + 1, + description: "Synthetic context ".repeat(80), + })), + ); + await db.$client.unsafe("analyze"); + await writeFile(root + "/seed.json", JSON.stringify({ companyId, agentId })); + await closeRegisteredClients(u); +} +async function sampleStats(dbName: string) { + const [stat] = + await admin`select xact_commit,xact_rollback,deadlocks,blks_read,blks_hit,tup_inserted,tup_updated,tup_deleted,temp_bytes from pg_stat_database where datname=${dbName}`; + const [wal] = await admin`select pg_current_wal_insert_lsn()::text as lsn`; + const statements = + await admin`select query,calls,total_exec_time,rows,shared_blks_hit,shared_blks_read,wal_bytes from pg_stat_statements where dbid=(select oid from pg_database where datname=${dbName}) order by calls desc`; + return { stat, wal: wal.lsn, statements }; +} +async function run(variant: string, n: number, trial: number, durationMs: number) { + const label = `${String(trial).padStart(2, "0")}_${variant}_${n}`, + dbName = ownedPrefix + "_" + trial; + await admin.unsafe(`create database ${dbName} template ${templateName}`); + const u = databaseUrl(dbName), + db = createDb(u, { + maxConnections: 10, + applicationName: "benchmark-application", + }); + const seed = JSON.parse(await readFile(root + "/seed.json", "utf8")); + const owner = { + bootId: randomUUID(), + pid: process.pid, + processStartedAt: new Date(), + }; + const items: any[] = [], + metrics: any = { + label, + variant, + n, + durationMs, + setupMs: 0, + errors: [], + dropped: 0, + renewalLosses: 0, + controls: 0, + changedRows: 0, + changedBytes: 0, + checkpointTransactions: 0, + sqlCount: 0, + sqlCategories: {}, + normalRead: [], + normalWrite: [], + databaseAdmission: [], + controlLatency: [], + checkpointLatency: [], + normalSchedulingLag: [], + controlSchedulingLag: [], + normalOffered: 0, + normalCompleted: 0, + normalWritesOffered: 0, + normalWritesCompleted: 0, + admissionsOffered: 0, + admissionsCompleted: 0, + peakPending: 0, + samples: [], + }; + // postgres.js calls onexecute after selecting a connection and encoding BEGIN. + // Preserve that callback. Queue/admission and BEGIN response time are separate; + // SQL execution/lock waits are reported independently by Postgres statistics. + metrics.poolAdmission = []; + metrics.beginResponse = []; + metrics.poolQueueByPool = {}; + metrics.transactionLatencyByPool = {}; + for (const [pool, client] of [ + ["request", db.$client], + ["liveness", livenessDb(db).$client], + ["checkpoint", checkpointDb(db).$client], + ] as const) { + metrics.poolQueueByPool[pool] = []; + metrics.transactionLatencyByPool[pool] = []; + const originalBegin = client.begin, + originalUnsafe = client.unsafe; + let entering: { start: number; sent?: number } | null = null; + client.unsafe = (query, parameters, options) => { + const admission = entering; + if (!admission || !/^begin\b/i.test(query)) return originalUnsafe(query, parameters, options); + return originalUnsafe(query, parameters, { + ...options, + onexecute: (connection) => { + admission.sent = performance.now(); + if (measuring) metrics.poolQueueByPool[pool].push(admission.sent - admission.start); + return options.onexecute(connection); + }, + }); + }; + client.begin = (...args) => { + const admission = { + start: performance.now(), + sent: undefined as number | undefined, + }; + const index = args.length - 1, + work = args[index]; + args[index] = async (connection) => { + if (measuring) { + metrics.poolAdmission.push(performance.now() - admission.start); + if (admission.sent !== undefined) metrics.beginResponse.push(performance.now() - admission.sent); + } + return work(connection); + }; + entering = admission; + try { + return originalBegin(...args).finally(() => { + if (measuring) metrics.transactionLatencyByPool[pool].push(performance.now() - admission.start); + }); + } finally { + entering = null; + } + }; + } + metrics.snapshotLatency = []; + metrics.snapshotTransactions = 0; + let measuring = false; + const originalFsync = fs.fsyncSync; + metrics.fsyncDurations = []; + fs.fsyncSync = function (fd) { + const t = performance.now(); + try { + return originalFsync(fd); + } finally { + if (measuring) metrics.fsyncDurations.push(performance.now() - t); + } + }; + syncBuiltinESMExports(); + db.$client.options.debug = (_conn: any, query: string) => { + if (!measuring) return; + metrics.sqlCount++; + const c = /native_runner_checkpoints/.test(query) + ? "checkpoint" + : /native_run_finalizations/.test(query) + ? "ownership" + : /heartbeat_runs/.test(query) + ? "run" + : /issues/.test(query) + ? "task" + : "other"; + metrics.sqlCategories[c] = (metrics.sqlCategories[c] ?? 0) + 1; + }; + metrics.livenessSqlCount = 0; + metrics.livenessSqlKinds = {}; + { + const live = livenessDb(db); + live.$client.options.debug = (_connection, query) => { + if (!measuring) return; + metrics.livenessSqlCount++; + const kind = query.trim().split(/\s+/)[0].toLowerCase(); + metrics.livenessSqlKinds[kind] = (metrics.livenessSqlKinds[kind] ?? 0) + 1; + }; + } + metrics.checkpointSqlCount = 0; + { + checkpointDb(db).$client.options.debug = (_conn, query) => { + if (measuring) metrics.checkpointSqlCount++; + }; + } + const tSetup = performance.now(), + runNonce = randomUUID(); + for (let i = 0; i < n; i++) { + const runId = randomUUID(), + issueId = randomUUID(), + leaseOwner = randomUUID(), + runnerId = randomUUID(), + sessionId = randomUUID(), + contractId = randomUUID(); + await db.transaction(async (tx: any) => { + await tx.insert(issues).values({ + id: issueId, + companyId: seed.companyId, + title: "Active fixture " + i, + status: "in_progress", + assigneeAgentId: seed.agentId, + }); + await tx.insert(completionContracts).values({ + id: contractId, + companyId: seed.companyId, + issueId, + revision: 1, + schemaVersion: "fixture", + policyVersion: "fixture", + risk: "low", + completionAuthority: "agent", + incompleteCriteriaPolicy: "allow", + contractJson: {}, + canonicalSha256: "fixture", + createdByActorType: "system", + createdByActorId: "benchmark", + }); + await tx.insert(heartbeatRuns).values({ + id: runId, + companyId: seed.companyId, + agentId: seed.agentId, + status: "running", + runtimeMode: "native", + nativeIssueId: issueId, + nativeSessionId: sessionId, + runnerInstanceId: runnerId, + completionContractId: contractId, + completionContractSha256: "fixture", + contextSnapshot: { + issueId, + fixture: "normal-load", + prompt: "x".repeat(4096), + }, + }); + await tx.insert(nativeRunFinalizations).values({ + runId, + companyId: seed.companyId, + issueId, + phase: "observed", + attempt: 1, + leaseOwner, + controllerBootId: owner.bootId, + controllerPid: owner.pid, + controllerProcessStartedAt: owner.processStartedAt, + controllerGeneration: 1, + leaseExpiresAt: new Date(Date.now() + 1200000), + }); + await tx.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + }); + const stateDirectory = (process.env.BENCH_STATE_ROOT ?? root + "/states") + "/" + label + "/" + runNonce + "/" + i; + const Core = variant === "candidate" ? CandidateCore : BaselineCore; + const core = new Core({ + stateDirectory, + identity: { + runnerInstanceId: runnerId, + environmentLeaseId: randomUUID(), + runId, + normalizedSessionId: sessionId, + turnId: randomUUID(), + itemId: randomUUID(), + }, + expectedRunnerVersion: "fixture-v1", + expectedRunnerDigest: "sha256:" + "a".repeat(64), + }); + { + const writer = (variant === "candidate" ? CandidateWriter : BaselineWriter)(db, { + runId, + issueId, + companyId: seed.companyId, + authorityRunId: runId, + bootId: owner.bootId, + generation: 1, + leaseOwner, + }); + core.setCheckpointPersistence(async (cp: any) => { + const start = performance.now(); + await writer(cp); + if (measuring) { + metrics.checkpointTransactions++; + metrics.changedRows += cp.changedRecords.size; + metrics.changedBytes += [...cp.changedRecords.values()].reduce( + (s: number, v: any) => s + Buffer.byteLength(v), + 0, + ); + metrics.checkpointLatency.push(performance.now() - start); + } + }); + await core.flushCheckpoint(); + } + const identity = { + companyId: seed.companyId, + agentId: seed.agentId, + issueId, + runId, + sessionId, + }; + const snapshot = { + backendKind: "codex", + sessionId, + identity, + driverKind: "codex_app_server", + driverSessionId: "thread_" + runId, + providerSessionId: "thread_" + runId, + activeTurnId: null, + terminalTurns: [], + pendingRuntimeRequests: [], + }; + const Port = variant === "candidate" ? CandidatePort : BaselinePort; + const port = new Port( + db, + { + ...identity, + sourceInstanceId: runnerId, + controlPlaneSourceInstanceId: "controller", + completionContractId: contractId, + completionContractSha256: "fixture", + }, + variant === "candidate" + ? { + nativeRecovery: { + compatibility: { + configDigest: "sha256:" + "a".repeat(64), + processConfigDigest: null, + }, + owner: { + companyId: seed.companyId, + issueId, + runId, + authorityRunId: runId, + bootId: owner.bootId, + generation: 1, + leaseOwner, + }, + }, + } + : {}, + ); + await port.openRun({ + identity, + backendKind: "codex", + sourceInstanceId: runnerId, + }); + items.push({ + core, + port, + snapshot, + runId, + issueId, + leaseOwner, + companyId: seed.companyId, + agentId: seed.agentId, + }); + } + metrics.setupMs = performance.now() - tSetup; + const monitor = postgres(u, { + max: 1, + onnotice: () => {}, + connection: { application_name: "benchmark-observer" }, + }); + if (mode === "pilot") { + // Verify the observer against one deliberately held row lock before measured work. + const calibration = observeLockWaits(monitor); + let locked!: () => void, release!: () => void; + const acquired = new Promise(r => { locked = r; }); + const held = new Promise(r => { release = r; }); + const blocker = db.$client.begin(async (tx: any) => { + await tx`select 1 from native_run_finalizations where run_id=${items[0].runId} for update`; + locked(); + await held; + }); + await acquired; + const waiterStarted = performance.now(); + const waiter = checkpointDb(db).$client.begin(async (tx: any) => { + await tx`select 1 from native_run_finalizations where run_id=${items[0].runId} for update`; + }); + // postgres.js BEGIN returns a lazy Promise; observing it starts the contender. + const waiting = Promise.resolve(waiter); + await sleep(250); + const releasedAtMs = performance.now() - waiterStarted; + release(); + await Promise.all([blocker, waiting]); + await sleep(50); + const observed = await calibration.stop(); + metrics.lockCalibration = { intentionalHoldMs: 250, releasedAtMs, + transactionElapsedMs: performance.now() - waiterStarted, observed }; + assert(observed.episodes.some((e: any) => e.application_name === "paperclip-runner-checkpoint" && e.observedLowerMs >= 100), "Lock observer did not detect the deliberate row wait"); + } + await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() + 60000) }); + await db.$client.unsafe("analyze"); + // Warm normal queries before measuring, identical in each arm. + for (let i = 0; i < 100; i++) await loadAgentRunAuthentication(db, items[i % n]); + await admin.unsafe("select pg_stat_statements_reset()"); + const pgBefore = pgResources(); + const before = await sampleStats(dbName), + cpuBefore = process.cpuUsage(), + el = monitorEventLoopDelay({ resolution: 10 }); + el.enable(); + const pending = new Set>(), + renewals: any[] = [], + timers: any[] = []; + function offer(action: () => Promise, out: number[], scheduled = performance.now()) { + if (pending.size >= 500) { + metrics.dropped++; + return; + } + const task = Promise.resolve() + .then(action) + .then(() => out.push(performance.now() - scheduled)) + .catch((e) => metrics.errors.push({ message: e.message, code: e.code })) + .finally(() => pending.delete(task)); + pending.add(task); + metrics.peakPending = Math.max(metrics.peakPending, pending.size); + } + measuring = true; + const lockWaitObserver = observeLockWaits(monitor); + const started = performance.now(), + startedAt = new Date().toISOString(); + await writeFile( + root + "/phase.json", + JSON.stringify({ + label, + phase: "running", + startedAt, + n, + variant, + durationMs, + }), + ); + console.log( + JSON.stringify({ + phase: "started", + label, + setupMs: metrics.setupMs, + startedAt, + }), + ); + for (const item of items) { + const input = { db, ...item, attempt: 1, controller: owner }; + renewals.push( + startNativeSessionExecutionLeaseRenewal({ + ...input, + remote: true, + onLost: async () => { + metrics.renewalLosses++; + }, + }), + ); + } + // Both arms receive the same offered workload. No await-before-next-tick pacing. + let tick = 0; + const offerNormal = () => { + const due = Math.min(Math.floor(durationMs / 100), Math.floor((performance.now() - started) / 100)); + while (tick < due) { + tick++; + metrics.normalSchedulingLag.push(Math.max(0, performance.now() - started - tick * 100)); + for (let k = 0; k < (mode === "burst" ? 10 : 50); k++) { + const idx = (tick * 50 + k) % n, + item = items[idx]; + metrics.normalOffered++; + offer( + async () => { + if (k % 10 < 7) { + const rows = await loadAgentRunAuthentication(db, item); + assert.equal(rows.length, 1); + assert.equal(rows[0].status, "running"); + } else if (k % 10 < 9) { + const rows = await db + .select({ + id: issues.id, + title: issues.title, + status: issues.status, + }) + .from(issues) + .where(and(eq(issues.companyId, seed.companyId), eq(issues.status, "backlog"))) + .orderBy(desc(issues.createdAt)) + .limit(50); + assert.equal(rows.length, 50); + } else { + const rows = await db + .select({ + runId: nativeRunFinalizations.runId, + phase: nativeRunFinalizations.phase, + }) + .from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, item.runId)); + assert.equal(rows[0]?.phase, "observed"); + } + metrics.normalCompleted++; + }, + metrics.normalRead, + started + tick * 100, + ); + } + for (let k = 0; k < (mode === "burst" ? 2 : 10); k++) { + const item = items[(tick * 10 + k) % n]; + metrics.normalWritesOffered++; + offer( + async () => { + await db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, item.issueId)); + metrics.normalWritesCompleted++; + }, + metrics.normalWrite, + started + tick * 100, + ); + } + if (tick % 50 === 0) { + metrics.admissionsOffered++; + offer( + async () => { + const runId = randomUUID(), + issueId = randomUUID(); + await db.transaction(async (tx: any) => { + await tx.insert(issues).values({ + id: issueId, + companyId: seed.companyId, + title: "Fresh admission database probe", + status: "in_progress", + }); + await tx.insert(heartbeatRuns).values({ + id: runId, + companyId: seed.companyId, + agentId: seed.agentId, + status: "queued", + runtimeMode: "native", + nativeIssueId: issueId, + }); + await tx.insert(nativeRunFinalizations).values({ + runId, + companyId: seed.companyId, + issueId, + phase: "observed", + }); + await tx.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + }); + metrics.admissionsCompleted++; + }, + metrics.databaseAdmission, + started + tick * 100, + ); + } + } + }; + const normal = setInterval(offerNormal, 100); + timers.push(normal); + // Same aggregate control rate, spread evenly across sessions for normal operation. + let offeredControls = 0; + const offerControl = () => { + const elapsed = performance.now() - started; + const due = Math.min( + Math.floor((durationMs * n) / 5000), + mode === "burst" ? Math.floor(elapsed / 5000) * n : Math.floor((elapsed * n) / 5000), + ); + while (offeredControls < due) { + const index = offeredControls++, + item = items[index % n], + seq = Math.floor(index / n) + 1; + const scheduled = started + (mode === "burst" ? seq * 5000 : ((index + 1) * 5000) / n); + metrics.controlSchedulingLag.push(Math.max(0, performance.now() - scheduled)); + offer( + async () => { + item.core.queueCommand( + "semantic_tool.result", + { + callId: `fixture_call_${seq}`, + operationId: "fixture_read", + result: { body: "x".repeat(4096) }, + isError: false, + }, + `control_${seq}`, + ); + await item.core.flushCheckpoint(); + const snapshotStarted = performance.now(); + await item.port.checkpointSession(item.snapshot); + metrics.snapshotLatency.push(performance.now() - snapshotStarted); + metrics.snapshotTransactions++; + metrics.controls++; + }, + metrics.controlLatency, + scheduled, + ); + } + }; + const control = setInterval(offerControl, 10); + timers.push(control); + const sampler = setInterval(() => { + void (async () => { + const m = process.memoryUsage(); + const activity = + await monitor`select count(*)::int as connections,count(*) filter(where state='active')::int as active,count(*) filter(where wait_event_type='Lock')::int as locks from pg_stat_activity where datname=current_database() and application_name!='benchmark-observer'`; + metrics.samples.push({ + elapsedMs: performance.now() - started, + rss: m.rss, + heapUsed: m.heapUsed, + postgresMemoryBytes: pgResources().memoryBytes, + ...activity[0], + }); + })().catch((e) => metrics.errors.push({ observer: e.message })); + }, 1000); + timers.push(sampler); + await sleep(durationMs); + offerNormal(); + offerControl(); + for (const t of timers) clearInterval(t); + const drainStart = performance.now(); + await Promise.allSettled([...pending]); + await Promise.allSettled(renewals.map((r) => r.stop())); + metrics.drainMs = performance.now() - drainStart; + metrics.lockWait = await lockWaitObserver.stop(); + metrics.elapsedMs = performance.now() - started; + metrics.startedAt = startedAt; + metrics.endedAt = new Date().toISOString(); + measuring = false; + fs.fsyncSync = originalFsync; + syncBuiltinESMExports(); + metrics.fsyncTotalMs = metrics.fsyncDurations.reduce((a, b) => a + b, 0); + el.disable(); + metrics.nodeCpu = process.cpuUsage(cpuBefore); + metrics.eventLoopDelay = { + p95Ms: el.percentile(95) / 1e6, + p99Ms: el.percentile(99) / 1e6, + maxMs: el.max / 1e6, + }; + // Flush PG statistics after all application activity; observer SQL is tagged/excluded below. + await sleep(1100); + const after = await sampleStats(dbName); + const [wal] = await admin`select pg_wal_lsn_diff(${after.wal}::pg_lsn,${before.wal}::pg_lsn)::text as bytes`; + metrics.walBytes = Number(wal.bytes); + const pgAfter = pgResources(); + metrics.postgresCpuUsec = pgAfter.cpuUsec - pgBefore.cpuUsec; + metrics.postgresResources = { before: pgBefore, after: pgAfter, throttledUsec: pgAfter.throttledUsec - pgBefore.throttledUsec, throttledPeriods: pgAfter.throttledPeriods - pgBefore.throttledPeriods }; + metrics.postgresMemoryBytes = pgAfter.memoryBytes; + metrics.postgresPeakSampledMemoryBytes = Math.max( + pgBefore.memoryBytes, + pgAfter.memoryBytes, + ...metrics.samples.map((s) => s.postgresMemoryBytes), + ); + metrics.platform = { + node: process.version, + os: process.platform, + arch: process.arch, + }; + metrics.databaseStats = Object.fromEntries( + Object.keys(after.stat).map((k) => [k, Number(after.stat[k]) - Number(before.stat[k])]), + ); + metrics.statements = after.statements; + metrics.lockObserverSql = after.statements.filter((s: any) => /pg_locks.*waitstart|waitstart.*pg_locks/.test(s.query)); + metrics.tableStats = + await monitor`select relname,n_tup_ins,n_tup_upd,n_tup_del,n_dead_tup from pg_stat_user_tables where relname in ('native_runner_checkpoints','native_run_finalizations','heartbeat_runs','issues') order by relname`; + const [storage] = + await monitor`select count(*)::int as rows,coalesce(sum(pg_column_size(material)),0)::bigint::text as encrypted_bytes,pg_total_relation_size('native_runner_checkpoints')::text as relation_bytes from native_runner_checkpoints`; + metrics.checkpointStorage = storage; + for (const name of [ + "poolAdmission", + "beginResponse", + "snapshotLatency", + "normalRead", + "normalWrite", + "databaseAdmission", + "controlLatency", + "checkpointLatency", + "normalSchedulingLag", + "controlSchedulingLag", + "fsyncDurations", + ]) { + metrics[name + "Summary"] = summary(metrics[name]); + delete metrics[name]; + } + for (const pool of Object.keys(metrics.poolQueueByPool)) { + metrics.poolQueueByPool[pool] = summary(metrics.poolQueueByPool[pool]); + metrics.transactionLatencyByPool[pool] = summary(metrics.transactionLatencyByPool[pool]); + } + metrics.nodeCpuAverageOneCorePercent = + ((metrics.nodeCpu.user + metrics.nodeCpu.system) / 1000 / metrics.elapsedMs) * 100; + metrics.correctnessPassed = + metrics.errors.length === 0 && + metrics.lockWait.errors.length === 0 && + metrics.renewalLosses === 0 && + metrics.dropped === 0 && + metrics.normalCompleted === metrics.normalOffered && + metrics.normalWritesCompleted === metrics.normalWritesOffered && + metrics.admissionsCompleted === metrics.admissionsOffered; + metrics.absoluteGates = { + requestP99: metrics.normalReadSummary.p99 < 250, + drain: metrics.drainMs < 1000, + throughput: metrics.normalCompleted / metrics.normalOffered >= 0.99, + postgresCpu: metrics.postgresCpuUsec / 1000 / metrics.elapsedMs / 2 < 0.75, + postgresMemory: metrics.postgresPeakSampledMemoryBytes < 1382.4 * 1024 * 1024, + }; + metrics.passed = metrics.correctnessPassed && Object.values(metrics.absoluteGates).every(Boolean); + await writeFile(root + "/" + label + ".json", JSON.stringify(metrics, null, 2) + "\n"); + console.log( + JSON.stringify({ + phase: "complete", + label, + passed: metrics.passed, + reads: metrics.normalReadSummary, + writes: metrics.normalWriteSummary, + control: metrics.controlLatencySummary, + renewalLosses: metrics.renewalLosses, + dropped: metrics.dropped, + cpu: metrics.nodeCpuAverageOneCorePercent, + errors: metrics.errors.slice(0, 3), + }), + ); + await monitor.end(); + await closeRegisteredClients(u); + await admin.unsafe(`drop database ${dbName}`); + return metrics; +} +try { + await admin.unsafe("create extension if not exists pg_stat_statements"); + await setup(); + if (mode === "pilot") { + if (!(await run("candidate", 10, 9, 25000)).passed) process.exitCode = 1; + } else { + const profile = [ + ["baseline", 300], + ["candidate", 300], + ["candidate", 300], + ["baseline", 300], + ] as const; + const result = []; + for (const [variant, n] of profile) result.push(await run(variant, n, result.length + 1, 65000)); + const report = { + schema: "paperclip.normal-recovery-load/v1", + ...provenance, + scope: + "Paired baseline/candidate, including real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.", + hardware: { + postgresCpuLimit: 2, + postgresMemoryLimitMiB: 1536, + mainPool: 10, + livenessPool: 2, + candidateCheckpointPool: 3, + }, + workload: { + mode, + normalReadsPerSecond: mode === "burst" ? 100 : 500, + normalWritesPerSecond: mode === "burst" ? 20 : 100, + databaseAdmissionEverySeconds: 5, + checkpointControlEverySeconds: 5, + controlPayloadBytes: 4096, + phaseSeconds: 65, + snapshotTransactionsPerControl: 1, + baselineCheckpointPool: 3, + baselinePoolPolicy: "Controlled baseline uses the common current DB factory's checkpoint cap of 3; this is not the unmodified historical cap-2 baseline.", + baselineRenewalIntervalSeconds: 20, + candidateRenewalIntervalSeconds: 20, + historicalIssues: 10000, + }, + results: result, + }; + const perOperation = (r: any, key: string) => + r[key] / (r.normalCompleted + r.normalWritesCompleted + r.controls + r.admissionsCompleted); + const average = (values: number[]) => values.reduce((a, b) => a + b, 0) / values.length; + const comparable = + result.every((r) => r.correctnessPassed) && + result.every( + (r) => + r.controls === result[0].controls && + r.normalCompleted === result[0].normalCompleted && + r.normalWritesCompleted === result[0].normalWritesCompleted, + ); + const relative = {}; + for (const key of ["postgresCpuUsec", "walBytes", "requestP99"]) { + const value = (r: any) => (key === "requestP99" ? r.normalReadSummary.p99 : perOperation(r, key)); + const baseline = average(result.filter((r) => r.variant === "baseline").map(value)); + const candidate = average(result.filter((r) => r.variant === "candidate").map(value)); + relative[key] = { + baseline, + candidate, + increasePercent: comparable ? 100 * (candidate / baseline - 1) : null, + passed: comparable ? candidate <= baseline * 1.1 : null, + }; + } + Object.assign(report, { + relative, + comparable, + passed: result.every((r) => r.passed) && Object.values(relative).every((r: any) => r.passed), + measurementLimits: [ + "Database request latency is not HTTP latency", + "Pool queue includes client encoding; BEGIN response includes network time", + "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed", + "Peak memory is sampled once a second", + "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide", + ], + }); + await writeFile(root + "/report.json", JSON.stringify(report, null, 2) + "\n"); + if (!(report as typeof report & { passed: boolean }).passed) process.exitCode = 1; + } +} finally { + await admin.end({ timeout: 1 }); +} +// On failure, retain owned databases for diagnosis. Successful trials are dropped; +// the random template remains available for fixture inspection until cluster cleanup. diff --git a/server/src/__tests__/heartbeat-execution-configuration.test.ts b/server/src/__tests__/heartbeat-execution-configuration.test.ts index 2cd52f84849..a38d0151087 100644 --- a/server/src/__tests__/heartbeat-execution-configuration.test.ts +++ b/server/src/__tests__/heartbeat-execution-configuration.test.ts @@ -2,18 +2,19 @@ import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js"; import { createHash, randomUUID } from "node:crypto"; import { trace } from "@opentelemetry/api"; import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace"; -import { access, chmod, mkdir, readdir, readFile, mkdtemp, rm, stat } from "node:fs/promises"; +import { access, chmod, mkdir, readdir, readFile, mkdtemp, rm, stat, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; import { eq } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; -import { agents, authUsers, companies, companyMemberships, companySkills, companySkillTestRuns, createDb, heartbeatRunEvents, heartbeatRuns, issues, projects, plugins, environments } from "@paperclipai/db"; +import { agents, authUsers, companies, companyMemberships, companySkills, companySkillTestRuns, createDb, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, issues, projects, plugins, environments } from "@paperclipai/db"; import { companySkillService } from "../services/company-skills.js"; import { withSkillVersionCacheLock } from "../services/skill-version-cache.js"; import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js"; import { HttpError } from "../errors.js"; import { heartbeatService } from "../services/heartbeat.js"; import { bindHeartbeatRunnerHomeMode, prepareHeartbeatExecutionConfiguration, type HeartbeatExecutionConfigurationInput } from "../services/heartbeat/execution-configuration.js"; +import { buildEffectiveRunSessionConfigMetadata, resolveTaskSessionConfigFreshness } from "../services/heartbeat/workspaces.js"; import { aiRuntimeConnectionBindingSchema } from "@paperclipai/shared"; import { secretService } from "../services/secrets.js"; import { instanceSettingsService } from "../services/instance-settings.js"; @@ -24,6 +25,12 @@ import * as assignedMcp from "../services/native-runtime/assigned-mcp-tools.js"; import * as sentry from "../sentry.js"; import { waitForPendingRunFailureReports } from "../services/run-failure-report.js"; import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js"; +import { selectHeartbeatEnvironment, type HeartbeatEnvironmentSelectionInput } from "../services/heartbeat/environment-selection.js"; +import { environmentService } from "../services/environments.js"; +import { executionWorkspaceService } from "../services/execution-workspaces.js"; +import { buildExecutionWorkspaceAdapterConfig } from "../services/execution-workspace-policy.js"; +import { buildNativeExecutionInput } from "../services/native-runtime/native-execution-input.js"; +import { nativeRuntimeContextFixture } from "../services/native-runtime/runtime-context.test-fixture.js"; function deferred() { let resolve!: (value: T) => void; @@ -35,6 +42,44 @@ const skillFailures = vi.hoisted(() => new Map()); const skillWaits = vi.hoisted(() => new Map Promise>()); describe("Runner home mode during recovery preparation", () => { + const fingerprint = (effectiveAdapterConfig: Record) => buildEffectiveRunSessionConfigMetadata({ + adapterType: "paperclip_runner", effectiveAdapterConfig, agentRuntimeConfig: {}, issueOverrides: null, + workspaceConfig: {}, environment: null, environmentEnv: null, projectEnv: null, routineEnv: null, runtimeSkills: [], + }); + it.each([ + { schema: "paperclip.native-execution-input.v5" }, + { schema: "paperclip.native-execution-input.v7", homeMode: "isolated" }, + ])("preserves the implicit isolated fingerprint through recovery and the next run: %j", async saved => { + const config = { provider: "codex", model: "fixture-model" }; + // This is the effective configuration shape built by preparation, including + // its optional property when no explicit home mode was configured. + const prepare = (input: unknown) => ({ ...config, homeMode: bindHeartbeatRunnerHomeMode(config, input).homeMode }); + const fresh = await fingerprint(prepare(null)); + const recovered = await fingerprint(prepare(saved)); + const next = await fingerprint(prepare(null)); + expect(recovered.fingerprint).toBe(fresh.fingerprint); + expect(next.fingerprint).toBe(recovered.fingerprint); + expect(resolveTaskSessionConfigFreshness({ hasTaskSession: true, configuredModel: config.model, + taskSessionParams: { __paperclipConfiguredModel: config.model, __paperclipConfigFingerprint: recovered.fingerprint, + __paperclipConfigFingerprintVersion: recovered.version, __paperclipConfigCategoryFingerprints: recovered.categoryFingerprints }, + configMetadata: next })).toMatchObject({ reset: false, changedCategories: [] }); + expect(config).not.toHaveProperty("homeMode"); + }); + + it("pins the admitted isolated mode but invalidates a genuinely preconfigured next run", async () => { + const config = { provider: "codex", model: "fixture-model", homeMode: "preconfigured" }; + const recoveredConfig = bindHeartbeatRunnerHomeMode(config, { schema: "paperclip.native-execution-input.v5" }); + expect(recoveredConfig.homeMode).toBe("isolated"); + expect(bindHeartbeatRunnerHomeMode(config, null).homeMode).toBe("preconfigured"); + const recovered = await fingerprint(recoveredConfig); + const next = await fingerprint(config); + expect(next.fingerprint).not.toBe(recovered.fingerprint); + expect(resolveTaskSessionConfigFreshness({ hasTaskSession: true, configuredModel: config.model, + taskSessionParams: { __paperclipConfiguredModel: config.model, __paperclipConfigFingerprint: recovered.fingerprint, + __paperclipConfigFingerprintVersion: recovered.version, __paperclipConfigCategoryFingerprints: recovered.categoryFingerprints }, + configMetadata: next })).toMatchObject({ reset: true, changedCategories: ["adapterConfig"] }); + }); + it.each([ ["preconfigured", { schema: "paperclip.native-execution-input.v5" }, "isolated"], ["preconfigured", { schema: "paperclip.native-execution-input.v7", homeMode: "isolated" }, "isolated"], @@ -395,6 +440,118 @@ describePostgres("heartbeat execution configuration through adapter dispatch", ( expect(initial.configSnapshot).toEqual(reused.configSnapshot); }); + it("keeps shared provider workspace configuration across native recovery and the next turn", async () => { + const f = await seed(); + const selectedSkills = []; + for (const name of ["Alphabetically first long skill", "B longer selected skill", "C", "D small"]) { + selectedSkills.push(await companySkillService(db).createLocalSkill(f.companyId, { name, markdown: "# Keep the saved result\n" })); + } + const instructionsRootPath = path.join(home, f.companyId); + await mkdir(instructionsRootPath); + await writeFile(path.join(instructionsRootPath, "AGENTS.md"), "Preserve the workspace across controller replacement.\n"); + const config = { provider: "codex", model: "fixture-model", lifecycleMode: "warm", idleTimeoutMs: 300_000, + codexPermissionMode: "full", paperclipSkillSync: { desiredSkills: selectedSkills.map(skill => skill.key) }, + instructionsRootPath, instructionsFilePath: path.join(instructionsRootPath, "AGENTS.md"), + instructionsEntryFile: "AGENTS.md", instructionsBundleMode: "managed", env: { OPENAI_API_KEY: "fixture-direct-key" } }; + const [environment] = await db.insert(environments).values({ name: `Directory ${f.companyId}`, driver: "plugin", + config: { pluginKey: "test.directory", driverKey: "tasks" } }).returning(); + const [agent] = await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: config, + defaultEnvironmentId: environment.id }).where(eq(agents.id, f.agentId)).returning(); + const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, + issueId: f.issueId, scopeKind: "issue", status: "running", runtimeMode: "native", responsibleUserId: f.userId, contextSnapshot: {} }).returning(); + const workspaceId = run.id; + const [workspace] = await db.insert(executionWorkspaces).values({ id: workspaceId, companyId: f.companyId, + projectId: issue.projectId, sourceIssueId: issue.id, name: "Shared task workspace", mode: "shared_workspace", + strategyType: "project_primary", providerType: "adapter_managed", cwd: `/workspaces/${workspaceId}` }).returning(); + const issueSettings = { mode: "shared_workspace" as const }; + const sourceConfig = (current: Record) => buildExecutionWorkspaceAdapterConfig({ agentConfig: current, + projectPolicy: null, issueSettings, mode: "shared_workspace", legacyUseProjectWorkspace: null }); + const selectionInput: HeartbeatEnvironmentSelectionInput = { + run, agent, task: { issueRef: { ...issue, executionWorkspaceId: null }, issueContext: issue, issueId: issue.id, + executionProjectId: issue.projectId, workspaceProjectContext: { hasWorkspace: false }, boundSourceWorkspace: null, + hasSelectedWorkspaceSource: false, taskWorkspaceIntent: null, responsibleUserId: f.userId, context: {}, + freshWorkspace: null }, + policy: { trustPreset: { kind: "standard", preset: "standard", boundary: null, sourcePresets: {} }, + requestedExecutionWorkspaceMode: "shared_workspace", isolatedWorkspacesEnabled: true, + projectExecutionWorkspacePolicy: null, issueExecutionWorkspaceSettings: issueSettings }, + config: { config, sourceWorkspaceConfig: sourceConfig(config), issueAssigneeOverrides: null }, + services: { environmentsSvc: environmentService(db), executionWorkspacesSvc: executionWorkspaceService(db), + instanceSettings: instanceSettingsService(db), findSharedWorkspaceHolder: vi.fn(async () => null), + findExecutionWorkspaceHolder: vi.fn(async () => null) }, + }; + const prepare = async () => { + const selected = await selectHeartbeatEnvironment(db, selectionInput); + const result = await prepareHeartbeatExecutionConfiguration(db, { run, agent, + task: { issueId: issue.id, context: {}, responsibleUserId: f.userId, acceptedPlanContinuationWake: false, + acceptedPlanWakeRoutingDecision: null, taskProjectContext: null, + routineEnvContext: { routineId: null, env: null, responsibleUserId: null } }, + config: { sourceWorkspaceConfig: selectionInput.config.sourceWorkspaceConfig, aiBinding: undefined }, + policy: { trustPreset: selectionInput.policy.trustPreset, useIsolatedTaskDirectory: selected.useIsolatedTaskDirectory, + explicitTaskDirectory: selected.explicitTaskDirectory, requestedExecutionWorkspaceMode: selectionInput.policy.requestedExecutionWorkspaceMode }, + environment: selected, services: { secretsSvc: secretService(db), companySkills: companySkillService(db), + appendRunEvent: vi.fn(), finalizeAiConnectionBusyDeferral: vi.fn() }, + onFailureSecretsResolved: () => {}, onManagedAiRuntimeAcquired: () => {}, onIdentityRedactorPrepared: () => {} }); + if (!result.prepared) throw new Error("Configuration was not prepared"); + const metadata = await buildEffectiveRunSessionConfigMetadata({ adapterType: "paperclip_runner", + effectiveAdapterConfig: result.runtimeConfig, agentRuntimeConfig: {}, issueOverrides: null, workspaceConfig: {}, + environment: null, environmentEnv: null, projectEnv: null, routineEnv: null, runtimeSkills: result.runtimeSkillEntries }); + return { selected, result, metadata }; + }; + const first = await prepare(); + expect(first.selected.useIsolatedTaskDirectory).toBe(false); + expect(first.result.runtimeConfig.workspaceStrategy).toBeUndefined(); + // Preserve the pre-fix fresh name/key query order, including exact versions. + expect(first.result.runtimeConfig.paperclipSkillSync).toEqual({ desiredSkills: selectedSkills.map(skill => ({ + key: skill.key, versionId: skill.currentVersionId, + })) }); + // Read the actual JSONB snapshot rather than reusing the pre-persistence + // JavaScript map: PostgreSQL key order differs for these four real keys. + const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + run.contextSnapshot = saved.contextSnapshot; + selectionInput.task.freshWorkspace = null; + selectionInput.task.issueRef!.executionWorkspaceId = workspaceId; + selectionInput.task.issueRef!.executionWorkspacePreference = "reuse_existing"; + await db.update(issues).set({ executionWorkspaceId: workspaceId, executionWorkspacePreference: "reuse_existing" }).where(eq(issues.id, issue.id)); + selectionInput.run.runnerProfileJson = { nativeExecutionInput: buildNativeExecutionInput({ + companyId: f.companyId, agentId: agent.id, runId: run.id, normalizedSessionId: randomUUID(), provider: "codex", + issue: { id: issue.id, identifier: "CFG-1", title: issue.title, description: null, workMode: "standard" }, + taskPrompt: "Continue the admitted work", workspace: { id: workspaceId, cwd: workspace.cwd!, repoUrl: null, repoRef: null, branchName: null }, + completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", + contract: { revision: "1", objective: "Retain the directory", criteria: [{ id: "objective", requirement: "Preserve the workspace" }] } }, + runtimeContext: nativeRuntimeContextFixture(), + }) }; + const recovered = await prepare(); + expect(recovered.selected.nativeRecoveryExecutionWorkspaceId).toBe(workspaceId); + expect(recovered.selected.useIsolatedTaskDirectory).toBe(false); + expect(recovered.result.runtimeSkillEntries).toEqual(first.result.runtimeSkillEntries); + expect(resolveTaskSessionConfigFreshness({ hasTaskSession: true, configuredModel: config.model, + taskSessionParams: { __paperclipConfiguredModel: config.model, __paperclipConfigFingerprint: first.metadata.fingerprint, + __paperclipConfigFingerprintVersion: first.metadata.version, __paperclipConfigCategoryFingerprints: first.metadata.categoryFingerprints }, + configMetadata: recovered.metadata })).toMatchObject({ reset: false, changedCategories: [] }); + expect(recovered.result.runtimeConfig).toEqual(first.result.runtimeConfig); + expect(recovered.metadata.fingerprint).toBe(first.metadata.fingerprint); + selectionInput.run.runnerProfileJson = {}; + run.contextSnapshot = {}; + const next = await prepare(); + expect(next.result.runtimeConfig).toEqual(first.result.runtimeConfig); + expect(next.metadata.fingerprint).toBe(first.metadata.fingerprint); + + selectionInput.config.sourceWorkspaceConfig = sourceConfig({ ...config, + paperclipSkillSync: { desiredSkills: selectedSkills.slice(1).map(skill => skill.key) } }); + const fewerSkills = await prepare(); + expect(fewerSkills.result.runtimeSkillEntries).toHaveLength(3); + expect(fewerSkills.metadata.fingerprint).not.toBe(first.metadata.fingerprint); + + // A real model change must still invalidate the next admission. + selectionInput.config.config = { ...config, model: "changed-model" }; + selectionInput.config.sourceWorkspaceConfig = sourceConfig(selectionInput.config.config); + const changed = await prepare(); + expect(changed.selected.useIsolatedTaskDirectory).toBe(false); + expect(changed.result.runtimeConfig.model).toBe("changed-model"); + expect(changed.metadata.fingerprint).not.toBe(first.metadata.fingerprint); + }, 30_000); + it.each([ { mode: "shared_workspace" as const, config: { workspaceStrategy: { type: "git_worktree", baseRef: "main" } } }, { mode: "agent_default" as const, config: {} }, diff --git a/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts b/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts index c7a3301ceee..3e952c66126 100644 --- a/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts +++ b/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts @@ -2,6 +2,9 @@ import { createDb } from "@paperclipai/db"; import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js"; import { dockerCommandRunner } from "./docker-command-runner.js"; +// Separate TypeScript fixture import time from actual controller admission. +process.send?.({ ready: true }); + process.once("message", async (input: { connectionString: string; container: string; diff --git a/server/src/__tests__/native-harness-backup-expiration.test.ts b/server/src/__tests__/native-harness-backup-expiration.test.ts index 2437d9711b5..cd242bd6392 100644 --- a/server/src/__tests__/native-harness-backup-expiration.test.ts +++ b/server/src/__tests__/native-harness-backup-expiration.test.ts @@ -412,6 +412,15 @@ describeEmbeddedPostgres("native harness backup expiration", () => { expect(await exists(session.current)).toBe(false); }); + it("retains incomplete native snapshot publications even after the issue closes", async () => { + const session = await seedSession({ finalization: { phase: "committed" } }); + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeControllerRecovery: { + schema: "paperclip.native-controller-recovery.v1", checkpoint: null, + } } }).where(eq(heartbeatRuns.id, session.runId)); + expect(await service().tick()).toMatchObject({ expired: 0, skipped: { native_recovery_incomplete: 1 } }); + expect(await exists(session.current)).toBe(true); + }); + it.each([ ["committed", "resuming_session"], ["terminal_failure", "blocked"], diff --git a/server/src/__tests__/native-session-resumption.test.ts b/server/src/__tests__/native-session-resumption.test.ts index 03ae5e6691c..47b8e01dfdc 100644 --- a/server/src/__tests__/native-session-resumption.test.ts +++ b/server/src/__tests__/native-session-resumption.test.ts @@ -9,12 +9,14 @@ import { eq, inArray } from "drizzle-orm"; import { activityLog, agentWakeupRequests, + agentTaskSessions, agents, companies, authUsers, companyMemberships, completionContracts, createDb, + closeRegisteredClients, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, @@ -78,6 +80,9 @@ vi.mock("../sentry.js", async () => { return { ...actual, captureRunFailure: mockCaptureRunFailure }; }); +import { createIsolatedTestHome } from "./helpers/isolated-test-home.js"; +import { issueService } from "../services/issues.js"; +import * as heartbeatWorkspaces from "../services/heartbeat/workspaces.js"; import { heartbeatService } from "../services/heartbeat.js"; import { persistHeartbeatRunProcessMetadata } from "../services/heartbeat/run-lifecycle.js"; import { instanceSettingsService } from "../services/instance-settings.js"; @@ -1512,4 +1517,165 @@ describe.each(["unchanged", "newer_active", "stale_idle"] as const)( ); + +describe("goal-less native task-session publication", () => { + let database: Awaited> | undefined; + let db: ReturnType; + beforeAll(async () => { + const external = process.env.PAPERCLIP_TEST_DATABASE_URL; + if (external) db = createDb(external); + else { + database = await startEmbeddedPostgresTestDatabase("paperclip-native-pointer-"); + db = createDb(database.connectionString); + } + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + afterAll(async () => { + await closeRegisteredClients(process.env.PAPERCLIP_TEST_DATABASE_URL ?? database!.connectionString); + await database?.cleanup(); + }); + + it.each(["retained", "reset"] as const)("publishes the completed task session before provider cleanup and resumes an ordinary comment (%s)", async mode => { + const home = createIsolatedTestHome(); + const previousHome = process.env.PAPERCLIP_HOME; + process.env.PAPERCLIP_HOME = home.home; + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(); + const issuePrefix = `NPT${companyId.slice(0, 6).toUpperCase()}`; + const projectId = randomUUID(), projectWorkspaceId = randomUUID(), workProductId = randomUUID(); + let providerSessionId: string | undefined; + const repoRoot = fileURLToPath(new URL("../../../", import.meta.url)); + const evidenceRef = `work_product:${workProductId}`; + const inputs: NativeExecutionInput[] = []; + const configs: Record[] = []; + const buildMetadata = heartbeatWorkspaces.buildEffectiveRunSessionConfigMetadata; + const configuration = vi.spyOn(heartbeatWorkspaces, "buildEffectiveRunSessionConfigMetadata").mockImplementation(input => { + configs.push(structuredClone(input.effectiveAdapterConfig)); + return buildMetadata(input); + }); + const openSession = vi.fn(), recoverSession = vi.fn(), startTurn = vi.fn(); + let releaseClose!: () => void, notifyClose!: () => void; + const closeHeld = new Promise(resolve => { releaseClose = resolve; }); + const closeEntered = new Promise(resolve => { notifyClose = resolve; }); + let firstRunId: string | undefined; + const backendFactory = (execution: NativeExecutionInput): NativeSessionBackend => { + inputs.push(structuredClone(execution)); + firstRunId ??= execution.binding.runId; + const currentRunId = execution.binding.runId; + const currentProviderSessionId = `provider-${execution.session.normalizedSessionId}`; + providerSessionId ??= currentProviderSessionId; + const turnId = `turn-${currentRunId}`; + const identity = { companyId, agentId, issueId, runId: currentRunId, sessionId: execution.session.normalizedSessionId }; + const result = structuredClone(CONTROL_PLANE_CONFORMANCE_RESULT); + result.completionClaim.contractRevision = execution.completionContract.contract.revision; + result.completionClaim.criteria = execution.completionContract.contract.criteria.map(criterion => ({ + ...result.completionClaim.criteria[0]!, criterionId: criterion.id, evidenceRefs: [evidenceRef], + })); + result.evidence = [{ kind: "work_product", ref: evidenceRef }]; + result.verification[0]!.artifactRef = evidenceRef; + let started!: () => void; + const turnStarted = new Promise(resolve => { started = resolve; }); + const session: NativeSession = { + identity: () => identity, + async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; }, + async *events() { + await turnStarted; + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, currentRunId)); + yield { schema: "paperclip.prp.event.v1", sourceEventId: `${currentRunId}:terminal`, sourceSeq: 1, + sourceInstanceId: run.runnerInstanceId!, sourceKind: "runner", runId: currentRunId, + normalizedSessionId: identity.sessionId, turnId, eventType: "turn.completed", schemaVersion: 1, + priority: 0, emittedAt: new Date().toISOString(), payload: {} } satisfies PrpEvent; + }, + async startTurn(input) { startTurn(input); started(); return { turnId }; }, + async result() { await turnStarted; return { result, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId }; }, + async snapshot() { return { backendKind: "mock", sessionId: "driver-retained", identity, providerSessionId: currentProviderSessionId, + activeTurnId: null, pendingRuntimeRequests: [], lineage: [], cursor: "1" }; }, + async close() { if (currentRunId === firstRunId) { notifyClose(); await closeHeld; } }, + }; + return { + async descriptor() { return { kind: "mock", name: "task-pointer-backend", version: "1", + capabilities: { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }, + runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } }; }, + async openSession(input) { openSession(input); return session; }, + async recoverSession(snapshot) { recoverSession(snapshot); expect(snapshot.providerSessionId).toBe(currentProviderSessionId); + return { recovered: true, session }; }, + }; + }; + const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: backendFactory }); + try { + await instanceSettingsService(db).updateExperimental({ enableNativeRunner: true, enableIsolatedWorkspaces: true }); + await db.insert(authUsers).values({ id: "pointer-user", name: "Pointer user", email: "pointer@example.test", createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing(); + await db.insert(companies).values({ id: companyId, name: "Native pointer", issuePrefix, status: "active", defaultResponsibleUserId: "pointer-user" }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "pointer-user", status: "active", membershipRole: "member" }); + await db.insert(projects).values({ id: projectId, companyId, name: "Pointer project", status: "active" }); + await db.insert(projectWorkspaces).values({ id: projectWorkspaceId, companyId, projectId, name: "Pointer workspace", cwd: repoRoot, isPrimary: true }); + await db.insert(agents).values({ id: agentId, companyId, name: "Pointer agent", adapterType: "paperclip_runner", status: "active", + adapterConfig: { provider: "codex", workspaceStrategy: { type: "project_primary" } }, + runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }); + await db.insert(issues).values({ id: issueId, companyId, projectId, projectWorkspaceId, issueNumber: 1, + identifier: `${issuePrefix}-1`, title: "Deliver the saved artifact", status: "in_progress", assigneeAgentId: agentId, workMode: "standard", + // Declare the shared mode up front, as the normal profile does. The + // first workspace otherwise materializes this setting and legitimately + // changes the effective adapter strategy before the next admission. + executionWorkspaceSettings: { mode: "shared_workspace" } }); + await db.insert(issueWorkProducts).values({ id: workProductId, companyId, issueId, type: "artifact", provider: "paperclip", + title: "Saved artifact", status: "ready_for_review", reviewState: "approved" }); + const first = await heartbeat.wakeup(agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented", + requestedByActorType: "user", requestedByActorId: "pointer-user", payload: { issueId }, contextSnapshot: { issueId, taskId: issueId, skipIssueComment: true } }); + expect(first).not.toBeNull(); + await closeEntered; + const [persisted] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, first!.id)); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, first!.id))).toHaveLength(1); + expect(persisted.runnerProfileJson?.sessionCheckpoint).toMatchObject({ identity: { sessionId: persisted.nativeSessionId }, + providerSessionId, semanticResult: expect.any(Object), terminal: { runTerminalState: "succeeded" } }); + const pointer = await db.select().from(agentTaskSessions).where(eq(agentTaskSessions.agentId, agentId)); + expect(pointer).toEqual([expect.objectContaining({ lastRunId: first!.id, + sessionParamsJson: expect.objectContaining({ sessionId: persisted.nativeSessionId }), sessionDisplayId: providerSessionId })]); + if (mode === "reset") { + expect(await heartbeat.resetRuntimeSession(agentId, { taskKey: issueId })).toMatchObject({ clearedTaskSessions: 1 }); + expect(await heartbeat.listTaskSessions(agentId)).toEqual([]); + } + releaseClose(); + await heartbeat.drainActiveRunExecutions(); + if (mode === "reset") expect(await heartbeat.listTaskSessions(agentId)).toEqual([]); + // The normal comment route reopens a completed task before its user wake. + // Use those public services; a raw comment on a still-done task is stale. + const issuesSvc = issueService(db); + await issuesSvc.update(issueId, { status: "todo", actorUserId: "pointer-user" }); + const comment = await issuesSvc.addComment(issueId, "Continue using the saved artifact.", { userId: "pointer-user" }); + const second = await heartbeat.wakeup(agentId, { source: "automation", triggerDetail: "system", reason: "issue_reopened_via_comment", + requestedByActorType: "user", requestedByActorId: "pointer-user", payload: { issueId, commentId: comment.id, mutation: "comment", reopenedFrom: "done" }, + contextSnapshot: { issueId, taskId: issueId, wakeCommentId: comment.id, reopenedFrom: "done", skipIssueComment: true } }); + expect(second).not.toBeNull(); + await heartbeat.drainActiveRunExecutions(); + const after = await heartbeat.getRun(second!.id); + expect(after?.status).toBe("succeeded"); + expect(inputs).toHaveLength(2); + if (mode === "retained") { + const changedKeys = [...new Set(configs.flatMap(config => Object.keys(config)))].filter(key => + JSON.stringify(configs[0]?.[key]) !== JSON.stringify(configs.at(-1)?.[key])); + expect(changedKeys).toEqual([]); + expect(after).toMatchObject({ nativeSessionId: persisted.nativeSessionId, sessionIdAfter: providerSessionId }); + expect(openSession).toHaveBeenCalledOnce(); + expect(recoverSession).toHaveBeenCalledOnce(); + } else { + expect(after?.nativeSessionId).not.toBe(persisted.nativeSessionId); + expect(after?.sessionIdAfter).not.toBe(providerSessionId); + expect(openSession).toHaveBeenCalledTimes(2); + expect(recoverSession).not.toHaveBeenCalled(); + } + expect(startTurn).toHaveBeenCalledTimes(2); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, second!.id))).toHaveLength(1); + } finally { + releaseClose(); + try { + await heartbeat.drainActiveRunExecutions(); + } finally { + configuration.mockRestore(); + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + home.cleanup(); + } + } + }, 30_000); +}); + enableRunFailureReporting(); diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index 8ba95a27e99..c2f0d519b3e 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -6149,9 +6149,14 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (!versionId || (selectedKeys && !selectedKeys.has(key))) continue; const [version] = await db.select({ skillId: companySkillVersions.companySkillId }).from(companySkillVersions) .where(and(eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.id, versionId))); - skills = skills.filter(skill => skill.key !== key); + // A saved JSONB selection map has a different key order from the fresh + // name/key query. Replace existing rows in place so pinning versions does + // not reorder the server-produced preference (and its session fingerprint). + const index = skills.findIndex(skill => skill.key === key); + if (index !== -1) skills.splice(index, 1); const owner = version ? await getById(companyId, version.skillId) : null; - if (owner) skills.push({ ...owner, key, slug: owner.key === key ? owner.slug : key.split("/").at(-1)! }); + if (owner) skills.splice(index === -1 ? skills.length : index, 0, + { ...owner, key, slug: owner.key === key ? owner.slug : key.split("/").at(-1)! }); } await packages.prefetch(companyId, skills.flatMap(skill => { const id = options.versionSelections?.get(skill.key) ?? skill.currentVersionId; diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 20b8ebe11c4..4b449d2477d 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { NATIVE_RETAINED_OWNER_WAIT } from "./native-runtime/native-controller-recovery.js"; import { externalConversationSettlementWork } from "./external-conversation-settlement-work.js"; import { nativeRecoveryNoticeWork } from "./native-runtime/native-recovery-notice-work.js"; import { nativeRunProjectionWork } from "./native-runtime/native-run-projection-work.js"; @@ -606,6 +607,9 @@ const nativeSessionResumeDispatchTimers = new Map< string, ReturnType >(); +// Routes and the scheduler can drain/resume through different service facades. +// Rearm through the original owner so its database and dispatch mode survive. +const nativeRecoveryDeadlines = new Map void }>(); // Shared with HTTP admission so an idle hold fences work before inspection. export { computeTaskDrain, applyTaskDrain, startTaskDrain, stopTaskDrain } from "./task-admission.js"; @@ -1334,6 +1338,9 @@ export function heartbeatService( } function resumeQueuedRunsAfterDrain() { + for (const pending of nativeRecoveryDeadlines.values()) { + pending.rearm(); + } const promise = heartbeatQueue.resumeQueuedRunsAfterDrain(); activeWakeupPromises.add(promise); void promise.then( @@ -1709,10 +1716,15 @@ export function heartbeatService( } } + // Native ownership waits survive a temporary drain without adding a polling + // loop. A process restart rediscovers the same deadlines from coordinator rows. function scheduleNativeSessionResumeDispatch( runId: string, nextAttemptAt: Date, + mode?: "restart", ) { + const pending = { rearm: () => scheduleNativeSessionResumeDispatch(runId, nextAttemptAt, mode) }; + nativeRecoveryDeadlines.set(runId, pending); const prior = nativeSessionResumeDispatchTimers.get(runId); if (prior) clearTimeout(prior); const delayMs = Math.max(0, nextAttemptAt.getTime() - Date.now()); @@ -1721,6 +1733,12 @@ export function heartbeatService( nativeSessionResumeDispatchTimers.delete(runId); void (async () => { if ((await getSchedulingSuppression()).suppressed) return; + if (nativeRecoveryDeadlines.get(runId) !== pending) return; + nativeRecoveryDeadlines.delete(runId); + if (mode === "restart") { + await recoverNativeRunsAfterRestart(new Date(), [runId]); + return; + } await dispatchNativeSessionResumptions({ db, runnerInstanceId: @@ -1742,6 +1760,9 @@ export function heartbeatService( }, }); })().catch((error) => { + if (!nativeRecoveryDeadlines.has(runId)) { + scheduleNativeSessionResumeDispatch(runId, new Date(Date.now() + 30_000), mode); + } logger.error( { err: error, runId }, "failed to dispatch scheduled native session resume", @@ -3532,7 +3553,8 @@ export function heartbeatService( err.original instanceof Error ? err.original.message : String(err.original ?? ""); - const retryReasonCode = /native_finalization_missing/i.test( + const waitingForRetainedOwner = retryMessage === NATIVE_RETAINED_OWNER_WAIT; + const retryReasonCode = waitingForRetainedOwner ? NATIVE_RETAINED_OWNER_WAIT : /native_finalization_missing/i.test( retryMessage, ) ? "semantic_result_missing" @@ -3552,7 +3574,8 @@ export function heartbeatService( stream: "system", level: "warn", message: - retryReasonCode === "semantic_result_missing" + waitingForRetainedOwner ? "native continuation waiting for the retained controller lease" + : retryReasonCode === "semantic_result_missing" ? "provider turn completed without a semantic result; same-run disposition recovery persisted" : "native session transport interrupted; same-run resume persisted", payload: { diff --git a/server/src/services/heartbeat/execution-configuration.ts b/server/src/services/heartbeat/execution-configuration.ts index fb9626b932f..6d88d5d7795 100644 --- a/server/src/services/heartbeat/execution-configuration.ts +++ b/server/src/services/heartbeat/execution-configuration.ts @@ -49,7 +49,11 @@ export function bindHeartbeatRunnerHomeMode(config: Record, per const saved = parseObject(persistedExecution); if (Object.keys(saved).length === 0) return config; // Execution schemas before v7 always used isolated homes. - return { ...config, homeMode: saved.homeMode ?? "isolated" }; + const homeMode = saved.homeMode ?? "isolated"; + // Omission already means isolated. Recovery must not add a fingerprinted + // configuration property merely to restate that unchanged default. + if (homeMode === "isolated" && config.homeMode === undefined) return config; + return { ...config, homeMode }; } export interface HeartbeatExecutionConfigurationInput { diff --git a/server/src/services/heartbeat/native-recovery-deadlines.test.ts b/server/src/services/heartbeat/native-recovery-deadlines.test.ts new file mode 100644 index 00000000000..e39daad1b52 --- /dev/null +++ b/server/src/services/heartbeat/native-recovery-deadlines.test.ts @@ -0,0 +1,102 @@ +import type { Db } from "@paperclipai/db"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { heartbeatService, startTaskDrain, stopTaskDrain } from "../heartbeat.js"; +import type { HeartbeatRecoveryDependencies } from "./recovery.js"; + +const hooks = vi.hoisted(() => ({ + schedule: undefined as HeartbeatRecoveryDependencies["scheduleNativeSessionResumeDispatch"] | undefined, + restart: vi.fn(async () => []), + resume: vi.fn(async () => []), + queue: vi.fn(async () => undefined), +})); + +vi.mock("./recovery.js", async original => { + const module = await original(); + return { ...module, createHeartbeatRecovery: (db: Db, dependencies: HeartbeatRecoveryDependencies) => { + hooks.schedule = dependencies.scheduleNativeSessionResumeDispatch; + return { ...module.createHeartbeatRecovery(db, dependencies), recoverNativeRunsAfterRestart: hooks.restart }; + } }; +}); +vi.mock("./queue.js", async original => { + const module = await original(); + return { ...module, createHeartbeatQueue: (...args: Parameters) => ({ + ...module.createHeartbeatQueue(...args), resumeQueuedRunsAfterDrain: hooks.queue, + }) }; +}); +vi.mock("../native-runtime/index.js", async original => ({ + ...await original(), + dispatchNativeSessionResumptions: hooks.resume, +})); + +describe("native deadline drain and resume", () => { + let heartbeat: ReturnType; + beforeEach(() => { + vi.clearAllMocks(); + vi.useFakeTimers(); + stopTaskDrain(); + const db = new Proxy({}, { get: () => { throw new Error("Unexpected deadline database access"); } }) as Db; + heartbeat = heartbeatService(db, { runtimeEnv: {} }); + }); + afterEach(async () => { + stopTaskDrain(); + await heartbeat?.resumeQueuedRunsAfterDrain(); + await vi.runOnlyPendingTimersAsync(); + await heartbeat?.drainActiveRunExecutions(); + vi.useRealTimers(); + }); + + it.each([undefined, "restart"] as const)("retains an overdue %s wait through drain with its original dispatch mode", async mode => { + const deadline = new Date(Date.now() + 100); + startTaskDrain({}); + hooks.schedule!("waiting-run", deadline, mode); + await vi.advanceTimersByTimeAsync(100); + expect(hooks.restart).not.toHaveBeenCalled(); + expect(hooks.resume).not.toHaveBeenCalled(); + stopTaskDrain(); + await heartbeat.resumeQueuedRunsAfterDrain(); + await vi.advanceTimersByTimeAsync(0); + const called = mode === "restart" ? hooks.restart : hooks.resume; + const other = mode === "restart" ? hooks.resume : hooks.restart; + expect(called).toHaveBeenCalledOnce(); + expect(other).not.toHaveBeenCalled(); + await heartbeat.resumeQueuedRunsAfterDrain(); + await vi.advanceTimersByTimeAsync(60_000); + expect(called).toHaveBeenCalledOnce(); + expect(vi.getTimerCount()).toBe(0); + }); + + it("preserves a future follow-up deadline when draining cancels its timer", async () => { + const deadline = new Date(Date.now() + 100); + hooks.schedule!("future-run", deadline); + await heartbeat.drainActiveRunExecutions(); + await heartbeat.resumeQueuedRunsAfterDrain(); + await vi.advanceTimersByTimeAsync(99); + expect(hooks.resume).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(1); + expect(hooks.resume).toHaveBeenCalledOnce(); + expect(hooks.restart).not.toHaveBeenCalled(); + }); + + it("rearms the original owner when another service facade ends the drain", async () => { + hooks.schedule!("other-facade-run", new Date(Date.now() + 100)); + const db = new Proxy({}, { get: () => { throw new Error("Unexpected replacement facade database access"); } }) as Db; + const other = heartbeatService(db, { runtimeEnv: {} }); + await other.drainActiveRunExecutions(); + await vi.advanceTimersByTimeAsync(100); + await other.resumeQueuedRunsAfterDrain(); + await vi.advanceTimersByTimeAsync(0); + expect(hooks.resume).toHaveBeenCalledOnce(); + expect(hooks.restart).not.toHaveBeenCalled(); + }); + + it("retries a failed follow-up obligation without changing its dispatch mode", async () => { + hooks.resume.mockRejectedValueOnce(new Error("temporary database failure")); + hooks.schedule!("failed-follow-up", new Date()); + await vi.advanceTimersByTimeAsync(0); + expect(hooks.resume).toHaveBeenCalledOnce(); + await vi.advanceTimersByTimeAsync(30_000); + expect(hooks.resume).toHaveBeenCalledTimes(2); + expect(hooks.restart).not.toHaveBeenCalled(); + expect(vi.getTimerCount()).toBe(0); + }); +}); diff --git a/server/src/services/heartbeat/recovery.test.ts b/server/src/services/heartbeat/recovery.test.ts index 893a1326816..a92ceff5823 100644 --- a/server/src/services/heartbeat/recovery.test.ts +++ b/server/src/services/heartbeat/recovery.test.ts @@ -1158,11 +1158,12 @@ describePostgres("heartbeat recovery module ownership", () => { await Promise.all([...deps.activeRunExecutionPromises]); }); - it("rearms the persisted native retry deadline through the service callback", async () => { + it.each(["future", "overdue-owner"])("rearms the persisted native retry deadline through the service callback: %s", async scenario => { const run = await seedRun(); - const nextAttemptAt = new Date(Date.now() + 60_000); + const nextAttemptAt = new Date(Date.now() + (scenario === "future" ? 60_000 : -1_000)); await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, status: "failed" }).where(eq(heartbeatRuns.id, run.id)); - await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!, phase: "retryable_failure", nextAttemptAt }); + await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!, phase: "retryable_failure", nextAttemptAt, + failureCode: scenario === "overdue-owner" ? "native_controller_recovery_waiting_for_owner" : null }); const deps = callbacks(db); const result = await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart(); expect(result.scheduledRetryRunIds).toEqual([run.id]); @@ -1173,6 +1174,16 @@ describePostgres("heartbeat recovery module ownership", () => { await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); }); + it("retains a live remote owner's actual lease deadline without dispatching competing work", async () => { + const deps = callbacks(db); + const runId = randomUUID(), retryAt = new Date(Date.now() + 60_000); + native.claims.mockResolvedValue([{ kind: "awaiting_evidence", runId, reason: "remote_controller_lease_live", retryAt }]); + const result = await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart(); + expect(result.awaitingEvidenceRunIds).toEqual([runId]); + expect(deps.scheduleNativeSessionResumeDispatch).toHaveBeenCalledWith(runId, retryAt, "restart"); + expect(deps.executeRun).not.toHaveBeenCalled(); + }); + it("retains shutdown ownership until background native cleanup physically settles", async () => { const deps = callbacks(db); let settle!: () => void; diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index fd5969a2088..d4e4d65c599 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -1,3 +1,4 @@ +import { NATIVE_RETAINED_OWNER_WAIT } from "../native-runtime/native-controller-recovery.js"; import { nativeEnvironmentSettlementDisposition, nativeEnvironmentSettlementWork } from "../native-runtime/native-environment-settlement-work.js"; import { nativeWorkspaceFinalizationWork } from "../native-runtime/native-workspace-finalization-work.js"; import { attemptNativeWorkspaceFinalization } from "../native-runtime/native-workspace-finalizer.js"; @@ -126,7 +127,7 @@ export interface HeartbeatRecoveryDependencies extends Pick Promise; }) => Promise; - scheduleNativeSessionResumeDispatch: (runId: string, nextAttemptAt: Date) => void; + scheduleNativeSessionResumeDispatch: (runId: string, nextAttemptAt: Date, mode?: "restart") => void; cancelHeartbeatNativeRun: (input: { db: Db; runId: string; reason: string; runtimeMode: string | null; }) => Promise; @@ -956,7 +957,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD inArray(heartbeatRuns.status, ["running", "failed"]), isNull(nativeRunFinalizations.resultId), eq(nativeRunFinalizations.phase, "retryable_failure"), - gt(nativeRunFinalizations.nextAttemptAt, now), + or(gt(nativeRunFinalizations.nextAttemptAt, now), eq(nativeRunFinalizations.failureCode, NATIVE_RETAINED_OWNER_WAIT)), ), ); for (const scheduled of scheduledNativeRetries) { @@ -1004,6 +1005,9 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD disposition.kind === "bootstrap_incomplete", ); for (const disposition of dispositions) { + if (disposition.kind === "awaiting_evidence" && disposition.retryAt) { + scheduleNativeSessionResumeDispatch(disposition.runId, disposition.retryAt, "restart"); + } const run = await getRun(disposition.runId); if (run) { const isClaim = diff --git a/server/src/services/heartbeat/run-completion.test.ts b/server/src/services/heartbeat/run-completion.test.ts index 62ce16df460..9b8a9810b2a 100644 --- a/server/src/services/heartbeat/run-completion.test.ts +++ b/server/src/services/heartbeat/run-completion.test.ts @@ -309,9 +309,16 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)( it("retains native completion metadata when reconciliation already committed the same terminal status", async () => { const finishedAt = new Date("2026-10-09T12:00:00Z"); const f = await fixture({ runtimeMode: "native", status: "succeeded", finishedAt, resultJson: { durable: "accepted" }, usageJson: { durableReceipt: "native" } }); - f.input.adapterResult.nativeFinalization = { schema: "paperclip.native-finalization.v1", runtimeMode: "native", runId: f.run.id, companyId: f.run.companyId, issueId: "unused", result: {}, terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", runTerminalState: "succeeded", reportedWorkDisposition: "done" }, turnId: null, sourceInstanceId: "test", normalizedSessionId: "native-session", providerSessionId: null, driverKind: "test", driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, workspaceFinalizeStatus: "succeeded" }; + await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, f.agent.id)); + f.agent.adapterType = "paperclip_runner"; + f.input.sessionCodec = getAdapterSessionCodec(f.agent.adapterType); + const nativeSessionId = randomUUID(), providerSessionId = randomUUID(); + f.input.adapterResult.sessionId = nativeSessionId; + delete f.input.adapterResult.sessionParams; + f.input.adapterResult.sessionDisplayId = providerSessionId; + f.input.adapterResult.nativeFinalization = { schema: "paperclip.native-finalization.v1", runtimeMode: "native", runId: f.run.id, companyId: f.run.companyId, issueId: "unused", result: {}, terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", runTerminalState: "succeeded", reportedWorkDisposition: "done" }, turnId: null, sourceInstanceId: "test", normalizedSessionId: nativeSessionId, providerSessionId, driverKind: "test", driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, workspaceFinalizeStatus: "succeeded" }; await f.completion.completeRun(f.input); - expect(await read(f.run.id)).toMatchObject({ status: "succeeded", finishedAt, sessionIdAfter: "provider-session", resultJson: { durable: "accepted" }, usageJson: { durableReceipt: "native", inputTokens: 20 } }); + expect(await read(f.run.id)).toMatchObject({ status: "succeeded", finishedAt, sessionIdAfter: providerSessionId, resultJson: { durable: "accepted" }, usageJson: { durableReceipt: "native", inputTokens: 20 } }); expect(f.deps.updateRuntimeState).toHaveBeenCalledOnce(); }); diff --git a/server/src/services/heartbeat/run-completion.ts b/server/src/services/heartbeat/run-completion.ts index c82816f384b..9a0696a1d11 100644 --- a/server/src/services/heartbeat/run-completion.ts +++ b/server/src/services/heartbeat/run-completion.ts @@ -1,4 +1,5 @@ import { preserveAcceptedRunUsageProjectionSql } from "../usage-receipts.js"; +import { nativeTaskSessionPublicationHandled } from "../native-runtime/native-task-session-publication.js"; import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js"; import { measureRunPhase } from "../execution-preparation-timing.js"; import { PROJECT_ACCESS_DENIED, ProjectAccessDeniedError } from "../project-access-denial.js"; @@ -489,6 +490,13 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC normalizedUsage, ); if (!taskKey) return; + if (nativeTaskSessionPublicationHandled(finalizedRun.runnerProfileJson, { + runId: finalizedRun.id, + companyId: finalizedRun.companyId, + agentId: agent.id, + taskKey, + adapterType: agent.adapterType, + })) return; if ( adapterResult.clearSession || (!nextSessionState.params && !nextSessionState.displayId) diff --git a/server/src/services/heartbeat/runtime-execution.test.ts b/server/src/services/heartbeat/runtime-execution.test.ts index 4f5b9c349dc..f39b180fbde 100644 --- a/server/src/services/heartbeat/runtime-execution.test.ts +++ b/server/src/services/heartbeat/runtime-execution.test.ts @@ -1,3 +1,5 @@ +import { NativeControllerRecoveryWaitingError } from "../native-runtime/native-controller-recovery.js"; + import { createNativeUsageDelivery } from "../native-runtime/native-usage-delivery.js"; import type { PrpEvent } from "../../vendor/paperclip-runner/index.js"; import { randomUUID } from "node:crypto"; @@ -10,7 +12,7 @@ import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vites import { trace } from "@opentelemetry/api"; import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace"; import { createRunPreparationTiming, measureRunPhase, measureWorkspaceSyncPhase, withRunExecutionTiming } from "../execution-preparation-timing.js"; -import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { recoveryPostgresSupport as support, startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js"; import * as pluginWorkspaces from "../plugin-workspaces.js"; import * as legacyRecovery from "../legacy-execution-recovery.js"; import { createAgentIdentityRedactor } from "../agent-identity-redaction.js"; @@ -43,15 +45,14 @@ vi.mock("./run-preparation.js", async original => ({ revokeHeartbeatRunGatewayTokens: vi.fn(async () => {}), })); -const support = await getEmbeddedPostgresTestSupport(); describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () => { - let database: Awaited>; + let database: Awaited>; let db: Db; let home: string; beforeAll(async () => { home = await mkdtemp(join(tmpdir(), "paperclip-runtime-execution-")); vi.stubEnv("PAPERCLIP_HOME", home); - database = await startEmbeddedPostgresTestDatabase("paperclip-runtime-execution"); + database = await startRecoveryPostgres("paperclip-runtime-execution"); db = createDb(database.connectionString); await db.execute(sql`set client_min_messages = warning`); }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); @@ -283,6 +284,18 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () = expectCleanup(input); }); + it("preserves the retained sandbox while native admission waits for its current owner", async () => { + const { input } = await fixture(true); + const error = new NativeControllerRecoveryWaitingError(new Date(Date.now() + 60_000)); + vi.mocked(executePaperclipNativeSession).mockRejectedValueOnce(error); + await expect(executeHeartbeatRuntime(db, input)).rejects.toMatchObject({ + name: "NativeSessionResumeScheduledError", original: error, + }); + expect(input.effects.onNativeSessionResumeScheduled).toHaveBeenCalledOnce(); + expect(input.effects.onProviderResourceDisposition).not.toHaveBeenCalledWith("stop_and_retain"); + expect(await finalizations()).toHaveLength(0); + }); + it("records native restoration inside the workspace finalization operation", async () => { const { input, run } = await fixture(true); const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT; diff --git a/server/src/services/heartbeat/runtime-execution.ts b/server/src/services/heartbeat/runtime-execution.ts index b5b64abc948..092639894cf 100644 --- a/server/src/services/heartbeat/runtime-execution.ts +++ b/server/src/services/heartbeat/runtime-execution.ts @@ -1,3 +1,5 @@ +import { NativeControllerRecoveryWaitingError } from "../native-runtime/native-controller-recovery.js"; + import { usesNativeUsageDelivery } from "../native-runtime/native-usage-accounting.js"; import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; import { writeExecutionControl } from "../execution-control-notifications.js"; @@ -559,6 +561,20 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe sessionGoalControl.requestId, )); if (completedGoalControl) sessionGoalControl = null; + // Goal checkpoints already publish their task session. Goal-less turns + // need the same durable pointer before cleanup can delay host completion. + const taskSessionPublication = taskKey && !sessionGoalControl && + context.resumeSessionGoalHeartbeat !== true && !completedGoalControl + ? { + taskKey, + adapterType: agent.adapterType, + sessionParamsJson: attachPaperclipSessionMetadataToSessionParams( + { ...runtimeSessionParamsForAdapter, cwd: executionWorkspace.cwd }, + configuredModel, + sessionConfigMetadata, + ), + } + : undefined; const nativeDispatchAtMs = Date.now(); const runCreatedAtMs = run.createdAt.getTime(); const runStartedAtMs = (run.startedAt ?? run.createdAt).getTime(); @@ -608,6 +624,7 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe resumeSessionGoalHeartbeat: context.resumeSessionGoalHeartbeat === true || completedGoalControl, + taskSessionPublication, onGoalCheckpoint: async (snapshot) => { if (!taskKey) return; const params = @@ -1026,6 +1043,10 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe preserveRunGatewayTokens = true; throw adapterErr; } + if (adapterErr instanceof NativeControllerRecoveryWaitingError) { + input.effects.onNativeSessionResumeScheduled(); + throw new NativeSessionResumeScheduledError(adapterErr); + } if (adapterErr instanceof NativeRunnerOwnershipUnverifiedError) { input.effects.onNativeOwnershipHeld(); throw adapterErr; diff --git a/server/src/services/heartbeat/upgrade-readiness.ts b/server/src/services/heartbeat/upgrade-readiness.ts index 1994ce5555e..a3036a4ee5f 100644 --- a/server/src/services/heartbeat/upgrade-readiness.ts +++ b/server/src/services/heartbeat/upgrade-readiness.ts @@ -4,6 +4,7 @@ import { inspectNativeRunnerUpgradeHeader } from "../native-runtime/native-runne import { readNativeWorkspaceSyncReference } from "../native-runtime/native-workspace-sync.js"; import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "../native-runtime/native-runner-recovery-contract.js"; import { currentNativeControllerIdentity } from "../native-runtime/native-restart-recovery.js"; +import { readNativeControllerRecoveryInventory } from "../native-runtime/native-controller-recovery-inventory.js"; const MAX_UPGRADE_RUNS = 1_000; type Counters = { activeRuns: number; pendingWakes: number; draining: boolean }; @@ -60,8 +61,16 @@ export async function readUpgradeReadiness(db: Db, input: { && identity.runnerInstanceId === row.runnerId && identity.normalizedSessionId === row.sessionId && identity.environmentLeaseId === (row.pluginLeaseId ?? workspace?.workspaceId)) }; }); - return summarizeUpgradeReadiness({ rows: candidates, before, after: input.counters(), + // Additive evidence does not strengthen or redefine the existing ready flag. + const retained = await readNativeControllerRecoveryInventory(db).catch(() => ({ + version: 1 as const, observedAt: new Date().toISOString(), coverage: "unknown" as const, + reasons: { inventory_unavailable: 1 }, fullPayloadsValidated: false as const, + })); + const summary = summarizeUpgradeReadiness({ rows: candidates, before, after: input.counters(), activeRunIds: [...new Set([...beforeIds, ...input.activeRunIds])], controllerBootId: owner.bootId }); + return { ...summary, nativeControllerRecovery: { activeExecutionAdoption: { + recoverableRuns: summary.recoverableRuns, blockingRuns: summary.blockingRuns, truncated: summary.truncated, + }, retained } }; } export function summarizeUpgradeReadiness(input: { diff --git a/server/src/services/native-runtime/native-controller-recovery-inventory.ts b/server/src/services/native-runtime/native-controller-recovery-inventory.ts new file mode 100644 index 00000000000..88f0077f9bd --- /dev/null +++ b/server/src/services/native-runtime/native-controller-recovery-inventory.ts @@ -0,0 +1,61 @@ +import type { Db } from "@paperclipai/db"; +import { sql } from "drizzle-orm"; +import { inspectNativeRunnerUpgradeHeader } from "./native-runner-checkpoint.js"; + +/** An explicit, paginated diagnostic. Never called by an idle/background scan. + * This inventories selected retained task-session pointers, not all history. + * Header evidence is preliminary: full snapshot/journal validation still occurs + * only when ordinary admission selects its exact continuation source. */ +export async function readNativeControllerRecoveryInventory(db: Db, input: { afterId?: string; limit?: number } = {}) { + if (input.limit !== undefined && !Number.isFinite(input.limit)) throw new Error("native_recovery_inventory_limit_invalid"); + const limit = Math.min(128, Math.max(1, Math.floor(input.limit ?? 128))); + const rows = await db.transaction(async tx => { + await tx.execute(sql`set local statement_timeout = '2s'`); + return tx.execute<{ + id: string; runId: string | null; companyId: string; issueId: string | null; sessionId: string | null; + runnerId: string | null; metadata: Record | null; material: Record | null; + pending: boolean; + }>(sql`with selected as ( + select id, company_id, agent_id, last_run_id from agent_task_sessions + where adapter_type = 'paperclip_runner' ${input.afterId ? sql`and id > ${input.afterId}::uuid` : sql``} + order by id limit ${limit + 1} + ) select s.id, s.company_id as "companyId", r.id as "runId", r.native_issue_id as "issueId", + r.native_session_id as "sessionId", r.runner_instance_id as "runnerId", + case when pg_column_size(r.runner_profile_json->'nativeControllerRecovery') <= 4096 + then r.runner_profile_json->'nativeControllerRecovery' else null end as metadata, + case when pg_column_size(c.material) <= 262144 and sum(pg_column_size(c.material)) over () <= 8388608 + then c.material else null end as material, + (r.finished_at is null or r.status not in ('succeeded', 'failed', 'cancelled', 'interrupted', 'timed_out') + or f.run_id is null or f.lease_owner is not null or f.recovery_state is not null + or f.phase not in ('committed', 'terminal_failure')) is true as pending + from selected s left join heartbeat_runs r on r.id = s.last_run_id and r.company_id = s.company_id and r.agent_id = s.agent_id + left join native_run_finalizations f on f.run_id = r.id and f.company_id = s.company_id + left join native_runner_checkpoints c on c.run_id = r.id and c.company_id = s.company_id + and c.issue_id = r.native_issue_id and c.part = 'header' + order by s.id`); + }, { accessMode: "read only" }); + const truncated = rows.length > limit; + const page = rows.slice(0, limit); + const reasons: Record = {}; + let selectedRetainedEvidence = 0, outstandingObligations = 0; + for (const row of page) { + let reason: string | null = null; + if (row.pending) { outstandingObligations++; reason = "persistence_or_handoff_pending"; } + else if (!row.runId || !row.issueId || !row.sessionId) reason = "selected_binding_missing"; + else if (row.metadata?.schema !== "paperclip.native-controller-recovery.v1") reason = "metadata_unsupported_or_missing"; + else { + const checkpoint = row.metadata.checkpoint as Record | null; + const header = row.material ? inspectNativeRunnerUpgradeHeader({ companyId: row.companyId, issueId: row.issueId, runId: row.runId }, row.material) : null; + if (!checkpoint || checkpoint.runId !== row.runId || checkpoint.runnerInstanceId !== row.runnerId + || checkpoint.normalizedSessionId !== row.sessionId || !header + || header.runnerInstanceId !== row.runnerId || header.normalizedSessionId !== row.sessionId) reason = "checkpoint_binding_missing"; + else selectedRetainedEvidence++; + } + if (reason) reasons[reason] = (reasons[reason] ?? 0) + 1; + } + return { version: 1 as const, observedAt: new Date().toISOString(), inspected: page.length, + coverage: truncated || input.afterId ? "partial" as const : "selected_task_session_pointers" as const, + truncated, nextAfterId: truncated ? page.at(-1)!.id : null, + selectedRetainedEvidence, outstandingObligations, reasons, + fullPayloadsValidated: false as const }; +} diff --git a/server/src/services/native-runtime/native-controller-recovery.integration.test.ts b/server/src/services/native-runtime/native-controller-recovery.integration.test.ts new file mode 100644 index 00000000000..2accfe2a6ca --- /dev/null +++ b/server/src/services/native-runtime/native-controller-recovery.integration.test.ts @@ -0,0 +1,165 @@ +import { randomUUID } from "node:crypto"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agentTaskSessions, agents, companies, createDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults } from "@paperclipai/db"; +import { DurablePrpControlPlane, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; +import { startRecoveryPostgres, recoveryPostgresSupport } from "../../__tests__/helpers/recovery-postgres.js"; +import { createIsolatedTestHome } from "../../__tests__/helpers/isolated-test-home.js"; +import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; +import { nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js"; +import { createNativeControllerRecovery, deferNativeControllerRecovery, importNativeControllerRecovery, NATIVE_RETAINED_OWNER_WAIT, resolveNativeControllerRecovery } from "./native-controller-recovery.js"; +import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js"; +import { claimNativeRestartRecoveries } from "./native-restart-recovery.js"; +import { readNativeControllerRecoveryInventory } from "./native-controller-recovery-inventory.js"; + +describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable discovery and import", () => { + let database: Awaited>; + let db: ReturnType; + let home: ReturnType; + let root: string; + const previousHome = process.env.PAPERCLIP_HOME; + const companyId = randomUUID(), agentId = randomUUID(); + const compatibility = { configDigest: `sha256:${"a".repeat(64)}`, processConfigDigest: null }; + beforeAll(async () => { + home = createIsolatedTestHome(); + process.env.PAPERCLIP_HOME = home.home; + root = await mkdtemp(join(tmpdir(), "native-controller-discovery-")); + database = await startRecoveryPostgres("native-controller-discovery-"); + db = createDb(database.connectionString); + await db.insert(companies).values({ id: companyId, name: "Controller discovery", issuePrefix: "NCD" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Runner", adapterType: "paperclip_runner", adapterConfig: { provider: "codex" } }); + }, 60_000); + afterAll(async () => { + await database?.cleanup(); + if (root) await rm(root, { recursive: true, force: true }); + home?.cleanup(); + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + }); + + async function seed() { + const issueId = randomUUID(), runId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Continue retained execution", assigneeAgentId: agentId, status: "in_progress" }); + const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: issueId }).returning(); + const native = await prepareNativeHeartbeatRun({ db, run, issue: { id: issueId, title: "Continue retained execution", description: null, reviewPolicy: null }, environmentLeaseId: randomUUID() }); + const owner = { companyId, issueId, runId, authorityRunId: runId, bootId: randomUUID(), generation: 1, leaseOwner: randomUUID() }; + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "observed", controllerBootId: owner.bootId, + controllerGeneration: 1, leaseOwner: owner.leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000) }); + const core = new DurablePrpControlPlane({ stateDirectory: join(root, runId), + identity: { runId, normalizedSessionId: native.normalizedSessionId, runnerInstanceId: native.runnerInstanceId, + environmentLeaseId: native.environmentLeaseId, turnId: native.turnId, itemId: native.itemId }, + expectedRunnerVersion: "discovery-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + core.setCheckpointPersistence(nativeRunnerCheckpointWriter(db, owner)); + await core.flushCheckpoint(); + await core.stop(); + const snapshot: PersistedNativeSession = { backendKind: "mock", sessionId: native.normalizedSessionId, + identity: { companyId, agentId, issueId, runId, sessionId: native.normalizedSessionId } }; + const [prepared] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const [result] = await db.insert(nativeRunResults).values({ companyId, issueId, runId, + completionContractId: prepared.completionContractId!, serverFingerprint: "fixture", schemaStatus: "accepted", + resultJson: {}, canonicalSha256: "fixture" }).returning(); + await db.update(nativeRunFinalizations).set({ phase: "committed", resultId: result.id, leaseOwner: null, leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, runId)); + const profile = { ...prepared.runnerProfileJson, sessionCheckpoint: snapshot }; + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date(), runnerProfileJson: profile }).where(eq(heartbeatRuns.id, runId)); + const scope = { companyId, agentId, issueId, runId, normalizedSessionId: native.normalizedSessionId }; + return { scope, native, profile, snapshot, owner, + supervisor: { schema: "paperclip.native-session-supervisor.v1", ...compatibility, snapshot } }; + } + + it("imports a retained source once, rejects stale input, and resolves it without any local projection", async () => { + const fixture = await seed(); + const input = { ...fixture.scope, expectedConfigDigest: compatibility.configDigest, supervisor: fixture.supervisor }; + expect(await importNativeControllerRecovery(db, { ...input, supervisor: { ...input.supervisor, snapshot: { ...fixture.snapshot, providerSessionId: "stale" } } })) + .toEqual({ kind: "missing_evidence", reason: "stale_supervisor_snapshot" }); + expect(await importNativeControllerRecovery(db, input)).toEqual({ kind: "imported" }); + expect(await importNativeControllerRecovery(db, input)).toEqual({ kind: "unchanged" }); + await rm(join(root, fixture.scope.runId), { recursive: true }); + const currentRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: currentRunId, companyId, agentId, nativeIssueId: fixture.scope.issueId, + nativeSessionId: fixture.scope.normalizedSessionId, status: "running", runtimeMode: "native" }); + const recovered = await resolveNativeControllerRecovery(db, { ...fixture.scope, currentRunId }); + expect(recovered.kind).toBe("recoverable"); + if (recovered.kind !== "recoverable") throw new Error(JSON.stringify(recovered)); + expect(recovered.snapshot).toEqual(fixture.snapshot); + expect(recovered.checkpoint.recoveryEvidence.runnerInstanceId).toBe(fixture.native.runnerInstanceId); + const deadline = new Date(Date.now() + 60_000); + await db.update(nativeRunFinalizations).set({ leaseOwner: "other-controller", leaseExpiresAt: deadline }).where(eq(nativeRunFinalizations.runId, fixture.scope.runId)); + expect(await importNativeControllerRecovery(db, input)).toEqual({ kind: "waiting_for_owner", deadline }); + expect(await resolveNativeControllerRecovery(db, { ...fixture.scope, currentRunId })).toEqual({ kind: "waiting_for_owner", deadline }); + }); + + it("holds mismatched snapshots and runner bindings instead of falling back to fresh execution", async () => { + const fixture = await seed(), currentRunId = randomUUID(); + const metadata = createNativeControllerRecovery(fixture.snapshot, compatibility, { runId: fixture.scope.runId, + runnerInstanceId: fixture.native.runnerInstanceId, normalizedSessionId: fixture.scope.normalizedSessionId }); + await db.insert(heartbeatRuns).values({ id: currentRunId, companyId, agentId, nativeIssueId: fixture.scope.issueId, + nativeSessionId: fixture.scope.normalizedSessionId, status: "running", runtimeMode: "native" }); + for (const envelope of [{ ...metadata, snapshotSha256: "wrong" }, { ...metadata, checkpoint: { ...metadata.checkpoint!, runnerInstanceId: randomUUID() } }]) { + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...fixture.profile, nativeControllerRecovery: envelope } }).where(eq(heartbeatRuns.id, fixture.scope.runId)); + expect(await resolveNativeControllerRecovery(db, { ...fixture.scope, currentRunId })) + .toMatchObject({ kind: "missing_evidence", reason: "snapshot_digest_or_checkpoint_binding" }); + } + }); + + it("keeps an unstarted follow-up on the same run until the retained owner's deadline", async () => { + const fixture = await seed(), runId = randomUUID(); + const scope = { ...fixture.scope, runId }; + const deadline = new Date(Date.now() + 60_000); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: scope.issueId, + nativeSessionId: scope.normalizedSessionId, status: "running", runtimeMode: "native", + // The admission boundary has already stored its input; the deferral + // and resumption owners never execute this transport-independent fixture. + runnerProfileJson: { nativeExecutionInput: { binding: scope } } }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, scope.issueId)); + await db.insert(nativeRunFinalizations).values({ companyId, issueId: scope.issueId, runId, phase: "observed" }); + await deferNativeControllerRecovery(db, { ...scope, retryAt: deadline }); + const read = () => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + expect((await read())[0]).toMatchObject({ phase: "retryable_failure", attempt: 0, leaseOwner: null, + nextAttemptAt: deadline, failureCode: NATIVE_RETAINED_OWNER_WAIT }); + expect(await claimNativeSessionResumptions({ db, runnerInstanceId: "successor", runIds: [runId] })).toEqual([]); + // Even after expiry, restart adoption cannot reinterpret this pre-provider + // wait as a crashed provider. The ordinary resumption owner consumes it. + expect(await claimNativeRestartRecoveries({ db, runIds: [runId], restartKind: "hard", now: deadline })).toEqual([]); + const claims = await claimNativeSessionResumptions({ db, runnerInstanceId: "successor", runIds: [runId], now: deadline }); + expect(claims).toHaveLength(1); + const nextDeadline = new Date(deadline.getTime() + 60_000); + await deferNativeControllerRecovery(db, { ...scope, leaseOwner: claims[0].leaseOwner, retryAt: nextDeadline }); + expect((await read())[0]).toMatchObject({ attempt: 0, leaseOwner: null, nextAttemptAt: nextDeadline }); + await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, runId)); + expect(await claimNativeSessionResumptions({ db, runnerInstanceId: "successor", runIds: [runId], now: nextDeadline })).toEqual([]); + }); + + it.each(["live-owner", "expired-owner", "provider-attempt", "stopped", "operator-stop"])("does not publish an admission wait after authority changes: %s", async scenario => { + const fixture = await seed(), runId = randomUUID(); + const scope = { ...fixture.scope, runId }; + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: scope.issueId, + status: scenario === "stopped" ? "cancelled" : "running", runtimeMode: "native", + resultJson: scenario === "operator-stop" ? { cancellation: { source: "operator", expected: true, + initiator: { type: "user" }, reason: "Stop", recordedAt: new Date().toISOString() } } : null, + runnerProfileJson: { nativeExecutionInput: { binding: scope } } }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, scope.issueId)); + await db.insert(nativeRunFinalizations).values({ companyId, issueId: scope.issueId, runId, phase: "observed", + attempt: scenario === "provider-attempt" ? 1 : 0, + leaseOwner: scenario.endsWith("-owner") ? "other-controller" : null, + leaseExpiresAt: scenario === "expired-owner" ? new Date(Date.now() - 1_000) : null }); + await expect(deferNativeControllerRecovery(db, { ...scope, + leaseOwner: scenario === "expired-owner" ? "other-controller" : undefined, + retryAt: new Date(Date.now() + 60_000) })) + .rejects.toThrow("wait_authority_changed"); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0].phase).toBe("observed"); + }); + + it("reports bounded retained-pointer coverage without loading historical payloads", async () => { + const fixture = await seed(); + await importNativeControllerRecovery(db, { ...fixture.scope, expectedConfigDigest: compatibility.configDigest, supervisor: fixture.supervisor }); + await db.insert(agentTaskSessions).values([0, 1].map(index => ({ companyId, agentId, taskKey: `${fixture.scope.issueId}-${index}`, + adapterType: "paperclip_runner", lastRunId: fixture.scope.runId }))); + const first = await readNativeControllerRecoveryInventory(db, { limit: 1 }); + expect(first).toMatchObject({ inspected: 1, coverage: "partial", truncated: true, selectedRetainedEvidence: 1, fullPayloadsValidated: false }); + const next = await readNativeControllerRecoveryInventory(db, { limit: 1, afterId: first.nextAfterId! }); + expect(next).toMatchObject({ inspected: 1, coverage: "partial", truncated: false, selectedRetainedEvidence: 1 }); + }); +}); diff --git a/server/src/services/native-runtime/native-controller-recovery.test.ts b/server/src/services/native-runtime/native-controller-recovery.test.ts new file mode 100644 index 00000000000..261763192fb --- /dev/null +++ b/server/src/services/native-runtime/native-controller-recovery.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, it } from "vitest"; +import { createNativeControllerRecovery, readNativeControllerRecovery } from "./native-controller-recovery.js"; +import { nativeSha256 } from "./canonical.js"; +import { redactRegisteredSecretValues } from "../run-secret-redaction.js"; +import type { PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; + +const snapshot: PersistedNativeSession = { backendKind: "mock", sessionId: "provider", + identity: { companyId: "company", agentId: "agent", issueId: "issue", runId: "run", sessionId: "session" } }; +const compatibility = { configDigest: `sha256:${"a".repeat(64)}`, processConfigDigest: null }; +const binding = { runId: "run", runnerInstanceId: "runner", normalizedSessionId: "session" }; + +describe("native controller recovery envelope", () => { + it("binds the stored snapshot, including redaction, without serializing its contents twice", () => { + const stored = { ...snapshot, providerSessionId: "[REDACTED]" }; + const envelope = createNativeControllerRecovery(stored, compatibility, binding); + expect(envelope.snapshotSha256).toBe(nativeSha256(stored)); + expect(readNativeControllerRecovery(envelope, JSON.parse(JSON.stringify(stored)))).toEqual(envelope); + expect(readNativeControllerRecovery(envelope, { ...stored, providerSessionId: "secret-before-redaction" })).toBeNull(); + expect(JSON.stringify(envelope)).not.toContain("[REDACTED]"); + expect(Buffer.byteLength(JSON.stringify(envelope))).toBeLessThan(4096); + }); + + it("does not certify a missing PRP publication or unknown format", () => { + const pending = createNativeControllerRecovery(snapshot, compatibility, null); + expect(readNativeControllerRecovery(pending, snapshot)).toBeNull(); + const published = createNativeControllerRecovery(snapshot, compatibility, binding); + expect(readNativeControllerRecovery({ ...published, schema: "future" }, snapshot)).toBeNull(); + expect(readNativeControllerRecovery({ ...published, processConfigDigest: "unverified" }, snapshot)).toBeNull(); + }); + + it("enforces the metadata byte bound on both write and read", () => { + expect(() => createNativeControllerRecovery(snapshot, { ...compatibility, configDigest: "x".repeat(4096) }, binding)) + .toThrow("metadata_too_large"); + expect(readNativeControllerRecovery({ ...createNativeControllerRecovery(snapshot, compatibility, binding), extra: "x".repeat(4096) }, snapshot)) + .toBeNull(); + }); + + it("keeps compatibility metadata out of public run projections", () => { + expect(redactRegisteredSecretValues({ runnerProfileJson: { preserved: true, + nativeControllerRecovery: createNativeControllerRecovery(snapshot, compatibility, binding) } }, [])) + .toEqual({ runnerProfileJson: { preserved: true } }); + }); +}); diff --git a/server/src/services/native-runtime/native-controller-recovery.ts b/server/src/services/native-runtime/native-controller-recovery.ts new file mode 100644 index 00000000000..4c5fd4ac419 --- /dev/null +++ b/server/src/services/native-runtime/native-controller-recovery.ts @@ -0,0 +1,270 @@ +import { and, eq, ne, sql } from "drizzle-orm"; +import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db"; +import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; +import type { DurablePrpCheckpoint, PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; +import { readRunCancellation } from "../run-cancellation.js"; +import { nativeSha256 } from "./canonical.js"; +import { findNativeSessionResumeRun, isNativeSessionId, isUnusedNativeSessionBootstrap, nativeSessionProviderEvidence } from "./native-session-resume.js"; +import { loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; +import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js"; + +export const NATIVE_CONTROLLER_RECOVERY_MAX_BYTES = 4 * 1024; + +/** Private run-profile metadata. No credentials or controller-local paths. */ +export interface NativeControllerRecovery { + schema: "paperclip.native-controller-recovery.v1"; + snapshotSha256: string; + configDigest: string; + processConfigDigest: string | null; + checkpoint: { + runId: string; + runnerInstanceId: string; + normalizedSessionId: string; + } | null; +} + +export type NativeControllerCompatibility = Pick; + +export function createNativeControllerRecovery( + snapshot: PersistedNativeSession, + compatibility: NativeControllerCompatibility, + checkpoint: NativeControllerRecovery["checkpoint"], +): NativeControllerRecovery { + const value: NativeControllerRecovery = { + schema: "paperclip.native-controller-recovery.v1", + snapshotSha256: nativeSha256(snapshot), + configDigest: compatibility.configDigest, + processConfigDigest: compatibility.processConfigDigest, + checkpoint, + }; + if (Buffer.byteLength(JSON.stringify(value)) > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES) { + throw new Error("native_controller_recovery_metadata_too_large"); + } + return value; +} + +function record(value: unknown): Record { + return value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} + +export function readNativeControllerRecovery(value: unknown, snapshot: unknown): NativeControllerRecovery | null { + const envelope = record(value), checkpoint = record(envelope.checkpoint); + if (Buffer.byteLength(JSON.stringify(value) ?? "null") > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES + || envelope.schema !== "paperclip.native-controller-recovery.v1" + || typeof envelope.snapshotSha256 !== "string" || envelope.snapshotSha256 !== nativeSha256(snapshot) + || typeof envelope.configDigest !== "string" || !/^sha256:[a-f0-9]{64}$/.test(envelope.configDigest) + || !(envelope.processConfigDigest === null || (typeof envelope.processConfigDigest === "string" && /^[a-f0-9]{64}$/.test(envelope.processConfigDigest))) + || ![checkpoint.runId, checkpoint.runnerInstanceId, checkpoint.normalizedSessionId].every(v => typeof v === "string" && v.length > 0)) return null; + return envelope as unknown as NativeControllerRecovery; +} + +export type NativeControllerRecoveryOutcome = + | { kind: "recoverable"; snapshot: PersistedNativeSession; metadata: NativeControllerRecovery; checkpoint: DurablePrpCheckpoint } + | { kind: "waiting_for_owner"; deadline: Date } + | { kind: "missing_evidence" | "unsupported" | "dependency_unavailable"; reason: string; cause?: unknown } + | { kind: "no_prior_execution" }; + +export const NATIVE_RETAINED_OWNER_WAIT = "native_controller_recovery_waiting_for_owner"; + +export class NativeControllerRecoveryWaitingError extends Error { + constructor(readonly retryAt: Date) { + super(NATIVE_RETAINED_OWNER_WAIT); + this.name = "NativeControllerRecoveryWaitingError"; + } +} + +/** Admission has not crossed the provider boundary. Retain this same business + * run on the existing retry owner; never turn an ownership wait into a failure + * or invent a new provider attempt. The usual resumption gates recheck Stop, + * permissions and task ownership when this deadline arrives. */ +export async function deferNativeControllerRecovery(db: Db, input: { + companyId: string; agentId: string; issueId: string; runId: string; + leaseOwner?: string; retryAt: Date; +}): Promise { + await db.transaction(async tx => { + const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues) + .where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update"); + const [owner] = await tx.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), + eq(nativeRunFinalizations.runId, input.runId))).for("update"); + const [run] = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId))).for("update"); + const [authority] = await tx.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints) + .where(and(eq(nativeRunnerCheckpoints.companyId, input.companyId), eq(nativeRunnerCheckpoints.runId, input.runId))).limit(1); + const progress = await nativeSessionProviderEvidence(tx as unknown as Db, [input.runId]); + const cancellation = record(run?.resultJson?.nativeCancellation); + const stop = readRunCancellation(run?.resultJson); + const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`); + if (!run || run.runtimeMode !== "native" || run.status !== "running" || task?.executionRunId !== input.runId + || !owner || owner.phase !== "observed" || owner.resultId || owner.attempt !== 0 + || owner.controllerGeneration !== 0 || owner.leaseOwner !== (input.leaseOwner ?? null) + || (input.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt <= new Date(clock.now))) + || run.processPid || run.processGroupId || run.processStartedAt || authority || progress.size + || !record(run.runnerProfileJson).nativeExecutionInput || run.resultJson?.startupCancellation + || (stop?.source === "operator" && stop.expected) + || (["run", "issue"].includes(String(cancellation.scope)) && ["pending", "acknowledged"].includes(String(cancellation.dispatchState)))) { + throw new Error("native_controller_recovery_wait_authority_changed"); + } + await notifyNativeFinalizationWork(tx, input); + await tx.update(nativeRunFinalizations).set({ + phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null, + nextAttemptAt: input.retryAt, recoveryState: "resuming_session", failureCode: NATIVE_RETAINED_OWNER_WAIT, + failureDetail: { originalFailureCode: NATIVE_RETAINED_OWNER_WAIT, + message: "Waiting for the retained native controller owner to release its lease.", + recoveryMode: record(run.runnerProfileJson).sessionCheckpoint ? "same_session_resume" : "bootstrap_retry", + providerSessionEstablished: false, providerEventsExist: false }, + updatedAt: new Date(), + }).where(eq(nativeRunFinalizations.runId, input.runId)); + }); +} + +/** Bind a read of retained Runner lifecycle to the surviving sandbox. The + * ordinary admission owner still checks current permissions before execution. */ +export async function assertNativeRetainedSandboxBinding(db: Db, input: { + companyId: string; issueId: string; runId: string; + target: Extract; +}): Promise { + const { target } = input; + if (target.transport !== "sandbox" || target.sandboxLeaseAcquisition?.outcome !== "resumed" + || !target.providerKey || !target.environmentId || !target.leaseId) { + throw new Error("native_controller_recovery_dependency_unavailable:retained_sandbox"); + } + const [owner] = await db.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), + eq(nativeRunFinalizations.runId, input.runId))); + const leases = await db.select({ metadata: environmentLeases.metadata, status: environmentLeases.status }).from(environmentLeases).where(and( + eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.issueId, input.issueId), + eq(environmentLeases.heartbeatRunId, input.runId), eq(environmentLeases.environmentId, target.environmentId), + eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, target.sandboxLeaseAcquisition.providerLeaseId))); + if (!owner || owner.leaseOwner || owner.phase !== "committed" || !owner.resultId || owner.recoveryState + || leases.length !== 1 || leases[0].metadata?.remoteCwd !== target.remoteCwd + || !["active", "released", "expired"].includes(leases[0].status)) { + throw new Error("native_controller_recovery_missing_evidence:retained_sandbox_binding"); + } +} + +/** Only the existing bounded selector chooses history. Never search backwards + * again when its selected record is incomplete or incompatible. This operation + * reads evidence; it grants neither command nor acknowledgement authority. */ +export async function resolveNativeControllerRecovery(db: Db, input: { + companyId: string; agentId: string; issueId: string; normalizedSessionId: string; currentRunId: string; +}): Promise { + if (!isNativeSessionId(input.normalizedSessionId)) return { kind: "missing_evidence", reason: "session_binding" }; + const [current] = await db.select({ createdAt: heartbeatRuns.createdAt, processPid: heartbeatRuns.processPid, + processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt, + runnerProfileJson: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.currentRunId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId), + eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))); + if (!current) return { kind: "missing_evidence", reason: "current_binding" }; + const source = await findNativeSessionResumeRun(db, { ...input, beforeCreatedAt: current.createdAt, includeProgressBarrier: true }); + if (!source) { + // An exhausted candidate window is unknown history, not fresh-bootstrap + // proof. Read only IDs, and retain the selector's existing 32-row boundary. + const prior = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + ne(heartbeatRuns.id, input.currentRunId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId), + eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).limit(33); + if (prior.length > 32) return { kind: "missing_evidence", reason: "candidate_window_exhausted" }; + const progress = await nativeSessionProviderEvidence(db, [input.currentRunId]); + if (!isUnusedNativeSessionBootstrap(current, progress.has(input.currentRunId))) { + return { kind: "missing_evidence", reason: "current_execution_progress" }; + } + // The normal admission and remote untouched-session claim still execute. + return { kind: "no_prior_execution" }; + } + const [owner] = await db.select({ leaseOwner: nativeRunFinalizations.leaseOwner, leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt }).from(nativeRunFinalizations) + .where(and(eq(nativeRunFinalizations.runId, source.id), eq(nativeRunFinalizations.companyId, input.companyId), + eq(nativeRunFinalizations.issueId, input.issueId))); + if (owner?.leaseOwner && owner.leaseExpiresAt && owner.leaseExpiresAt.getTime() > Date.now()) { + return { kind: "waiting_for_owner", deadline: owner.leaseExpiresAt }; + } + if (source.runtimeMode !== "native" || !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(source.status) + || source.createdAt > current.createdAt) return { kind: "missing_evidence", reason: "newer_progress_barrier" }; + const profile = record(source.runnerProfileJson), snapshot = record(profile.sessionCheckpoint); + const identity = record(snapshot.identity); + if (identity.companyId !== input.companyId || identity.agentId !== input.agentId + || identity.issueId !== input.issueId || identity.runId !== source.id || identity.sessionId !== input.normalizedSessionId) { + return { kind: "missing_evidence", reason: "snapshot_binding" }; + } + if (record(profile.nativeControllerRecovery).schema !== "paperclip.native-controller-recovery.v1") { + return { kind: "unsupported", reason: "metadata_format" }; + } + const metadata = readNativeControllerRecovery(profile.nativeControllerRecovery, snapshot); + if (!metadata?.checkpoint || metadata.checkpoint.runId !== source.id + || metadata.checkpoint.normalizedSessionId !== input.normalizedSessionId + || metadata.checkpoint.runnerInstanceId !== source.runnerInstanceId) { + return { kind: "missing_evidence", reason: "snapshot_digest_or_checkpoint_binding" }; + } + let checkpoint: DurablePrpCheckpoint | null; + try { + checkpoint = await loadNativeRunnerCheckpoint(db, { companyId: input.companyId, issueId: input.issueId, runId: source.id }); + } catch (cause) { + return { kind: "dependency_unavailable", reason: "checkpoint_authority_unreadable", cause }; + } + if (!checkpoint || checkpoint.recoveryEvidence.runnerInstanceId !== metadata.checkpoint.runnerInstanceId + || checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId) { + return { kind: "missing_evidence", reason: "checkpoint_binding" }; + } + return { kind: "recoverable", snapshot: snapshot as unknown as PersistedNativeSession, metadata, checkpoint }; +} + +/** Bounded internal migration of ONE retained source while its disk is available. + * The migration owner supplies compatibility computed from the admitted input. + * This never repairs/replaces the saved snapshot or claims a live execution. */ +export async function importNativeControllerRecovery(db: Db, input: { + companyId: string; agentId: string; issueId: string; runId: string; normalizedSessionId: string; + expectedConfigDigest: string; + supervisor: unknown; +}): Promise<{ kind: "imported" | "unchanged" } | Exclude> { + const supervisor = record(input.supervisor); + if (supervisor.schema !== "paperclip.native-session-supervisor.v1" || supervisor.configDigest !== input.expectedConfigDigest) { + return { kind: "unsupported", reason: "supervisor_compatibility" }; + } + return db.transaction(async tx => { + const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues) + .where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("share"); + const [owner] = await tx.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), + eq(nativeRunFinalizations.runId, input.runId))).for("update"); + if (!task || !owner) return { kind: "missing_evidence" as const, reason: "owner_binding" }; + if (owner.leaseOwner || task.executionRunId) { + return owner.leaseExpiresAt && owner.leaseExpiresAt > new Date() + ? { kind: "waiting_for_owner" as const, deadline: owner.leaseExpiresAt } + : { kind: "missing_evidence" as const, reason: "normal_ownership_recovery_required" }; + } + if (owner.phase !== "committed" || !owner.resultId || owner.recoveryState) { + return { kind: "missing_evidence" as const, reason: "pending_finalization" }; + } + const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), + eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId), + eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).for("update"); + const profile = record(run?.runnerProfileJson), snapshot = record(profile.sessionCheckpoint), identity = record(snapshot.identity); + if (run?.runtimeMode !== "native" || run.status !== "succeeded" || !run.finishedAt + || identity.runId !== input.runId || identity.companyId !== input.companyId || identity.issueId !== input.issueId + || identity.agentId !== input.agentId || identity.sessionId !== input.normalizedSessionId + || nativeSha256(snapshot) !== nativeSha256(redactRunnerApiPayloads(supervisor.snapshot))) { + return { kind: "missing_evidence" as const, reason: "stale_supervisor_snapshot" }; + } + const checkpoint = await loadNativeRunnerCheckpoint(tx, input); + if (!checkpoint || checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId + || checkpoint.recoveryEvidence.runnerInstanceId !== run.runnerInstanceId) { + return { kind: "missing_evidence" as const, reason: "checkpoint_binding" }; + } + const metadata = createNativeControllerRecovery(snapshot as unknown as PersistedNativeSession, { + configDigest: input.expectedConfigDigest, + processConfigDigest: typeof supervisor.processConfigDigest === "string" ? supervisor.processConfigDigest : null, + }, { runId: input.runId, runnerInstanceId: checkpoint.recoveryEvidence.runnerInstanceId, normalizedSessionId: input.normalizedSessionId }); + if (!readNativeControllerRecovery(metadata, snapshot)) return { kind: "unsupported" as const, reason: "metadata_format" }; + if (profile.nativeControllerRecovery) { + return nativeSha256(profile.nativeControllerRecovery) === nativeSha256(metadata) + ? { kind: "unchanged" as const } + : { kind: "missing_evidence" as const, reason: "existing_metadata_conflict" }; + } + await tx.update(heartbeatRuns).set({ runnerProfileJson: { ...profile, nativeControllerRecovery: metadata }, updatedAt: new Date() }) + .where(eq(heartbeatRuns.id, input.runId)); + return { kind: "imported" as const }; + }); +} diff --git a/server/src/services/native-runtime/native-harness-backup-expiration.ts b/server/src/services/native-runtime/native-harness-backup-expiration.ts index 8ea9a242629..34402b8ad2d 100644 --- a/server/src/services/native-runtime/native-harness-backup-expiration.ts +++ b/server/src/services/native-runtime/native-harness-backup-expiration.ts @@ -181,6 +181,8 @@ async function sessionRetentionBlocker( id: heartbeatRuns.id, status: heartbeatRuns.status, finishedAt: heartbeatRuns.finishedAt, + incompleteRecovery: sql`(${heartbeatRuns.runnerProfileJson}->'nativeControllerRecovery'->>'schema' = 'paperclip.native-controller-recovery.v1' + and ${heartbeatRuns.runnerProfileJson}->'nativeControllerRecovery'->'checkpoint' = 'null'::jsonb) is true`, issueId: sql`coalesce(${heartbeatRuns.nativeIssueId}, ${heartbeatRuns.issueId})`, }) .from(heartbeatRuns) @@ -194,6 +196,7 @@ async function sessionRetentionBlocker( if (runs.some((run) => LIVE_RUN_STATUSES.includes(run.status) || !run.finishedAt)) { return { reason: "live_run" }; } + if (runs.some(run => run.incompleteRecovery)) return { reason: "native_recovery_incomplete" }; if (input.checkpoint && runs.some(run => run.finishedAt! >= input.closedBefore)) return { reason: "recent_run" }; const issueIds = [...new Set(runs.map((run) => run.issueId).filter((id): id is string => Boolean(id)))]; if (issueIds.length === 0) return { reason: "issue_unknown" }; diff --git a/server/src/services/native-runtime/native-recovery-usage-delivery.test.ts b/server/src/services/native-runtime/native-recovery-usage-delivery.test.ts new file mode 100644 index 00000000000..c2fdaec2e21 --- /dev/null +++ b/server/src/services/native-runtime/native-recovery-usage-delivery.test.ts @@ -0,0 +1,92 @@ +import { randomUUID } from "node:crypto"; +import { PgDialect } from "drizzle-orm/pg-core"; +import type { SQL } from "drizzle-orm"; +import type { Db } from "@paperclipai/db"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import type { DurableRecoveryIdentity, PrpEvent } from "../../vendor/paperclip-runner/index.js"; +import { buildNativeExecutionInput } from "./native-execution-input.js"; +import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; +import { createNativeRecoveryUsageDelivery } from "./native-recovery-usage-delivery.js"; + +const accounting = vi.hoisted(() => ({ open: vi.fn(), capture: vi.fn() })); +vi.mock("./native-usage-delivery.js", () => ({ createNativeUsageContinuationDelivery: accounting.open })); +beforeEach(() => { vi.clearAllMocks(); accounting.capture.mockResolvedValue(undefined); accounting.open.mockResolvedValue({ capture: accounting.capture }); }); + +function fixture() { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), sessionId = randomUUID(); + const source: DurableRecoveryIdentity = { runId: randomUUID(), runnerInstanceId: randomUUID(), environmentLeaseId: randomUUID(), + normalizedSessionId: sessionId, turnId: randomUUID(), itemId: randomUUID() }; + const build = (runId: string) => buildNativeExecutionInput({ companyId, agentId, runId, normalizedSessionId: sessionId, + provider: "codex", issue: { id: issueId, identifier: "REC-1", title: "Continue", description: null, workMode: "standard" }, + taskPrompt: "Continue once", workspace: { id: randomUUID(), cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null }, + completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", + contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Preserve the result" }] } }, + runtimeContext: nativeRuntimeContextFixture() }); + const execution = build(randomUUID()), previous = build(source.runId); + const row = { run: { status: "succeeded", finishedAt: new Date() as Date | null, runtimeMode: "native", runnerInstanceId: source.runnerInstanceId, + nativeSessionId: sessionId, runnerProfileJson: { nativeExecutionInput: previous } }, + owner: { phase: "committed", resultId: randomUUID() as string | null, leaseOwner: null as string | null, companyId, issueId } }; + const where = vi.fn(async (_predicate: SQL) => [row]); + const select = vi.fn(() => ({ from: () => ({ innerJoin: () => ({ where }) }) })); + const db = { select } as unknown as Db; + const matchesExecution = vi.fn(() => true); + const deliver = () => createNativeRecoveryUsageDelivery({ db, execution, sourceIdentity: { ...source }, matchesExecution }); + const event: PrpEvent = { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", sourceInstanceId: source.runnerInstanceId, + sourceEventId: "source-usage", sourceSeq: 1, runId: source.runId, normalizedSessionId: sessionId, turnId: source.turnId, + eventType: "usage.reported", priority: 0, emittedAt: new Date().toISOString(), payload: {} }; + return { source, execution, previous, row, select, where, matchesExecution, deliver, event }; +} + +describe("source accounting recovery authority", () => { + it("pins the relational source tuple and hydrates once without binding target accounting", async () => { + const f = fixture(), deliver = f.deliver(); + await deliver(f.event, f.source); + await deliver({ ...f.event, sourceEventId: "source-terminal", sourceSeq: 2, eventType: "turn.completed" }, f.source); + expect(f.select).toHaveBeenCalledOnce(); + expect(new PgDialect().sqlToQuery(f.where.mock.calls[0][0]).params).toEqual([ + f.source.runId, f.execution.binding.companyId, f.execution.binding.agentId, f.execution.binding.issueId, + ]); + expect(f.matchesExecution).toHaveBeenCalledWith(f.previous); + expect(accounting.open).toHaveBeenCalledExactlyOnceWith(expect.anything(), { + companyId: f.execution.binding.companyId, runId: f.source.runId, runnerInstanceId: f.source.runnerInstanceId, provider: f.previous.provider, + }); + expect(accounting.capture).toHaveBeenCalledTimes(2); + expect(accounting.capture).toHaveBeenCalledWith(f.event); + }); + + it.each(["runId", "runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "turnId", "itemId"] as const)( + "rejects a changed pinned %s before source hydration", async key => { + const f = fixture(); + await expect(f.deliver()(f.event, { ...f.source, [key]: randomUUID() })).rejects.toThrow("native_recovery_usage_source_unproven"); + expect(f.select).not.toHaveBeenCalled(); + expect(accounting.open).not.toHaveBeenCalled(); + }); + + it.each(["runId", "sourceInstanceId", "normalizedSessionId"] as const)("rejects mismatched event %s before source hydration", async key => { + const f = fixture(); + await expect(f.deliver()({ ...f.event, [key]: randomUUID() }, f.source)).rejects.toThrow("native_recovery_usage_source_unproven"); + expect(f.select).not.toHaveBeenCalled(); + }); + + it.each(["nonterminal", "unfinished", "uncommitted", "no-result", "held", "company", "issue", "scope"])( + "keeps an unproven source held: %s", async change => { + const f = fixture(); + if (change === "nonterminal") f.row.run.status = "running"; + if (change === "unfinished") f.row.run.finishedAt = null; + if (change === "uncommitted") f.row.owner.phase = "observed"; + if (change === "no-result") f.row.owner.resultId = null; + if (change === "held") f.row.owner.leaseOwner = "live-owner"; + if (change === "company") f.row.owner.companyId = randomUUID(); + if (change === "issue") f.row.owner.issueId = randomUUID(); + if (change === "scope") f.matchesExecution.mockReturnValue(false); + await expect(f.deliver()(f.event, f.source)).rejects.toThrow("native_recovery_usage_source_unproven"); + expect(accounting.open).not.toHaveBeenCalled(); + expect(accounting.capture).not.toHaveBeenCalled(); + }); + + it("keeps a durable source receipt failure unacknowledged", async () => { + const f = fixture(); + accounting.capture.mockRejectedValue(new Error("native_usage_after_finalization_requires_reconciliation")); + await expect(f.deliver()(f.event, f.source)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation"); + }); +}); diff --git a/server/src/services/native-runtime/native-recovery-usage-delivery.ts b/server/src/services/native-runtime/native-recovery-usage-delivery.ts new file mode 100644 index 00000000000..9401c72dd71 --- /dev/null +++ b/server/src/services/native-runtime/native-recovery-usage-delivery.ts @@ -0,0 +1,47 @@ +import { and, eq } from "drizzle-orm"; +import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { parseNativeExecutionInput, type DurableRecoveryIdentity, type NativeExecutionInput, type PrpEvent } from "../../vendor/paperclip-runner/index.js"; +import { createNativeUsageContinuationDelivery } from "./native-usage-delivery.js"; + +const identityKeys = ["runId", "runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "turnId", "itemId"] as const; + +/** A cold handoff may discover accounting that belongs to its completed source. + * Hydrate that producer once; never route its receipts through the new run. */ +export function createNativeRecoveryUsageDelivery(input: { + db: Db; + execution: NativeExecutionInput; + sourceIdentity: Record; + matchesExecution(previous: NativeExecutionInput): boolean; +}) { + let delivery: ReturnType | undefined; + const deny = (): never => { throw new Error("native_recovery_usage_source_unproven"); }; + const open = async (source: DurableRecoveryIdentity) => { + const { binding } = input.execution; + const [prior] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations }) + .from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) + .where(and(eq(heartbeatRuns.id, source.runId), eq(heartbeatRuns.companyId, binding.companyId), + eq(heartbeatRuns.agentId, binding.agentId), eq(heartbeatRuns.nativeIssueId, binding.issueId))); + if (!prior || prior.run.status !== "succeeded" || prior.run.finishedAt === null || + prior.run.runtimeMode !== "native" || prior.run.runnerInstanceId !== source.runnerInstanceId || + prior.run.nativeSessionId !== source.normalizedSessionId || prior.owner.phase !== "committed" || + !prior.owner.resultId || prior.owner.leaseOwner || prior.owner.companyId !== binding.companyId || + prior.owner.issueId !== binding.issueId) return deny(); + const previous = parseNativeExecutionInput(prior.run.runnerProfileJson?.nativeExecutionInput); + if (previous.binding.companyId !== binding.companyId || previous.binding.agentId !== binding.agentId || + previous.binding.issueId !== binding.issueId || previous.binding.runId !== source.runId || + !input.matchesExecution(previous)) return deny(); + return createNativeUsageContinuationDelivery(input.db, { + companyId: binding.companyId, runId: source.runId, + runnerInstanceId: source.runnerInstanceId, provider: previous.provider, + }); + }; + return async (event: PrpEvent, source: DurableRecoveryIdentity) => { + if (source.runId === input.execution.binding.runId || + identityKeys.some(key => typeof input.sourceIdentity[key] !== "string" || + !input.sourceIdentity[key] || source[key] !== input.sourceIdentity[key]) || + event.runId !== source.runId || event.sourceInstanceId !== source.runnerInstanceId || + event.normalizedSessionId !== source.normalizedSessionId) return deny(); + delivery ??= open(source); + await (await delivery).capture(event); + }; +} diff --git a/server/src/services/native-runtime/native-restart-recovery.test.ts b/server/src/services/native-runtime/native-restart-recovery.test.ts index f032829c639..fd79245b6e5 100644 --- a/server/src/services/native-runtime/native-restart-recovery.test.ts +++ b/server/src/services/native-runtime/native-restart-recovery.test.ts @@ -8,6 +8,18 @@ import { } from "./native-restart-recovery.js"; describe("native restart recovery classification", () => { + it.each([false, true])("recognizes verified staged remote evidence with target snapshot=%s", hasCheckpoint => { + const evidence = { remoteSandbox: true, runnerPidAlive: false, runnerGroupAlive: false, + processStartMatches: false, hasCheckpoint, hasProviderEvidence: false }; + expect(classifyNativeRunnerRecoveryEvidence(evidence).claimKind).toBeNull(); + expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, stagedAttachmentVerified: true })).toEqual({ + claimKind: "reattach_remote_runner", reason: "staged_remote_attachment_requires_sandbox_verification", + }); + expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, stagedAttachmentVerified: true, + checkpointFailed: true }).claimKind).toBeNull(); + expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, remoteSandbox: false, + stagedAttachmentVerified: true }).claimKind).toBe(hasCheckpoint ? null : "bootstrap_incomplete"); + }); it.each([true, false])("uses the remote fenced lease instead of the old container PID (live=%s)", async live => { const now = new Date(); const probe = vi.fn(() => { throw new Error("must not probe another container's PID"); }); diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index 8ac4d9bd918..53ccc1b07f7 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -1,3 +1,4 @@ +import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js"; import { notifyNativeEnvironmentSettlement } from "../native-environment-settlement-notifications.js"; import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js"; import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; @@ -6,7 +7,8 @@ import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-pre import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { preserveNativeFailedWorkspace } from "./native-failed-workspace-recovery.js"; -import { hasNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; +import { hasNativeRunnerCheckpoint, loadNativeRunnerCheckpointHeader } from "./native-runner-checkpoint.js"; +import { resolveNativeStagedAttachmentRecovery, type NativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js"; import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js"; import { activeNativeExecutionRunIds, NATIVE_PREPARATION_FENCING_VERSION } from "./native-active-runs.js"; import { @@ -15,7 +17,7 @@ import { recordNativeLocalProcessStop, } from "../native-local-process-stop.js"; import { randomUUID } from "node:crypto"; -import { and, desc, eq, inArray, isNotNull, isNull, lte, or, sql } from "drizzle-orm"; +import { and, desc, eq, inArray, isNotNull, isNull, lte, ne, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { agents, @@ -68,6 +70,8 @@ export type NativeRestartRecoveryClaim = recoveryRequestId: string | null; remote: { kind?: "sandbox"; providerLeaseId: string; remoteCwd: string } | { kind: "plugin_task"; leaseId: string }; + /** Discovery evidence only; executor revalidates the encrypted header. */ + stagedAttachment?: Pick; } | { kind: "resume_dead_runner"; @@ -94,6 +98,7 @@ export type NativeRestartRecoveryDisposition = kind: "awaiting_evidence" | "blocked" | "already_finalized"; runId: string; reason: string; + retryAt?: Date; }; export function nextNativeProviderAttempt( @@ -320,6 +325,7 @@ export function classifyNativeRunnerRecoveryEvidence(input: { checkpointIdentityMatches?: boolean; hasProviderEvidence: boolean; checkpointFailed?: boolean; + stagedAttachmentVerified?: boolean; providerAttempt?: number; }): { claimKind: NativeRestartRecoveryClaim["kind"] | null; @@ -334,6 +340,9 @@ export function classifyNativeRunnerRecoveryEvidence(input: { // inspection; the executor must prove the original lease and authenticate // the surviving runner before it can resume delivery. if (input.remoteSandbox) { + if (input.stagedAttachmentVerified === true) { + return { claimKind: "reattach_remote_runner", reason: "staged_remote_attachment_requires_sandbox_verification" }; + } if (input.unclaimedBootstrap === true && !input.hasCheckpoint && !input.hasProviderEvidence) { return { claimKind: "bootstrap_incomplete", reason: "remote_execution_never_claimed" }; } @@ -514,6 +523,9 @@ export async function claimNativeRestartRecoveries(input: { eq(agents.adapterType, "paperclip_runner"), inArray(heartbeatRuns.status, ["running", "failed"]), isNull(nativeRunFinalizations.resultId), + // A pre-provider retained-owner wait belongs to normal same-run + // resumption, including when its deadline elapsed while we were down. + or(isNull(nativeRunFinalizations.failureCode), ne(nativeRunFinalizations.failureCode, NATIVE_RETAINED_OWNER_WAIT)), or( eq(nativeRunFinalizations.phase, "observed"), and( @@ -668,9 +680,21 @@ export async function claimNativeRestartRecoveries(input: { kind: "awaiting_evidence", runId: row.run.id, reason: takeover.reason, + ...(durableAuthority && row.coordinator.leaseExpiresAt ? { retryAt: row.coordinator.leaseExpiresAt } : {}), } as const; } + // Only the classic remote path stages attachments. After the owner is + // eligible for takeover, inspect its bounded header even if activation + // already published a runtime snapshot. Never scan command/event payloads. + const stagedHeader = remoteWorkspace && !pluginTask && durableAuthority + ? await loadNativeRunnerCheckpointHeader(tx, { + companyId: row.run.companyId, issueId: row.coordinator.issueId, runId: row.run.id, + }) : null; + const stagedAttachment = stagedHeader ? await resolveNativeStagedAttachmentRecovery({ + db: tx, currentRun: row.run, header: stagedHeader, + }) : null; + const runnerPidAlive = !remoteProcess && processIsAlive(row.run.processPid); const runnerGroupAlive = @@ -859,6 +883,7 @@ export async function claimNativeRestartRecoveries(input: { knownProviderProcessIdentityAmbiguous: providerProcesses.ambiguousLivePids.length > 0, hasCheckpoint, + stagedAttachmentVerified: stagedAttachment !== null, checkpointIdentityMatches: checkpointIdentityMatches && (!remoteSandbox || remoteWorkspace !== null), @@ -1109,6 +1134,10 @@ export async function claimNativeRestartRecoveries(input: { return { kind: claimKind, ...common, + ...(stagedAttachment ? { stagedAttachment: { + sourceIdentity: stagedAttachment.sourceIdentity, targetIdentity: stagedAttachment.targetIdentity, + templateDigest: stagedAttachment.templateDigest, + } } : {}), remote: pluginTask ? { kind: "plugin_task", leaseId: pluginTask.id } : { providerLeaseId: remoteWorkspace!.providerLeaseId, remoteCwd: remoteWorkspace!.remoteCwd, diff --git a/server/src/services/native-runtime/native-runner-checkpoint.ts b/server/src/services/native-runtime/native-runner-checkpoint.ts index 3cdef6040e2..00859cfd3f0 100644 --- a/server/src/services/native-runtime/native-runner-checkpoint.ts +++ b/server/src/services/native-runtime/native-runner-checkpoint.ts @@ -5,7 +5,7 @@ import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provid import { legacyControllerBootId } from "../legacy-controller-lease.js"; type Binding = { companyId: string; issueId: string; runId: string }; -type Owner = Binding & { bootId: string; leaseOwner: string; generation: number; authorityRunId: string }; +export type NativeRunnerCheckpointOwner = Binding & { bootId: string; leaseOwner: string; generation: number; authorityRunId: string }; function encryptParts(binding: Binding, records: ReadonlyMap) { if (!records.size) return []; const cipher = createLocalEncryptedCipher("write"); @@ -25,49 +25,79 @@ function decryptPart(cipher: ReturnType, bind return envelope.contents; } -export function nativeRunnerCheckpointWriter(db: Db, owner: Owner) { +type CheckpointTransaction = Parameters[0]>[0]; + +/** A dependent run-lock query can consume this CTE to preserve coordinator-first + * lock order without adding a second database round trip. */ +export function nativeRunnerCheckpointOwnerLock(tx: CheckpointTransaction, owner: NativeRunnerCheckpointOwner) { + return tx.$with("checkpoint_owner").as(tx.select({ + leaseOwner: nativeRunFinalizations.leaseOwner, + leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt, + resultId: nativeRunFinalizations.resultId, + }).from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, owner.companyId), eq(nativeRunFinalizations.issueId, owner.issueId), + eq(nativeRunFinalizations.runId, owner.authorityRunId), eq(nativeRunFinalizations.controllerBootId, owner.bootId), + eq(nativeRunFinalizations.controllerGeneration, owner.generation))).for("update")); +} + +/** Locks the coordinator before callers lock a run. A caller that subsequently + * waits for another lock must recheck the fence before writing. */ +export async function lockNativeRunnerCheckpointOwner( + tx: CheckpointTransaction, owner: NativeRunnerCheckpointOwner, +) { + // A locking CTE acquires ownership before the outer projection samples + // wall time. Sampling in the locking SELECT can precede its lock wait. + // Keep both steps in one statement without another database round trip. + const locked = nativeRunnerCheckpointOwnerLock(tx, owner); + const [owned] = await tx.with(locked).select({ leaseOwner: locked.leaseOwner, + leaseExpiresAt: locked.leaseExpiresAt, resultId: locked.resultId, now: sql`clock_timestamp()` }).from(locked); + if (!owned) throw new Error("native_runner_checkpoint_owner_lost"); + return owned; +} + +export async function validateNativeRunnerCheckpointOwner( + tx: CheckpointTransaction, owner: NativeRunnerCheckpointOwner, + owned: Awaited>, newWork: boolean, +): Promise { + const current = owned; + const activeOwner = current.leaseOwner === owner.leaseOwner + && current.leaseExpiresAt !== null && current.leaseExpiresAt > new Date(owned.now); + if (!activeOwner) { + const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues) + .where(and(eq(issues.companyId, owner.companyId), eq(issues.id, owner.issueId))); + const [successor] = task?.executionRunId && task.executionRunId !== owner.runId + ? await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.runId, task.executionRunId), eq(nativeRunFinalizations.companyId, owner.companyId), + eq(nativeRunFinalizations.issueId, owner.issueId), eq(nativeRunFinalizations.controllerBootId, owner.bootId), + gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`))) : []; + const [preparingSuccessor] = !successor && task?.executionRunId && task.executionRunId !== owner.runId + ? await tx.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, task.executionRunId), eq(heartbeatRuns.companyId, owner.companyId), + eq(heartbeatRuns.issueId, owner.issueId), eq(heartbeatRuns.runtimeMode, "legacy"), + eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.executionStage, "preparing"), + eq(heartbeatRuns.controllerBootId, legacyControllerBootId), + gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`), + notExists(tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, heartbeatRuns.id))))) : []; + if (!current.resultId || newWork || !task || (task.executionRunId && task.executionRunId !== owner.runId && !successor && !preparingSuccessor)) { + throw new Error("native_runner_checkpoint_owner_lost"); + } + } +} + +export async function assertNativeRunnerCheckpointOwner(tx: CheckpointTransaction, owner: NativeRunnerCheckpointOwner, newWork: boolean) { + const locked = await lockNativeRunnerCheckpointOwner(tx, owner); + await validateNativeRunnerCheckpointOwner(tx, owner, locked, newWork); +} + +export function nativeRunnerCheckpointWriter(db: Db, owner: NativeRunnerCheckpointOwner) { const persistenceDb = checkpointDb(db); let previousWork = new Set(); return async (checkpoint: DurablePrpCheckpoint): Promise => { if (checkpoint.runId !== owner.runId) throw new Error("native_runner_checkpoint_binding_mismatch"); const changed = encryptParts(owner, checkpoint.changedRecords); await persistenceDb.transaction(async tx => { - // A locking CTE acquires ownership before the outer projection samples - // wall time. Sampling in the locking SELECT can precede its lock wait. - // Keep both steps in one statement without another database round trip. - const locked = tx.$with("checkpoint_owner").as(tx.select().from(nativeRunFinalizations) - .where(and(eq(nativeRunFinalizations.companyId, owner.companyId), - eq(nativeRunFinalizations.issueId, owner.issueId), eq(nativeRunFinalizations.runId, owner.authorityRunId), - eq(nativeRunFinalizations.controllerBootId, owner.bootId), - eq(nativeRunFinalizations.controllerGeneration, owner.generation))).for("update")); - const [owned] = await tx.with(locked).select({ leaseOwner: locked.leaseOwner, - leaseExpiresAt: locked.leaseExpiresAt, resultId: locked.resultId, now: sql`clock_timestamp()` }).from(locked); - if (!owned) throw new Error("native_runner_checkpoint_owner_lost"); - const current = owned; - const activeOwner = current.leaseOwner === owner.leaseOwner - && current.leaseExpiresAt !== null && current.leaseExpiresAt > new Date(owned.now); - if (!activeOwner) { - const newWork = checkpoint.workCommandIds.some(id => !previousWork.has(id)); - const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues) - .where(and(eq(issues.companyId, owner.companyId), eq(issues.id, owner.issueId))); - const [successor] = task?.executionRunId && task.executionRunId !== owner.runId - ? await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and( - eq(nativeRunFinalizations.runId, task.executionRunId), eq(nativeRunFinalizations.companyId, owner.companyId), - eq(nativeRunFinalizations.issueId, owner.issueId), eq(nativeRunFinalizations.controllerBootId, owner.bootId), - gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`))) : []; - const [preparingSuccessor] = !successor && task?.executionRunId && task.executionRunId !== owner.runId - ? await tx.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).where(and( - eq(heartbeatRuns.id, task.executionRunId), eq(heartbeatRuns.companyId, owner.companyId), - eq(heartbeatRuns.issueId, owner.issueId), eq(heartbeatRuns.runtimeMode, "legacy"), - eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.executionStage, "preparing"), - eq(heartbeatRuns.controllerBootId, legacyControllerBootId), - gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`), - notExists(tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations) - .where(eq(nativeRunFinalizations.runId, heartbeatRuns.id))))) : []; - if (!current.resultId || newWork || !task || (task.executionRunId && task.executionRunId !== owner.runId && !successor && !preparingSuccessor)) { - throw new Error("native_runner_checkpoint_owner_lost"); - } - } + await assertNativeRunnerCheckpointOwner(tx, owner, checkpoint.workCommandIds.some(id => !previousWork.has(id))); if (checkpoint.predecessor) { const [prior] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( eq(heartbeatRuns.id, checkpoint.predecessor.runId), eq(heartbeatRuns.companyId, owner.companyId), @@ -150,7 +180,17 @@ export async function registerNativeRunnerCheckpoint(input: { await input.authority.flushCheckpoint(); } -export async function loadNativeRunnerCheckpoint(db: Db, binding: Binding): Promise { +/** Exact-run bounded metadata read; never loads command/event payload records. */ +export async function loadNativeRunnerCheckpointHeader(db: Pick, binding: Binding): Promise { + const [row] = await db.select({ material: nativeRunnerCheckpoints.material }).from(nativeRunnerCheckpoints).where(and( + eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId), + eq(nativeRunnerCheckpoints.runId, binding.runId), eq(nativeRunnerCheckpoints.part, "header"))).limit(1); + if (!row) return null; + try { return decryptPart(createLocalEncryptedCipher("read"), binding, "header", row.material); } + catch { return null; } +} + +export async function loadNativeRunnerCheckpoint(db: Pick, binding: Binding): Promise { const rows = await db.select().from(nativeRunnerCheckpoints).where(and( eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId), eq(nativeRunnerCheckpoints.runId, binding.runId))); diff --git a/server/src/services/native-runtime/native-runner-execution-driver.test.ts b/server/src/services/native-runtime/native-runner-execution-driver.test.ts index 28bc0b34db1..51c79577e02 100644 --- a/server/src/services/native-runtime/native-runner-execution-driver.test.ts +++ b/server/src/services/native-runtime/native-runner-execution-driver.test.ts @@ -6,9 +6,12 @@ import { prepareNativeRemoteRunner, createNativeRunnerExecutionDriver } from "./ const transport = vi.hoisted(() => ({ register: vi.fn(), connect: vi.fn(), + checkpoint: vi.fn(), + checkpointRegistration: vi.fn(), })); vi.mock("../../realtime/runner-prp-ws.js", () => ({ registerRunnerPrpAuthority: transport.register })); vi.mock("../../realtime/runner-prp-outbound.js", () => ({ connectRunnerPrpIngress: transport.connect })); +vi.mock("./native-runner-checkpoint.js", () => ({ nativeRunnerCheckpointRegistration: transport.checkpointRegistration })); const binding = { companyId: "company", issueId: "issue", agentId: "agent", runId: "run-a" }; const authority = {} as Parameters["controlPlaneRegistration"]>>[0]; @@ -30,9 +33,41 @@ beforeEach(() => { vi.clearAllMocks(); transport.register.mockResolvedValue({ connectUrl: "ws://127.0.0.1/core", release: vi.fn() }); transport.connect.mockReturnValue({ ready: Promise.resolve(), close: vi.fn(async () => {}) }); + transport.checkpointRegistration.mockReturnValue(transport.checkpoint); + transport.checkpoint.mockResolvedValue(undefined); }); describe("shared Runner execution driver", () => { + it("installs both staged checkpoint writers under the business lease without acquiring endpoints", async () => { + const prepare = vi.fn(); + const persistence = { db: {} as never, bootId: "boot-c", authorityRunId: "run-c" }; + const driver = createNativeRunnerExecutionDriver({ target: sandbox(), binding: { ...binding, runId: "run-c" }, + controllerPersistence: persistence, prepareRemote: prepare }); + const sourceIdentity = { runnerInstanceId: "runner", environmentLeaseId: "lease", normalizedSessionId: "session", + runId: "run-a", turnId: "turn-a", itemId: "item-a" }; + const targetIdentity = { ...sourceIdentity, runId: "run-c", turnId: "turn-c", itemId: "item-c" }; + await driver.onStagedRunAttachment!(authority, { sourceIdentity, targetIdentity, templateDigest: `sha256:${"a".repeat(64)}` }); + expect(transport.checkpointRegistration).toHaveBeenCalledExactlyOnceWith({ ...persistence, + binding: { companyId: binding.companyId, issueId: binding.issueId, runId: "run-c" } }); + expect(transport.checkpoint.mock.calls).toEqual([[authority, sourceIdentity], [authority, targetIdentity]]); + expect(prepare).not.toHaveBeenCalled(); + expect(transport.register).not.toHaveBeenCalled(); + expect(transport.connect).not.toHaveBeenCalled(); + }); + + it("installs the target writer first when restoring an already persisted stage", async () => { + const sourceIdentity = { runnerInstanceId: "runner", environmentLeaseId: "lease", normalizedSessionId: "session", + runId: "run-a", turnId: "turn-a", itemId: "item-a" }; + const targetIdentity = { ...sourceIdentity, runId: "run-c", turnId: "turn-c", itemId: "item-c" }; + const restored = { store: { state: { stagedRunAttachment: { sourceIdentity, targetIdentity } } } } as unknown as typeof authority; + const driver = createNativeRunnerExecutionDriver({ target: sandbox(), binding: { ...binding, runId: "run-c" }, + controllerPersistence: { db: {} as never, bootId: "boot-c", authorityRunId: "run-c" } }); + await driver.onStagedRunAttachment!(restored, { sourceIdentity, targetIdentity, templateDigest: `sha256:${"a".repeat(64)}` }); + expect(transport.checkpoint.mock.calls).toEqual([[restored, targetIdentity], [restored, sourceIdentity]]); + expect(transport.register).not.toHaveBeenCalled(); + expect(transport.connect).not.toHaveBeenCalled(); + }); + it("keeps local execution on the registered loopback route", async () => { const driver = createNativeRunnerExecutionDriver({ target: { kind: "local" }, binding }); const connection = await driver.controlPlaneRegistration!(authority); diff --git a/server/src/services/native-runtime/native-runner-execution-driver.ts b/server/src/services/native-runtime/native-runner-execution-driver.ts index 6bf3b44ded5..0929d99d82a 100644 --- a/server/src/services/native-runtime/native-runner-execution-driver.ts +++ b/server/src/services/native-runtime/native-runner-execution-driver.ts @@ -92,6 +92,20 @@ export async function prepareNativeRemoteRunner(input: RemotePreparation) { export function createNativeRunnerExecutionDriver(input: NativeRunnerExecutionDriverInput): RunnerExecutionDriver { return { ...input.process, + onStagedRunAttachment: input.controllerPersistence && input.binding.issueId + ? async (authority, proof) => { + // Both journals use the current business lease. Install persistence + // before staging; endpoint acquisition still belongs to attachment. + const register = nativeRunnerCheckpointRegistration({ ...input.controllerPersistence!, + binding: { companyId: input.binding.companyId, issueId: input.binding.issueId!, runId: input.binding.runId }, + }); + // A restored stage already publishes under the target. Its writer + // must exist before registration flushes that restored revision. + const identities = authority.store?.state.stagedRunAttachment + ? [proof.targetIdentity, proof.sourceIdentity] : [proof.sourceIdentity, proof.targetIdentity]; + for (const identity of identities) await register(authority, identity); + } + : undefined, controlPlaneRegistration: async (authority, attachmentIdentity) => { await input.beforeConnect?.(attachmentIdentity, authority); if (input.controllerPersistence && input.binding.issueId) { diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 3dbcd10ab35..6711939a601 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -60,6 +60,7 @@ import { readProcessStartedAt } from "../hot-restart.js"; import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; import { buildNativeExecutionInput } from "./native-execution-input.js"; +import { buildNativeExecutionWithCheckpoint, findNativeSessionResumeRun, nativeToolContractFingerprintForTarget } from "./native-session-resume.js"; import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; import { materializeAsset } from "./runtime-context.js"; import { createRunnerTestTlsTunnel, dockerCommandRunner } from "../../__tests__/helpers/docker-command-runner.js"; @@ -321,8 +322,10 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( }; } - it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "legacy", "changed", "same-server"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { + it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "legacy", "changed", "same-server", "diskless", "diskless-per-turn", "diskless-per-turn-read-retry"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { const hardRestart = !["warm", "same-server"].includes(scenario); + const diskless = scenario.startsWith("diskless"); + const perTurn = scenario.startsWith("diskless-per-turn"); const container = process.env.PAPERCLIP_TEST_RUNNER_CONTAINER!; const tunnel = await createRunnerTestTlsTunnel({ port: Number(process.env.PAPERCLIP_TEST_RUNNER_PORT), key: process.env.PAPERCLIP_TEST_RUNNER_TLS_KEY!, cert: process.env.NODE_EXTRA_CA_CERTS! }); @@ -346,12 +349,13 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( taskPrompt: "Complete the first turn.", workspace: { id: randomUUID(), cwd: workspace, repoUrl: null, repoRef: null, branchName: null }, normalizedSessionId: fixture.native.normalizedSessionId, - lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 }, + lifecyclePolicy: perTurn ? { mode: "per_turn", idleTimeoutMs: null } : { mode: "warm", idleTimeoutMs: 60_000 }, runtimeContext: context, completionContract: await boundContract(fixture.runId), }); await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...fixture.preparedProfile, nativeExecutionInput: execution, + nativeToolContractFingerprint: nativeToolContractFingerprintForTarget("remote", execution.provider.kind), } }).where(eq(heartbeatRuns.id, fixture.runId)); const remoteCwd = `/tmp/paperclip-continuity-${fixture.issueId}`; expect((await runner.execute({ command: "mkdir", args: ["-p", remoteCwd] })).exitCode).toBe(0); @@ -385,6 +389,26 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( onLog: async (_stream: "stdout" | "stderr", text: string) => { logs.push(text); }, }; let controller: ReturnType | undefined; + const executeInController = (executionOptions: Parameters[0]) => new Promise<{ exitCode: number | null }>((resolveResult, reject) => { + controller = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), resolve("server/src/__tests__/helpers/remote-warm-controller.test-worker.ts")], + { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env }); + let timer = setTimeout(() => { controller?.kill("SIGKILL"); reject(new Error("Controller fixture imports exceeded 120 seconds")); }, 120_000); + controller.stdout?.on("data", data => logs.push(String(data))); + controller.stderr?.on("data", data => logs.push(String(data))); + controller.on("error", error => { clearTimeout(timer); reject(error); }); + controller.on("exit", (code, signal) => { clearTimeout(timer); reject(new Error(`Warm controller exited: ${code}/${signal}: ${logs.join("").slice(-3_000)}`)); }); + controller.on("message", (message: { ready?: boolean; log?: { text: string }; result?: { exitCode: number | null }; error?: string }) => { + if (message.ready) { + clearTimeout(timer); + timer = setTimeout(() => { controller?.kill("SIGKILL"); reject(new Error(`Warm controller admission exceeded 60 seconds: ${logs.join("").slice(-3_000)}`)); }, 60_000); + controller!.send?.({ connectionString: temporary.connectionString, container, ingressUrl: `wss://127.0.0.1:${tunnel.port}`, + options: { ...executionOptions, db: undefined, onLog: undefined, runnerExecutionTarget: { ...executionOptions.runnerExecutionTarget, runner: undefined } } }); + } + if (message.log) logs.push(message.log.text); + if (message.error) { clearTimeout(timer); reject(new Error(message.error)); } + if (message.result) { clearTimeout(timer); resolveResult(message.result); } + }); + }); try { const firstOptions = { ...options, execution, runnerExecutionTarget: target }; const executeFirst = async () => { @@ -454,28 +478,13 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( expect(nextProcess.stdout).toBe(priorProcess.stdout); return result; }; - const first = hardRestart ? await new Promise<{ exitCode: number | null }>((resolveResult, reject) => { - controller = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), resolve("server/src/__tests__/helpers/remote-warm-controller.test-worker.ts")], - { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env }); - const timer = setTimeout(() => { controller?.kill("SIGKILL"); reject(new Error("Warm controller startup exceeded 60 seconds")); }, 60_000); - controller.stdout?.on("data", data => logs.push(String(data))); - controller.stderr?.on("data", data => logs.push(String(data))); - controller.on("error", error => { clearTimeout(timer); reject(error); }); - controller.on("exit", (code, signal) => { clearTimeout(timer); reject(new Error(`Warm controller exited: ${code}/${signal}: ${logs.join("").slice(-3_000)}`)); }); - controller.on("message", (message: { log?: { text: string }; result?: { exitCode: number | null }; error?: string }) => { - if (message.log) logs.push(message.log.text); - if (message.error) { clearTimeout(timer); reject(new Error(message.error)); } - if (message.result) { clearTimeout(timer); resolveResult(message.result); } - }); - controller.send?.({ connectionString: temporary.connectionString, container, ingressUrl: `wss://127.0.0.1:${tunnel.port}`, - options: { ...firstOptions, db: undefined, onLog: undefined, runnerExecutionTarget: { ...target, runner: undefined } } }); - }) : await executeFirst(); + const first = hardRestart ? await executeInController(firstOptions) : await executeFirst(); expect(first.exitCode).toBe(0); await finalizeNativeRun({ db: fixture.db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); const snapshot = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] }); expect(snapshot.exitCode).toBe(0); const saved = JSON.parse(snapshot.stdout); - expect(saved.lifecycle).toBe("ready"); + expect(saved.lifecycle).toBe(perTurn ? "suspended" : "ready"); const processBefore = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); expect(processBefore.exitCode).toBe(0); const historyBefore = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/filesystem/codex-home/fake-codex-state.json`] }); @@ -502,12 +511,11 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( runnerExecutionTarget: { ...target, leaseId: nextLeaseId, sandboxLeaseAcquisition: { outcome: "resumed" as const, providerLeaseId: container } }, }; if (controller) { - await expect(executePaperclipNativeSession(nextOptions)).rejects.toThrow("prior_owner_terminal_state_indeterminate"); const exited = new Promise(resolveExit => controller!.once("exit", () => resolveExit())); - controller.kill("SIGKILL"); + (controller as ReturnType).kill("SIGKILL"); await exited; controller = undefined; - await expect(executePaperclipNativeSession({ ...nextOptions, + if (!perTurn) await expect(executePaperclipNativeSession({ ...nextOptions, runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget, sandboxLeaseAcquisition: { outcome: "replacement", providerLeaseId: "unrelated-sandbox" } }, })).rejects.toThrow("prior_owner_terminal_state_indeterminate"); @@ -526,10 +534,43 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( } expect(changed).toBe(1); } - const [nextPrepared] = await fixture.db.select({ profile: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(eq(heartbeatRuns.id, nextRunId)); - await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...nextPrepared.profile, - nativeExecutionInput: nextOptions.execution } }).where(eq(heartbeatRuns.id, nextRunId)); - const second = await executePaperclipNativeSession(nextOptions); + if (diskless) { + // Preserve sandbox/provider storage and workspace dependencies. Delete + // both native controller projections; admission must discover the source. + await rm(resolve(resolvePaperclipInstanceRoot(), "runtime", "paperclip-runner"), { recursive: true, force: true }); + } + // Exercise the existing admission selector/rebinder, which supplies per-turn + // runtime snapshots. It does not restore PRP or supervisor files. + const admitContinuation = async (current: typeof execution) => { + const [row] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, current.binding.runId)); + const selected = await findNativeSessionResumeRun(fixture.db, { ...current.binding, currentRunId: row.id, + normalizedSessionId: fixture.native.normalizedSessionId, beforeCreatedAt: row.createdAt }); + const admission = buildNativeExecutionWithCheckpoint({ previousRun: selected, + normalizedSessionId: fixture.native.normalizedSessionId, executionTargetKind: "remote", buildExecution: () => current }); + if (perTurn) expect(admission.checkpoint).not.toBeNull(); + await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...row.runnerProfileJson, + nativeExecutionInput: current, nativeToolContractFingerprint: nativeToolContractFingerprintForTarget("remote", current.provider.kind), + ...(perTurn ? { sessionCheckpoint: admission.checkpoint } : {}) } }).where(eq(heartbeatRuns.id, row.id)); + }; + await admitContinuation(nextOptions.execution); + if (scenario === "diskless-per-turn-read-retry") { + let rejectedReads = 0; + await expect(executePaperclipNativeSession({ ...nextOptions, + runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget, runner: { ...runner, + execute: async request => { + if (request.args?.some(arg => arg.includes("base64 <") && arg.includes("runner-state.json"))) { + rejectedReads++; + throw new Error("fixture_remote_state_timeout"); + } + return runner.execute(request); + }, + } }, + })).rejects.toThrow("fixture_remote_state_timeout"); + expect(rejectedReads).toBe(1); + // Retry with the partial PRP projection left by the failed read. No + // fixture writes a suspended receipt or changes the surviving Runner. + } + const second = diskless ? await executeInController(nextOptions) : await executePaperclipNativeSession(nextOptions); expect(second.exitCode).toBe(0); const historyAfter = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/filesystem/codex-home/fake-codex-state.json`] }); const providerAfter = JSON.parse(historyAfter.stdout); @@ -540,12 +581,19 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( expect(calls.stdout.split("\n").filter(line => line === "thread/start")).toHaveLength(1); const providerStarts = calls.stdout.split("\n").filter(line => line === "process-start").length; expect(providerStarts).toBeGreaterThanOrEqual(1); - expect(providerStarts).toBeLessThanOrEqual(["legacy", "changed"].includes(scenario) ? 3 : hardRestart ? 2 : 1); + expect(providerStarts).toBeLessThanOrEqual(perTurn || ["legacy", "changed"].includes(scenario) ? 3 : hardRestart ? 2 : 1); const processAfter = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); expect(processAfter.exitCode).toBe(0); - if (["legacy", "changed"].includes(scenario)) expect(processAfter.stdout).not.toBe(processBefore.stdout); + if (perTurn || ["legacy", "changed"].includes(scenario)) expect(processAfter.stdout).not.toBe(processBefore.stdout); else expect(processAfter.stdout).toBe(processBefore.stdout); await finalizeNativeRun({ db: fixture.db, runId: nextRunId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + if (diskless && controller) { + const exited = new Promise(resolveExit => controller!.once("exit", () => resolveExit())); + (controller as ReturnType).kill("SIGKILL"); + await exited; + controller = undefined; + await rm(resolve(resolvePaperclipInstanceRoot(), "runtime", "paperclip-runner"), { recursive: true, force: true }); + } const thirdRunId = randomUUID(); const thirdLeaseId = randomUUID(); const [thirdRun] = await fixture.db.insert(heartbeatRuns).values({ @@ -562,9 +610,11 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( issue: { id: fixture.issueId, title: "Continue the same conversation", description: null, reviewPolicy: null }, environmentLeaseId: thirdLeaseId }); await fixture.db.insert(nativeRunFinalizations).values({ runId: thirdRunId, companyId, issueId: fixture.issueId, phase: "observed" }); + const thirdExecution = { ...nextOptions.execution, completionContract: await boundContract(thirdRunId), + binding: { ...execution.binding, runId: thirdRunId }, task: { ...execution.task, prompt: "Complete the third conversation turn." } }; + await admitContinuation(thirdExecution); const third = await executePaperclipNativeSession({ ...nextOptions, runnerInstanceId: thirdNative.runnerInstanceId, - execution: { ...nextOptions.execution, completionContract: await boundContract(thirdRunId), - binding: { ...execution.binding, runId: thirdRunId }, task: { ...execution.task, prompt: "Complete the third conversation turn." } }, + execution: thirdExecution, runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget, leaseId: thirdLeaseId }, }); expect(third.exitCode).toBe(0); @@ -573,10 +623,11 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( const finalCalls = await runner.execute({ command: "cat", args: [`${remoteCwd}/provider-calls.log`] }); expect(finalCalls.stdout.split("\n").filter(line => line === "turn/start")).toHaveLength(3); expect(finalCalls.stdout.split("\n").filter(line => line === "thread/start")).toHaveLength(1); - expect(finalCalls.stdout.split("\n").filter(line => line === "process-start")).toHaveLength(providerStarts); + expect(finalCalls.stdout.split("\n").filter(line => line === "process-start")).toHaveLength(providerStarts + (perTurn ? 1 : 0)); const finalProcess = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); - expect(finalProcess.stdout).toBe(processAfter.stdout); - expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0 }); + if (perTurn) expect(finalProcess.stdout).not.toBe(processAfter.stdout); + else expect(finalProcess.stdout).toBe(processAfter.stdout); + expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: perTurn ? 0 : 1, busy: 0, failed: 0 }); const suspended = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] }); expect(suspended.exitCode).toBe(0); expect(JSON.parse(suspended.stdout).lifecycle).toBe("suspended"); @@ -588,7 +639,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( await closeIdleWarmNativeSessionsForRestart(); await tunnel.close(); } - }, 180_000); + }, 360_000); async function persistRestartEvidence(input: { fixture: Awaited>; @@ -1277,10 +1328,11 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( } finally { await authority.stop(); } }); - it.each(["idle", "preparing_successor", "unclaimed_successor", "claimed_successor", "expired_preparation", "dispatching_successor", "foreign_preparation", "foreign_native_preparation", "wrong_issue", "new_work", "replaced_controller"] as const)( - "settles a completed warm checkpoint during early retirement without granting work: %s", - async scenario => { - const fixture = await seedRun(`WARM-EARLY-RETIREMENT-${scenario}`); + it.each((["idle", "preparing_successor", "unclaimed_successor", "claimed_successor", "expired_preparation", "dispatching_successor", "foreign_preparation", "foreign_native_preparation", "wrong_issue", "new_work", "replaced_controller", "unaccepted"] as const).flatMap(scenario => + (["runner.suspend", "turn.stop"] as const).map(command => ({ scenario, command }))))( + "settles a completed warm checkpoint during early retirement without granting work: $scenario/$command", + async ({ scenario, command }) => { + const fixture = await seedRun(`WARM-EARLY-RETIREMENT-${scenario}-${command}`); const binding = { companyId, issueId: fixture.issueId, runId: fixture.runId }; const identity = transportOptions(fixture, runtimeRoot).prpIdentity; const authority = new DurablePrpControlPlane({ @@ -1305,7 +1357,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( }); await fixture.db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }) .where(eq(heartbeatRuns.id, fixture.runId)); - await fixture.db.update(nativeRunFinalizations).set({ phase: "committed", resultId, + await fixture.db.update(nativeRunFinalizations).set({ phase: "committed", resultId: scenario === "unaccepted" ? null : resultId, leaseOwner: null, leaseExpiresAt: null, ...(scenario === "replaced_controller" ? { controllerBootId: randomUUID(), controllerGeneration: 2 } : {}), }).where(eq(nativeRunFinalizations.runId, fixture.runId)); @@ -1314,7 +1366,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( if (scenario !== "idle") { nextRunId = randomUUID(); const preparing = scenario !== "claimed_successor" && scenario !== "unclaimed_successor"; - const otherIssue = scenario === "wrong_issue" ? (await seedRun("WARM-OTHER-ISSUE")).issueId : fixture.issueId; + const otherIssue = scenario === "wrong_issue" ? (await seedRun(`WARM-OTHER-ISSUE-${command}`)).issueId : fixture.issueId; await fixture.db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, issueId: otherIssue, scopeKind: "issue", contextSnapshot: { issueId: otherIssue }, ...(preparing ? { runtimeMode: "legacy", executionStage: scenario === "dispatching_successor" ? "dispatching" : "preparing", @@ -1331,7 +1383,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( } await fixture.db.update(issues).set({ executionRunId: nextRunId }).where(eq(issues.id, fixture.issueId)); const before = (await loadNativeRunnerCheckpoint(fixture.db, binding))!; - authority.queueCommand(scenario === "new_work" ? "turn.start" : "runner.suspend", {}, `early-retirement-${scenario}`); + authority.queueCommand(scenario === "new_work" ? "turn.start" : command, {}, `early-retirement-${scenario}`); if (!["idle", "preparing_successor", "claimed_successor"].includes(scenario)) { await expect(authority.flushCheckpoint()).rejects.toThrow("native_runner_checkpoint_owner_lost"); expect((await loadNativeRunnerCheckpoint(fixture.db, binding))?.records).toEqual(before.records); @@ -1342,7 +1394,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( saved.restoreSessionDirectory(restoredRoot); const restored = JSON.parse(await readFile(resolve(restoredRoot, "control-plane", "control-plane-state.json"), "utf8")); expect(restored.identity).toEqual(identity); - expect(restored.commands).toContainEqual(expect.objectContaining({ type: "runner.suspend", status: "pending" })); + expect(restored.commands).toContainEqual(expect.objectContaining({ type: command, status: "pending" })); } const [task] = await fixture.db.select({ executionRunId: issues.executionRunId }).from(issues) .where(eq(issues.id, fixture.issueId)); @@ -1351,6 +1403,104 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( }, ); + realProcessIt("durably retires an accepted warm Runner while its same-controller successor prepares", async () => { + const fixture = await seedRun("REAL-WARM-RETIREMENT"); + const binding = { companyId, issueId: fixture.issueId, runId: fixture.runId }; + const root = resolve(runtimeRoot, fixture.runId, "real-retirement"); + const options = transportOptions(fixture, root); + const retirementController = await currentNativeControllerIdentity(); + const diagnostics: string[] = [], checkpointFailures: unknown[] = []; + let authority!: DurablePrpControlPlane; + await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: retirementController.bootId, + controllerGeneration: 1, leaseOwner: "real-retirement-owner", leaseExpiresAt: new Date(Date.now() + 60_000), + }).where(eq(nativeRunFinalizations.runId, fixture.runId)); + const transport = createRunnerdCodexTransport({ ...options, + codexArgs: [...options.codexArgs, "--linger-after-turn-start"], + onDiagnostic: message => diagnostics.push(message), + controlPlaneRegistration: async core => { + authority = core; + const write = nativeRunnerCheckpointWriter(fixture.db, { ...binding, authorityRunId: fixture.runId, + bootId: retirementController.bootId, generation: 1, leaseOwner: "real-retirement-owner" }); + core.setCheckpointPersistence(async checkpoint => { + try { await write(checkpoint); } + catch (error) { checkpointFailures.push(error); throw error; } + }); + return registerRunnerPrpAuthority({ ...binding, authority: core }); + }, + }); + let runnerPid: number | null = null, providerPid: number | null = null; + try { + await transport.transport.request("thread/start", { cwd: root, dynamicTools: [] }); + await transport.transport.request("turn/start", { input: [{ type: "text", text: "Keep the provider alive after accepting its result." }] }); + runnerPid = transport.evidence().runnerPid; + providerPid = transport.evidence().providerPid; + expect(runnerPid).toEqual(expect.any(Number)); + expect(providerPid).toEqual(expect.any(Number)); + await waitForCondition("active real provider before accepted-result retirement", () => + authority.store.state.committedEvents.some(event => event.eventType === "turn.started")); + await authority.flushCheckpoint(); + const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const resultId = randomUUID(); + await fixture.db.insert(nativeRunResults).values({ id: resultId, ...binding, turnId: options.prpIdentity.turnId, + completionContractId: run.completionContractId!, serverFingerprint: "real-retirement-accepted", + schemaStatus: "accepted", canonicalSha256: "a".repeat(64), + resultJson: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "Accepted before provider cleanup." }, + }); + await fixture.db.update(nativeRunFinalizations).set({ phase: "committed", resultId, leaseOwner: null, leaseExpiresAt: null }) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + await fixture.db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }) + .where(eq(heartbeatRuns.id, fixture.runId)); + const nextRunId = randomUUID(); + const [nextRun] = await fixture.db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, + issueId: fixture.issueId, nativeIssueId: fixture.issueId, scopeKind: "issue", status: "running", runtimeMode: "native", + invocationSource: "assignment", contextSnapshot: { issueId: fixture.issueId } }).returning(); + const next = await prepareNativeHeartbeatRun({ db: fixture.db, run: nextRun, + issue: { id: fixture.issueId, title: "Next conversation", description: null }, environmentLeaseId: randomUUID() }); + await fixture.db.insert(nativeRunFinalizations).values({ runId: nextRunId, companyId, issueId: fixture.issueId, phase: "observed" }); + await fixture.db.update(issues).set({ executionRunId: nextRunId }).where(eq(issues.id, fixture.issueId)); + const [contract] = await fixture.db.select().from(completionContracts).where(eq(completionContracts.id, run.completionContractId!)); + const nextExecution = buildNativeExecutionInput({ companyId, agentId, runId: nextRunId, + issue: { id: fixture.issueId, identifier: "NRR-NATIVE-RETIREMENT", title: "Next conversation", description: null, workMode: "standard" }, + taskPrompt: "Start only after the previous provider durably suspends.", + workspace: { id: randomUUID(), cwd: root, repoUrl: null, repoRef: null, branchName: null }, + normalizedSessionId: next.normalizedSessionId, runtimeContext: nativeRuntimeContextFixture(), + completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: contract.schemaVersion, contract: contract.contractJson as never }, + }); + const fixtureStoppedAfterRealClaim = new Error("fixture stopped before next provider dispatch"); + await expect(executePaperclipNativeSession({ db: fixture.db, execution: nextExecution, runnerInstanceId: next.runnerInstanceId, + onExecutionLeaseAcquired: async () => { + const [claimed] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, nextRunId)); + expect(claimed.controllerBootId).toBe(retirementController.bootId); + expect(claimed.leaseOwner).toEqual(expect.any(String)); + expect(claimed.leaseExpiresAt!.getTime()).toBeGreaterThan(Date.now()); + await expect(transport.transport.close(), diagnostics.join("\n")).resolves.toBeUndefined(); + throw fixtureStoppedAfterRealClaim; + }, + })).rejects.toBe(fixtureStoppedAfterRealClaim); + expect(checkpointFailures).toEqual([]); + const state = JSON.parse(await readFile(resolve(root, "runner", "runner-state.json"), "utf8")); + expect(state).toMatchObject({ lifecycle: "suspended", runId: fixture.runId, + normalizedSessionId: options.prpIdentity.normalizedSessionId, runnerInstanceId: options.prpIdentity.runnerInstanceId }); + const saved = (await loadNativeRunnerCheckpoint(fixture.db, binding))!; + const commands = [...saved.records].filter(([key]) => key.startsWith("command:")).map(([, value]) => JSON.parse(value)); + for (const type of ["turn.stop", "runner.drain", "runner.suspend"]) { + expect(commands, diagnostics.join("\n")).toContainEqual(expect.objectContaining({ type, status: "completed" })); + } + expect(commands.filter(command => command.type === "turn.start")).toHaveLength(1); + const [task] = await fixture.db.select().from(issues).where(eq(issues.id, fixture.issueId)); + expect(task.executionRunId).toBe(nextRunId); + } catch (error) { + throw new Error(`${String(error)}; owned retirement diagnostics ${JSON.stringify({ + checkpointFailures: checkpointFailures.map(String), diagnostics, + commands: authority?.store.state.commands.map(({ type, status }) => ({ type, status })), + })}`); + } finally { + await transport.transport.close().catch(() => undefined); + if (providerPid && processAlive(providerPid)) process.kill(providerPid, "SIGKILL"); + await stopOwnedProcessGroup(runnerPid, root).catch(() => undefined); + } + }, 45_000); + it("applies checkpoint deltas in Postgres without rewriting authority and removes only evicted events", async () => { const fixture = await seedRun("CHECKPOINT-DELTA"); const root = resolve(runtimeRoot, fixture.runId); @@ -1463,8 +1613,8 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( realProcessIt("keeps checkpoint SQL independent of streamed output volume with a real Postgres writer", async () => { const samples: Array<{ chunks: number; checkpointWrites: number; transactions: number; changedRows: number; checkpointBytes: number; walBytes: number; elapsedMs: number; idleSql: number; - streamSql: number; streamTransactions: number; streamChangedRows: number }> = []; - for (const chunks of [8, 512]) { + streamSql: number; streamTransactions: number; streamChangedRows: number; renewalSql: number }> = []; + for (const chunks of [8, 512, 8192]) { const fixture = await seedRun(`CHECKPOINT-COST-${chunks}`); const root = resolve(runtimeRoot, fixture.runId); const releaseOutput = resolve(root, "release-output"); @@ -1472,19 +1622,31 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( const owner = { ...binding, authorityRunId: fixture.runId, bootId: successor.bootId, generation: 1, leaseOwner: "cost-controller" }; await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: owner.bootId, - controllerGeneration: 1, leaseOwner: owner.leaseOwner, leaseExpiresAt: new Date(Date.now() + 120_000), + controllerPid: successor.pid, controllerProcessStartedAt: successor.processStartedAt, + controllerGeneration: 1, leaseOwner: owner.leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000), }).where(eq(nativeRunFinalizations.runId, fixture.runId)); let authority!: DurablePrpControlPlane; let measuring = false; let transactions = 0, changedRows = 0, checkpointBytes = 0; const statements: string[] = []; - const capturedClients = [fixture.db.$client, checkpointDb(fixture.db).$client]; + const renewalStatements: string[] = []; + let leaseLost = false; + const renewal = startNativeSessionExecutionLeaseRenewal({ db: fixture.db, ...binding, leaseOwner: owner.leaseOwner, + attempt: 0, controller: successor, remote: true, onLost: async () => { leaseLost = true; } }); + const capturedClients = [fixture.db.$client, checkpointDb(fixture.db).$client, livenessDb(fixture.db).$client]; const priorDebug = capturedClients.map(client => client.options.debug); // Capture statements only, never SQL parameters (encrypted state or secrets). - for (const client of capturedClients) { - client.options.debug = (_connection, query) => { if (measuring) statements.push(query); }; - } + capturedClients.forEach((client, index) => { + client.options.debug = (_connection, query) => { + if (!measuring) return; + // Renewals are existing liveness traffic, reported separately. A + // checkpoint query on any pool still counts against the zero oracle. + if (index === 2 && !query.includes("native_runner_checkpoints")) renewalStatements.push(query); + else statements.push(query); + }; + }); const write = nativeRunnerCheckpointWriter(fixture.db, owner); + let costFailure: unknown; const transport = createRunnerdCodexTransport({ ...transportOptions(fixture, root), codexArgs: ["--state-file", resolve(root, "provider.json"), "--linger-after-turn-start", "--held-output", "--held-output-count", String(chunks), "--held-output-release-file", releaseOutput], @@ -1516,7 +1678,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( const beforeStream = { sql: statements.length, transactions, changedRows }; await writeFile(releaseOutput, "release\n", { flag: "wx" }); await waitForCondition(`the final scripted text chunk of ${chunks}`, () => authority.store.state.committedEvents - .some(event => event.eventType === "item.delta" && JSON.stringify(event.envelope).includes(`still computing ${chunks - 1} `)), 60_000) + .some(event => event.eventType === "item.delta" && JSON.stringify(event.envelope).includes(`still computing ${chunks - 1} `)), chunks > 4096 ? 480_000 : 60_000) .catch(error => { console.info("checkpoint-cost-stream-diagnostic", JSON.stringify({ chunks, cursor: authority.store.state.ackedSourceSeq, receivedTextEvents: authority.store.state.committedEvents.filter(event => event.eventType === "item.delta").length })); throw error; }); @@ -1528,13 +1690,14 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( && query.includes("native_runner_checkpoints")).length; expect(checkpointWrites).toBeGreaterThan(0); // The collector must see real control writes. const snapshot = { chunks, checkpointWrites, transactions, changedRows, checkpointBytes, - walBytes: Number(walEnd.bytes), elapsedMs, idleSql: 0, + walBytes: Number(walEnd.bytes), elapsedMs, idleSql: 0, renewalSql: renewalStatements.length, streamSql: statements.length - beforeStream.sql, streamTransactions: transactions - beforeStream.transactions, streamChangedRows: changedRows - beforeStream.changedRows }; // Setup control messages may coalesce differently. The released text-only // interval must add exactly zero work, independent of that scheduling. expect(snapshot).toMatchObject({ streamSql: 0, streamTransactions: 0, streamChangedRows: 0 }); + expect(leaseLost).toBe(false); statements.length = 0; measuring = true; await new Promise(done => setTimeout(done, 100)); @@ -1547,20 +1710,34 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( expect(statements.some(query => /insert into "native_runner_checkpoints"/i.test(query))).toBe(true); measuring = false; samples.push(snapshot); + console.info("native-checkpoint-cost-sample", JSON.stringify(snapshot)); + } catch (error) { + costFailure = error; + throw error; } finally { measuring = false; capturedClients.forEach((client, index) => { client.options.debug = priorDebug[index]; }); - await transport.transport.close(); + try { await transport.transport.close(); } + catch (error) { + if (costFailure) throw new AggregateError([costFailure, error], "Checkpoint cost measurement and cleanup failed"); + throw error; + } + finally { + await renewal.stop().catch(error => { + if (costFailure) throw new AggregateError([costFailure, error], "Checkpoint cost measurement and lease renewal failed"); + throw error; + }); + } } } const report = { schema: "paperclip.checkpoint-cost/v1", samples, scope: "isolated Postgres; control operations included; WAL is diagnostic; no production capacity claim" }; if (process.env.PAPERCLIP_TEST_CHECKPOINT_COST_REPORT) await writeFile(process.env.PAPERCLIP_TEST_CHECKPOINT_COST_REPORT, `${JSON.stringify(report, null, 2)}\n`, { flag: "wx", mode: 0o600 }); - // Both stream sizes were checked independently; total setup counts remain + // Each stream size was checked independently; total setup counts remain // diagnostic because control-message batching is scheduling-dependent. console.info("native-checkpoint-cost", JSON.stringify(report)); - }, 120_000); + }, 540_000); realProcessIt.each([false, true])("adopts the same runner from Postgres after losing the controller filesystem (prepared task: %s)", async preparedTask => { const fixture = await seedRun(`EMPTY-CONTROLLER-DISK-${preparedTask}`); @@ -1579,9 +1756,10 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( heartbeatRunId: fixture.runId, issueId: fixture.issueId, status: "active", leasePolicy: "ephemeral", provider: "plugin", providerLeaseId: "saved-task", metadata: { agentId, runnerTaskLease: true, runnerTaskDispatched: true } }); } + const originalLeaseExpiresAt = new Date(Date.now() + 60_000); await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: bootId, controllerGeneration: 1, controllerPid: process.pid, controllerProcessStartedAt: new Date(), - leaseOwner: "original-controller", leaseExpiresAt: new Date(Date.now() + 60_000), + leaseOwner: "original-controller", leaseExpiresAt: originalLeaseExpiresAt, }).where(eq(nativeRunFinalizations.runId, fixture.runId)); const options = { ...transportOptions(fixture, controllerRoot), @@ -1647,9 +1825,13 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( await new Promise(resolveOutage => setTimeout(resolveOutage, outageMs)); expect(processAlive(runnerPid)).toBe(true); expect(processAlive(providerPid)).toBe(true); - await fixture.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1) }) - .where(eq(nativeRunFinalizations.runId, fixture.runId)); bootId = randomUUID(); + if (originalLeaseExpiresAt.getTime() > Date.now()) { + const [waiting] = await claimNativeRestartRecoveries({ db: fixture.db, controller: { ...successor, bootId }, + restartKind: "hard", runIds: [fixture.runId] }); + expect(waiting).toMatchObject({ kind: "awaiting_evidence", reason: "remote_controller_lease_live", retryAt: originalLeaseExpiresAt }); + await new Promise(resolveExpiry => setTimeout(resolveExpiry, Math.max(0, originalLeaseExpiresAt.getTime() - Date.now()) + 20)); + } const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: { ...successor, bootId }, restartKind: "hard", runIds: [fixture.runId] }); expect(claim).toMatchObject({ kind: "reattach_remote_runner", providerAttempt: 0, controllerGeneration: 2 }); diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 96732286a2d..fea3271f66d 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -1,6 +1,8 @@ import { onRetainedNativeSessionCleanupRelease } from "../retained-native-session-cleanup-notifications.js"; import { readIdleLocalWork, markIdleStartupComplete, markIdleIngressTracked } from "../idle-local-work.js"; import * as runnerCheckpoint from "./native-runner-checkpoint.js"; +import * as stagedAttachmentRecovery from "./native-staged-attachment-recovery.js"; +import * as controllerRecovery from "./native-controller-recovery.js"; import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js"; import * as environmentTaskRuntime from "../environment-task-runtime.js"; import * as pluginTaskRuntime from "./plugin-runner-task.js"; @@ -419,6 +421,10 @@ import { import { onNativeRunnerEvidence } from "./native-runner-evidence-signal.js"; beforeEach(() => { + // These transport-wiring fixtures use a partial Db double and synthetic IDs. + // The real selector/bootstrap proof is exercised with Postgres by the durable + // discovery and diskless admission suites, not by this backend unit boundary. + vi.spyOn(controllerRecovery, "resolveNativeControllerRecovery").mockResolvedValue({ kind: "no_prior_execution" }); state.drainMaintenance.mockReset().mockResolvedValue(undefined); usageRecorder.capture.mockReset().mockImplementation(async receipt => receipt); usageRecorder.create.mockReset().mockResolvedValue({ capture: usageRecorder.capture, @@ -9294,6 +9300,44 @@ describe("native warm session supervision", () => { expect(state.execute).toHaveBeenCalledTimes(2); }); + it("claims native execution authority before retiring a superseded warm identity", async () => { + const { base, next, run } = fixture("identity-native-claim-order"); + const updates: Array<{ table: unknown; values: Record }> = []; + let inspected = false; + const close = vi.fn(async () => { + if (inspected) return; + inspected = true; + expect(updates.some(update => update.table === nativeRunFinalizations + && update.values.leaseOwner && update.values.controllerBootId)).toBe(true); + }); + state.execute.mockReset().mockImplementationOnce(async options => { + options.onSession?.({ close }); return result; + }).mockImplementationOnce(async () => result); + await run(base); + // Normal runtime selection already converted preparation to native and + // inserted an unclaimed observed coordinator before calling this facade. + await executePaperclipNativeSession({ db: leaseDb(next, {}, {}, updates), + execution: next, runnerInstanceId: "runner", useRunnerd: true }); + expect(close).toHaveBeenCalledOnce(); + expect(state.execute).toHaveBeenCalledTimes(2); + }); + + it.each(["cancelled", "foreign-owner"] as const)("does not retire or stage a warm successor after claim denial: %s", async denial => { + const { base, next, run } = fixture(`identity-native-claim-denied-${denial}`); + const close = vi.fn(async () => undefined); + state.execute.mockReset().mockImplementationOnce(async options => { + options.onSession?.({ close }); return result; + }); + await run(base); + state.stageNativeRunnerWakeAttachments.mockClear(); + const db = leaseDb(next, denial === "foreign-owner" ? { leaseOwner: "different-controller" } : {}, + {}, [], {}, denial === "cancelled" ? "cancelled" : "running"); + await expect(executePaperclipNativeSession({ db, execution: next, runnerInstanceId: "runner", useRunnerd: true })).rejects.toThrow(); + expect(close).not.toHaveBeenCalled(); + expect(state.stageNativeRunnerWakeAttachments).not.toHaveBeenCalled(); + expect(state.execute).toHaveBeenCalledTimes(1); + }); + it("does not retire an active turn when a new session identity arrives", async () => { const { base, next, run } = fixture("identity-active"); let finish!: () => void; @@ -12274,6 +12318,55 @@ describe("runnerd provider runtime wiring", () => { ).toThrow("scope_closed"); }); + it.each(["attached", "changed-admission", "changed-template", "changed-template-digest", "changed-preparation"] as const)( + "verifies frozen staged admission after normal wake attachment preparation: %s", async mode => { + const current = scopedNativeExecution(`staged-attachment-${mode}`); + current.task.prompt = "Inspect attachment ref:original"; + const root = await initializeScopedRunnerRoot(current, "attachment-runner"); + const sourceIdentity = { runnerInstanceId: "attachment-runner", environmentLeaseId: "attachment-lease", + runId: "attachment-source", normalizedSessionId: current.session.normalizedSessionId!, turnId: "source-turn", itemId: "source-item" }; + const targetIdentity = { ...sourceIdentity, runId: current.binding.runId, turnId: "target-turn", itemId: "target-item" }; + const controller = new DurablePrpControlPlane({ stateDirectory: join(root, "control-plane"), identity: sourceIdentity, + expectedRunnerVersion: "fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + controller.stageRunAttachment(targetIdentity, { provider: { kind: "codex", instructions: "prepared attachment grant" } }); + const path = join(root, "control-plane/control-plane-state.json"); + const saved = JSON.parse(await readFile(path, "utf8")); + const canonicalTemplateDigest = saved.stagedRunAttachment.templateDigest; + if (mode === "changed-template" || mode === "changed-template-digest") { + saved.stagedRunAttachment.runAttachTemplate.provider.instructions = "substituted grant"; + if (mode === "changed-template-digest") saved.stagedRunAttachment.templateDigest = `sha256:${nativeSha256(saved.stagedRunAttachment.runAttachTemplate)}`; + await writeFile(path, JSON.stringify(saved)); + } + state.stageNativeRunnerWakeAttachments.mockImplementationOnce(async () => { + if (mode === "changed-preparation") current.task.prompt = "Inspect attachment ref:mutated-during-preparation"; + return { attachments: [{ id: "attachment-original" }], cleanup: async () => undefined } as never; + }); + state.renderNativeRunnerStagedAttachmentPrompt.mockReturnValueOnce("Materialized attachment: .paperclip-inbound/current/original"); + const admission = structuredClone(current); + if (mode === "changed-admission") admission.task.prompt = "Inspect attachment ref:changed"; + const reached = new Error("verified staged admission reached recheck"); + const restore = vi.spyOn(runnerCheckpoint, "restoreNativeRunnerCheckpoint").mockResolvedValue(null); + const header = vi.spyOn(runnerCheckpoint, "loadNativeRunnerCheckpointHeader").mockResolvedValue("staged header"); + const verify = vi.spyOn(stagedAttachmentRecovery, "resolveNativeStagedAttachmentRecovery") + .mockResolvedValueOnce({ currentExecution: admission, sourceIdentity, targetIdentity, + templateDigest: canonicalTemplateDigest } as never) + .mockRejectedValueOnce(reached); + try { + const resume = executePaperclipNativeSession({ db: leaseDb(current), execution: current, useRunnerd: true, + runnerInstanceId: "attachment-runner", restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, + leaseOwner: "controller", controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, + remote: { kind: "sandbox", leaseId: "attachment-lease", providerLeaseId: "sandbox", remoteCwd: "/workspace" } } as never, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", leaseId: "attachment-lease", remoteCwd: "/workspace", + runner: { execute: vi.fn() }, sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never }); + if (mode === "attached") { await expect(resume).rejects.toBe(reached); expect(verify).toHaveBeenCalledTimes(2); } + else await expect(resume).rejects.toThrow(mode === "changed-template" ? "staged_attachment_binding_mismatch" + : mode === "changed-preparation" ? "native_attachment_admission_changed" : "staged_attachment_recovery_unproven"); + expect(state.stageNativeRunnerWakeAttachments).toHaveBeenCalledWith(expect.objectContaining({ binding: expect.objectContaining({ + companyId: current.binding.companyId, issueId: current.binding.issueId, agentId: current.binding.agentId, runId: current.binding.runId, + }) })); + } finally { restore.mockRestore(); header.mockRestore(); verify.mockRestore(); await controller.stop(); } + }); + it("passes the run checkpoint active turn into restart recovery", async () => { state.createBackend.mockClear(); state.createTransport.mockClear(); @@ -12304,6 +12397,27 @@ describe("runnerd provider runtime wiring", () => { ); }); + it.each(["warm", "per_turn"] as const)("defers a %s retained-owner wait before claiming a provider attempt", async mode => { + const fresh = scopedNativeExecution(`retained-owner-${mode}`); + fresh.session.lifecyclePolicy = mode === "warm" + ? { mode: "warm", idleTimeoutMs: 300_000 } : { mode: "per_turn", idleTimeoutMs: null }; + const deadline = new Date(Date.now() + 60_000); + vi.mocked(controllerRecovery.resolveNativeControllerRecovery).mockResolvedValueOnce({ kind: "waiting_for_owner", deadline }); + const defer = vi.spyOn(controllerRecovery, "deferNativeControllerRecovery").mockResolvedValueOnce(); + const onExecutionLeaseAcquired = vi.fn(); + const updates: Array<{ table: unknown; values: Record }> = []; + const db = leaseDb(fresh, {}, {}, updates); + await expect(executePaperclipNativeSession({ db, execution: fresh, runnerInstanceId: "waiting-controller", + useRunnerd: true, onExecutionLeaseAcquired, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", + environmentId: "retained-environment", leaseId: "retained-lease" }, + })).rejects.toMatchObject({ name: "NativeControllerRecoveryWaitingError", retryAt: deadline }); + expect(defer).toHaveBeenCalledWith(db, expect.objectContaining({ ...fresh.binding, retryAt: deadline })); + expect(onExecutionLeaseAcquired).not.toHaveBeenCalled(); + expect(updates.some(update => update.table === nativeRunFinalizations && update.values.attempt !== undefined)).toBe(false); + expect(state.createBackend).not.toHaveBeenCalled(); + }); + it("rejects overlapping runs for the same runnerd provider session scope", async () => { const first = { ...execution, @@ -12664,7 +12778,7 @@ describe("runnerd provider runtime wiring", () => { } as NativeExecutionInputV1; const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!, runnerInstanceId: `admitted-${provider}`, environmentLeaseId: "admitted-lease", - turnId: `turn-${current.binding.runId}`, itemId: `item-${current.binding.runId}` }; + turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` }; const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId); await mkdir(join(root, "control-plane"), { recursive: true }); await writeFile(join(root, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity))); @@ -12694,9 +12808,56 @@ describe("runnerd provider runtime wiring", () => { runnerArtifact: artifact, adoptExistingRunner: { pid: 123, startedAt: "2026-09-19T10:00:00.000Z" } }); expect(state.resolveRemoteRunnerBinary).not.toHaveBeenCalled(); expect(execute.mock.calls.some(([request]) => request.args.includes("paperclip-runner-launch") || request.args.includes("--build-metadata"))).toBe(false); + const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise }; + await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined(); + expect(execute.mock.calls.some(([request]) => request.args.includes("paperclip-runner-launch") + || request.args.includes("paperclip-runner-claim-unstarted-session"))).toBe(false); }); - it.each(["claude", "codex"].flatMap(provider => (["per_turn", "warm"] as const).flatMap(policy => ["canonical", "local-receipt", "local-receipt-missing-terminal", "local-receipt-pending-command", "quarantined", "empty-runner", "empty-runner-quarantined", "nonempty-runner", "wrong-identity", "active", "foreign-scope", "no-stop-proof", "not-suspended", "pending-outbox", "pending-command", "missing-terminal", "ambiguous", "contradictory"].map(mode => [provider, policy, mode] as const))))( + it.each(["verified", "no-proof", "dead", "stale"] as const)("reuses only verified idle warm process during actual remote preparation: %s", async mode => { + const previous = scopedNativeExecution(`warm-live-preparation-${mode}`); + previous.session.lifecyclePolicy = { mode: "warm", idleTimeoutMs: 300_000 }; + const current = { ...previous, binding: { ...previous.binding, runId: `${previous.binding.runId}-next` } }; + const identity = { runId: previous.binding.runId, normalizedSessionId: previous.session.normalizedSessionId!, + runnerInstanceId: "retained-warm-runner", environmentLeaseId: "retained-lease", turnId: "prior-turn", itemId: "prior-item" }; + const root = await initializeScopedRunnerRoot(previous, identity.runnerInstanceId); + await writeRunnerAuthority(root, identity, "suspended"); + const projections = vi.spyOn(controllerRecovery, "resolveNativeControllerRecovery").mockResolvedValue({ kind: "no_prior_execution" }); + onTestFinished(() => projections.mockRestore()); + const db = leaseDb(current), ordinarySelect = db.select.bind(db); + db.select = ((fields?: Record) => fields?.status === heartbeatRuns.status && fields?.runnerProfileJson === heartbeatRuns.runnerProfileJson + ? priorNativeRunDb(previous).select(fields as never) : ordinarySelect(fields as never)) as Db["select"]; + let alive = true; + const execute = vi.fn(async (request: { args: string[] }) => ({ exitCode: request.args[2] === "paperclip-runner-recovery-check" && !alive ? 45 : 0, + timedOut: false, stderr: "", stdout: request.args[0] === "--build-metadata" ? JSON.stringify({ + schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd", + packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2, + durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"], + }) : request.args[0] === "--version" ? "codex-cli 0.160.0" : request.args[1]?.includes("base64") + ? Buffer.from(JSON.stringify({ ...identity, lifecycle: "ready", outbox: [], pendingTerminalDelivery: null })).toString("base64") + : request.args[1]?.includes("cat --") ? `nonce\n123\n2026-09-19T10:00:00.000Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : "" })); + const revalidate = vi.fn(async () => undefined); + const syncIn = vi.fn(async () => undefined); + await createRunnerdBackend({ db, execution: current, runnerInstanceId: identity.runnerInstanceId, + runnerIngressAuthorized: true, + ...(mode === "no-proof" ? {} : { remoteWarmRecovery: { identity, reuseProcess: true, previousExecution: previous, + checkpoint: {} as never, remote: { providerLeaseId: "retained-sandbox", remoteCwd: "/workspace" }, revalidate } }), + runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", environmentId: "environment", + leaseId: identity.environmentLeaseId, providerKey: "daytona", effectiveCapabilities: { runnerWebSocketIngress: true }, + sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "retained-sandbox" }, runner: { execute, syncIn } } as never }); + state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); + const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise }; + if (mode === "dead") alive = false; + if (mode === "stale") revalidate.mockRejectedValueOnce(new Error("fixture_warm_owner_changed")); + if (mode === "verified") await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined(); + else await expect(options.prepareExternalRunnerState()).rejects.toThrow(mode === "no-proof" ? "resumed_not_suspended" + : mode === "dead" ? "runner_remote_process_identity_unavailable" : "fixture_warm_owner_changed"); + expect(syncIn).not.toHaveBeenCalled(); + expect(execute.mock.calls.some(([request]) => request.args.includes("paperclip-runner-launch") + || request.args.includes("paperclip-runner-claim-unstarted-session"))).toBe(false); + }); + + it.each(["claude", "codex"].flatMap(provider => (["per_turn", "warm"] as const).flatMap(policy => ["canonical", "resumed", "resumed-timeout", "local-receipt", "local-receipt-missing-terminal", "local-receipt-pending-command", "quarantined", "empty-runner", "empty-runner-quarantined", "nonempty-runner", "wrong-identity", "active", "foreign-scope", "no-stop-proof", "not-suspended", "pending-outbox", "pending-command", "missing-terminal", "ambiguous", "contradictory"].filter(mode => policy === "per_turn" || !mode.startsWith("resumed")).map(mode => [provider, policy, mode] as const))))( "recovers a blocked %s %s plugin checkpoint on a replacement lease: %s", async (provider, policy, mode) => { const { permissionPolicy: _permissionPolicy, ...claudeProfile } = QUALIFIED_ACPX_PROFILES.claude; const first = { ...scopedNativeExecution(`plugin-${provider}-${policy}-${mode}`), @@ -12744,8 +12905,13 @@ describe("runnerd provider runtime wiring", () => { if (mode === "no-stop-proof") throw new Error("Plugin Runner retained session is not authorized"); return "old-task"; }); + let timedOut = false; const task = vi.fn(async (operation: any) => { environmentTaskOperationSchema.parse(operation); + if (mode === "resumed-timeout" && operation.kind === "read_state" && !timedOut) { + timedOut = true; + throw new Error("fixture_remote_state_timeout"); + } if (operation.kind === "read_state") return { kind: "state", taskId: "replacement-task", state: { ...durableRunnerState(identity, mode === "not-suspended" ? "ready" : "suspended"), outbox: mode === "pending-outbox" ? [{}] : [], pendingTerminalDelivery: null, @@ -12764,7 +12930,7 @@ describe("runnerd provider runtime wiring", () => { managedAiCredentialHome: home, managedAiCredentialIdentity: "revision", managedAiAccountIdentity: "account", runnerEnvironment: { ANTHROPIC_API_KEY: "synthetic-fixture" }, runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", leaseId: "replacement-lease", runnerTask: task, - sandboxLeaseAcquisition: { outcome: "replacement", providerLeaseId: "replacement-task" } } as never }; + sandboxLeaseAcquisition: { outcome: mode.startsWith("resumed") ? "resumed" : "replacement", providerLeaseId: "replacement-task" } } as never }; if (throughExecutor) { const validated = validatePrpStructuredRunResult({ schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "done", completionClaim: { contractRevision: next.completionContract.contract.revision, objectiveSatisfied: true, criteria: [], remainingWork: [] }, @@ -12774,12 +12940,19 @@ describe("runnerd provider runtime wiring", () => { normalizedSessionId: next.session.normalizedSessionId, providerSessionId: "original-provider-history", driverKind: next.session.driverKind, driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null }); } + if (mode === "resumed-timeout") { + await expect(createRunnerdBackend(launch)).rejects.toThrow("fixture_remote_state_timeout"); + expect(existsSync(join(scoped, "plugin-runner-state.json"))).toBe(false); + expect(await readFile(join(scoped, "control-plane/control-plane-state.json"), "utf8")).toBe(journal); + } const backend = throughExecutor ? executePaperclipNativeSession({ ...launch, useRunnerd: true }) : createRunnerdBackend(launch); - if (["canonical", "local-receipt", "quarantined", "empty-runner", "empty-runner-quarantined"].includes(mode)) { + if (["canonical", "resumed", "resumed-timeout", "local-receipt", "quarantined", "empty-runner", "empty-runner-quarantined"].includes(mode)) { await expect(backend).resolves.toBeDefined(); expect(state.createBackend).toHaveBeenCalledOnce(); - expect(owner).toHaveBeenCalledWith(expect.objectContaining({ resumeFrom: "old-task", priorIdentity: identity })); - expect(resume).toHaveBeenCalledWith(db, expect.objectContaining({ leaseId: "replacement-lease", priorIdentity: identity, inspectCheckpoint: true })); + expect(owner).toHaveBeenCalledWith(expect.objectContaining({ priorIdentity: identity, + ...(!mode.startsWith("resumed") ? { resumeFrom: "old-task" } : {}) })); + if (!mode.startsWith("resumed")) expect(resume).toHaveBeenCalledWith(db, expect.objectContaining({ leaseId: "replacement-lease", priorIdentity: identity, inspectCheckpoint: true })); + else expect(task).toHaveBeenCalledWith({ kind: "read_state", identity }); expect(JSON.parse(await readFile(join(scoped, "plugin-runner-state.json"), "utf8"))).toMatchObject({ ...identity, lifecycle: "suspended" }); expect(await readFile(join(scoped, "control-plane/control-plane-state.json"), "utf8")).toBe(journal); } else { diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index cf183108d3a..604bff67972 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,5 +1,6 @@ import { rehydratePinnedSkillAssets } from "./runtime-context.js"; import { createNativeUsageDelivery } from "./native-usage-delivery.js"; +import { createNativeRecoveryUsageDelivery } from "./native-recovery-usage-delivery.js"; import { createNativeTurnAccounting, nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage, usesNativeUsageDelivery, isNativeUsageAccountingEvent } from "./native-usage-accounting.js"; export { nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage } from "./native-usage-accounting.js"; import { measureRunnerOperation, recordRunnerTimingReceipt } from "./runner-operation-timing.js"; @@ -105,8 +106,9 @@ import { } from "node:fs"; import { tmpdir, userInfo } from "node:os"; import { DatabaseSync } from "node:sqlite"; -import { nativeRunnerCheckpointPersistence, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; +import { loadNativeRunnerCheckpointHeader, nativeRunnerCheckpointPersistence, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; import { REMOTE_CONTROLLER_LEASE_MS, REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runner-recovery-contract.js"; +import type { NativeTaskSessionPublicationContext } from "./native-task-session-publication.js"; import { basename, dirname, join, posix, resolve } from "node:path"; import type { AdapterExecutionResult, @@ -139,6 +141,7 @@ import { createRunnerdCodexTransport, executeNativeSession, applyNativeSessionGoalControl, + inspectStagedRunAttachment, inspectWarmRunTransition, parseNativeExecutionInput, parsePaperclipQuestionSet, @@ -185,6 +188,8 @@ import { nativeRunResults, } from "@paperclipai/db"; import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js"; +import { assertNativeRetainedSandboxBinding, deferNativeControllerRecovery, NativeControllerRecoveryWaitingError, resolveNativeControllerRecovery } from "./native-controller-recovery.js"; +import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js"; import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { appendNativeCredentialUpdate } from "./native-credential-updates.js"; import { nativeSha256 } from "./canonical.js"; @@ -210,6 +215,7 @@ import { } from "./remote-runner-recovery-diagnostics.js"; import { readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS } from "./remote-runner-recovery-probe.js"; import { inspectRemoteWarmSessionRecovery, type RemoteWarmSessionRecovery } from "./remote-warm-session-recovery.js"; +import { resolveNativeStagedAttachmentRecovery, type NativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js"; import { emitStartupReuse, environmentStartupReuse } from "@paperclipai/adapter-utils/startup-reuse"; import { persistActivity, publishActivity } from "../activity-log.js"; import { commitNativeStatusDecision } from "./status-decision-committer.js"; @@ -561,6 +567,7 @@ export type NativeInstructionWorkingCopy = { type WarmNativeSession = { onReleased: () => void; + sessionCheckpointPersisted?: (snapshot: PersistedNativeSession) => void; pluginTaskId?: string; observabilityCompatibilityKey?: string; runId: string; @@ -1802,6 +1809,14 @@ class SessionAuthorityEpoch { this.#assertCurrent(); } + async acceptRecoveryAccountingEvent(event: PrpEvent, capture: () => Promise) { + if (!isNativeUsageAccountingEvent(event)) return; + this.#assertCurrent(); + if (event.runId === this.runId) throw new Error("native_recovery_usage_source_unproven"); + await capture(); + this.#assertCurrent(); + } + definitions() { this.#assertCurrent(); return this.#authority.definitions(); @@ -5076,6 +5091,7 @@ async function recoverPluginRunnerCheckpoint(input: { async function migrateRunnerdStateRootForExecution(input: { db: Db; execution: NativeExecutionInput; + preparedExecutionAdmission?: NativePreparedExecutionAdmission; allowVerifiedBackup: boolean; retainedWarmOwnerRunId?: string; allowLocalRecovery: boolean; @@ -5111,6 +5127,12 @@ async function migrateRunnerdStateRootForExecution(input: { if (!isSafeNativeStateDirectory(scoped)) { throw new Error("runner_state_directory_unsafe"); } + const staged = await verifyStagedAttachmentRestart(input); + if (staged) { + // Physical inspection belongs to backend admission after lease renewal + // starts. This phase only validates the immutable source/target proof. + return; + } if (hasRetainedWarmTransitionEvidence(scoped)) { // A copied receipt is not restart/process authority. Keep both valid // unsupported and invalid forward evidence out of destructive legacy @@ -5623,6 +5645,11 @@ function readRunnerdDurableIdentity( ).toString("utf8"), ), ); + if (state.schema === "paperclip.runner.durable.control-plane-state.staged-attachment.v1") { + const staged = inspectStagedRunAttachment(state); + if (!staged) throw new Error("native_runner_staged_attachment_binding_mismatch"); + return { ...staged.sourceIdentity }; + } if (state.schema !== RUNNERD_CONTROL_PLANE_STATE_SCHEMA && !(allowExternal && state.schema === "paperclip.runner.external-control-plane-state.v1")) return null; return record(state.identity); @@ -5631,6 +5658,62 @@ function readRunnerdDurableIdentity( } } +function readStagedRunnerdAttachment(root: string) { + const path = resolve(root, "control-plane", "control-plane-state.json"); + if (!existsSync(path)) return null; + let state: Record; + try { + state = record(JSON.parse(readBoundedNativeFile(path, NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, + "runner_durable_identity_too_large").toString("utf8"))); + } catch { + // Detection must not preempt the existing forward-evidence fence or + // unreadable legacy identity quarantine. Neither path admits this state. + return null; + } + if (state.stagedRunAttachment === undefined && + state.schema !== "paperclip.runner.durable.control-plane-state.staged-attachment.v1") return null; + const staged = inspectStagedRunAttachment(state); + if (!staged) throw new Error("native_runner_staged_attachment_binding_mismatch"); + return staged; +} + +/** A staged attachment has not published a target session. Re-establish its + * exact source and target from private durable evidence before any remote RPC. */ +type NativePreparedExecutionAdmission = Readonly<{ admissionSha256: string; preparedSha256: string }>; + +async function verifyStagedAttachmentRestart(input: { + db: Db; execution: NativeExecutionInput; restartRecovery?: NativeRestartRecoveryClaim; + preparedExecutionAdmission?: NativePreparedExecutionAdmission; + runnerExecutionTarget?: AdapterExecutionTarget | null; +}, expected?: NativeStagedAttachmentRecovery): Promise { + const root = scopedRunnerdStateRoot(input.execution); + const localStage = readStagedRunnerdAttachment(root); + const staged = expected ?? localStage; + if (!staged) return null; + const deny = (): never => { throw new Error("native_runner_staged_attachment_recovery_unproven"); }; + const target = input.runnerExecutionTarget; + if (input.restartRecovery?.kind !== "reattach_remote_runner" || input.restartRecovery.remote.kind === "plugin_task" || + target?.kind !== "remote" || target.transport !== "sandbox" || target.runnerTask || + target.sandboxLeaseAcquisition?.outcome !== "resumed") return deny(); + const [current] = await input.db.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.execution.binding.runId), eq(heartbeatRuns.companyId, input.execution.binding.companyId), + eq(heartbeatRuns.agentId, input.execution.binding.agentId), eq(heartbeatRuns.nativeIssueId, input.execution.binding.issueId))); + const header = await loadNativeRunnerCheckpointHeader(input.db, input.execution.binding); + if (!current || current.status !== "running" || !header) return deny(); + const verified = await resolveNativeStagedAttachmentRecovery({ db: input.db, currentRun: current, header, + expectedTarget: staged.targetIdentity, + matchesExecution: (previous, execution) => nativeSessionScopeKey(previous) === nativeSessionScopeKey(execution) && + nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(execution, "remote"), + }); + const executionSha256 = nativeSha256(input.execution); + if (!verified || nativeSha256(verified.currentExecution) !== (input.preparedExecutionAdmission?.admissionSha256 ?? executionSha256) || + (input.preparedExecutionAdmission && executionSha256 !== input.preparedExecutionAdmission.preparedSha256) || + verified.templateDigest !== staged.templateDigest || + nativeSha256(verified.sourceIdentity) !== nativeSha256(staged.sourceIdentity) || + (localStage && nativeSha256(readStagedRunnerdAttachment(root)) !== nativeSha256(localStage))) return deny(); + return verified; +} + function durableIdentityMatchesExecution( identity: Record | null, execution: NativeExecutionInput, @@ -5784,12 +5867,43 @@ async function restorePerTurnRunnerdController(input: { const root = scopedRunnerdStateRoot(input.execution); const identity = readRunnerdDurableIdentity(root); const reference = identity ?? readRunnerdDurableIdentity(root, true); - if (!durableIdentityMatchesSession(reference, input.execution)) return; - if (identity?.runId === input.execution.binding.runId) return; - const restored = await restoreNativeRunnerCheckpoint(input.db, - { companyId: input.execution.binding.companyId, issueId: input.execution.binding.issueId, runId: reference.runId }, - root, nativeSessionKey(input.execution)); - if (!restored && !identity) throw new Error("native_runner_checkpoint_missing"); + if (!reference) { + if (!await restoreNativeControllerProjections(input.db, input.execution, root)) return; + } else { + if (!durableIdentityMatchesSession(reference, input.execution)) return; + if (identity?.runId === input.execution.binding.runId) return; + const restored = await restoreNativeRunnerCheckpoint(input.db, + { companyId: input.execution.binding.companyId, issueId: input.execution.binding.issueId, runId: reference.runId }, + root, nativeSessionKey(input.execution)); + if (!restored && !identity) throw new Error("native_runner_checkpoint_missing"); + } + const target = input.runnerExecutionTarget; + const prior = readRunnerdDurableIdentity(root); + // PRP and lifecycle evidence have different owners. A failed remote read can + // leave a valid PRP projection, so its presence must not suppress this retry. + // Never overwrite contradictory local Runner evidence with a remote receipt. + if (target.transport !== "sandbox" || target.sandboxLeaseAcquisition?.outcome !== "resumed" + || !durableIdentityMatchesSession(prior, input.execution) || prior.runId === input.execution.binding.runId + || runnerdAuthorityLifecycle(root, prior) !== "absent") return; + let state: Record; + if (target.runnerTask) { + // The admitted task owner binds this read to the surviving physical task; + // its suspended identity still belongs to the previous business turn. + const result = environmentTaskResultSchema.parse(await target.runnerTask({ kind: "read_state", identity: prior })); + if (result.kind !== "state") throw new Error("Plugin Runner checkpoint unavailable"); + state = result.state; + } else { + if (!target.runner) throw new Error("native_controller_recovery_dependency_unavailable:runner_state_reader"); + await assertNativeRetainedSandboxBinding(input.db, { ...input.execution.binding, runId: prior.runId, target }); + const stateDirectory = posix.join(target.remoteCwd, ".paperclip-runtime", "paperclip-runner", "sessions", + createHash("sha256").update(nativeSessionKey(input.execution)).digest("hex"), "runner"); + state = await readRemoteRunnerState({ runner: target.runner, stateDirectory }); + } + if (state.lifecycle !== "suspended" || !Array.isArray(state.outbox) || state.outbox.length !== 0 + || state.pendingTerminalDelivery != null || state.pendingProviderCleanup != null || state.warmTransition != null) { + throw new Error("native_controller_recovery_missing_evidence:runner_not_settled"); + } + savePluginRunnerCheckpoint(root, state, prior); } /** Materialize opaque controller state before ordinary warm admission reads it. @@ -5805,6 +5919,9 @@ async function restoreWarmRunnerdController(input: { const root = scopedRunnerdStateRoot(input.execution); const identity = readRunnerdDurableIdentity(root); const reference = identity ?? readRunnerdDurableIdentity(root, true); + if (!idleWarmNativeSessionOwnerRunId(input) && (!reference || !existsSync(nativeSessionCheckpointPath(input.execution)))) { + if (await restoreNativeControllerProjections(input.db, input.execution, root)) return; + } if (!durableIdentityMatchesSession(reference, input.execution)) return; const retainedRunId = idleWarmNativeSessionOwnerRunId(input); let runId = retainedRunId ?? reference.runId; @@ -5831,6 +5948,23 @@ async function restoreWarmRunnerdController(input: { if (!restored && !identity) throw new Error("native_runner_checkpoint_missing"); } +async function restoreNativeControllerProjections(db: Db, execution: NativeExecutionInput, root: string): Promise { + const recovered = await resolveNativeControllerRecovery(db, { + ...execution.binding, currentRunId: execution.binding.runId, normalizedSessionId: nativeSessionKey(execution), + }); + if (recovered.kind === "no_prior_execution") return false; + if (recovered.kind === "waiting_for_owner") { + throw new NativeControllerRecoveryWaitingError(recovered.deadline); + } + if (recovered.kind !== "recoverable") throw new Error(`native_controller_recovery_${recovered.kind}:${recovered.reason}`, { cause: recovered.cause }); + recovered.checkpoint.restoreSessionDirectory(root); + if (execution.session.lifecyclePolicy.mode === "warm") { + persistWarmNativeCheckpoint(execution, recovered.metadata.configDigest, recovered.snapshot, + recovered.metadata.processConfigDigest ?? undefined); + } + return true; +} + function nativeHarnessEnvironmentFingerprint( execution: NativeExecutionInput, ): string { @@ -7830,6 +7964,8 @@ export async function executePaperclipNativeSession(input: { refreshTools?: boolean; db: Db; execution: NativeExecutionInput; + /** Private server preparation proof; never persisted or accepted from a tool. */ + preparedExecutionAdmission?: NativePreparedExecutionAdmission; runnerInstanceId: string; /** Trusted task identity from the heartbeat orchestration. */ conversationMode?: boolean; @@ -7860,6 +7996,8 @@ export async function executePaperclipNativeSession(input: { /** Persist task-level continuity before a durable goal can outlive this run. */ onUsage?: (receipt: AdapterUsageCheckpoint) => Promise; onGoalCheckpoint?: (snapshot: PersistedNativeSession) => Promise; + /** Prepared continuity metadata; published with the accepted terminal checkpoint. */ + taskSessionPublication?: NativeTaskSessionPublicationContext; sessionGoalControl?: NativeSessionGoalControl | null; resumeSessionGoalHeartbeat?: boolean; preparationSpans?: NativeRunHistoricalSpan[]; @@ -7957,7 +8095,7 @@ export async function executePaperclipNativeSession(input: { throw new Error("native_session_supervisor_busy"); } executingNativeOwnerScopes.set(ownerScope, ownerToken); - await retireSupersededWarmNativeSessions( + if (!input.useRunnerd) await retireSupersededWarmNativeSessions( ownerScope, nativeSessionScopeKey(input.execution), nativeSessionKey(input.execution), ); if ((input.execution.schema === "paperclip.native-execution-input.v6" || input.execution.schema === "paperclip.native-execution-input.v8") && isCloudManagedInstance() @@ -7990,53 +8128,65 @@ export async function executePaperclipNativeSession(input: { } const workspaceRoot = input.execution.workspace.cwd; if (workspaceRoot === null) throw new Error("native_workspace_required"); - const targetKind = input.runnerExecutionTarget?.kind ?? "local"; - const chatAttachmentReadScope = new NativeChatAttachmentReadScope({ - db: input.db, - binding: input.execution.binding, - workspaceRoot, - executionTargetKind: targetKind, - }); - preparedInput = { ...input, chatAttachmentReadScope }; - cleanupStagedAttachments = () => chatAttachmentReadScope.close(); - const attachmentStage = await stageNativeRunnerWakeAttachments({ - db: input.db, - binding: { - companyId: input.execution.binding.companyId, - issueId: input.execution.binding.issueId, - runId: input.execution.binding.runId, - agentId: input.execution.binding.agentId, + const needsWarmRetirement = [...warmNativeSessions].some(([scope, entry]) => + scope !== sessionScopeId && entry.ownerScope === ownerScope); + const prepareAttachments = async () => { + const admissionSha256 = nativeSha256(input.execution); + const targetKind = input.runnerExecutionTarget?.kind ?? "local"; + const chatAttachmentReadScope = new NativeChatAttachmentReadScope({ + db: input.db, + binding: input.execution.binding, workspaceRoot, executionTargetKind: targetKind, - }, - }); - cleanupStagedAttachments = async () => { - const cleanupResults = await Promise.allSettled([ - chatAttachmentReadScope.close(), - attachmentStage.cleanup(), - ]); - if (cleanupResults.some((result) => result.status === "rejected")) { - throw new Error("paperclip_runner_attachment_staging_cleanup_failed"); - } - }; - const stagedPrompt = renderNativeRunnerStagedAttachmentPrompt( - attachmentStage.attachments, - ); - if (stagedPrompt) { - preparedInput = { - ...preparedInput, - execution: parseNativeExecutionInput({ - ...input.execution, - task: { - ...input.execution.task, - prompt: `${input.execution.task.prompt}\n\n${stagedPrompt}`, - }, - }), + }); + preparedInput = { ...input, chatAttachmentReadScope }; + cleanupStagedAttachments = () => chatAttachmentReadScope.close(); + const attachmentStage = await stageNativeRunnerWakeAttachments({ + db: input.db, + binding: { + companyId: input.execution.binding.companyId, + issueId: input.execution.binding.issueId, + runId: input.execution.binding.runId, + agentId: input.execution.binding.agentId, + workspaceRoot, + executionTargetKind: targetKind, + }, + }); + cleanupStagedAttachments = async () => { + const cleanupResults = await Promise.allSettled([ + chatAttachmentReadScope.close(), + attachmentStage.cleanup(), + ]); + if (cleanupResults.some((result) => result.status === "rejected")) { + throw new Error("paperclip_runner_attachment_staging_cleanup_failed"); + } }; - } - return await executePaperclipNativeSessionWithinScope(preparedInput); + const stagedPrompt = renderNativeRunnerStagedAttachmentPrompt( + attachmentStage.attachments, + ); + if (nativeSha256(input.execution) !== admissionSha256) throw new Error("native_attachment_admission_changed"); + if (stagedPrompt) { + const preparedExecution = parseNativeExecutionInput({ + ...input.execution, + task: { ...input.execution.task, prompt: `${input.execution.task.prompt}\n\n${stagedPrompt}` }, + }); + preparedInput = { + ...preparedInput, + execution: preparedExecution, + preparedExecutionAdmission: { admissionSha256, preparedSha256: nativeSha256(preparedExecution) }, + }; + } + return preparedInput; + }; + return needsWarmRetirement + ? await executePaperclipNativeSessionWithinScope(input, prepareAttachments) + : await executePaperclipNativeSessionWithinScope(await prepareAttachments()); } catch (error) { executionFailure = error; + if (error instanceof NativeControllerRecoveryWaitingError) { + await deferNativeControllerRecovery(input.db, { ...input.execution.binding, + leaseOwner: input.leaseOwner, retryAt: error.retryAt }); + } const remoteRunnerRecovery = readRemoteRunnerRecoveryError(error); if (remoteRunnerRecovery) { await input.onEvent?.({ @@ -8097,6 +8247,7 @@ export async function executePaperclipNativeSession(input: { async function executePaperclipNativeSessionWithinScope( input: Parameters[0], + prepareAfterClaim?: () => Promise[0]>, ): Promise { if (input.execution.provider.kind === "acpx" && input.execution.provider.agent === "antigravity") { const candidate = resolveAntigravityCandidateAdmission({ @@ -8120,6 +8271,8 @@ async function executePaperclipNativeSessionWithinScope( const restoredController = input.restartRecovery?.kind === "reattach_remote_runner" ? await restoreNativeRunnerCheckpoint(input.db, input.execution.binding, scopedRunnerdStateRoot(input.execution)) : null; + const stagedRestart = input.restartRecovery?.kind === "reattach_remote_runner" + ? await verifyStagedAttachmentRestart(input) : null; if (input.useRunnerd) await restoreWarmRunnerdController(input); const identityRedactor = createAgentIdentityRedactor(input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY); const redactIdentityText = (text: string) => redactSensitiveText( @@ -8240,6 +8393,7 @@ async function executePaperclipNativeSessionWithinScope( retainedTransition = await migrateRunnerdStateRootForExecution({ db: input.db, execution: input.execution, + preparedExecutionAdmission: input.preparedExecutionAdmission, allowVerifiedBackup: input.runnerExecutionTarget?.kind === "remote", retainedWarmOwnerRunId: idleWarmNativeSessionOwnerRunId(input) ?? remoteWarmRecovery?.identity.runId, allowLocalRecovery: input.runnerExecutionTarget?.kind !== "remote", @@ -8274,6 +8428,7 @@ async function executePaperclipNativeSessionWithinScope( const executionLeaseTtlMs = input.runnerExecutionTarget?.kind === "remote" ? REMOTE_CONTROLLER_LEASE_MS : NATIVE_SESSION_EXECUTION_LEASE_TTL_MS; let attempt: number; + let controllerGeneration = 0; try { attempt = await trace.measure( "native.coordinator.claim", @@ -8454,6 +8609,7 @@ async function executePaperclipNativeSessionWithinScope( .returning({ runId: nativeRunFinalizations.runId }) .then((rows) => rows[0] ?? null); if (!claimed) throw new Error("native_session_lease_lost"); + controllerGeneration = nextControllerGeneration; await tx .update(heartbeatRuns) .set({ @@ -8488,6 +8644,15 @@ async function executePaperclipNativeSessionWithinScope( && nativeRunsDetachingForRestart.get(input.execution.binding.runId) === startup.priorRestartDetach) { nativeRunsDetachingForRestart.delete(input.execution.binding.runId); } + // Runtime selection has already retired the legacy preparation lease. + // Claim current native authority before the old warm checkpoint writer + // must prove its same-controller successor. Shared staging stays after + // strict predecessor suspension; it cannot run on a failed claim/Stop. + if (input.useRunnerd) await retireSupersededWarmNativeSessions( + nativeSessionOwnerScope(input.execution, input.runnerExecutionTarget?.environmentId ?? null), + nativeSessionScopeKey(input.execution), nativeSessionKey(input.execution), + ); + if (prepareAfterClaim) input = await prepareAfterClaim(); await trace.end(taskPrepareScope); } catch (error) { await trace.end(taskPrepareScope, { outcome: "failed" }); @@ -8675,6 +8840,10 @@ async function executePaperclipNativeSessionWithinScope( && nativeSessionStartups.get(input.execution.binding.runId) === startupForAdmission) return; assertControllerActive(); }; + const controllerCompatibility = { + configDigest: nativeSessionConfigDigest(input.execution, input.runnerExecutionTarget?.kind ?? "local"), + processConfigDigest: warmNativeProcessConfigDigest(input) ?? null, + }; const controlPlane = new PaperclipControlPlanePort( input.db, { @@ -8692,6 +8861,16 @@ async function executePaperclipNativeSessionWithinScope( privateKeyPem: input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY, assertControllerActive, assertRunAdmissionActive, + taskSessionPublication: input.taskSessionPublication, + taskSessionPublicationOwner: input.taskSessionPublication ? { + ...input.execution.binding, authorityRunId: input.execution.binding.runId, + bootId: controller.bootId, leaseOwner, generation: controllerGeneration, + } : undefined, + nativeRecovery: input.useRunnerd && !input.backend && input.runnerExecutionTarget?.kind === "remote" ? { + compatibility: controllerCompatibility, + owner: { ...input.execution.binding, authorityRunId: input.execution.binding.runId, + bootId: controller.bootId, leaseOwner, generation: controllerGeneration }, + } : undefined, onResultCommitted: async () => { const at=Date.now(); await trace.record({name:"completion.result_committed",parentName:"native.session.execute",startedAtMs:at,endedAtMs:at}); @@ -8988,10 +9167,7 @@ async function executePaperclipNativeSessionWithinScope( : null; const warmConfigDigest = lifecyclePolicy.mode === "warm" - ? nativeSessionConfigDigest( - input.execution, - input.runnerExecutionTarget?.kind ?? "local", - ) + ? controllerCompatibility.configDigest : null; const configuredEnvironmentDigest = createHash("sha256") .update(JSON.stringify(configuredEnvironment(input.runnerEnvironment))) @@ -9004,7 +9180,8 @@ async function executePaperclipNativeSessionWithinScope( let managedCredentialSession: NativeSession | undefined; let githubAccess: NativeGitHubAccess | undefined; let releaseGitHubRun: (() => void) | undefined; - let persistedWarmSession: PersistedNativeSession | null | undefined; + let persistedWarmSession: PersistedNativeSession | null | undefined = stagedRestart && !stagedRestart.hasTargetCheckpoint + ? stagedRestart.checkpoint : undefined; if (warmSessionId !== null && warmConfigDigest !== null && !input.restartRecovery) { const entry = warmNativeSessions.get(warmSessionId); if (entry) { @@ -9264,6 +9441,13 @@ async function executePaperclipNativeSessionWithinScope( persistedWarmSession = null; } else { await recovered.session.close({ reason: "recovered warm session requires current process configuration" }); + // Closing an externally persisted controller leaves an identity marker, + // not its attachment template. The next backend must rotate the saved + // PRP authority from the committed suspension checkpoint. + const suspendedController = await restoreNativeRunnerCheckpoint(input.db, + remoteWarmRecovery.previousExecution.binding, scopedRunnerdStateRoot(input.execution), + nativeSessionKey(input.execution)); + if (!suspendedController) throw new Error("native_runner_checkpoint_missing_after_suspension"); persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution, nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote"), await recovered.session.snapshot()); persistedWarmSession = loadWarmNativeCheckpoint(input.execution, warmConfigDigest!, "remote"); @@ -9379,17 +9563,24 @@ async function executePaperclipNativeSessionWithinScope( requireSessionCloseBeforeReturn: runnerdBackend !== null || input.instructionWorkingCopy !== undefined, onSessionClosed: input.instructionWorkingCopy?.collectStopped, onCheckpoint: async (snapshot) => { + // The runtime invokes this only after its existing database + // snapshot transaction commits. Retirement coalesces into the + // next PRP command; this notification performs no persistence. + const checkpointOwner = warmSessionId === null ? undefined + : warmNativeSessions.get(warmSessionId)?.sessionCheckpointPersisted; + (checkpointOwner ?? runnerdBackend?.sessionCheckpointPersisted)?.(snapshot); if (snapshot.governedWait) { const startup = nativeSessionStartups.get(input.execution.binding.runId); if (startup) startup.governedWait = true; } snapshot = identityRedactor.redact(snapshot); + if (input.runnerExecutionTarget?.kind === "remote") snapshot = redactRunnerApiPayloads(snapshot); if (warmSessionId !== null && warmConfigDigest !== null) { await persistWarmNativeCheckpoint( input.execution, warmConfigDigest, snapshot, - warmNativeProcessConfigDigest(input), + controllerCompatibility.processConfigDigest ?? undefined, ); } if ( @@ -9474,11 +9665,15 @@ async function executePaperclipNativeSessionWithinScope( if (existing.ownerToken !== warmSessionOwnerToken) { throw new Error("native_session_supervisor_busy"); } + if (existing.session !== session) { + existing.sessionCheckpointPersisted = runnerdBackend?.sessionCheckpointPersisted; + } existing.session = session; existing.runId = input.execution.binding.runId; existing.issueId = input.execution.binding.issueId; } else warmNativeSessions.set(warmSessionId, { + sessionCheckpointPersisted: runnerdBackend?.sessionCheckpointPersisted, onReleased: () => releaseRetainedNativeSessionCleanup(input.db, { companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId, }), @@ -11501,6 +11696,62 @@ export async function verifyRemoteRunnerReattachment(input: { }; } +/** A crash before source bootstrap leaves a suspended Runner; a crash during + * its drain leaves the same live source. Neither permits a replacement until + * the physical process and all six persisted identity fields are proven. */ +async function verifyStagedRemoteRunner(input: { + recovery: NativeStagedAttachmentRecovery; + execution: NativeExecutionInput; + target: AdapterExecutionTarget | null | undefined; + claim: Extract; +}) { + const { target } = input; + if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask) { + throw new Error("native_runner_staged_attachment_profile_unsupported"); + } + const sourceIdentity = input.recovery.sourceIdentity; + const stateDirectory = posix.join(target.remoteCwd, ".paperclip-runtime", "paperclip-runner", "sessions", + createHash("sha256").update(sourceIdentity.normalizedSessionId).digest("hex"), "runner"); + const read = () => readRemoteRunnerState({ runner: target.runner!, stateDirectory }); + const state = await read(); + const control = record(JSON.parse(readBoundedNativeFile(resolve(scopedRunnerdStateRoot(input.execution), + "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, + "runner_durable_identity_too_large").toString("utf8"))); + const warm = inspectWarmRunTransition({ controlPlaneState: control, runnerState: state, + expectedNewIdentity: input.recovery.targetIdentity, expectedRunnerVersion: input.recovery.runnerVersion, + expectedRunnerDigest: input.recovery.runnerDigest }); + if (!warm && (control.warmTransition !== undefined || state.warmTransition !== undefined)) { + throw new Error("native_runner_staged_attachment_recovery_unproven"); + } + const identity = warm?.runnerIdentity ?? input.recovery.protocolIdentity; + const sameIdentity = (value: Record) => + Object.entries(identity).every(([key, expected]) => value[key] === expected); + if (!sameIdentity(state)) throw new Error("runner_remote_recovery_identity_mismatch"); + const claim = { ...input.claim, runId: identity.runId }; + const verify = () => verifyRemoteRunnerReattachment({ claim, target, identity: { ...identity }, + runId: identity.runId, normalizedSessionId: identity.normalizedSessionId }); + if (state.lifecycle !== "suspended") return verify(); + const settled = (value: Record) => sameIdentity(value) && value.lifecycle === "suspended" && + Array.isArray(value.outbox) && value.outbox.length === 0 && value.pendingTerminalDelivery == null && + value.pendingProviderCleanup == null && value.warmTransition == null; + if (!settled(state)) throw new Error("native_runner_staged_attachment_source_unsettled"); + try { + const retiring = await verify(); + const deadline = Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS; + while (await retiring.isAlive()) { + if (Date.now() >= deadline) throw new Error("native_runner_staged_attachment_source_retiring"); + await new Promise(resolveWait => setTimeout(resolveWait, 100)); + } + } catch (error) { + const diagnostic = readRemoteRunnerRecoveryError(error); + if (diagnostic?.stage !== "process_verification" || diagnostic.reason !== "process_not_running") throw error; + } + // An absent/unsafe marker or a changed process generation never proves exit. + // The package rechecks this same suspended source before launching it. + if (!settled(await read())) throw new Error("native_runner_staged_attachment_source_unsettled"); + return undefined; +} + async function waitForRemoteRunnerProcessIdentity(input: { runner: CommandManagedRuntimeRunner; identityPath: string; @@ -11880,6 +12131,7 @@ export async function createRunnerdBackend(input: { controllerPersistence?: { bootId: string; runnerArtifact?: { version: string; digest: string } }; onAccountingEvent?: (event: PrpEvent) => Promise; execution: NativeExecutionInput; + preparedExecutionAdmission?: NativePreparedExecutionAdmission; runnerInstanceId: string; chatAttachmentReadScope?: NativeChatAttachmentReadScope; restartRecovery?: NativeRestartRecoveryClaim; @@ -11929,7 +12181,7 @@ export async function createRunnerdBackend(input: { contextSnapshot: Record; }, ) => Promise; -}): Promise { +}): Promise { const sessionScopeId = nativeSessionScopeKey(input.execution); const scopeOwner = executingRunnerdSessionScopes.get(sessionScopeId); if (scopeOwner && scopeOwner !== input.execution.binding.runId) { @@ -11954,6 +12206,7 @@ export async function createRunnerdBackend(input: { retainedTransition = await migrateRunnerdStateRootForExecution({ db: input.db, execution: input.execution, + preparedExecutionAdmission: input.preparedExecutionAdmission, allowVerifiedBackup: input.runnerExecutionTarget?.kind === "remote", retainedWarmOwnerRunId: undefined, allowLocalRecovery: input.runnerExecutionTarget?.kind !== "remote", @@ -12079,11 +12332,16 @@ async function prepareRemoteExternalRunner({ input, target, runner, root, identi return { runnerBinary, readProviderState, ...executionDriver }; } +type ManagedRunnerdBackend = NativeSessionBackend & { + bindManagedSession(session: NativeSession): NativeSession; + sessionCheckpointPersisted?(snapshot: PersistedNativeSession): void; +}; + async function createRunnerdBackendWithinSessionClaim( input: Parameters[0], sessionScopeId: string, retainedTransition?: VerifiedWarmTransitionBinding, -): Promise { +): Promise { const timeController = createRunPreparationTiming(input.execution.binding.runId, "native.controller"); let recoveryPending = retainedTransition !== undefined; const target = input.runnerExecutionTarget ?? { kind: "local" as const }; @@ -12216,6 +12474,17 @@ async function createRunnerdBackendWithinSessionClaim( : durableIdentityMatchesSession(durableIdentity, input.execution) ? durableIdentity : null; + const stagedRecovery = input.restartRecovery?.kind === "reattach_remote_runner" + ? await verifyStagedAttachmentRestart(input) : null; + let stagedRecoveryPending = stagedRecovery !== null; + const recoveryUsageIdentity = stagedRecovery?.sourceIdentity ?? input.remoteWarmRecovery?.identity ?? durableIdentity; + const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity + ? createNativeRecoveryUsageDelivery({ + db: input.db, execution: input.execution, sourceIdentity: { ...recoveryUsageIdentity }, + matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) && + nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind), + }) + : undefined; const effectiveRunnerInstanceId = durableBinding?.runnerInstanceId ?? input.runnerInstanceId; const effectiveEnvironmentLeaseId = @@ -13048,7 +13317,20 @@ async function createRunnerdBackendWithinSessionClaim( bytesTransferred: 0, }, }); - if (sandboxLeaseAcquisition?.outcome === "resumed") { + if (sandboxLeaseAcquisition?.outcome === "resumed" && adoptedProcess && + (input.restartRecovery?.kind === "reattach_remote_runner" || input.remoteWarmRecovery)) { + // Active recovery and a verified idle warm handoff adopt the + // exact surviving process, rather than restoring a suspended + // per-turn backup. Identity/lease/PID generation were verified + // above; refresh that proof after any preparation wait. The + // transport still authenticates the durable PRP authority. + await measureNativeRunnerSpan(input.trace, "sandbox.lease.resume", async () => { + await input.remoteWarmRecovery?.revalidate(); + if (!await adoptedProcess.isAlive()) { + throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "process_not_running" }); + } + }); + } else if (sandboxLeaseAcquisition?.outcome === "resumed") { const reuseStartedAtMs = Date.now(); const state = await measureNativeRunnerSpan( input.trace, @@ -13606,7 +13888,9 @@ async function createRunnerdBackendWithinSessionClaim( ? input.restartRecovery.process : null; const adoptedProcess = - input.remoteWarmRecovery + stagedRecovery && input.restartRecovery?.kind === "reattach_remote_runner" + ? await verifyStagedRemoteRunner({ recovery: stagedRecovery, execution: input.execution, target, claim: input.restartRecovery }) + : input.remoteWarmRecovery ? await (async () => { await input.remoteWarmRecovery!.revalidate(); return verifyRemoteRunnerReattachment({ @@ -13641,6 +13925,10 @@ async function createRunnerdBackendWithinSessionClaim( ) => { const current = sessionToolAuthorityEpochs.get(sessionScopeId); if (!current) throw new Error("native_session_tool_authority_unavailable"); + const attachmentRunId = runnerControlPlane?.store?.state.identity.runId; + if (attachmentRunId && attachmentRunId !== current.runId) { + throw new Error("native_runner_source_semantic_reconciliation_required"); + } return current.execute(call); }; if (requiresRemoteProviderPack) { @@ -13688,8 +13976,23 @@ async function createRunnerdBackendWithinSessionClaim( onAccountingEvent: input.onAccountingEvent ? async event => { const current = sessionToolAuthorityEpochs.get(sessionScopeId); if (!current) throw new Error("native_session_authority_unavailable"); + if (input.remoteWarmRecovery && event.runId === input.remoteWarmRecovery.identity.runId) { + if (!recoveryUsage) throw new Error("native_recovery_usage_source_unproven"); + const source = input.remoteWarmRecovery.identity; + await current.acceptRecoveryAccountingEvent(event, () => recoveryUsage(event, { + runnerInstanceId: source.runnerInstanceId, environmentLeaseId: source.environmentLeaseId, + runId: source.runId, normalizedSessionId: source.normalizedSessionId, + turnId: source.turnId, itemId: source.itemId, + })); + return; + } await current.acceptAccountingEvent(event); } : undefined, + onRecoveryAccountingEvent: input.onAccountingEvent ? async (event, source) => { + const current = sessionToolAuthorityEpochs.get(sessionScopeId); + if (!current || !recoveryUsage) throw new Error("native_recovery_usage_source_unproven"); + await current.acceptRecoveryAccountingEvent(event, () => recoveryUsage(event, source)); + } : undefined, onCredentialsUpdated: (event) => appendNativeCredentialUpdate(input.db, { companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId, @@ -13866,13 +14169,23 @@ async function createRunnerdBackendWithinSessionClaim( input.execution.interactionResponses.length > 0 ? "allow_replacement_after_governed_wait" : undefined), - prpIdentity: input.remoteWarmRecovery ? { + prpIdentity: stagedRecovery ? stagedRecovery.targetIdentity : input.remoteWarmRecovery ? { runnerInstanceId: input.remoteWarmRecovery.identity.runnerInstanceId, environmentLeaseId: input.remoteWarmRecovery.identity.environmentLeaseId, runId: input.remoteWarmRecovery.identity.runId, normalizedSessionId: input.remoteWarmRecovery.identity.normalizedSessionId, turnId: input.remoteWarmRecovery.identity.turnId, itemId: input.remoteWarmRecovery.identity.itemId, + } : input.restartRecovery?.kind === "reattach_remote_runner" ? { + // Reattachment verified all six durable fields against the surviving + // Runner. Warm handoff uses opaque attachment turn/item IDs; deriving + // replacements from runId would request a different same-run authority. + runnerInstanceId: durableIdentity!.runnerInstanceId as string, + environmentLeaseId: durableIdentity!.environmentLeaseId as string, + runId: durableIdentity!.runId as string, + normalizedSessionId: durableIdentity!.normalizedSessionId as string, + turnId: durableIdentity!.turnId as string, + itemId: durableIdentity!.itemId as string, } : { runnerInstanceId: effectiveRunnerInstanceId, environmentLeaseId: effectiveEnvironmentLeaseId, @@ -13883,10 +14196,19 @@ async function createRunnerdBackendWithinSessionClaim( turnId: `turn-${input.execution.binding.runId}`, itemId: `item-${input.execution.binding.runId}`, }, - warmTransitionRegistrationMode: retainedTransition - ? "routed_connect" - : undefined, - authorizeWarmTransitionRecovery: retainedTransition + warmTransitionRegistrationMode: stagedRecovery && remoteTarget?.transport === "sandbox" && + remoteTarget.effectiveCapabilities?.runnerWebSocketIngress === true + ? "routed_same_endpoint" + : retainedTransition || stagedRecovery ? "routed_connect" : undefined, + authorizeWarmTransitionRecovery: stagedRecovery + ? async () => { + if (!stagedRecoveryPending) return; + const current = await verifyStagedAttachmentRestart(input, stagedRecovery); + if (!current || current.templateDigest !== stagedRecovery.templateDigest) { + throw new Error("native_runner_staged_attachment_recovery_unproven"); + } + } + : retainedTransition ? async ( stage: "before_bootstrap" | "before_spawn" | "before_authentication", @@ -13961,6 +14283,13 @@ async function createRunnerdBackendWithinSessionClaim( beforeConnect: async (_identity, authority) => { runnerControlPlane = authority; await input.remoteWarmRecovery?.revalidate(); + if (stagedRecovery && authority.store.state.stagedRunAttachment) { + const current = await verifyStagedAttachmentRestart(input, stagedRecovery); + if (!current || current.templateDigest !== stagedRecovery.templateDigest || + nativeSha256(current.targetIdentity) !== nativeSha256(stagedRecovery.targetIdentity)) { + throw new Error("native_runner_staged_attachment_recovery_unproven"); + } + } if (retainedTransition && recoveryPending) { const current = await verifyWarmTransitionRestart(input); if (current.stateFingerprint !== retainedTransition.stateFingerprint) { @@ -14052,6 +14381,15 @@ async function createRunnerdBackendWithinSessionClaim( }; return { bindManagedSession: wrapManagedSession, + sessionCheckpointPersisted: snapshot => { + const staged = runnerControlPlane?.store.state.stagedRunAttachment; + if (staged && snapshot.identity.runId === staged.targetIdentity.runId && + snapshot.identity.sessionId === staged.targetIdentity.normalizedSessionId) { + if (runnerControlPlane!.markStagedTargetCheckpointPublished(staged.targetIdentity)) { + stagedRecoveryPending = false; + } + } + }, descriptor: () => backend.descriptor(), openSession: async (sessionInput) => { const session = await backend.openSession(sessionInput); @@ -14074,7 +14412,7 @@ async function createRunnerdBackendWithinSessionClaim( await recordAdmittedProviderPack(); return wrapManagedSession(session); }, - } satisfies NativeSessionBackend & { bindManagedSession(session: NativeSession): NativeSession }; + } satisfies ManagedRunnerdBackend; } export function createRemoteNativeArtifactPreparation(input: { diff --git a/server/src/services/native-runtime/native-session-resume-projection.test.ts b/server/src/services/native-runtime/native-session-resume-projection.test.ts index 08242f2a690..502bcaf68f3 100644 --- a/server/src/services/native-runtime/native-session-resume-projection.test.ts +++ b/server/src/services/native-runtime/native-session-resume-projection.test.ts @@ -5,6 +5,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import { findNativeSessionResumeRun } from "./native-session-resume.js"; +import { resolveNativeControllerRecovery } from "./native-controller-recovery.js"; const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TEST_DATABASE_URL"); const supported = Boolean(externalUrl) || (await getEmbeddedPostgresTestSupport()).supported; @@ -58,7 +59,7 @@ const supported = Boolean(externalUrl) || (await getEmbeddedPostgresTestSupport( const saved = source(); await db.insert(heartbeatRuns).values([saved, run(1), run(2)]); const result = await select(); - expect(result).toEqual({ ...saved, runnerProfileJson: { nativeExecutionInput, sessionCheckpoint: checkpoint, nativeToolContractFingerprint: "fingerprint" }, + expect(result).toEqual({ ...saved, runnerProfileJson: { nativeExecutionInput, sessionCheckpoint: checkpoint, nativeToolContractFingerprint: "fingerprint", nativeControllerRecovery: null }, contextSnapshot: { paperclipWake, paperclipIssue } }); const stored = await db.select({ profile: heartbeatRuns.runnerProfileJson, context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(eq(heartbeatRuns.id, saved.id)); expect(stored[0].profile).toEqual(profile); @@ -80,6 +81,16 @@ const supported = Boolean(externalUrl) || (await getEmbeddedPostgresTestSupport( } }); + it("does not call missing or newer incompatible recovery material a fresh session", async () => { + const saved = source(); + await db.insert(heartbeatRuns).values([saved, run(30, { id: currentRunId, status: "running" })]); + const { beforeCreatedAt: _before, ...binding } = input; + expect(await resolveNativeControllerRecovery(db, binding)).toEqual({ kind: "missing_evidence", reason: "snapshot_binding" }); + // The newest progress barrier remains authoritative even without a snapshot. + await db.insert(heartbeatRuns).values(run(10, { runtimeMode: "legacy" })); + expect(await resolveNativeControllerRecovery(db, binding)).toEqual({ kind: "missing_evidence", reason: "newer_progress_barrier" }); + }); + it("skips only unused SQL-null, JSON-null, scalar, array and checkpoint-null profiles", async () => { const saved = source(), newer = run(1, { runnerProfileJson: null }); await db.insert(heartbeatRuns).values([saved, newer]); diff --git a/server/src/services/native-runtime/native-session-resume.ts b/server/src/services/native-runtime/native-session-resume.ts index 25463b68c78..f297fa95057 100644 --- a/server/src/services/native-runtime/native-session-resume.ts +++ b/server/src/services/native-runtime/native-session-resume.ts @@ -223,6 +223,8 @@ export async function findNativeSessionResumeRun( normalizedSessionId: string; currentRunId: string; beforeCreatedAt: Date; + /** Recovery discovery must retain an ineligible newest progress barrier. */ + includeProgressBarrier?: boolean; }, ) { if (!isNativeSessionId(input.normalizedSessionId)) return null; @@ -287,10 +289,12 @@ export async function findNativeSessionResumeRun( nativeToolContractFingerprint: unknown; nativeExecutionInput: unknown; sessionCheckpoint: unknown; + nativeControllerRecovery: unknown; }>`jsonb_build_object( 'nativeToolContractFingerprint', ${heartbeatRuns.runnerProfileJson}->'nativeToolContractFingerprint', 'nativeExecutionInput', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput', - 'sessionCheckpoint', ${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint' + 'sessionCheckpoint', ${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint', + 'nativeControllerRecovery', ${heartbeatRuns.runnerProfileJson}->'nativeControllerRecovery' )`, contextSnapshot: sql<{ paperclipWake: unknown; @@ -306,14 +310,16 @@ export async function findNativeSessionResumeRun( .from(heartbeatRuns) .innerJoin(barrier, eq(heartbeatRuns.id, barrier.id)) .where(and( - eq(barrier.runtimeMode, "native"), - inArray(barrier.status, terminalStatuses), - eq(barrier.hasCheckpoint, true), + ...(input.includeProgressBarrier ? [] : [ + eq(barrier.runtimeMode, "native"), + inArray(barrier.status, terminalStatuses), + eq(barrier.hasCheckpoint, true), + sql`date_trunc('milliseconds', ${barrier.createdAt}) <= ${input.beforeCreatedAt.toISOString()}::timestamptz`, + ]), sql`${heartbeatRuns.companyId}::text = ${input.companyId}`, sql`${heartbeatRuns.agentId}::text = ${input.agentId}`, sql`${heartbeatRuns.nativeIssueId}::text = ${input.issueId}`, sql`${heartbeatRuns.nativeSessionId}::text = ${input.normalizedSessionId}`, - sql`date_trunc('milliseconds', ${barrier.createdAt}) <= ${input.beforeCreatedAt.toISOString()}::timestamptz`, )); return run ?? null; } diff --git a/server/src/services/native-runtime/native-staged-attachment-recovery.test.ts b/server/src/services/native-runtime/native-staged-attachment-recovery.test.ts new file mode 100644 index 00000000000..1c872bba7ab --- /dev/null +++ b/server/src/services/native-runtime/native-staged-attachment-recovery.test.ts @@ -0,0 +1,187 @@ +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { Db } from "@paperclipai/db"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { DurablePrpControlPlane, type DurablePrpCheckpoint } from "../../vendor/paperclip-runner/index.js"; +import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provider.js"; +import { buildNativeExecutionInput } from "./native-execution-input.js"; +import { nativeToolContractFingerprintForTarget } from "./native-session-resume.js"; +import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; +import { resolveNativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js"; + +const companyId = "10000000-0000-4000-8000-000000000001"; +const issueId = "20000000-0000-4000-8000-000000000002"; +const agentId = "30000000-0000-4000-8000-000000000003"; +const sourceRunId = "40000000-0000-4000-8000-000000000004"; +const currentRunId = "50000000-0000-4000-8000-000000000005"; +const normalizedSessionId = "60000000-0000-4000-8000-000000000006"; +const runnerInstanceId = "70000000-0000-4000-8000-000000000007"; +const sourceIdentity = { runId: sourceRunId, normalizedSessionId, runnerInstanceId, + environmentLeaseId: "environment-1", turnId: "opaque-source-turn", itemId: "opaque-source-item" }; +const targetIdentity = { ...sourceIdentity, runId: currentRunId, turnId: "opaque-target-turn", itemId: "opaque-target-item" }; +const roots: string[] = []; +afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); }); + +function execution(runId: string) { + return buildNativeExecutionInput({ companyId, runId, agentId, normalizedSessionId, + issue: { id: issueId, identifier: "TEST-1", title: "Continue", description: null, workMode: "standard" }, + taskPrompt: "Continue normally", provider: "codex", workspace: { id: "workspace-1", cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null }, + completionContract: { id: "80000000-0000-4000-8000-000000000008", sha256: `sha256:${"a".repeat(64)}`, + schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: "Continue", criteria: [{ id: "done", requirement: "Answer" }] } }, + runtimeContext: nativeRuntimeContextFixture(), + }); +} + +async function fixture() { + const root = await mkdtemp(join(tmpdir(), "native-staged-proof-")); roots.push(root); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity: sourceIdentity, + expectedRunnerVersion: "0.0.0", expectedRunnerDigest: `sha256:${"f".repeat(64)}` }); + let saved!: DurablePrpCheckpoint; + core.setCheckpointPersistence(async checkpoint => { saved = checkpoint; }); + core.persistRunAttachTemplate({ provider: { provider: "codex", command: "codex" }, authorizedTools: {} }); + await core.flushCheckpoint(); + const sourceHeader = saved.records.get("header")!; + core.stageRunAttachment(targetIdentity, { provider: { provider: "codex", command: "codex" }, authorizedTools: {} }); + await core.flushCheckpoint(); + const header = saved.records.get("header")!; + const base = { companyId, agentId, issueId, nativeIssueId: issueId, runnerInstanceId, + nativeSessionId: normalizedSessionId, runtimeMode: "native", status: "running", finishedAt: null as Date | null }; + const currentRun = { ...base, id: currentRunId, runnerProfileJson: { nativeExecutionInput: execution(currentRunId) } }; + const sourceRun = { ...base, id: sourceRunId, status: "succeeded", finishedAt: new Date(), runnerProfileJson: { + nativeExecutionInput: execution(sourceRunId), nativeToolContractFingerprint: nativeToolContractFingerprintForTarget("remote", "codex"), + sessionCheckpoint: { sessionId: "provider-thread-1", providerSessionId: "provider-thread-1", identity: { + companyId, issueId, agentId, runId: sourceRunId, sessionId: normalizedSessionId }, + }, + } }; + const source = { run: sourceRun, coordinator: { runId: sourceRunId, companyId, issueId, + phase: "committed", resultId: "result-1", leaseOwner: null, leaseExpiresAt: null }, resultId: "result-1", resultStatus: "accepted" }; + function database(sourceRows = [source], contents = sourceHeader, envelope = {}) { + const material = createLocalEncryptedCipher("write").encrypt(JSON.stringify({ companyId, issueId, runId: sourceRunId, part: "header", contents, ...envelope })); + const batches: unknown[][] = [sourceRows, [{ material }]]; + const select = vi.fn(() => { + const chain = { from: () => chain, innerJoin: () => chain, where: () => chain, + limit: vi.fn(async () => batches.shift() ?? []) }; + return chain; + }); + return { db: { select } as unknown as Pick, select }; + } + return { header, sourceHeader, source, sourceRun, currentRun, database }; +} + +describe("private staged attachment restart evidence", () => { + it("proves both frozen run bindings before the target has a session snapshot", async () => { + const f = await fixture(); const { db, select } = f.database(); + const matchesExecution = vi.fn(() => true); + const result = await resolveNativeStagedAttachmentRecovery({ db, header: f.header, currentRun: f.currentRun, + expectedTarget: targetIdentity, matchesExecution }); + expect(result).toMatchObject({ sourceIdentity, targetIdentity, sourceRun: { id: sourceRunId }, hasTargetCheckpoint: false, targetCheckpoint: null, + protocolIdentity: sourceIdentity, phase: "source_bootstrap", receipt: null }); + expect(matchesExecution).toHaveBeenCalledOnce(); + expect(select).toHaveBeenCalledTimes(2); // one source point lookup and one encrypted header point lookup + expect(f.currentRun.runnerProfileJson).not.toHaveProperty("sessionCheckpoint"); + }); + + it("retains staged evidence when the target runtime has already published its snapshot", async () => { + const f = await fixture(); + const snapshot = structuredClone(f.sourceRun.runnerProfileJson.sessionCheckpoint); + snapshot.identity.runId = currentRunId; + Object.assign(snapshot, { cursor: "target-cursor", activeTurnId: "target-current-turn", pendingRuntimeRequests: [{ requestId: "target-request" }] }); + Object.assign(f.currentRun.runnerProfileJson, { sessionCheckpoint: snapshot }); + const result = await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun }); + expect(result).toMatchObject({ sourceIdentity, targetIdentity, checkpoint: { identity: { runId: currentRunId } }, + hasTargetCheckpoint: true, targetCheckpoint: snapshot }); + expect(f.currentRun.runnerProfileJson).toHaveProperty("sessionCheckpoint", snapshot); + }); + + it("returns the codec's queued phase without assuming the Runner has activated the target", async () => { + const f = await fixture(); const header = JSON.parse(f.header); + // Header inspection deliberately does not certify separate command records. + // Full restoration/authentication owns that later gate; no RPC runs here. + header.state.stagedRunAttachment.commandId = "staged-attach-command"; + const result = await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: JSON.stringify(header), currentRun: f.currentRun }); + expect(result).toMatchObject({ commandId: "staged-attach-command", protocolIdentity: sourceIdentity, + phase: "attachment_queued", receipt: null, targetIdentity }); + }); + + it.each(["runId", "companyId", "issueId", "agentId", "sessionId", "providerSessionId"])( + "rejects a present target checkpoint with changed %s instead of replaying the source", async key => { + const f = await fixture(); const snapshot = structuredClone(f.sourceRun.runnerProfileJson.sessionCheckpoint); + snapshot.identity.runId = currentRunId; + if (key === "providerSessionId") snapshot.providerSessionId = "foreign"; + else Object.assign(snapshot.identity, { [key]: "foreign" }); + Object.assign(f.currentRun.runnerProfileJson, { sessionCheckpoint: snapshot }); + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull(); + }); + + it.each(["runId", "runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "turnId", "itemId"] as const)( + "rejects a different expected target %s before querying the source", async key => { + const f = await fixture(); const { db, select } = f.database(); + expect(await resolveNativeStagedAttachmentRecovery({ db, header: f.header, currentRun: f.currentRun, + expectedTarget: { ...targetIdentity, [key]: "foreign" } })).toBeNull(); + expect(select).not.toHaveBeenCalled(); + }); + + it.each(["agentId", "companyId", "nativeIssueId", "nativeSessionId", "runnerInstanceId"] as const)( + "rejects mixed source DB %s", async key => { + const f = await fixture(); Object.assign(f.source.run, { [key]: "foreign" }); + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull(); + }); + + it.each(["binding", "schema", "runner", "session"])("rejects current frozen evidence mismatch: %s", async kind => { + const f = await fixture(); const { db, select } = f.database(); + if (kind === "binding") f.currentRun.runnerProfileJson.nativeExecutionInput.binding.agentId = sourceRunId; + if (kind === "schema") Object.assign(f.currentRun.runnerProfileJson.nativeExecutionInput, { schema: "unsupported" }); + if (kind === "runner") f.currentRun.runnerInstanceId = sourceRunId; + if (kind === "session") f.currentRun.nativeSessionId = sourceRunId; + expect(await resolveNativeStagedAttachmentRecovery({ db, header: f.header, currentRun: f.currentRun })).toBeNull(); + expect(select).not.toHaveBeenCalled(); + }); + + it.each(["phase", "leaseOwner", "resultStatus", "status", "finishedAt", "frozenRun", "workspace", "provider", "sourceTuple", "artifactVersion", "artifactDigest"])( + "rejects unsupported or stale source evidence: %s", async kind => { + const f = await fixture(); let sourceHeader = f.sourceHeader; + if (kind === "phase") f.source.coordinator.phase = "observed"; + if (kind === "leaseOwner") Object.assign(f.source.coordinator, { leaseOwner: "another-owner", leaseExpiresAt: new Date(0) }); + if (kind === "resultStatus") f.source.resultStatus = "rejected"; + if (kind === "status") f.source.run.status = "running"; + if (kind === "finishedAt") Object.assign(f.source.run, { finishedAt: null }); + if (kind === "frozenRun") f.source.run.runnerProfileJson.nativeExecutionInput.binding.runId = currentRunId; + if (kind === "workspace") f.source.run.runnerProfileJson.nativeExecutionInput.binding.executionWorkspaceId = "foreign"; + if (kind === "provider") Object.assign(f.source.run.runnerProfileJson.nativeExecutionInput.provider, { model: "another-model" }); + if (kind === "sourceTuple") { const h = JSON.parse(sourceHeader); h.state.identity.turnId = "foreign"; sourceHeader = JSON.stringify(h); } + if (kind === "artifactVersion") { const h = JSON.parse(sourceHeader); h.runnerVersion = "other-version"; sourceHeader = JSON.stringify(h); } + if (kind === "artifactDigest") { const h = JSON.parse(sourceHeader); h.runnerDigest = `sha256:${"d".repeat(64)}`; sourceHeader = JSON.stringify(h); } + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database([f.source], sourceHeader), header: f.header, currentRun: f.currentRun })).toBeNull(); + }); + + it("keeps private executor scope/config verification mandatory when supplied", async () => { + const f = await fixture(); + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun, + matchesExecution: () => false })).toBeNull(); + }); + + it.each(["companyId", "issueId", "runId", "part"])("rejects encrypted source header with another %s", async key => { + const f = await fixture(); + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database([f.source], f.sourceHeader, { [key]: "foreign" }), + header: f.header, currentRun: f.currentRun })).toBeNull(); + }); + + it("does not accept a source lacking the current tool contract or provider identity", async () => { + const f = await fixture(); + f.source.run.runnerProfileJson.nativeToolContractFingerprint = "old-contract"; + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull(); + f.source.run.runnerProfileJson.nativeToolContractFingerprint = nativeToolContractFingerprintForTarget("remote", "codex"); + f.source.run.runnerProfileJson.sessionCheckpoint.providerSessionId = ""; + expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull(); + }); + + it("rejects corrupted or ordinary headers instead of promoting them to staged evidence", async () => { + const f = await fixture(); + for (const header of [f.sourceHeader, "{}", "not-json"]) { + const { db, select } = f.database(); + expect(await resolveNativeStagedAttachmentRecovery({ db, header, currentRun: f.currentRun })).toBeNull(); + expect(select).not.toHaveBeenCalled(); + } + }); +}); diff --git a/server/src/services/native-runtime/native-staged-attachment-recovery.ts b/server/src/services/native-runtime/native-staged-attachment-recovery.ts new file mode 100644 index 00000000000..e64bdf3402c --- /dev/null +++ b/server/src/services/native-runtime/native-staged-attachment-recovery.ts @@ -0,0 +1,124 @@ +import { isDeepStrictEqual } from "node:util"; +import { and, eq } from "drizzle-orm"; +import { heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { + inspectDurablePrpCheckpointHeader, + inspectStagedRunAttachmentHeader, + parseNativeExecutionInput, + type DurableRecoveryIdentity, + type NativeExecutionInput, + type PersistedNativeSession, + type StagedRunAttachment, +} from "../../vendor/paperclip-runner/index.js"; +import { loadNativeRunnerCheckpointHeader } from "./native-runner-checkpoint.js"; +import { rebindNativeSessionCheckpoint } from "./native-session-resume.js"; + +type FrozenRun = Pick; + +type StagedHeaderProof = NonNullable>; +export type NativeStagedAttachmentRecovery = Omit & + Pick & { + sourceRun: FrozenRun; + sourceExecution: NativeExecutionInput; + currentExecution: NativeExecutionInput; + /** In-memory input only. It is not a published/resumable target snapshot. */ + checkpoint: PersistedNativeSession; + /** A published target snapshot takes precedence over the source rebound. */ + hasTargetCheckpoint: boolean; + targetCheckpoint: PersistedNativeSession | null; + runnerVersion: string; + runnerDigest: string; +}; + +function record(value: unknown): Record { + return value !== null && typeof value === "object" && !Array.isArray(value) + ? value as Record : {}; +} + +function boundTargetCheckpoint(run: FrozenRun, source: PersistedNativeSession): PersistedNativeSession | null { + const snapshot = record(run.runnerProfileJson?.sessionCheckpoint), identity = record(snapshot.identity); + if (typeof snapshot.sessionId !== "string" || !snapshot.sessionId || identity.runId !== run.id || + identity.companyId !== run.companyId || identity.issueId !== run.nativeIssueId || identity.agentId !== run.agentId || + identity.sessionId !== run.nativeSessionId) return null; + // A target checkpoint may advance cursor/goal/tool progress, but cannot change + // the provider session retained by this attachment. Do not rebound it again. + if ((source.providerSessionId != null && source.providerSessionId !== snapshot.providerSessionId) || + (source.providerIdentity != null && !isDeepStrictEqual(source.providerIdentity, snapshot.providerIdentity))) return null; + return structuredClone(snapshot) as unknown as PersistedNativeSession; +} + +function frozenExecution(run: FrozenRun): NativeExecutionInput | null { + try { + const execution = parseNativeExecutionInput(run.runnerProfileJson?.nativeExecutionInput); + return run.runtimeMode === "native" && execution.binding.runId === run.id && + execution.binding.companyId === run.companyId && execution.binding.agentId === run.agentId && + execution.binding.issueId === run.nativeIssueId && execution.session.normalizedSessionId === run.nativeSessionId + ? execution : null; + } catch { return null; } +} + +function matchesRun(identity: DurableRecoveryIdentity, run: FrozenRun): boolean { + return identity.runId === run.id && identity.runnerInstanceId === run.runnerInstanceId && + identity.normalizedSessionId === run.nativeSessionId; +} + +/** Staging proves which prior attachment may be inspected, not that it is alive + * or that work may be dispatched. Classification reads only two indexed point + * lookups: the named source/result and its encrypted header. The executor must + * re-read this proof with its private scope/config matcher before any RPC, then + * independently prove the physical Runner and authenticate the target tuple. */ +export async function resolveNativeStagedAttachmentRecovery(input: { + db: Pick; + currentRun: FrozenRun; + header: string; + expectedTarget?: DurableRecoveryIdentity; + matchesExecution?: (previous: NativeExecutionInput, current: NativeExecutionInput) => boolean; +}): Promise { + const proof = inspectStagedRunAttachmentHeader(input.header); + const current = frozenExecution(input.currentRun); + if (!proof || !current || current.provider.kind !== "codex" || !["running", "failed"].includes(input.currentRun.status) || + !matchesRun(proof.attachment.targetIdentity, input.currentRun) || + (input.expectedTarget && !isDeepStrictEqual(proof.attachment.targetIdentity, input.expectedTarget))) return null; + const { attachment } = proof; + const [source] = await input.db.select({ run: heartbeatRuns, coordinator: nativeRunFinalizations, + resultId: nativeRunResults.id, resultStatus: nativeRunResults.schemaStatus }).from(heartbeatRuns) + .innerJoin(nativeRunFinalizations, and(eq(nativeRunFinalizations.runId, heartbeatRuns.id), + eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId), eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId))) + .innerJoin(nativeRunResults, and(eq(nativeRunResults.id, nativeRunFinalizations.resultId), + eq(nativeRunResults.runId, heartbeatRuns.id), eq(nativeRunResults.companyId, heartbeatRuns.companyId), + eq(nativeRunResults.issueId, heartbeatRuns.nativeIssueId))) + .where(and(eq(heartbeatRuns.id, attachment.sourceIdentity.runId), eq(heartbeatRuns.companyId, input.currentRun.companyId), + eq(heartbeatRuns.agentId, input.currentRun.agentId), eq(heartbeatRuns.nativeIssueId, current.binding.issueId))) + .limit(1); + if (!source || source.run.companyId !== current.binding.companyId || source.run.agentId !== current.binding.agentId || + source.run.nativeIssueId !== current.binding.issueId || !matchesRun(attachment.sourceIdentity, source.run) || + source.run.status !== "succeeded" || source.run.finishedAt === null || source.coordinator.phase !== "committed" || + source.coordinator.companyId !== current.binding.companyId || source.coordinator.issueId !== current.binding.issueId || + source.coordinator.runId !== source.run.id || !source.coordinator.resultId || + source.coordinator.resultId !== source.resultId || source.resultStatus !== "accepted" || + source.coordinator.leaseOwner !== null || source.coordinator.leaseExpiresAt !== null) return null; + const previous = frozenExecution(source.run); + if (!previous || (input.matchesExecution && !input.matchesExecution(previous, current))) return null; + // Reuse the existing provider, tool-contract, workspace, runtime-context and + // normalized-session compatibility owner; no second set of digest formulas. + const checkpoint = rebindNativeSessionCheckpoint({ previousRun: source.run, currentExecution: current, executionTargetKind: "remote" }); + if (!checkpoint || !(typeof checkpoint.providerSessionId === "string" && checkpoint.providerSessionId.length > 0 || + checkpoint.providerIdentity && Object.keys(checkpoint.providerIdentity).length > 0)) return null; + const hasTargetCheckpoint = input.currentRun.runnerProfileJson?.sessionCheckpoint != null; + const targetCheckpoint = hasTargetCheckpoint ? boundTargetCheckpoint(input.currentRun, checkpoint) : null; + if (hasTargetCheckpoint && !targetCheckpoint) return null; + const sourceHeader = await loadNativeRunnerCheckpointHeader(input.db, { + companyId: source.run.companyId, issueId: current.binding.issueId, runId: source.run.id, + }); + if (!sourceHeader || !isDeepStrictEqual(inspectDurablePrpCheckpointHeader(sourceHeader), attachment.sourceIdentity)) return null; + // The codec above has validated this bounded header's artifact fields. A + // staged target cannot silently select different bytes from its source. + const sourceArtifact = JSON.parse(sourceHeader) as { runnerVersion: string; runnerDigest: string }; + if (sourceArtifact.runnerVersion !== proof.runnerVersion || sourceArtifact.runnerDigest !== proof.runnerDigest) return null; + return { ...attachment, sourceRun: source.run, sourceExecution: previous, currentExecution: current, + checkpoint, hasTargetCheckpoint, targetCheckpoint, + protocolIdentity: proof.protocolIdentity, phase: proof.phase, receipt: proof.receipt, + runnerVersion: proof.runnerVersion, runnerDigest: proof.runnerDigest }; +} diff --git a/server/src/services/native-runtime/native-task-session-publication.test.ts b/server/src/services/native-runtime/native-task-session-publication.test.ts new file mode 100644 index 00000000000..4ab4c73c059 --- /dev/null +++ b/server/src/services/native-runtime/native-task-session-publication.test.ts @@ -0,0 +1,319 @@ +import { createRequire } from "node:module"; +import { randomUUID } from "node:crypto"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { eq, sql } from "drizzle-orm"; +import { drizzle } from "drizzle-orm/postgres-js"; +import { agents, agentTaskSessions, companies, completionContracts, createDb, heartbeatRuns, issues, + nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { CONTROL_PLANE_CONFORMANCE_OPEN, CONTROL_PLANE_CONFORMANCE_RESULT, + CONTROL_PLANE_CONFORMANCE_TERMINAL } from "../../vendor/paperclip-runner/testing.js"; +import type { PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; +import { startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js"; +import { redactRegisteredSecretValues } from "../run-secret-redaction.js"; +import { nativeSha256 } from "./canonical.js"; +import { PaperclipControlPlanePort, type PaperclipControlPlaneBinding } from "./paperclip-control-plane-port.js"; +import { nativeTaskSessionPublicationHandled, publishNativeTaskSessionInTransaction, NATIVE_TASK_SESSION_PUBLICATION_KEY, + type NativeTaskSessionPublicationContext } from "./native-task-session-publication.js"; + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_PUBLICATION_TEST_DATABASE_URL"); +describe("accepted native task session publication", () => { + let temporary: Awaited> | null = null; + let db: Db; + let measured: Db; + let client: Db["$client"]; + const statements: string[] = []; + beforeAll(async () => { + if (!external) temporary = await startRecoveryPostgres("native-publication-"); + const url = external ?? temporary!.connectionString; + db = createDb(url, { maxConnections: 2 }); + // Test-only wire observer. Production still uses the existing Db/pools. + const postgres = createRequire(import.meta.resolve("@paperclipai/db"))("postgres") as + (url: string, options: object) => Db["$client"]; + client = postgres(url, { max: 1, prepare: false, connection: { application_name: "native-publication-observer" }, + debug: (_id: number, query: string) => { statements.push(query); } }); + measured = drizzle(client) as unknown as Db; + }); + afterAll(async () => { + await client?.end({ timeout: 1 }); + await db?.$client.end({ timeout: 1 }); + await temporary?.cleanup(); + }); + + async function fixture(source: "runner" | "control" = "runner", accepted = true) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + const sessionId = randomUUID(), contractId = randomUUID(), runner = randomUUID(), control = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Native pointer", issuePrefix: `P${companyId.slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Native pointer", adapterType: "codex_local", status: "running" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Native pointer", status: "in_progress", assigneeAgentId: agentId }); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "standalone-v1", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: "standalone-v1", objective: "Conformance", criteria: [{ id: "objective", requirement: "Complete" }] }, + canonicalSha256: "publication-contract", createdByActorType: "system", createdByActorId: "test" }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", + nativeIssueId: issueId, nativeSessionId: sessionId, runnerInstanceId: runner, completionContractId: contractId, + completionContractSha256: "publication-contract", startedAt: new Date(Date.now() - 2_000), runnerProfileJson: {} }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const owner = { companyId, issueId, runId, authorityRunId: runId, bootId: "publication-boot", generation: 1, leaseOwner: "publication-owner" }; + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", + controllerBootId: owner.bootId, controllerGeneration: owner.generation, leaseOwner: owner.leaseOwner, + leaseExpiresAt: new Date(Date.now() + 60_000) }); + const binding: PaperclipControlPlaneBinding = { companyId, agentId, issueId, runId, sessionId, + completionContractId: contractId, completionContractSha256: "publication-contract", sourceInstanceId: runner, + controlPlaneSourceInstanceId: control }; + const context: NativeTaskSessionPublicationContext = { taskKey: issueId, adapterType: "codex_local", + sessionParamsJson: { cwd: "/working", paperclipSessionConfigFingerprint: "prepared-once", opaque: "private-prepared-param" } }; + const snapshot: PersistedNativeSession = { backendKind: "mock", sessionId, identity: { companyId, agentId, issueId, runId, sessionId }, + providerSessionId: "provider-thread", activeTurnId: null, pendingRuntimeRequests: [], goal: null, + semanticResult: CONTROL_PLANE_CONFORMANCE_RESULT, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL }; + const port = new PaperclipControlPlanePort(measured, binding, { taskSessionPublication: context, taskSessionPublicationOwner: owner }); + await port.openRun({ ...CONTROL_PLANE_CONFORMANCE_OPEN, identity: snapshot.identity, sourceInstanceId: runner }); + if (accepted) await new PaperclipControlPlanePort(db, binding).completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, + terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId: "publication-turn" }); + await db.insert(agentTaskSessions).values({ companyId, agentId, adapterType: context.adapterType, taskKey: issueId, + goalSourceId: `${source === "runner" ? runner : control}:${runId}`, goalSourceCursor: 4 }); + const read = async () => { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const [session] = await db.select().from(agentTaskSessions).where(eq(agentTaskSessions.taskKey, issueId)); + return { run, session }; + }; + return { binding, owner, context, snapshot, port, read }; + } + + it.each(["runner", "control"] as const)("publishes from the exact accepted %s epoch in one checkpoint transaction", async source => { + const f = await fixture(source); statements.length = 0; + await f.port.checkpointSession(f.snapshot); + const queries = [...statements], { run, session } = await f.read(); + expect(session).toMatchObject({ sessionParamsJson: { ...f.context.sessionParamsJson, sessionId: f.binding.sessionId }, + sessionDisplayId: "provider-thread", lastRunId: f.binding.runId }); + expect(nativeTaskSessionPublicationHandled(run.runnerProfileJson, { ...f.binding, ...f.context })).toBe(true); + expect(queries.filter(query => /^begin\b/i.test(query.trim()))).toHaveLength(1); + expect(queries.filter(query => /^commit\b/i.test(query.trim()))).toHaveLength(1); + expect(queries.filter(query => /native_run_results/i.test(query))).toHaveLength(1); + expect(queries.filter(query => /^select .*from "agent_task_sessions"/i.test(query.trim()))).toHaveLength(1); + expect(queries.filter(query => /^update "agent_task_sessions"/i.test(query.trim()))).toHaveLength(1); + expect(queries).toHaveLength(7); + expect(JSON.stringify(run.runnerProfileJson)).not.toContain("private-prepared-param"); + expect(JSON.stringify({ [NATIVE_TASK_SESSION_PUBLICATION_KEY]: run.runnerProfileJson![NATIVE_TASK_SESSION_PUBLICATION_KEY] }).length).toBeLessThan(4_096); + expect(redactRegisteredSecretValues(run.runnerProfileJson, [])).not.toHaveProperty(NATIVE_TASK_SESSION_PUBLICATION_KEY); + expect(nativeTaskSessionPublicationHandled(run.runnerProfileJson, { ...f.binding, ...f.context, companyId: randomUUID() })).toBe(false); + expect(nativeTaskSessionPublicationHandled(run.runnerProfileJson, { ...f.binding, ...f.context, agentId: randomUUID() })).toBe(false); + const malformed = structuredClone(run.runnerProfileJson!); + (malformed[NATIVE_TASK_SESSION_PUBLICATION_KEY] as Record).resultId = "not-an-id"; + expect(nativeTaskSessionPublicationHandled(malformed, { ...f.binding, ...f.context })).toBe(false); + }); + + it("does not recreate a reset pointer or query task state on repeated terminal maintenance", async () => { + const f = await fixture(); await f.port.checkpointSession(f.snapshot); + await db.delete(agentTaskSessions).where(eq(agentTaskSessions.taskKey, f.binding.issueId)); statements.length = 0; + await f.port.checkpointSession(f.snapshot); + const repeated = [...statements]; + expect((await f.read()).session).toBeUndefined(); + expect(repeated.filter(query => /agent_task_sessions|native_run_results/i.test(query))).toHaveLength(0); + expect(repeated.filter(query => /^begin\b/i.test(query.trim()))).toHaveLength(1); + expect(repeated.filter(query => /^update /i.test(query.trim()))).toHaveLength(1); + }); + + it("persists a pre-acceptance terminal observation and publishes only at the existing accepted checkpoint", async () => { + const f = await fixture("runner", false); + // Driver event/startup checkpoints can expose a semantic terminal observation + // before the runtime calls completeRun. That observation is not acceptance. + await f.port.checkpointSession(f.snapshot); + const observed = await f.read(); + expect(observed.run.runnerProfileJson?.sessionCheckpoint).toMatchObject({ semanticResult: CONTROL_PLANE_CONFORMANCE_RESULT }); + expect(observed.run.runnerProfileJson).not.toHaveProperty(NATIVE_TASK_SESSION_PUBLICATION_KEY); + expect(observed.session?.sessionParamsJson).toBeNull(); + expect(nativeTaskSessionPublicationHandled(observed.run.runnerProfileJson, { ...f.binding, ...f.context })).toBe(false); + await f.port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, + terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId: "publication-turn" }); + expect((await f.read()).session?.sessionParamsJson).toBeNull(); + await f.port.checkpointSession(f.snapshot); + const completed = await f.read(); + expect(completed.session?.lastRunId).toBe(f.binding.runId); + expect(nativeTaskSessionPublicationHandled(completed.run.runnerProfileJson, { ...f.binding, ...f.context })).toBe(true); + }); + + it("does not hold the issue parent lock while a result replay waits for the status coordinator", async () => { + const f = await fixture(); + let committed = 0; + const port = new PaperclipControlPlanePort(db, f.binding, { nativeRecovery: { owner: f.owner, + compatibility: { configDigest: nativeSha256("config"), processConfigDigest: null } }, + onResultCommitted: async () => { committed++; } }); + await port.openRun({ ...CONTROL_PLANE_CONFORMANCE_OPEN, identity: f.snapshot.identity, sourceInstanceId: f.binding.sourceInstanceId }); + let replay: Promise | undefined; + let statusLockError: unknown = null; + try { + await db.transaction(async tx => { + // This is commitNativeStatusDecision's actual lock prefix. Hold its + // coordinator first, then overlap an identical remote result replay. + await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.binding.runId)).for("update"); + const [backend] = await tx.execute(sql`select pg_backend_pid() as pid`); + replay = port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, + terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId: "publication-turn" }).then(() => null, error => error); + let waiting = false; + for (let attempt = 0; attempt < 100; attempt++) { + const [state] = await measured.execute(sql`select exists ( + select 1 from pg_stat_activity where ${backend.pid} = any(pg_blocking_pids(pid))) as waiting`); + if (state.waiting) { waiting = true; break; } + await new Promise(resolve => setTimeout(resolve, 10)); + } + expect(waiting).toBe(true); + // A bounded test-local timeout reports the inverse-order cycle without + // relying on which transaction PostgreSQL chooses as deadlock victim. + await tx.execute(sql`set local lock_timeout = '500ms'`); + await tx.select().from(issues).where(eq(issues.id, f.binding.issueId)).for("update"); + }); + } catch (error) { statusLockError = error; } + const replayError = await replay; + expect(statusLockError).toBeNull(); + expect(replayError).toBeNull(); + expect(committed).toBe(0); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.binding.runId))).toHaveLength(1); + await db.update(nativeRunFinalizations).set({ controllerGeneration: 2 }).where(eq(nativeRunFinalizations.runId, f.binding.runId)); + await expect(port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, + terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId: "publication-turn" })).rejects.toThrow("checkpoint_owner_lost"); + }); + + it.each(["missing", "foreign_epoch", "newer_pointer", "newer_execution", "generation_reset", "stop"])("fences %s and records a handled suppression", async kind => { + const f = await fixture(); + if (kind === "missing") await db.delete(agentTaskSessions).where(eq(agentTaskSessions.taskKey, f.binding.issueId)); + if (kind === "foreign_epoch") await db.update(agentTaskSessions).set({ goalSourceId: `${randomUUID()}:${f.binding.runId}` }).where(eq(agentTaskSessions.taskKey, f.binding.issueId)); + if (kind === "newer_pointer" || kind === "newer_execution") { + const newer = randomUUID(); + await db.insert(heartbeatRuns).values({ id: newer, companyId: f.binding.companyId, agentId: f.binding.agentId, + status: "running", issueId: f.binding.issueId, startedAt: new Date() }); + if (kind === "newer_pointer") await db.update(agentTaskSessions).set({ lastRunId: newer, sessionParamsJson: { sessionId: "newer" } }).where(eq(agentTaskSessions.taskKey, f.binding.issueId)); + else await db.update(issues).set({ executionRunId: newer }).where(eq(issues.id, f.binding.issueId)); + } + if (kind === "generation_reset") await db.update(issues).set({ conversationAgentId: f.binding.agentId, + conversationUserId: "publication-test-user", conversationState: "active", conversationSessionGeneration: 2 }).where(eq(issues.id, f.binding.issueId)); + if (kind === "stop") await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.binding.runId)); + await f.port.checkpointSession(f.snapshot); + const { run, session } = await f.read(); + expect(nativeTaskSessionPublicationHandled(run.runnerProfileJson, { ...f.binding, ...f.context })).toBe(true); + expect(session?.sessionParamsJson?.sessionId).not.toBe(f.binding.sessionId); + }); + + it("keeps unsupported task keys and missing remote recovery headers outside handled publication", async () => { + const f = await fixture(); + const context = { ...f.context, taskKey: "custom-task" }; + const port = new PaperclipControlPlanePort(measured, f.binding, { taskSessionPublication: context, taskSessionPublicationOwner: f.owner }); + await port.openRun({ ...CONTROL_PLANE_CONFORMANCE_OPEN, identity: f.snapshot.identity, sourceInstanceId: f.binding.sourceInstanceId }); + await port.checkpointSession(f.snapshot); + const { run, session } = await f.read(); + expect(session?.sessionParamsJson).toBeNull(); + expect(nativeTaskSessionPublicationHandled(run.runnerProfileJson, { ...f.binding, ...context })).toBe(false); + const remote = await fixture(); + const remotePort = new PaperclipControlPlanePort(measured, remote.binding, { taskSessionPublication: remote.context, + taskSessionPublicationOwner: remote.owner, nativeRecovery: { owner: remote.owner, + compatibility: { configDigest: nativeSha256("config"), processConfigDigest: null } } }); + await remotePort.openRun({ ...CONTROL_PLANE_CONFORMANCE_OPEN, identity: remote.snapshot.identity, sourceInstanceId: remote.binding.sourceInstanceId }); + await remotePort.checkpointSession(remote.snapshot); + expect((await remote.read()).session?.sessionParamsJson).toBeNull(); + expect(nativeTaskSessionPublicationHandled((await remote.read()).run.runnerProfileJson, { ...remote.binding, ...remote.context })).toBe(false); + }); + + it("rejects stale owners and a different accepted result without partially publishing", async () => { + const stale = await fixture(); + await db.update(nativeRunFinalizations).set({ controllerGeneration: 2 }).where(eq(nativeRunFinalizations.runId, stale.binding.runId)); + await expect(stale.port.checkpointSession(stale.snapshot)).rejects.toThrow("checkpoint_owner_lost"); + expect((await stale.read()).session?.sessionParamsJson).toBeNull(); + const wrong = await fixture(); + await expect(wrong.port.checkpointSession({ ...wrong.snapshot, semanticResult: { ...CONTROL_PLANE_CONFORMANCE_RESULT, summary: "different" } })).rejects.toThrow("publication_result_mismatch"); + expect((await wrong.read()).session?.sessionParamsJson).toBeNull(); + const foreign = await fixture(); + const foreignResult = await fixture(); + const [otherCoordinator] = await db.select({ resultId: nativeRunFinalizations.resultId }).from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, foreignResult.binding.runId)); + await expect(db.update(nativeRunFinalizations).set({ resultId: otherCoordinator.resultId }) + .where(eq(nativeRunFinalizations.runId, foreign.binding.runId))) + .rejects.toMatchObject({ cause: { constraint_name: "native_run_finalizations_result_owner_fk" } }); + expect((await foreign.read()).session?.sessionParamsJson).toBeNull(); + await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, foreign.binding.runId)); + // The CP defers preacceptance snapshots, while the publication helper itself + // still requires accepted-result authority if called directly. + await expect(publishNativeTaskSessionInTransaction({} as never, { binding: foreign.binding, + context: foreign.context, snapshot: foreign.snapshot, run: (await foreign.read()).run, + issue: null, resultId: null })).rejects.toThrow("publication_result_missing"); + }); + + it("adds no publication SQL for nonterminal or goal snapshots", async () => { + const f = await fixture(); statements.length = 0; + await f.port.checkpointSession({ ...f.snapshot, semanticResult: null, terminal: null }); + expect(statements.some(query => /agent_task_sessions|native_run_results|publication_issue/i.test(query))).toBe(false); + statements.length = 0; + await f.port.checkpointSession({ ...f.snapshot, goal: {} as NonNullable }); + expect(statements.some(query => /agent_task_sessions|native_run_results|publication_issue/i.test(query))).toBe(false); + expect((await f.read()).session?.sessionParamsJson).toBeNull(); + }); + + it.each(["completed_predecessor", "expired_accepted_lease"])("preserves legitimate %s maintenance", async kind => { + const f = await fixture(); + if (kind === "completed_predecessor") { + const prior = randomUUID(); + await db.insert(heartbeatRuns).values({ id: prior, companyId: f.binding.companyId, agentId: f.binding.agentId, + issueId: f.binding.issueId, status: "succeeded", finishedAt: new Date(Date.now() - 5_000) }); + await db.update(agentTaskSessions).set({ lastRunId: prior, sessionParamsJson: { sessionId: "prior" } }) + .where(eq(agentTaskSessions.taskKey, f.binding.issueId)); + } else { + await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) }) + .where(eq(nativeRunFinalizations.runId, f.binding.runId)); + } + await f.port.checkpointSession(f.snapshot); + expect((await f.read()).session?.lastRunId).toBe(f.binding.runId); + }); + + it("does not block completeRun's issue KEY SHARE while publishing", async () => { + const f = await fixture(); let release!: () => void; let locked!: () => void; + const ready = new Promise(resolve => { locked = resolve; }); + const hold = new Promise(resolve => { release = resolve; }); + const blocker = db.transaction(async tx => { + await tx.select().from(issues).where(eq(issues.id, f.binding.issueId)).for("key share"); locked(); await hold; + }); + await ready; + try { await f.port.checkpointSession(f.snapshot); } + finally { release(); await blocker; } + expect((await f.read()).session?.lastRunId).toBe(f.binding.runId); + }); + + it("rechecks the owner clock and issue authority after an issue lock wait", async () => { + const f = await fixture(); let release!: () => void; let locked!: () => void; + await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() + 500) }).where(eq(nativeRunFinalizations.runId, f.binding.runId)); + const ready = new Promise(resolve => { locked = resolve; }); + const hold = new Promise(resolve => { release = resolve; }); + const newer = randomUUID(); + await db.insert(heartbeatRuns).values({ id: newer, companyId: f.binding.companyId, agentId: f.binding.agentId, + issueId: f.binding.issueId, status: "running", startedAt: new Date() }); + const blocker = db.transaction(async tx => { + await tx.select().from(issues).where(eq(issues.id, f.binding.issueId)).for("no key update"); + await tx.update(issues).set({ executionRunId: newer }).where(eq(issues.id, f.binding.issueId)); + locked(); await hold; + }); + await ready; + const publication = f.port.checkpointSession(f.snapshot); + const rejected = expect(publication).rejects.toThrow("checkpoint_owner_lost"); + try { + let waiting = false; + for (let attempt = 0; attempt < 100; attempt++) { + const rows = await db.execute(sql`select 1 from pg_stat_activity where application_name = 'native-publication-observer' + and wait_event_type = 'Lock' and query like '%publication_issue%'`); + if (rows.length) { waiting = true; break; } + await new Promise(resolve => setTimeout(resolve, 10)); + } + expect(waiting).toBe(true); + await new Promise(resolve => setTimeout(resolve, 600)); + } finally { release(); await blocker; } + await rejected; + expect((await f.read()).session?.sessionParamsJson).toBeNull(); + }); + + it("rolls back the pointer if combined private metadata exceeds its existing bound", async () => { + const f = await fixture(); + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeControllerRecovery: { padding: "x".repeat(4_096) } } }).where(eq(heartbeatRuns.id, f.binding.runId)); + await expect(f.port.checkpointSession(f.snapshot)).rejects.toThrow("publication_metadata_too_large"); + const { run, session } = await f.read(); + expect(session?.sessionParamsJson).toBeNull(); + expect(run.runnerProfileJson).not.toHaveProperty(NATIVE_TASK_SESSION_PUBLICATION_KEY); + }); +}); diff --git a/server/src/services/native-runtime/native-task-session-publication.ts b/server/src/services/native-runtime/native-task-session-publication.ts new file mode 100644 index 00000000000..ee61fc3e74e --- /dev/null +++ b/server/src/services/native-runtime/native-task-session-publication.ts @@ -0,0 +1,145 @@ +import { and, eq } from "drizzle-orm"; +import { agentTaskSessions, heartbeatRuns, nativeRunResults, type Db, type issues } from "@paperclipai/db"; +import { validatePrpStructuredRunResult, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; +import { nativeSha256 } from "./canonical.js"; +import type { PaperclipControlPlaneBinding } from "./paperclip-control-plane-port.js"; +import { readRunCancellation } from "../run-cancellation.js"; + +/** Prepared once by heartbeat; never copied into recovery metadata or receipts. */ +export interface NativeTaskSessionPublicationContext { + taskKey: string; + adapterType: string; + sessionParamsJson: Record | null; +} + +export const NATIVE_TASK_SESSION_PUBLICATION_KEY = "nativeTaskSessionPublication"; +type Transaction = Parameters[0]>[0]; +type Outcome = "published" | "suppressed_reset" | "suppressed_newer" | "suppressed_authority" | "missing_stub" | "unsupported"; +interface Receipt { + schema: "paperclip.native-task-session-publication.v1"; + companyId: string; + issueId: string; + agentId: string; + runId: string; + taskKey: string; + adapterType: string; + sessionId: string; + resultId: string; + outcome: Outcome; +} + +function record(value: unknown): Record { + return value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} +const uuid = (value: unknown): value is string => typeof value === "string" && /^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(value); +const outcomes: readonly string[] = ["published", "suppressed_reset", "suppressed_newer", "suppressed_authority", "missing_stub", "unsupported"]; + +function receiptFromProfile(profile: unknown): Receipt | null { + const root = record(profile), value = record(root[NATIVE_TASK_SESSION_PUBLICATION_KEY]); + if (value.schema !== "paperclip.native-task-session-publication.v1" + || ![value.companyId, value.issueId, value.agentId, value.runId, value.sessionId, value.resultId].every(uuid) + || typeof value.taskKey !== "string" || value.taskKey.length > 512 + || typeof value.adapterType !== "string" || !value.adapterType.length || value.adapterType.length > 128 + || !outcomes.includes(String(value.outcome)) || Object.keys(value).length !== 10) return null; + const snapshot = record(root.sessionCheckpoint), identity = record(snapshot.identity); + if (identity.runId !== value.runId || identity.companyId !== value.companyId || identity.issueId !== value.issueId + || identity.agentId !== value.agentId || identity.sessionId !== value.sessionId) return null; + return value as unknown as Receipt; +} + +/** Only the exact server-written outcome may suppress heartbeat's late fallback. */ +export function nativeTaskSessionPublicationHandled(profile: unknown, binding: { + companyId: string; agentId: string; runId: string; taskKey: string; adapterType: string; +}): boolean { + const receipt = receiptFromProfile(profile); + return !!receipt && receipt.companyId === binding.companyId && receipt.agentId === binding.agentId + && receipt.runId === binding.runId && receipt.taskKey === binding.taskKey + && receipt.adapterType === binding.adapterType && ["published", "suppressed_reset", "suppressed_newer", "suppressed_authority", "missing_stub"].includes(receipt.outcome); +} + +export function eligibleNativeTaskSessionPublication(snapshot: PersistedNativeSession): boolean { + return snapshot.goal == null && snapshot.semanticResult != null && snapshot.terminal?.runTerminalState === "succeeded" + && snapshot.activeTurnId == null && !(snapshot.pendingRuntimeRequests?.length); +} + +export type NativeTaskSessionPublicationRun = Pick; +export type NativeTaskSessionPublicationIssue = Pick; + +/** Caller holds coordinator → issue NO KEY UPDATE → run. This helper owns only + * the task-row lock and UPDATE; never starts a transaction or creates a stub. */ +export async function publishNativeTaskSessionInTransaction(tx: Transaction, input: { + binding: PaperclipControlPlaneBinding; + context: NativeTaskSessionPublicationContext; + snapshot: PersistedNativeSession; + run: NativeTaskSessionPublicationRun; + issue: NativeTaskSessionPublicationIssue | null; + resultId: string | null; + recoveryReady?: boolean; +}): Promise { + const { binding, context, snapshot, run, issue } = input; + const existingReceipt = receiptFromProfile(run.runnerProfileJson); + if (existingReceipt) { + if (existingReceipt.companyId !== binding.companyId || existingReceipt.issueId !== binding.issueId + || existingReceipt.agentId !== binding.agentId || existingReceipt.runId !== binding.runId + || existingReceipt.sessionId !== snapshot.identity.sessionId || existingReceipt.taskKey !== context.taskKey + || existingReceipt.adapterType !== context.adapterType || existingReceipt.resultId !== input.resultId) { + throw new Error("native_task_session_publication_receipt_mismatch"); + } + return existingReceipt; + } + if (record(run.runnerProfileJson)[NATIVE_TASK_SESSION_PUBLICATION_KEY] !== undefined) { + throw new Error("native_task_session_publication_receipt_invalid"); + } + if (!input.resultId) throw new Error("native_task_session_publication_result_missing"); + const [accepted] = await tx.select({ id: nativeRunResults.id, canonicalSha256: nativeRunResults.canonicalSha256, + turnId: nativeRunResults.turnId }).from(nativeRunResults).where(and( + eq(nativeRunResults.id, input.resultId), eq(nativeRunResults.runId, binding.runId), + eq(nativeRunResults.companyId, binding.companyId), eq(nativeRunResults.issueId, binding.issueId), + eq(nativeRunResults.completionContractId, binding.completionContractId), eq(nativeRunResults.schemaStatus, "accepted"), + )).limit(1); + const validated = validatePrpStructuredRunResult(snapshot.semanticResult); + if (!accepted || !validated.ok || accepted.canonicalSha256 !== nativeSha256({ + binding, result: validated.result, terminal: snapshot.terminal, turnId: accepted.turnId ?? null, + })) throw new Error("native_task_session_publication_result_mismatch"); + const receipt = (outcome: Outcome): Receipt => ({ schema: "paperclip.native-task-session-publication.v1", + companyId: binding.companyId, issueId: binding.issueId, agentId: binding.agentId, runId: binding.runId, + taskKey: context.taskKey, adapterType: context.adapterType, sessionId: snapshot.identity.sessionId, + resultId: accepted.id, outcome }); + if (context.taskKey !== binding.issueId || input.recoveryReady === false) return receipt("unsupported"); + const cancellation = record(run.resultJson?.nativeCancellation), stop = readRunCancellation(run.resultJson); + if (!issue || ["cancelled", "interrupted", "failed", "timed_out"].includes(run.status) || issue.companyId !== binding.companyId + || run.resultJson?.startupCancellation || (stop?.source === "operator" && stop.expected) + || (["run", "issue"].includes(String(cancellation.scope)) && ["pending", "acknowledged"].includes(String(cancellation.dispatchState))) + || (issue.conversationAgentId && issue.conversationAgentId !== binding.agentId)) return receipt("suppressed_authority"); + if (issue.conversationAgentId + && run.contextSnapshot?.conversationSessionGeneration !== issue.conversationSessionGeneration) return receipt("suppressed_reset"); + if (issue.executionRunId && issue.executionRunId !== binding.runId) return receipt("suppressed_newer"); + const [session] = await tx.select({ id: agentTaskSessions.id, goalSourceId: agentTaskSessions.goalSourceId, + lastRunId: agentTaskSessions.lastRunId }).from(agentTaskSessions).where(and( + eq(agentTaskSessions.companyId, binding.companyId), eq(agentTaskSessions.agentId, binding.agentId), + eq(agentTaskSessions.adapterType, context.adapterType), eq(agentTaskSessions.taskKey, context.taskKey), + )).for("update").limit(1); + // Supported native turns synchronously project a stub before this terminal + // snapshot. Absence can be an operator deletion; never recreate it later. + if (!session) return receipt("missing_stub"); + const boundSources = [binding.sourceInstanceId, binding.controlPlaneSourceInstanceId] + .filter((value): value is string => typeof value === "string" && value.length > 0) + .map(source => `${source}:${binding.runId}`); + if (!session.goalSourceId || !boundSources.includes(session.goalSourceId)) return receipt("suppressed_newer"); + if (session.lastRunId && session.lastRunId !== binding.runId) { + const [prior] = await tx.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: heartbeatRuns.issueId, nativeIssueId: heartbeatRuns.nativeIssueId, finishedAt: heartbeatRuns.finishedAt, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, session.lastRunId)).limit(1); + if (!prior || prior.companyId !== binding.companyId || prior.agentId !== binding.agentId + || (prior.nativeIssueId ?? prior.issueId) !== binding.issueId || !prior.finishedAt || !run.startedAt + || prior.finishedAt > run.startedAt) return receipt("suppressed_newer"); + } + await tx.update(agentTaskSessions).set({ + sessionParamsJson: { ...context.sessionParamsJson, sessionId: snapshot.identity.sessionId }, + sessionDisplayId: snapshot.providerSessionId ?? snapshot.sessionId, lastRunId: binding.runId, + lastError: null, updatedAt: new Date(), + }).where(eq(agentTaskSessions.id, session.id)); + return receipt("published"); +} diff --git a/server/src/services/native-runtime/native-usage-delivery.integration.test.ts b/server/src/services/native-runtime/native-usage-delivery.integration.test.ts index 21eab93cb2c..e59ccb2a965 100644 --- a/server/src/services/native-runtime/native-usage-delivery.integration.test.ts +++ b/server/src/services/native-runtime/native-usage-delivery.integration.test.ts @@ -12,8 +12,8 @@ import { agents, budgetPolicies, companies, costEvents, createDb, heartbeatRuns, import { type PrpEvent } from "../../vendor/paperclip-runner/index.js"; import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { accountRunCost } from "../run-cost-accounting.js"; -import { usageReceiptSpoolPath } from "../usage-receipts.js"; -import { createNativeUsageDelivery } from "./native-usage-delivery.js"; +import { usageReceiptSpoolPath, type UsageReceiptEnvelope } from "../usage-receipts.js"; +import { createNativeUsageContinuationDelivery, createNativeUsageDelivery } from "./native-usage-delivery.js"; const runnerRoot = fileURLToPath(new URL("../../../../packages/paperclip-runner/runner/", import.meta.url)); const fixtureProfile = process.env.CI ? "release" : "debug"; @@ -44,6 +44,89 @@ describe("producer-owned native usage", () => { payload: type === "usage.reported" ? { runDeltaAvailable: true, runDeltaBasis: "run", runDelta: { inputTokens: 120, outputTokens: 10, providerCostUsd: amount } } : {}, } as PrpEvent); + const finalizedSource = async () => { + const input = await seed(); + await db.update(heartbeatRuns).set({ runnerInstanceId: input.runnerInstanceId }).where(eq(heartbeatRuns.id, input.runId)); + const delivery = await createNativeUsageDelivery(db, input); + const report = event(input, 1, "usage.reported"), done = event(input, 2, "turn.completed"); + await delivery.capture(report); + await delivery.capture(done); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, input.runId)); + expect(await accountRunCost(db, input.runId)).toBe(true); + const { billing: _billing, ...source } = input; + return { input, source, report, done }; + }; + + it("continues exact finalized source receipts without acquiring ownership or resetting accounting", async () => { + const { input, source, report, done } = await finalizedSource(); + const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)); + await expect(createNativeUsageDelivery(db, input)).rejects.toThrow("native_usage_run_not_available"); + const continuation = await createNativeUsageContinuationDelivery(db, source); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before); + await continuation.capture(report); + await continuation.capture(done); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before); + expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId))).toHaveLength(2); + expect(await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, input.runId))).toHaveLength(1); + await expect(continuation.capture({ ...report, sourceInstanceId: randomUUID() })).rejects.toThrow("native_usage_source_mismatch"); + await expect(continuation.capture({ ...report, payload: { ...report.payload as object, runDeltaAvailable: false } })) + .rejects.toThrow("native_usage_source_conflict"); + }); + + it("journals an equal finalized source tail using the source billing without repricing", async () => { + const { input, source } = await finalizedSource(); + const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)); + // Even a structurally supplied extra current-run classification cannot own + // the old receipt: the continuation derives billing from saved source data. + const withCurrentBilling = { ...source, billing: { provider: "anthropic", biller: "current-grant", billingType: "subscription_included" } }; + const continuation = await createNativeUsageContinuationDelivery(db, withCurrentBilling); + await continuation.capture(event(input, 3, "usage.reported")); + const receipts = await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId)); + expect(receipts).toHaveLength(3); + expect(receipts.find(row => row.sequence === 3)?.receiptJson.receipt).toMatchObject(billing); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before); + expect((await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, input.runId)))[0].costCents).toBe(25); + }); + + it.each(["tokens", "cost", "incomplete", "unpriceable", "basis"])( + "holds a differing finalized source tail before acknowledgement: %s", async change => { + const { input, source } = await finalizedSource(); + const continuation = await createNativeUsageContinuationDelivery(db, source); + const late = event(input, 3, "usage.reported"); + if (change === "incomplete") (late.payload as Record).runDeltaAvailable = false; + if (change === "basis") delete (late.payload as Record).runDeltaBasis; + if (["tokens", "cost", "unpriceable"].includes(change)) (late.payload as Record).runDelta = { + inputTokens: change === "tokens" ? 150 : change === "unpriceable" ? "unknown" : 120, + outputTokens: 10, providerCostUsd: change === "cost" ? 0.5 : 0.25, + }; + await expect(continuation.capture(late)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation"); + await expect(continuation.capture(late)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation"); + expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId))).toHaveLength(2); + expect((await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, input.runId)))[0].costCents).toBe(25); + }); + + it.each(["company", "run", "runner", "source"])("rejects a changed continuation binding without mutation: %s", async change => { + const { input, source } = await finalizedSource(); + if (change === "source") await db.update(heartbeatRuns).set({ usageJson: sql`${heartbeatRuns.usageJson} || ${JSON.stringify({ accountingReceiptSourceId: randomUUID() })}::jsonb` }) + .where(eq(heartbeatRuns.id, input.runId)); + const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)); + await expect(createNativeUsageContinuationDelivery(db, { ...source, + ...(change === "company" ? { companyId: randomUUID() } : {}), + ...(change === "run" ? { runId: randomUUID() } : {}), + ...(change === "runner" ? { runnerInstanceId: randomUUID() } : {}), + })).rejects.toThrow("native_usage_continuation_source_mismatch"); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before); + }); + + it("rejects changed immutable source receipt facts during hydration", async () => { + const { input, source } = await finalizedSource(); + const [receipt] = await db.select().from(runUsageReceipts).where(and(eq(runUsageReceipts.runId, input.runId), eq(runUsageReceipts.sequence, 1))); + const changed = structuredClone(receipt.receiptJson) as UsageReceiptEnvelope; + changed.producerEvent!.usage!.runDelta = { inputTokens: 999, outputTokens: 10 }; + await db.update(runUsageReceipts).set({ receiptJson: changed }).where(eq(runUsageReceipts.id, receipt.id)); + await expect(createNativeUsageContinuationDelivery(db, source)).rejects.toThrow("native_usage_source_conflict"); + }); + it("retains the same Runner report through a failed Postgres receipt and an empty replacement receiver", async () => { const input = await seed(); expect(existsSync(usageReceiptSpoolPath())).toBe(false); diff --git a/server/src/services/native-runtime/native-usage-delivery.ts b/server/src/services/native-runtime/native-usage-delivery.ts index 2bbf52d0d1b..0d6f05d7de8 100644 --- a/server/src/services/native-runtime/native-usage-delivery.ts +++ b/server/src/services/native-runtime/native-usage-delivery.ts @@ -16,11 +16,13 @@ const uuid = (identity: string) => { return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-5${hex.slice(13, 16)}-a${hex.slice(17, 20)}-${hex.slice(20, 32)}`; }; -export async function createNativeUsageDelivery(db: Db, input: { +type NativeUsageDeliveryInput = { companyId: string; runId: string; runnerInstanceId: string; provider: NativeExecutionInput["provider"]; billing: Pick; -}) { +}; + +export async function createNativeUsageDelivery(db: Db, input: NativeUsageDeliveryInput) { const sourceId = uuid(`${input.runId}:${input.runnerInstanceId}`); const [bound] = await db.update(heartbeatRuns).set({ usageJson: sql` coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || '{"accountingReceiptOwner":"native_producer"}'::jsonb || @@ -29,9 +31,39 @@ export async function createNativeUsageDelivery(db: Db, input: { }).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId), isNull(heartbeatRuns.costAccountedAt))).returning({ id: heartbeatRuns.id }); if (!bound) throw new Error("native_usage_run_not_available"); - const saved = await db.select({ receipt: runUsageReceipts.receiptJson }).from(runUsageReceipts).where(and( + return createNativeUsageReceiver(db, input, sourceId, await loadNativeUsageReceipts(db, input, sourceId)); +} + +/** Drain the authenticated previous producer's accounting without acquiring or + * resetting its ownership. Finalized sources retain normal receipt equality + * and reconciliation rules; this grants no business/tool execution authority. */ +export async function createNativeUsageContinuationDelivery(db: Db, input: Omit) { + const sourceId = uuid(`${input.runId}:${input.runnerInstanceId}`); + const [source] = await db.select({ runnerInstanceId: heartbeatRuns.runnerInstanceId, usage: heartbeatRuns.usageJson }) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))); + if (source?.runnerInstanceId !== input.runnerInstanceId || source.usage?.accountingReceiptOwner !== "native_producer" + || source.usage.accountingReceiptSourceId !== sourceId) throw new Error("native_usage_continuation_source_mismatch"); + const saved = await loadNativeUsageReceipts(db, input, sourceId); + if (!saved.length) throw new Error("native_usage_continuation_receipts_missing"); + const classification = (receipt: AdapterUsageCheckpoint) => ({ + provider: receipt.provider, biller: receipt.biller, billingType: receipt.billingType, + }); + const receipts = saved.map(row => (row.receipt as UsageReceiptEnvelope).receipt as AdapterUsageCheckpoint); + const billing = classification(receipts[0]); + if (receipts.some(receipt => receiptFingerprint(classification(receipt)) !== receiptFingerprint(billing) + || receipt.model !== (input.provider.model ?? "unknown"))) throw new Error("native_usage_source_conflict"); + return createNativeUsageReceiver(db, { ...input, billing }, sourceId, saved, true); +} + +function loadNativeUsageReceipts(db: Db, input: Pick, sourceId: string) { + return db.select({ receipt: runUsageReceipts.receiptJson, hash: runUsageReceipts.receiptHash, + sequence: runUsageReceipts.sequence }).from(runUsageReceipts).where(and( eq(runUsageReceipts.companyId, input.companyId), eq(runUsageReceipts.runId, input.runId), eq(runUsageReceipts.sourceId, sourceId))).orderBy(asc(runUsageReceipts.sequence)); +} + +function createNativeUsageReceiver(db: Db, input: NativeUsageDeliveryInput, sourceId: string, + saved: Awaited>, verifySaved = false) { const accepted = new Map(); const turnAccounting = ["opencode", "acpx"].includes(input.provider.kind) ? createNativeTurnAccounting(input.provider) : null; let observedUsage: Record | null = null; @@ -60,6 +92,12 @@ export async function createNativeUsageDelivery(db: Db, input: { for (const row of saved) { const envelope = row.receipt as UsageReceiptEnvelope; if (!envelope.producerEvent) throw new Error("native_usage_receipt_missing_producer"); + if (verifySaved && (envelope.companyId !== input.companyId || envelope.runId !== input.runId + || envelope.sourceId !== sourceId || envelope.sequence !== row.sequence || envelope.adapterType !== "paperclip_runner" + || envelope.id !== uuid(`${input.runId}:${input.runnerInstanceId}:${envelope.producerEvent.sourceEventId}`) + || receiptFingerprint(envelope) !== row.hash || accepted.has(envelope.producerEvent.sourceEventId))) { + throw new Error("native_usage_source_conflict"); + } accepted.set(envelope.producerEvent.sourceEventId, envelope); observe(envelope); } diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts index 2edc3e1b718..ec45dca6437 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts @@ -48,6 +48,7 @@ import { import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js"; import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js"; +import { nativeSha256 } from "./canonical.js"; import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js"; import { finalizeNativeRun, repairCommittedNativeChatResponse } from "./native-run-finalizer.js"; import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; @@ -313,6 +314,63 @@ describe("PaperclipControlPlanePort conformance", () => { } }); + it("publishes native recovery metadata in the existing snapshot transaction and fences stale generations", async () => { + const identity = { ...CONTROL_PLANE_CONFORMANCE_OPEN.identity, runId: randomUUID(), sessionId: randomUUID() }; + const runnerId = randomUUID(); + const owner = { ...identity, authorityRunId: identity.runId, bootId: randomUUID(), generation: 1, leaseOwner: "snapshot-owner" }; + await db.insert(heartbeatRuns).values({ id: identity.runId, companyId: identity.companyId, agentId: identity.agentId, + status: "running", runtimeMode: "native", nativeIssueId: identity.issueId, nativeSessionId: identity.sessionId, + runnerInstanceId: runnerId, completionContractId: contractId, completionContractSha256: contractSha, + runnerProfileJson: { unrelated: "preserve" } }); + await db.insert(nativeRunFinalizations).values({ runId: identity.runId, companyId: identity.companyId, issueId: identity.issueId, + phase: "observed", + controllerBootId: owner.bootId, controllerGeneration: owner.generation, leaseOwner: owner.leaseOwner, + leaseExpiresAt: new Date(Date.now() + 60_000) }); + const port = new PaperclipControlPlanePort(db, { ...identity, completionContractId: contractId, + completionContractSha256: contractSha, sourceInstanceId: runnerId, controlPlaneSourceInstanceId: "snapshot-control" }, { + nativeRecovery: { owner, compatibility: { configDigest: `sha256:${"a".repeat(64)}`, processConfigDigest: null } }, + }); + await port.openRun({ identity, backendKind: "mock", sourceInstanceId: runnerId }); + const snapshot = { backendKind: "mock" as const, sessionId: identity.sessionId, identity }; + const statements: string[] = []; + const previousDebug = db.$client.options.debug; + db.$client.options.debug = (_connection, statement) => { statements.push(statement); }; + try { await port.checkpointSession(snapshot); } + finally { db.$client.options.debug = previousDebug; } + expect(statements.filter(statement => /^begin\b/i.test(statement.trim()))).toHaveLength(1); + expect(statements.filter(statement => /^update\s+"heartbeat_runs"/i.test(statement.trim()))).toHaveLength(1); + expect(statements.filter(statement => /^(?:with|select)\b/i.test(statement.trim()))).toHaveLength(1); + const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, identity.runId)); + expect(saved.runnerProfileJson).toMatchObject({ unrelated: "preserve", sessionCheckpoint: snapshot, + nativeControllerRecovery: { schema: "paperclip.native-controller-recovery.v1", snapshotSha256: nativeSha256(saved.runnerProfileJson?.sessionCheckpoint), + checkpoint: null, processConfigDigest: null } }); + // Missing PRP is an explicit incomplete publication, never fabricated authority. + await db.update(nativeRunFinalizations).set({ controllerGeneration: 2 }).where(eq(nativeRunFinalizations.runId, identity.runId)); + await expect(port.checkpointSession({ ...snapshot, providerSessionId: "stale" })).rejects.toThrow("checkpoint_owner_lost"); + await expect(port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL })) + .rejects.toThrow("checkpoint_owner_lost"); + const [unchanged] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, identity.runId)); + expect(unchanged.runnerProfileJson).toEqual(saved.runnerProfileJson); + // A valid coordinator lock does not certify a lease that expires while the + // subsequent run-row lock waits. Recheck time at the actual write boundary. + await db.update(nativeRunFinalizations).set({ controllerGeneration: 1, leaseExpiresAt: new Date(Date.now() + 150) }) + .where(eq(nativeRunFinalizations.runId, identity.runId)); + let locked!: () => void, release!: () => void; + const acquired = new Promise(resolve => { locked = resolve; }); + const held = new Promise(resolve => { release = resolve; }); + const blocker = db.transaction(async tx => { + await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, identity.runId)).for("update"); + locked(); + await held; + }); + await acquired; + const waiting = port.checkpointSession(snapshot); + const rejected = expect(waiting).rejects.toThrow("checkpoint_owner_lost"); + try { await new Promise(resolve => setTimeout(resolve, 200)); } + finally { release(); await blocker; } + await rejected; + }); + it("permits only read-only admission while an existing startup drains", async () => { const identity = { ...CONTROL_PLANE_CONFORMANCE_OPEN.identity, runId: randomUUID(), sessionId: randomUUID() }; const runnerId = randomUUID(); diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.ts b/server/src/services/native-runtime/paperclip-control-plane-port.ts index 2ee3446f568..f7bac1622d3 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.ts @@ -1,7 +1,7 @@ import { notifyNativeResultCommit } from "../native-result-commit-notifications.js"; import { notifyNativeWorkspaceFinalization } from "../native-workspace-finalization-notifications.js"; import { createAgentIdentityRedactor } from "../agent-identity-redaction.js"; -import { and, asc, eq, gt, or } from "drizzle-orm"; +import { and, asc, eq, gt, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { heartbeatRunEvents, @@ -9,6 +9,7 @@ import { issues, nativeRunFinalizations, nativeRunResults, + nativeRunnerCheckpoints, } from "@paperclipai/db"; import type { CompleteControlPlaneRunInput, @@ -30,6 +31,10 @@ import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { publishLiveEvent } from "../live-events.js"; import { nativeSha256 } from "./canonical.js"; import { redactRunnerApiEvent, redactRunnerApiPayloads } from "./runner-api-log-redaction.js"; +import { createNativeControllerRecovery, NATIVE_CONTROLLER_RECOVERY_MAX_BYTES, type NativeControllerCompatibility } from "./native-controller-recovery.js"; +import { eligibleNativeTaskSessionPublication, publishNativeTaskSessionInTransaction, NATIVE_TASK_SESSION_PUBLICATION_KEY, + type NativeTaskSessionPublicationContext, type NativeTaskSessionPublicationIssue } from "./native-task-session-publication.js"; +import { lockNativeRunnerCheckpointOwner, nativeRunnerCheckpointOwnerLock, validateNativeRunnerCheckpointOwner, inspectNativeRunnerUpgradeHeader, type NativeRunnerCheckpointOwner } from "./native-runner-checkpoint.js"; export interface PaperclipControlPlaneBinding { companyId: string; @@ -90,6 +95,9 @@ export class PaperclipControlPlanePort implements ControlPlanePort { readonly #onEventPersistence?: (event: PrpEvent, startedAtMs: number, endedAtMs: number) => Promise; readonly #onCommittedEvent?: (event: PrpEvent, cursor: number) => Promise; readonly #onDuplicateEvent?: (event: PrpEvent, cursor: number) => Promise; + readonly #nativeRecovery?: { compatibility: NativeControllerCompatibility; owner: NativeRunnerCheckpointOwner }; + readonly #taskSessionPublication?: NativeTaskSessionPublicationContext; + readonly #taskSessionPublicationOwner?: NativeRunnerCheckpointOwner; constructor( db: Db, @@ -105,6 +113,9 @@ export class PaperclipControlPlanePort implements ControlPlanePort { onEventPersistence?: (event: PrpEvent, startedAtMs: number, endedAtMs: number) => Promise; onCommittedEvent?: (event: PrpEvent, cursor: number) => Promise; onDuplicateEvent?: (event: PrpEvent, cursor: number) => Promise; + nativeRecovery?: { compatibility: NativeControllerCompatibility; owner: NativeRunnerCheckpointOwner }; + taskSessionPublication?: NativeTaskSessionPublicationContext; + taskSessionPublicationOwner?: NativeRunnerCheckpointOwner; } = {}, ) { this.#db = db; @@ -116,6 +127,9 @@ export class PaperclipControlPlanePort implements ControlPlanePort { this.#onEventPersistence = options.onEventPersistence; this.#onResultCommitted = options.onResultCommitted; this.#onDuplicateEvent = options.onDuplicateEvent; + this.#nativeRecovery = options.nativeRecovery; + this.#taskSessionPublication = options.taskSessionPublication ? structuredClone(options.taskSessionPublication) : undefined; + this.#taskSessionPublicationOwner = options.taskSessionPublicationOwner; } #assertActive(options?: { signal: AbortSignal }): void { @@ -123,7 +137,9 @@ export class PaperclipControlPlanePort implements ControlPlanePort { options?.signal.throwIfAborted(); } - #matchesPersistedBinding(run: typeof heartbeatRuns.$inferSelect): boolean { + #matchesPersistedBinding(run: Pick): boolean { return run.companyId === this.#binding.companyId && run.agentId === this.#binding.agentId && run.runtimeMode === "native" @@ -200,19 +216,105 @@ export class PaperclipControlPlanePort implements ControlPlanePort { || identity.agentId !== this.#binding.agentId || this.#sessionId !== identity.sessionId ) throw new Error("native_session_checkpoint_binding_mismatch"); + const publication = this.#taskSessionPublication && eligibleNativeTaskSessionPublication(snapshot) + ? this.#taskSessionPublication : null; + if (publication && !this.#taskSessionPublicationOwner) throw new Error("native_task_session_publication_owner_required"); await this.#db.transaction(async (tx) => { - const run = await tx.select().from(heartbeatRuns) - .where(eq(heartbeatRuns.id, this.#binding.runId)).for("update").limit(1) + const recovery = this.#nativeRecovery; + const owner = recovery?.owner ?? (publication ? this.#taskSessionPublicationOwner : undefined); + if (publication && recovery && nativeSha256(recovery.owner) !== nativeSha256(this.#taskSessionPublicationOwner)) { + throw new Error("native_task_session_publication_owner_mismatch"); + } + const ownerLock = owner ? nativeRunnerCheckpointOwnerLock(tx, owner) : null; + // NO KEY UPDATE serializes reset/admission fields while remaining compatible + // with completeRun's parent KEY SHARE. Keep coordinator → issue → run order. + const publicationIssue = publication ? tx.$with("publication_issue").as(tx.select({ + id: issues.id, companyId: issues.companyId, executionRunId: issues.executionRunId, + conversationAgentId: issues.conversationAgentId, conversationSessionGeneration: issues.conversationSessionGeneration, + }).from(issues).where(and(eq(issues.id, this.#binding.issueId), eq(issues.companyId, this.#binding.companyId), + ownerLock ? sql`exists (select 1 from ${ownerLock})` : undefined)).for("no key update")) : null; + const lockedRun = tx.$with("snapshot_run").as(tx.select({ + id: heartbeatRuns.id, status: heartbeatRuns.status, + contextSnapshot: (publication ? sql>`jsonb_build_object('conversationSessionGeneration', ${heartbeatRuns.contextSnapshot}->'conversationSessionGeneration')` : sql`null`).as("publication_context"), + resultJson: (publication ? sql>`jsonb_build_object('nativeCancellation', ${heartbeatRuns.resultJson}->'nativeCancellation', + 'cancellation', ${heartbeatRuns.resultJson}->'cancellation', 'startupCancellation', ${heartbeatRuns.resultJson}->'startupCancellation')` : sql`null`).as("publication_cancellation"), + startedAt: heartbeatRuns.startedAt, createdAt: heartbeatRuns.createdAt, + companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, nativeIssueId: heartbeatRuns.nativeIssueId, + nativeSessionId: heartbeatRuns.nativeSessionId, runnerInstanceId: heartbeatRuns.runnerInstanceId, + completionContractId: heartbeatRuns.completionContractId, + completionContractSha256: heartbeatRuns.completionContractSha256, + runnerProfileJson: heartbeatRuns.runnerProfileJson, + }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.id, this.#binding.runId), ownerLock + ? sql`exists (select 1 from ${ownerLock})` : undefined, + publicationIssue ? sql`exists (select 1 from ${publicationIssue})` : undefined)).for("update")); + // Project the locking CTE's tuples, including any update committed while + // the lock waited. A second scan of heartbeat_runs could see an old MVCC tuple. + // The run CTE depends on the coordinator CTE: both locks precede the + // outer clock sample, including when either lock waits. + const selected = await tx.with(...(ownerLock ? [ownerLock] : []), ...(publicationIssue ? [publicationIssue] : []), lockedRun).select({ run: { + id: lockedRun.id, status: lockedRun.status, contextSnapshot: lockedRun.contextSnapshot, resultJson: lockedRun.resultJson, + startedAt: lockedRun.startedAt, createdAt: lockedRun.createdAt, + companyId: lockedRun.companyId, agentId: lockedRun.agentId, + runtimeMode: lockedRun.runtimeMode, nativeIssueId: lockedRun.nativeIssueId, + nativeSessionId: lockedRun.nativeSessionId, runnerInstanceId: lockedRun.runnerInstanceId, + completionContractId: lockedRun.completionContractId, + completionContractSha256: lockedRun.completionContractSha256, + runnerProfileJson: lockedRun.runnerProfileJson, + }, now: sql`clock_timestamp()`, ...(ownerLock ? { + leaseOwner: sql`(select ${ownerLock.leaseOwner} from ${ownerLock})`, + leaseExpiresAt: sql`(select ${ownerLock.leaseExpiresAt} from ${ownerLock})`.mapWith(nativeRunFinalizations.leaseExpiresAt), + resultId: sql`(select ${ownerLock.resultId} from ${ownerLock})`, + } : {}), ...(recovery ? { + checkpointHeader: sql | null>`(select material from ${nativeRunnerCheckpoints} + where ${nativeRunnerCheckpoints.runId} = ${this.#binding.runId} + and ${nativeRunnerCheckpoints.companyId} = ${this.#binding.companyId} + and ${nativeRunnerCheckpoints.issueId} = ${this.#binding.issueId} + and ${nativeRunnerCheckpoints.part} = 'header')`, + } : {}), ...(publicationIssue ? { publicationIssue: sql`(select jsonb_build_object( + 'id', ${publicationIssue.id}, 'companyId', ${publicationIssue.companyId}, 'executionRunId', ${publicationIssue.executionRunId}, + 'conversationAgentId', ${publicationIssue.conversationAgentId}, 'conversationSessionGeneration', ${publicationIssue.conversationSessionGeneration}) from ${publicationIssue})` } : {}), + }).from(lockedRun).limit(1) .then((rows) => rows[0] ?? null); + const run = selected?.run; this.#assertActive(options); + if (owner && !selected) throw new Error("native_runner_checkpoint_owner_lost"); if (!run || !this.#matchesPersistedBinding(run)) { throw new Error("native_session_checkpoint_binding_mismatch"); } - await tx.update(heartbeatRuns).set({ - runnerProfileJson: { + // The run row lock may have waited beyond our lease. Generation cannot + // change while we hold the coordinator, but wall-clock expiry still can. + if (owner && selected) await validateNativeRunnerCheckpointOwner(tx, owner, + { leaseOwner: selected.leaseOwner ?? null, leaseExpiresAt: selected.leaseExpiresAt ?? null, + resultId: selected.resultId ?? null, now: selected.now }, !snapshot.semanticResult); + const header = recovery && selected?.checkpointHeader + ? inspectNativeRunnerUpgradeHeader(this.#binding, selected.checkpointHeader) : null; + if (header && (header.normalizedSessionId !== identity.sessionId || header.runnerInstanceId !== this.#binding.sourceInstanceId)) { + throw new Error("native_session_checkpoint_prp_binding_mismatch"); + } + const profile: Record = { ...record(run.runnerProfileJson), sessionCheckpoint: structuredClone(snapshot) as unknown as Record, - }, + ...(recovery ? { nativeControllerRecovery: createNativeControllerRecovery(snapshot, recovery.compatibility, + header ? { runId: this.#binding.runId, runnerInstanceId: header.runnerInstanceId, + normalizedSessionId: header.normalizedSessionId } : null) } : {}), + }; + // Driver/live snapshots may already describe a terminal result before + // completeRun accepts it. Persist that observation without a handled + // receipt; the existing post-acceptance checkpoint publishes the pointer. + if (publication && selected?.resultId) { + const receipt = await publishNativeTaskSessionInTransaction(tx, { binding: this.#binding, context: publication, + snapshot, run, issue: selected.publicationIssue ?? null, resultId: selected.resultId ?? null, + recoveryReady: !recovery || header !== null }); + profile[NATIVE_TASK_SESSION_PUBLICATION_KEY] = receipt; + if (Buffer.byteLength(JSON.stringify({ nativeControllerRecovery: profile.nativeControllerRecovery, + [NATIVE_TASK_SESSION_PUBLICATION_KEY]: receipt })) > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES) { + throw new Error("native_task_session_publication_metadata_too_large"); + } + } + await tx.update(heartbeatRuns).set({ + runnerProfileJson: profile, sessionIdAfter: snapshot.providerSessionId ?? snapshot.sessionId, nativePhase: snapshot.semanticResult ? "workspace_finalizing" : "observed", nativePhaseUpdatedAt: new Date(), @@ -386,9 +488,12 @@ export class PaperclipControlPlanePort implements ControlPlanePort { }); const insertedResult = await this.#db.transaction(async (tx) => { - // Result insertion checks the task foreign key. Acquire that parent lock - // before the run, matching task mutations that subsequently update a run. - // Otherwise concurrent chat/status writes can deadlock after generation. + // Status commit and snapshot publication lock coordinator → issue → run. + // A replay must follow that order too: holding the issue while waiting + // for the coordinator deadlocks a status commit's issue UPDATE lock. + const recovery = this.#nativeRecovery; + const owner = recovery ? await lockNativeRunnerCheckpointOwner(tx, recovery.owner) : null; + // Result insertion checks the task FK. Keep its parent lock before run. await tx.select({ id: issues.id }).from(issues) .where(and(eq(issues.id, this.#binding.issueId), eq(issues.companyId, this.#binding.companyId))) .for("key share"); @@ -399,6 +504,10 @@ export class PaperclipControlPlanePort implements ControlPlanePort { if (!run || !this.#matchesPersistedBinding(run)) { throw new Error("native_result_binding_mismatch"); } + if (recovery && owner) { + const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`); + await validateNativeRunnerCheckpointOwner(tx, recovery.owner, { ...owner, now: clock.now }, false); + } const callerConditions = [eq(nativeRunResults.serverFingerprint, serverFingerprint)]; if (value.callerResultId) callerConditions.push(eq(nativeRunResults.callerResultId, value.callerResultId)); @@ -411,6 +520,7 @@ export class PaperclipControlPlanePort implements ControlPlanePort { if (existing.canonicalSha256 !== canonicalSha256) throw new Error("structured_result_replay_conflict"); return false; } + if (owner?.resultId) throw new Error("structured_result_replay_conflict"); const resultJson = { result: validated.result, terminal: value.terminal, diff --git a/server/src/services/native-runtime/remote-warm-session-recovery.test.ts b/server/src/services/native-runtime/remote-warm-session-recovery.test.ts new file mode 100644 index 00000000000..91317d2af4b --- /dev/null +++ b/server/src/services/native-runtime/remote-warm-session-recovery.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, it } from "vitest"; +import { isSettledRemoteWarmControlState } from "./remote-warm-session-recovery.js"; + +const identity = { runId: "run", normalizedSessionId: "session", runnerInstanceId: "runner", environmentLeaseId: "lease", turnId: "turn", itemId: "item" }; +const terminal = { eventType: "run.terminal", envelope: { ...identity } }; +const goal = { schema: "paperclip.session_goal.snapshot.v1", goal: null, workingNow: false }; +const notification = { schema: "paperclip.prp.event.v1", eventType: "session.goal.snapshot", runId: identity.runId, + normalizedSessionId: identity.normalizedSessionId, sourceInstanceId: "runner", sourceKind: "runner", payload: goal }; +const snapshot = { eventType: "session.goal.snapshot", envelope: { ...identity, payload: notification } }; +const state = (events: unknown[]) => ({ identity, commands: [{ status: "completed" }], committedEvents: events }); + +describe("settled remote warm control evidence", () => { + it.each(["paperclip.prp.event.v1", "paperclip.prp.event.v2"])("accepts the terminal and an authenticated %s idle goal-read suffix without changing its order", schema => { + const suffix = { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, schema } } }; + const control = state([terminal, suffix]); + expect(isSettledRemoteWarmControlState(control, identity)).toBe(true); + expect(control.committedEvents).toEqual([terminal, suffix]); + expect(isSettledRemoteWarmControlState(state([terminal]), identity)).toBe(true); + }); + + it.each([ + ["active goal", { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, payload: { ...goal, goal: { status: "active" } } } } }], + ["provider work", { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, payload: { ...goal, workingNow: true } } } }], + ["unknown event", { ...snapshot, eventType: "unknown" }], + ["unknown protocol version", { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, schema: "paperclip.prp.event.v3" } } }], + ["new turn", { ...snapshot, eventType: "turn.started" }], + ["foreign run", { ...snapshot, envelope: { ...snapshot.envelope, runId: "other" } }], + ["foreign session", { ...snapshot, envelope: { ...snapshot.envelope, normalizedSessionId: "other" } }], + ["foreign source", { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, sourceInstanceId: "other" } } }], + ["foreign lease", { ...snapshot, envelope: { ...snapshot.envelope, environmentLeaseId: "other" } }], + ["foreign notification", { ...snapshot, envelope: { ...snapshot.envelope, payload: { ...notification, runId: "other" } } }], + ["missing payload", { ...snapshot, envelope: { ...identity } }], + ])("holds a %s suffix", (_name, suffix) => { + expect(isSettledRemoteWarmControlState(state([terminal, suffix]), identity)).toBe(false); + }); + + it("holds missing terminals, pending commands and mixed core identities", () => { + expect(isSettledRemoteWarmControlState(state([snapshot]), identity)).toBe(false); + expect(isSettledRemoteWarmControlState({ ...state([terminal, snapshot]), commands: [{ status: "dispatched" }] }, identity)).toBe(false); + expect(isSettledRemoteWarmControlState({ ...state([terminal, snapshot]), identity: { ...identity, runnerInstanceId: "other" } }, identity)).toBe(false); + expect(isSettledRemoteWarmControlState(state([{ ...terminal, envelope: { ...identity, runId: "other" } }, snapshot]), identity)).toBe(false); + }); +}); diff --git a/server/src/services/native-runtime/remote-warm-session-recovery.ts b/server/src/services/native-runtime/remote-warm-session-recovery.ts index 2d91c49f6c7..893dfefaef6 100644 --- a/server/src/services/native-runtime/remote-warm-session-recovery.ts +++ b/server/src/services/native-runtime/remote-warm-session-recovery.ts @@ -3,6 +3,7 @@ import { agents, companies, environmentLeases, heartbeatRuns, issues, nativeRunF import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import { parseNativeExecutionInput, type NativeExecutionInput, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; import { currentNativeControllerIdentity, evaluateNativeControllerTakeover } from "./native-restart-recovery.js"; +import { readNativeControllerRecovery } from "./native-controller-recovery.js"; export type RemoteWarmSessionRecovery = { identity: Record; @@ -13,6 +14,36 @@ export type RemoteWarmSessionRecovery = { revalidate: (leaseOwner: string) => Promise; }; +/** A goal read can publish an idle snapshot after the terminal. It carries no + * new work; every other suffix still requires reconciliation before handoff. */ +export function isSettledRemoteWarmControlState(control: Record, identity: Record): boolean { + const record = (value: unknown): Record => + value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; + const commands = control.commands; + const events = control.committedEvents; + if (!Array.isArray(commands) || commands.some(command => record(command).status !== "completed") || + !Array.isArray(events) || Object.entries(identity).some(([key, value]) => record(control.identity)[key] !== value)) return false; + const terminalIndex = events.findLastIndex(event => record(event).eventType === "run.terminal"); + if (terminalIndex < 0) return false; + const matchesSession = (event: unknown) => { + const envelope = record(record(event).envelope); + return envelope.runId === identity.runId && envelope.normalizedSessionId === identity.normalizedSessionId; + }; + if (!matchesSession(events[terminalIndex])) return false; + return events.slice(terminalIndex + 1).every(event => { + const envelope = record(record(event).envelope); + const notification = record(envelope.payload); + const payload = record(notification.payload); + return record(event).eventType === "session.goal.snapshot" && matchesSession(event) && + Object.entries(identity).every(([key, value]) => envelope[key] === value) && + (notification.schema === "paperclip.prp.event.v1" || notification.schema === "paperclip.prp.event.v2") && + notification.eventType === "session.goal.snapshot" && + notification.sourceInstanceId === identity.runnerInstanceId && notification.sourceKind === "runner" && + notification.runId === identity.runId && notification.normalizedSessionId === identity.normalizedSessionId && + payload.schema === "paperclip.session_goal.snapshot.v1" && payload.goal === null && payload.workingNow === false; + }); +} + export async function inspectRemoteWarmSessionRecovery(input: { db: Db; execution: NativeExecutionInput; @@ -52,15 +83,14 @@ export async function inspectRemoteWarmSessionRecovery(input: { checkpoint.driverKind !== previous.session.driverKind || !checkpoint.providerSessionId || checkpoint.activeTurnId !== null || checkpoint.pendingRuntimeRequests?.length !== 0) return deny(); const control = input.readControlState(); - const commands = control.commands; - const events = control.committedEvents; - const terminal = Array.isArray(events) ? events.at(-1) : undefined; - if (!Array.isArray(commands) || commands.some(command => command.status !== "completed") || - terminal?.eventType !== "run.terminal" || terminal?.envelope?.runId !== identity.runId || - terminal?.envelope?.normalizedSessionId !== identity.normalizedSessionId || - Object.entries(identity).some(([key, value]) => (control.identity as Record)?.[key] !== value)) return deny(); + if (!isSettledRemoteWarmControlState(control, identity)) return deny(); const assertPriorStopped = async () => { - const stopped = await evaluateNativeControllerTakeover({ owner: prior.owner, now: new Date(), requireStoppedController: true }); + const persistedRecovery = readNativeControllerRecovery(prior.run.runnerProfileJson?.nativeControllerRecovery, + prior.run.runnerProfileJson?.sessionCheckpoint); + // A PID in another webserver container cannot prove life or death. New + // durable controllers are fenced by the database lease and current turn. + const stopped = await evaluateNativeControllerTakeover({ owner: prior.owner, now: new Date(), requireStoppedController: true, + remoteLeaseFenced: persistedRecovery !== null }); if (!stopped.allowed) return deny(); }; await assertPriorStopped(); diff --git a/server/src/services/run-secret-redaction.ts b/server/src/services/run-secret-redaction.ts index ad07a402920..597d12a9bd9 100644 --- a/server/src/services/run-secret-redaction.ts +++ b/server/src/services/run-secret-redaction.ts @@ -52,7 +52,7 @@ export function redactRegisteredSecretValues(input: T, values: string[], cred if (!record) return input; return Object.fromEntries( Object.entries(record) - .filter(([key]) => key !== REGISTRY_KEY) + .filter(([key]) => key !== REGISTRY_KEY && key !== "nativeControllerRecovery" && key !== "nativeTaskSessionPublication") .map(([key, value]) => [key, key === "credentialAccess" ? (credentialConnectionName ? { connectionName: redactText(credentialConnectionName, values) } : {}) : redactRegisteredSecretValues(value, values, credentialConnectionName)]), diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index 6e84dd0fbbd..63a918160ea 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -94,6 +94,9 @@ export const RUNNER_PRP_PROTOCOL_MIN = runner.RUNNER_PRP_PROTOCOL_MIN; export const RUNNER_PRP_PROTOCOL_MAX = runner.RUNNER_PRP_PROTOCOL_MAX; export const runnerCodexDynamicToolsFit = runner.runnerCodexDynamicToolsFit; export const inspectWarmRunTransition = runner.inspectWarmRunTransition; +export const inspectStagedRunAttachment = runner.inspectStagedRunAttachment; +export const inspectStagedRunAttachmentHeader = runner.inspectStagedRunAttachmentHeader; +export type StagedRunAttachment = import("@paperclipai/paperclip-runner").StagedRunAttachment; export const readRunnerdArtifactBinding = runner.readRunnerdArtifactBinding; export const readLinuxProcessStartedAt = runner.readLinuxProcessStartedAt; export const NativeSessionCloseUnrecoverableError = diff --git a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md index dbb11f6b884..1d7bf6f6fcb 100644 --- a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md +++ b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md @@ -114,6 +114,13 @@ effects with different tool IDs cannot repeat. `remote-restart-before.json`, `remote-restart-after.json` and `remote-restart-final.json` retain receipt identities and hashes. They exclude provider text, tool arguments, command output, and raw provider sandbox IDs. +The dedicated explicit-only `daytona-restart-continuity` definition 7 additionally +replaces the controller after completion, then posts its unchanged saved-project +follow-up through the normal authenticated task-comment API. Exactly one matching +persisted user comment, consumed wake, same native/provider identity, new completed +tool and useful output are required. Its initial browser journey remains; post-C +readback is API-only and does not qualify browser reconnection or rendered status. +The definition 6 browser timeout remains failed with an unknown browser cause. The suite definition version and harness digest change with this oracle. Its deterministic calibration is not a claim that a live Daytona run passed. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 01049d9e5a2..fbfbe27169c 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -10,13 +10,45 @@ qualification. The explicit-only `daytona-restart-continuity` suite runs the existing `recover-controller` journey with the recorded Codex and Claude staging models -on fresh Daytona fixtures. After a completed task tool in an active revision -turn, it keeps the owned development controller offline for 125 seconds, beyond -the controller's two-minute reconnect budget. The existing independent oracle +on fresh reusable warm Daytona fixtures. The sandbox, Runner and provider history +must survive both controller replacements. After a completed task tool in an active revision +turn, it keeps the owned development controller offline for 300 seconds. This +exceeds the 60-second ownership lease and remains within the Runner's 420-second +reconnect grace. The existing independent oracle requires the original process, native session, sandbox lease and receipt prefix, one completion and successful workspace finalization, plus verified downloaded -work. It permits one attempt per cell and retains failed evidence. This is +work. After completion, it replaces the controller again with its native cache +unavailable and asks a new turn to execute the saved project. The continuation +oracle requires the same retained provider session and a new useful result. +It permits one attempt per cell and retains failed evidence. This is development API-key acceptance, distinct from staging subscription recovery. +The five-minute outage is a resilience test, not an HTTP downtime allowance. +This suite gives its Daytona sandbox a 15-minute provider auto-stop interval, +leaving headroom for the outage and replacement startup. Native agent idle timing, +ownership/reconnect limits, archive/delete backstops and owned cleanup remain unchanged. +The retained continuation accepts whitespace between its exact marker and the +exact `queue_r` output token; readiness and final grading use the same matcher. + +Definition version 7 uses the existing warm reusable environment with a dedicated +configuration key. Only the post-completion replacement follow-up uses the normal +board-authenticated task-comment API, with one persisted user comment and the +unchanged follow-up request. Initial creation, active outage and queued continuation +retain their browser journey. Post-C settlement, task status and useful reply are +read through production APIs, without post-C browser navigation or rendered-status +assertions. The required final screenshot captures the actual retained Chromium +page, which may be stale or show an error; it is labeled as unqualified browser +evidence and does not establish fresh product state. The same strict native/provider +identity, new tool, +saved-project output and finalization checks remain required. Definition 6's +observed post-C browser navigation timeout remains failed; its cause is unknown. +This definition does not claim that browser reconnection was fixed. +The prior version 5 per-turn/no-reuse measurement **failed** +retained native/provider identity and remains failed; it does not qualify this +warm fixture. Cache retirement moves the entire controller Runner directory, +including provider failover-backup bytes. Once a per-turn sandbox is disposed, +its DB checkpoint alone cannot replace those bytes. Per-turn application +qualification under this disk loss awaits the provider-backup owner's durable +storage contract; native controller recovery does not waive that dependency. The explicit-only [GitHub review bot journey](GITHUB-REVIEW-JOURNEY.md) tests attended bot setup and an automatic PR review against an attached deployment. @@ -596,6 +628,15 @@ inputs, or a tracked env file. ## Local commands +When the host cannot start embedded Postgres, set +`PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL` to an explicitly supplied loopback +test Postgres administrator. The launcher creates a new `runner_e2e_` +database per attempt; the serving app applies its migrations. This does not +reuse the administrator URL's database. The administrator variable is not +forwarded to the server or browser. The launcher checks the private ownership +receipt and deletes only its database after confirmed process/resource cleanup. +Unknown remote cleanup preserves that database and the private fixture for recovery. + Install dependencies and Chromium once. Native local cells also need the local runner binaries: @@ -2507,7 +2548,13 @@ The same four installed-package pins also admit three existing local native Codex cells: `first-task.runner-codex.local.task-card-accept`, `everyday-workflows.runner-codex.local.build-revise`, and `agent-chat.runner-codex.local.continuity-restart`. Select exact IDs; other -Codex cells, legacy profiles and remote environments remain outside this lane. +Codex cells and legacy profiles remain outside this lane. The two explicit +Daytona controller-recovery cells are also admitted: +`everyday-workflows.runner-codex.daytona.recover-controller` and +`daytona-restart-continuity.runner-codex.daytona.recover-controller`. +The latter uses the 300-second outage and retained-session follow-up. Daytona +cells require the pinned installed plugin described below. Admission support +does not establish that either cell has passed against installed artifacts. These reuse the onboarding, independently checked file/revision, and restart graders. They require separately authorized provider access and are not executed by the provider-free release smoke. diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index a8ba74083b5..f7b1d783ee8 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -33,6 +33,47 @@ import { } from "./selectors.js"; describe("runner E2E catalog", () => { + it("keeps the Daytona controller outage on a reusable warm sandbox inside its idle lifetime", () => { + const suite = runnerSuites.find(candidate => candidate.id === "daytona-restart-continuity")!; + const secretRefs = { + OPENAI_API_KEY: { type: "secret_ref" as const, secretId: "22222222-2222-4222-8222-222222222222", version: "latest" as const }, + ANTHROPIC_API_KEY: { type: "secret_ref" as const, secretId: "33333333-3333-4333-8333-333333333333", version: "latest" as const }, + DAYTONA_API_KEY: { type: "secret_ref" as const, secretId: "44444444-4444-4444-8444-444444444444", version: "latest" as const }, + }; + const environmentInput = { executionId: "controller-outage", secretRefs, daytonaImage: `fixture@sha256:${"a".repeat(64)}` }; + const base = runnerEnvironments.find(environment => environment.id === "daytona")!; + const baseConfig = base.buildEnvironment(environmentInput).config as Record; + const warmConfig = daytonaWarmEnvironment.buildEnvironment(environmentInput).config as Record; + const cells = runnerMatrix.filter(entry => entry.suite.id === suite.id); + expect(cells).toHaveLength(2); + expect(cells.map(cell => cell.id)).toEqual([ + "daytona-restart-continuity.runner-codex.daytona.recover-controller", + "daytona-restart-continuity.runner-acpx-claude.daytona.recover-controller", + ]); + for (const cell of cells) { + const config = cell.environment.buildEnvironment(environmentInput).config as Record; + expect(config).toMatchObject({ reuseLease: true, runnerLifecycleMode: "warm" }); + expect(config.autoStopInterval).toBe(15); + expect(Number(config.autoStopInterval) * 60_000).toBeGreaterThan(420_000); + expect(config).toEqual({ ...warmConfig, autoStopInterval: 15 }); + expect(cell.environment.configurationKey).toBe("warm-controller-outage-v1"); + expect(cell.environment.label).toContain("warm reusable"); + const agentInput = { ...environmentInput, environmentId: "env-1", environmentFixtureId: "daytona" as const, workspacePath: "/workspace" }; + const adapterConfig = cell.profile.buildAgent(agentInput).adapterConfig as Record; + const original = runnerProfiles.find(profile => profile.id === cell.profile.id)!.buildAgent(agentInput).adapterConfig as Record; + expect(adapterConfig.idleTimeoutMs).toBe(300_000); + expect(adapterConfig.lifecycleMode).toBe(original.lifecycleMode); + expect(cell.suite.manualOnly).toBe(true); + expect(cell.task.automaticRetryPolicy).toBe("single_attempt"); + expect(selectRunnerExecutions(parseRunnerSelectors(["--id", cell.id])).map(entry => entry.id)).toEqual([cell.id]); + expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(entry => entry.id === cell.id)).toBe(false); + } + expect(baseConfig).toMatchObject({ autoStopInterval: 5, reuseLease: false, runnerLifecycleMode: "per_turn" }); + expect(warmConfig).toMatchObject({ autoStopInterval: 5, reuseLease: true, runnerLifecycleMode: "warm" }); + expect(suite.definitionMetadata).toMatchObject({ version: 7, controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, + retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", maximumAttemptsPerCell: 1 }); + }); + it("selects non-reused Kimi lifecycle leases without changing its warm or local cases", () => { const suite = runnerSuites.find(row => row.id === "kimi-qualification")!; const cells = runnerMatrix.filter(row => row.suite.id === suite.id); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 9ee243f3e8e..8b6fae6296c 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1160,13 +1160,21 @@ const repositoryModelProfiles = [ export const runnerSuites: readonly RunnerSuiteFixture[] = [ { - id: "daytona-restart-continuity", label: "Daytona controller outage continuity", manualOnly: true, - description: "Keep the original active native runner, task tools and saved files through more than two minutes without a controller, then commit the result without replay.", + id: "daytona-restart-continuity", label: "Daytona warm controller outage continuity", manualOnly: true, + description: "Keep the original active native runner, reusable warm sandbox, task tools and saved files through five minutes without a controller, then continue the completed session after another replacement.", groups: [], profiles: repositoryModelProfiles.map(({ native }) => productionStoryProfile(native)), - environments: runnerEnvironments.filter(environment => environment.id === "daytona"), + environments: [{ + ...daytonaWarmEnvironment, + configurationKey: "warm-controller-outage-v1", + buildEnvironment(input: EnvironmentFixtureBuildInput) { + const payload = daytonaWarmEnvironment.buildEnvironment(input); + // Keep provider idle stop beyond the outage and replacement budget. + return { ...payload, config: { ...(payload.config as Record), autoStopInterval: 15 } }; + }, + }], tasks: everydayTasks.filter(task => task.id === "recover-controller") - .map(task => ({ ...task, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2, - definitionMetadata: { version: 2, scheduling: "explicit-only", controllerDowntimeMs: 125_000, + .map(task => ({ ...task, expectedRunCount: task.expectedRunCount + 1, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2, + definitionMetadata: { version: 7, scheduling: "explicit-only", controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", maximumAttemptsPerCell: 1, budgetMonthlyCents: 1000, instructions: "production", grading: "same-active-process-lease-native-prefix-and-workspace-receipt", authentication: "api-key", stagingAuthentication: "subscription", stagingQualification: false }, }, diff --git a/tests/runner-e2e/controller-outage.test.ts b/tests/runner-e2e/controller-outage.test.ts index d8bb79b9743..edb1c9a5718 100644 --- a/tests/runner-e2e/controller-outage.test.ts +++ b/tests/runner-e2e/controller-outage.test.ts @@ -25,7 +25,7 @@ describe("controller outage qualification", () => { expect(executions).toHaveLength(2); for (const execution of executions) { const outage = controllerOutageForExecution(execution, {}); - expect(outage).toEqual({ signal: "SIGKILL", holdMs: 125_000 }); + expect(outage).toEqual({ signal: "SIGKILL", holdMs: 300_000 }); expect(controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "300000" })) .toEqual({ signal: "SIGKILL", holdMs: 300_000 }); expect(() => controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "1000" })) diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index 9f24264026f..cbba89e4db7 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -8,7 +8,7 @@ import { gradeConnectionGuidanceDecline, hasConnectionGuidanceDeclineReply, expl import { expect, type Page } from "@playwright/test"; import { runnerApiToolsEnabled } from "../../server/src/services/native-runtime/runner-api-rollout.js"; import { spawn } from "node:child_process"; -import { createHash } from "node:crypto"; +import { createHash, randomUUID } from "node:crypto"; import { mkdir, readFile, readdir } from "node:fs/promises"; import path from "node:path"; import { isDeepStrictEqual } from "node:util"; @@ -26,6 +26,7 @@ import { hasPersistedSource, isSavedSourceCheckpoint } from "./everyday-interrup import { collectRunEvents } from "./run-observations.js"; import { gradeRemoteRestart, observeRemoteRestartBoundary, projectRemoteRestartEvents, projectRemoteRestartRun, waitForRemoteRestartBoundary, type RemoteRestartEvidence } from "./everyday-remote-restart.js"; import { controllerOutageForExecution, type ControllerOutageReceipt } from "./controller-outage.js"; +import { gradeRetainedControllerContinuation, retainedControllerRun, retainedControllerFollowupTransport, submitRetainedControllerComment, onePersistedRetainedControllerComment } from "./retained-controller-continuation.js"; import { setupAggregatorFixture } from "./aggregator-fixture.js"; import { gradeProviderChoice, gradeProviderOutcome, requireProviderAccessCard } from "./connection-routing-evidence.js"; import { setupConnectionReview } from "./connection-reviews.js"; @@ -44,6 +45,8 @@ import { storyLifecycleChecks, storyRepliesConsumed, storyHasAgentReply, + storyAgentReplyMatches, + retainedControllerReplyMatcher, storyHasPendingHumanInteraction, storyReviewContinuationTimeoutDetail, storyHasStrandedBlockedLeaf, @@ -59,6 +62,7 @@ import { type StoryCheck, type StoryIssue, type StoryRun, + type StoryReplyExpectation, } from "./everyday-observations.js"; type Row = Record; @@ -187,6 +191,8 @@ export async function runEverydayFlow(input: Input) { let providerAccessDecision: { id: string; connectionId: string } | undefined; let stoppedWorkspace: Record | undefined; let settledAgentReply: Row | undefined; + let postControllerApiFollowup = false; + let retainedFollowupCommentId: string | undefined; let reviewHandoffBoundary: { childId: string; assigneeAgentId: string | null | undefined; @@ -280,7 +286,7 @@ export async function runEverydayFlow(input: Input) { ), ]; } - async function reply(message: string, target: StoryIssue = parent!) { + async function reply(message: string, target: StoryIssue = parent!, transport: "browser" | "task_comment_api" = "browser") { const priorFailures = ev.checks.filter((c) => !c.passed); if (priorFailures.length) throw new Error( @@ -288,7 +294,9 @@ export async function runEverydayFlow(input: Input) { ); const before = await api.get(`/api/issues/${target.id}/comments`); lastSubmissionAt = Date.now(); - await submitTaskReply(page, message); + const submitted = transport === "task_comment_api" + ? await submitRetainedControllerComment(api, target.id, message, randomUUID()) + : (await submitTaskReply(page, message), undefined); const after = await pollUntil({ label: "one persisted user reply", deadlineAt: Date.now() + 30_000, @@ -303,11 +311,12 @@ export async function runEverydayFlow(input: Input) { ); check( `reply-${ev.timeline.length}-stored-once`, - added.length === 1, - "A composer submission creates exactly one user message.", + transport === "task_comment_api" ? onePersistedRetainedControllerComment({ added, commentId: submitted?.id ?? "", body: message }) : added.length === 1, + transport === "task_comment_api" ? "The production comment API creates exactly one matching user message." : "A composer submission creates exactly one user message.", ); submittedCommentIds.push(...added.map((c) => c.id)); - note("composer-message-persisted", { + if (submitted) retainedFollowupCommentId = submitted.id; + note(transport === "task_comment_api" ? "task-api-message-persisted" : "composer-message-persisted", { issueId: target.id, commentIds: added.map((c) => c.id), }); @@ -324,7 +333,7 @@ export async function runEverydayFlow(input: Input) { sameConnections: false, }; } - async function settled(expectedAgentReply?: string) { + async function settled(expectedAgentReply?: StoryReplyExpectation, readback: "browser" | "api" = "browser") { const settledState = await pollUntil({ label: `everyday ${caseId} settled`, deadlineAt: input.deadlineAt, @@ -389,16 +398,14 @@ export async function runEverydayFlow(input: Input) { if (expectedAgentReply) { const issue = settledState.issues.find((candidate) => candidate.id === parent?.id); const reply = issue?.comments?.find( - (comment) => - comment.authorAgentId === fixtures.agent.id && - String(comment.body ?? "").includes(expectedAgentReply), + (comment) => storyAgentReplyMatches(comment, fixtures.agent.id, expectedAgentReply), ); if (reply) { // Keep the exact snapshot that satisfied the readiness predicate. A // later refresh may return a newer projection and must not change the // evidence used by the assertion below. settledAgentReply = { ...reply }; - note("expected-agent-reply-visible-at-settlement", { + note(readback === "api" ? "expected-agent-reply-persisted-at-settlement" : "expected-agent-reply-visible-at-settlement", { issueId: issue?.id, commentId: reply.id, authorAgentId: reply.authorAgentId, @@ -407,14 +414,16 @@ export async function runEverydayFlow(input: Input) { }); } } - await openParent(); - await expect( - page.getByTestId("issue-detail-header").getByRole("button", { - name: "Change status (current: Done)", - exact: true, - }), - ).toBeVisible(); - note("all-tasks-done"); + if (readback === "browser") { + await openParent(); + await expect( + page.getByTestId("issue-detail-header").getByRole("button", { + name: "Change status (current: Done)", + exact: true, + }), + ).toBeVisible(); + } + note("all-tasks-done", { readback }); } async function downloadDelegated( mode: "base" | "separator" | "max-length", @@ -574,6 +583,7 @@ export async function runEverydayFlow(input: Input) { "everyday-delivery.ts", "everyday-observations.ts", "everyday-remote-restart.ts", + "retained-controller-continuation.ts", "controller-outage.ts", "controller-recovery-observer.ts", "upgrade-artifacts.ts", @@ -1440,13 +1450,39 @@ export async function runEverydayFlow(input: Input) { artifactGradeModeForPhase("reviewed-delivery"), "reviewed-delivery", ); - } else if (caseId === "recover-controller") + } else if (caseId === "recover-controller") { await download( parent!.id, artifactGradeModeForPhase("recovered-delivery"), "recovered-delivery", ); - else if (caseId === "stop-redirect") + if (execution.suite.definitionMetadata?.retainedControllerContinuation === true) { + const prior = ev.runs.filter(run => run.status === "succeeded" && run.agentId === fixtures.agent.id) + .sort((a, b) => Date.parse(b.finishedAt ?? "") - Date.parse(a.finishedAt ?? ""))[0]; + if (!prior) throw new Error("Retained continuation requires a completed source run"); + const before = retainedControllerRun(prior), priorIds = new Set(ev.runs.map(run => run.id)); + const outage = await input.restart(); + const followupTransport = retainedControllerFollowupTransport(execution); + postControllerApiFollowup = followupTransport === "task_comment_api"; + if (!postControllerApiFollowup) await openParent(); + note("retained-controller-followup-entry", { transport: followupTransport, + browserNavigationVerified: !postControllerApiFollowup }); + const requestedAt = new Date().toISOString(), marker = `CONTINUED_${nonce}`; + const expectedReply = retainedControllerReplyMatcher(marker); + await reply(`Run the existing delivered slugify.py on " Queue--Ready!! " with --separator _ and --max-length 7. Report its output after the marker ${marker}. Use the saved project; do not rebuild it.`, parent!, followupTransport); + await settled(expectedReply, postControllerApiFollowup ? "api" : "browser"); + const after = ev.runs.filter(run => !priorIds.has(run.id) && run.status === "succeeded" && run.agentId === fixtures.agent.id) + .sort((a, b) => Date.parse(b.finishedAt ?? "") - Date.parse(a.finishedAt ?? ""))[0]; + if (!after) throw new Error("Retained continuation produced no new successful run"); + const events = await collectRunEvents((afterSeq, limit) => api.get(`/api/heartbeat-runs/${after.id}/events?afterSeq=${afterSeq}&limit=${limit}`)); + const evidence = { before, after: retainedControllerRun(after), outage: outage || undefined, requestedAt, + followupTransport, followupCommentId: retainedFollowupCommentId, + completedTool: events.some(row => row.payload?.prpEvent?.eventType === "tool.execution.completed" && row.payload.prpEvent.payload?.status === "completed"), + usefulReply: storyAgentReplyMatches(settledAgentReply, fixtures.agent.id, expectedReply) }; + ev.checks.push(...gradeRetainedControllerContinuation(evidence)); + await input.evidence("retained-controller-continuation.json", evidence); + } + } else if (caseId === "stop-redirect") check( "new-direction-delivered", settledAgentReply?.authorAgentId === fixtures.agent.id && @@ -1480,7 +1516,7 @@ export async function runEverydayFlow(input: Input) { ev.issues.find((i) => i.id === parent!.id)!.documents = bodies; } } - await page.reload(); + if (!postControllerApiFollowup) await page.reload(); await refresh(); check( "submitted-replies-consumed", @@ -1575,25 +1611,30 @@ export async function runEverydayFlow(input: Input) { ), "No completion confirmation or unanswered interaction remains.", ); - if (providerChoice || nativeProviderCase) await openParent(); - else { - await waitForTaskIdentity(page, taskUrl(parent!), parent!.identifier ?? parent!.id); - await waitForTaskChatRendered(page); + if (!postControllerApiFollowup) { + if (providerChoice || nativeProviderCase) await openParent(); + else { + await waitForTaskIdentity(page, taskUrl(parent!), parent!.identifier ?? parent!.id); + await waitForTaskChatRendered(page); + } + const latestAgentComment = ev.issues + .find((i) => i.id === parent!.id) + ?.comments?.filter((c: Row) => c.authorAgentId) + .at(-1); + if (latestAgentComment && !providerChoice && !nativeProviderCase) { + const response = page.locator(`[id="comment-${latestAgentComment.id}"]`); + await expect(response).toBeVisible(); + await response.scrollIntoViewIfNeeded(); + } + await input.capture( + "final-state", + "Finished everyday workflow", + "final-state.png", + ); + } else { + note("final-state-api-readback", { browserNavigationVerified: false, renderedStatusVerified: false }); + await input.capture("final-state", "Browser state retained after API-verified native continuation; browser reconnection unqualified", "final-state.png"); } - const latestAgentComment = ev.issues - .find((i) => i.id === parent!.id) - ?.comments?.filter((c: Row) => c.authorAgentId) - .at(-1); - if (latestAgentComment && !providerChoice && !nativeProviderCase) { - const response = page.locator(`[id="comment-${latestAgentComment.id}"]`); - await expect(response).toBeVisible(); - await response.scrollIntoViewIfNeeded(); - } - await input.capture( - "final-state", - "Finished everyday workflow", - "final-state.png", - ); const failed = ev.checks.filter((c) => !c.passed); if (failed.length) throw new Error( diff --git a/tests/runner-e2e/everyday-observations.ts b/tests/runner-e2e/everyday-observations.ts index 40ab35e4a44..798b85e5899 100644 --- a/tests/runner-e2e/everyday-observations.ts +++ b/tests/runner-e2e/everyday-observations.ts @@ -344,17 +344,34 @@ export function storyRepliesConsumed( ); } +export type StoryReplyExpectation = string | ((body: string) => boolean); + +export function storyAgentReplyMatches( + comment: StoryComment | undefined, + agentId: string, + expected: StoryReplyExpectation, +): boolean { + if (!comment || comment.authorAgentId !== agentId) return false; + const body = String(comment.body ?? ""); + return typeof expected === "string" ? body.includes(expected) : expected(body); +} + +/** The requested output is one exact token after this continuation's marker. */ +export function retainedControllerReplyMatcher(marker: string): (body: string) => boolean { + const escaped = marker.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + const pattern = new RegExp(`(?:^|\\s)${escaped}\\s+queue_r(?=\\s|$)`); + return body => pattern.test(body); +} + /** A terminal task is only ready for response assertions after its agent reply is visible. */ export function storyHasAgentReply( issue: StoryIssue | undefined, agentId: string, - expectedText: string, + expectedText: StoryReplyExpectation, ): boolean { return ( issue?.comments?.some( - (comment) => - comment.authorAgentId === agentId && - String(comment.body ?? "").includes(expectedText), + (comment) => storyAgentReplyMatches(comment, agentId, expectedText), ) ?? false ); } diff --git a/tests/runner-e2e/everyday.test.ts b/tests/runner-e2e/everyday.test.ts index 88cbecddd72..c9880edaa23 100644 --- a/tests/runner-e2e/everyday.test.ts +++ b/tests/runner-e2e/everyday.test.ts @@ -9,6 +9,8 @@ import { storyLifecycleChecks, storyRepliesConsumed, storyHasAgentReply, + storyAgentReplyMatches, + retainedControllerReplyMatcher, storyHasPendingAgentReview, storyHasPendingAgentWake, storyHasPendingAgentWork, @@ -385,6 +387,49 @@ describe("lifecycle oracle calibrated failures", () => { }); describe("reply completion boundary", () => { + it("accepts an explicit reply matcher for continuation output on a new line", () => { + const issue = { id: "parent", companyId: "company", title: "Continue", status: "done", + comments: [{ authorAgentId: "lead", body: "CONTINUED_nonce\nqueue_r" }] }; + const expected = (body: string) => body === "CONTINUED_nonce\nqueue_r"; + expect(storyHasAgentReply(issue, "lead", expected)).toBe(true); + }); + + it.each([" ", "\n", "\r\n", "\t", " \n"])("accepts continuation output separated by %j", separator => { + const expected = retainedControllerReplyMatcher("CONTINUED_nonce.1"); + const comment = { authorAgentId: "lead", body: `CONTINUED_nonce.1${separator}queue_r` }; + expect(storyHasAgentReply({ id: "task", companyId: "company", title: "Continue", status: "done", comments: [comment] }, "lead", expected)).toBe(true); + expect(storyAgentReplyMatches(comment, "lead", expected)).toBe(true); + }); + + it.each(["\nI used the saved project.", " I used the saved project.", "\r\nqueue_ready"])("accepts explanation after the exact continuation token %j", suffix => { + const expected = retainedControllerReplyMatcher("CONTINUED_nonce.1"); + const comment = { authorAgentId: "lead", body: `CONTINUED_nonce.1\nqueue_r${suffix}` }; + expect(storyHasAgentReply({ id: "task", companyId: "company", title: "Continue", status: "done", comments: [comment] }, "lead", expected)).toBe(true); + expect(storyAgentReplyMatches(comment, "lead", expected)).toBe(true); + }); + + it.each([ + "CONTINUED_nonce.1 queue_ready", "CONTINUED_nonce.1 queue_r_extra", "CONTINUED_nonce.1 wrong", + "CONTINUED_other queue_r", "CONTINUED_nonceX1 queue_r", "queue_r", "CONTINUED_nonce.1queue_r", + "OTHER_CONTINUED_nonce.1 queue_r", "CONTINUED_nonce.1 queue_r-extra", "CONTINUED_nonce.1 queue_r.", + ])("rejects an incorrect continuation reply %j", body => { + const expected = retainedControllerReplyMatcher("CONTINUED_nonce.1"); + const comment = { authorAgentId: "lead", body }; + expect(storyHasAgentReply({ id: "task", companyId: "company", title: "Continue", status: "done", comments: [comment] }, "lead", expected)).toBe(false); + expect(storyAgentReplyMatches(comment, "lead", expected)).toBe(false); + }); + + it("preserves literal string expectations and continuation author checks", () => { + const expected = retainedControllerReplyMatcher("CONTINUED_nonce"); + const comment = { authorAgentId: "other", body: "CONTINUED_nonce\nqueue_r" }; + expect(storyAgentReplyMatches(comment, "lead", expected)).toBe(false); + const issue = { id: "task", companyId: "company", title: "Continue", status: "done", comments: [comment] }; + expect(storyHasAgentReply(issue, "lead", expected)).toBe(false); + expect(storyAgentReplyMatches({ ...comment, authorAgentId: "lead" }, "lead", "CONTINUED_nonce queue_r")).toBe(false); + expect(storyAgentReplyMatches({ authorAgentId: "lead", body: "a.b" }, "lead", "a.b")).toBe(true); + expect(storyAgentReplyMatches({ authorAgentId: "lead", body: "axb" }, "lead", "a.b")).toBe(false); + }); + const run: StoryRun = { id: "old", companyId: "company", diff --git a/tests/runner-e2e/external-controller-database.test.ts b/tests/runner-e2e/external-controller-database.test.ts new file mode 100644 index 00000000000..5570b38d6fb --- /dev/null +++ b/tests/runner-e2e/external-controller-database.test.ts @@ -0,0 +1,52 @@ +import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { expect, it } from "vitest"; +import { createDb } from "../../packages/db/src/index.js"; +import { createExternalControllerDatabase } from "./external-controller-database.js"; +import { assertEmbeddedDatabaseIsolation } from "./instance-isolation.js"; +import { createRunnerE2ETemporaryRoot } from "./server-config.js"; +import { buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment } from "./harness-env.js"; + +it("keeps the database administrator out of both child environments", () => { + const source = { PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL: "postgres://test-secret@localhost/postgres" }; + expect(buildRunnerE2EProcessEnvironment(source, []).PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL).toBeUndefined(); + expect(buildPaperclipServerEnvironment(source).PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL).toBeUndefined(); +}); + +it("rejects a nonlocal administrator before creating a fixture", async () => { + await expect(createExternalControllerDatabase({ adminUrl: "postgres://example.invalid/postgres", + temporaryRoot: "/unused", configPath: "/unused/config.json", serverPort: 3100 })) + .rejects.toThrow("explicit loopback test server"); +}); + +it.skipIf(!process.env.PAPERCLIP_TEST_DATABASE_URL)("owns one empty database, rejects a changed binding, and deletes only that database", async () => { + const root = await createRunnerE2ETemporaryRoot(), configPath = path.join(root, "instance", "config.json"); + const admin = createDb(process.env.PAPERCLIP_TEST_DATABASE_URL!, { maxConnections: 1 }); + let fixture: Awaited> | undefined; + let client: ReturnType | undefined; + try { + fixture = await createExternalControllerDatabase({ adminUrl: process.env.PAPERCLIP_TEST_DATABASE_URL!, + temporaryRoot: root, configPath, serverPort: 3100 }); + const config = JSON.parse(await readFile(configPath, "utf8")); + const name = new URL(config.database.connectionString).pathname.slice(1); + expect(name).toMatch(/^runner_e2e_[a-f0-9]{32}$/); + client = createDb(config.database.connectionString); + expect(await client.$client`select tablename from pg_tables where schemaname = 'public'`).toEqual([]); + await mkdir(path.dirname(config.secrets.localEncrypted.keyFilePath), { recursive: true }); + await writeFile(config.secrets.localEncrypted.keyFilePath, "fixture-key"); + await expect(assertEmbeddedDatabaseIsolation(configPath, root)).resolves.toBeUndefined(); + config.database.connectionString = process.env.PAPERCLIP_TEST_DATABASE_URL; + await writeFile(configPath, JSON.stringify(config)); + await expect(assertEmbeddedDatabaseIsolation(configPath, root)).rejects.toThrow("fixture-owned database"); + await client.$client.end(); client = undefined; + await fixture.cleanup(); + await fixture.cleanup(); + expect(await admin.$client`select datname from pg_database where datname=${name}`).toEqual([]); + expect(await admin.$client`select current_database() as name`).toHaveLength(1); + } finally { + await client?.$client.end({ timeout: 1 }); + await fixture?.cleanup(); + await admin.$client.end({ timeout: 1 }); + await rm(root, { recursive: true, force: true }); + } +}); diff --git a/tests/runner-e2e/external-controller-database.ts b/tests/runner-e2e/external-controller-database.ts new file mode 100644 index 00000000000..5a538bd1f80 --- /dev/null +++ b/tests/runner-e2e/external-controller-database.ts @@ -0,0 +1,47 @@ +/** Launcher-owned disposable database. The admin credential never enters the + * server/browser environment. Unknown remote cleanup keeps this database alive. */ +import { randomUUID } from "node:crypto"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { createDb } from "../../packages/db/src/index.js"; +import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js"; +import { recordControllerDatabaseOwnership } from "./instance-isolation.js"; +import { prepareRunnerE2EServerConfig } from "./server-config.js"; + +export async function createExternalControllerDatabase(input: { + adminUrl: string; temporaryRoot: string; configPath: string; serverPort: number; +}) { + const url = new URL(input.adminUrl); + if (!["postgres:", "postgresql:"].includes(url.protocol) + || !["127.0.0.1", "localhost", "[::1]"].includes(url.hostname)) { + throw new Error("E2E Postgres administrator must use an explicit loopback test server"); + } + await (await prepareRunnerE2EServerConfig(input))?.close(); + const config = paperclipConfigSchema.parse(JSON.parse(await readFile(input.configPath, "utf8"))); + if (config.database.mode !== "embedded-postgres") throw new Error("E2E database fixture already configured"); + const databaseName = `runner_e2e_${randomUUID().replaceAll("-", "")}`; + const admin = createDb(input.adminUrl, { maxConnections: 1 }); + let created = false; + const cleanup = async () => { + if (!created) return; + const connection = createDb(input.adminUrl, { maxConnections: 1 }); + try { + // No FORCE: a remaining controller keeps its evidence and fails cleanup. + await connection.$client.unsafe(`drop database "${databaseName}"`); + created = false; + } finally { await connection.$client.end({ timeout: 1 }); } + }; + try { + await admin.$client.unsafe(`create database "${databaseName}"`); + created = true; + url.pathname = `/${databaseName}`; + config.database = { ...config.database, mode: "postgres", connectionString: url.toString() }; + await mkdir(config.database.embeddedPostgresDataDir, { recursive: true, mode: 0o700 }); + await recordControllerDatabaseOwnership({ ...input, dataDirectory: config.database.embeddedPostgresDataDir, + connectionString: url.toString(), externalDatabaseName: databaseName }); + await writeFile(input.configPath, `${JSON.stringify(config, null, 2)}\n`, { mode: 0o600 }); + return { cleanup }; + } catch (error) { + await cleanup(); + throw error; + } finally { await admin.$client.end({ timeout: 1 }); } +} diff --git a/tests/runner-e2e/harness-env.ts b/tests/runner-e2e/harness-env.ts index 0f9de2d0c3f..b33e3f2125b 100644 --- a/tests/runner-e2e/harness-env.ts +++ b/tests/runner-e2e/harness-env.ts @@ -136,6 +136,7 @@ export function buildRunnerE2EProcessEnvironment( const installed = usesInstalledCli(source); if (installed) assertInstalledCliSelection(source, executions); const result = { ...source }; + delete result.PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL; // The supervisor's read-only recovery observer must use the same disposable // instance key as the serving child, never a launching shell's vault key. delete result.PAPERCLIP_SECRETS_MASTER_KEY; @@ -209,6 +210,7 @@ export function buildPaperclipServerEnvironment( overrides: NodeJS.ProcessEnv = {}, ): NodeJS.ProcessEnv { const result = { ...source }; + delete result.PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL; for (const key of Object.keys(result)) { if (PROVIDER_SECRET_KEY.test(key)) delete result[key]; } diff --git a/tests/runner-e2e/installed-cli.test.ts b/tests/runner-e2e/installed-cli.test.ts index 7b917353ba3..7f3074e0fd8 100644 --- a/tests/runner-e2e/installed-cli.test.ts +++ b/tests/runner-e2e/installed-cli.test.ts @@ -32,7 +32,7 @@ it("admits all26 current Pi cases through pinned installed public CLI/server wit }); it("admits existing Codex onboarding, useful-file revision and restart journeys without broadening other cells", async () => { const { env, server } = await fixture(); - for (const id of ["first-task.runner-codex.local.task-card-accept", "everyday-workflows.runner-codex.local.build-revise", "agent-chat.runner-codex.local.continuity-restart", "everyday-workflows.runner-codex.daytona.recover-controller"]) { + for (const id of ["first-task.runner-codex.local.task-card-accept", "everyday-workflows.runner-codex.local.build-revise", "agent-chat.runner-codex.local.continuity-restart", "everyday-workflows.runner-codex.daytona.recover-controller", "daytona-restart-continuity.runner-codex.daytona.recover-controller"]) { const cell = runnerMatrix.find(e => e.id === id)!; expect(cell).toBeDefined(); expect(await verifyInstalledCli(env, [cell])).toMatchObject({ serverRoot: server, defaultRuntimeResolution: true, qualificationOverride: false }); diff --git a/tests/runner-e2e/installed-cli.ts b/tests/runner-e2e/installed-cli.ts index 0e5fe3c55c5..820577869a5 100644 --- a/tests/runner-e2e/installed-cli.ts +++ b/tests/runner-e2e/installed-cli.ts @@ -17,6 +17,7 @@ const installedCodexCells = new Set([ "everyday-workflows.runner-codex.local.build-revise", "agent-chat.runner-codex.local.continuity-restart", "everyday-workflows.runner-codex.daytona.recover-controller", + "daytona-restart-continuity.runner-codex.daytona.recover-controller", ]); const installedCopilotCells = new Set([ "copilot-protection.runner-acpx-copilot.daytona.attached-async-settlement", diff --git a/tests/runner-e2e/instance-isolation.ts b/tests/runner-e2e/instance-isolation.ts index 9bd98b60a3e..1c2117dbdcd 100644 --- a/tests/runner-e2e/instance-isolation.ts +++ b/tests/runner-e2e/instance-isolation.ts @@ -6,6 +6,7 @@ const CONTROLLER_DATABASE_OWNER_FILE = ".runner-e2e-controller-owner.json"; /** Written only after the supervisor has started this fixture's Postgres. */ export async function recordControllerDatabaseOwnership(input: { configPath: string; temporaryRoot: string; dataDirectory: string; connectionString: string; + externalDatabaseName?: string; }) { const temporaryRoot = await realpath(input.temporaryRoot); const dataDirectory = await realpath(input.dataDirectory); @@ -13,7 +14,8 @@ export async function recordControllerDatabaseOwnership(input: { throw new Error("Controller database path escaped its fixture-owned root"); } await writeFile(path.join(dataDirectory, CONTROLLER_DATABASE_OWNER_FILE), JSON.stringify({ - version: 1, configPath: await realpath(input.configPath), temporaryRoot, + version: input.externalDatabaseName ? 2 : 1, externalDatabaseName: input.externalDatabaseName, + configPath: await realpath(input.configPath), temporaryRoot, dataDirectory, connectionString: input.connectionString, }), { mode: 0o600 }); } @@ -82,12 +84,16 @@ export async function assertEmbeddedDatabaseIsolation( if (!directory.startsWith(`${root}${path.sep}`)) throw new Error("Database escaped fixture"); const owner = JSON.parse(await readFile(path.join(directory, CONTROLLER_DATABASE_OWNER_FILE), "utf8")); const port = config.database.embeddedPostgresPort; - const expectedConnection = `postgres://paperclip:paperclip@127.0.0.1:${port}/paperclip`; - if (!Number.isInteger(port) || Number(port) < 1 || Number(port) > 65535 - || config.database.connectionString !== expectedConnection - || owner.version !== 1 || owner.temporaryRoot !== root || owner.dataDirectory !== directory - || owner.configPath !== await realpath(configPath) || owner.connectionString !== expectedConnection - || !(await stat(path.join(directory, "PG_VERSION"))).isFile()) { + const connection = new URL(config.database.connectionString!); + const external = owner.version === 2 && /^runner_e2e_[a-f0-9]{32}$/.test(owner.externalDatabaseName) + && ["postgres:", "postgresql:"].includes(connection.protocol) + && ["127.0.0.1", "localhost", "[::1]"].includes(connection.hostname) + && connection.pathname === `/${owner.externalDatabaseName}`; + const embedded = owner.version === 1 && Number.isInteger(port) && Number(port) > 0 && Number(port) <= 65535 + && config.database.connectionString === `postgres://paperclip:paperclip@127.0.0.1:${port}/paperclip` + && (await stat(path.join(directory, "PG_VERSION"))).isFile(); + if ((!external && !embedded) || owner.temporaryRoot !== root || owner.dataDirectory !== directory + || owner.configPath !== await realpath(configPath) || owner.connectionString !== config.database.connectionString) { throw new Error("Supervisor database ownership mismatch"); } } catch (cause) { diff --git a/tests/runner-e2e/launch.ts b/tests/runner-e2e/launch.ts index 215cbd3a863..3a278d79dbe 100644 --- a/tests/runner-e2e/launch.ts +++ b/tests/runner-e2e/launch.ts @@ -10,6 +10,7 @@ import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js"; import { assertInstalledCliSelection, assertInstalledStartupOnly, verifyInstalledCli } from "./installed-cli.js"; import { createProcessTreeOwner, stopOwnedProcessTree } from "./process-tree-owner.js"; import { createRunnerE2ETemporaryRoot } from "./server-config.js"; +import { createExternalControllerDatabase } from "./external-controller-database.js"; import { createHash, randomBytes } from "node:crypto"; import { prepareCodexCiSandbox, requiresCodexCiSandbox } from "./codex-ci-sandbox.js"; import { spawn } from "node:child_process"; @@ -508,6 +509,7 @@ async function runAttempt(input: { let attemptSecrets: string[] = []; let processCleanupFailed = false; let startupReceiptPath: string | undefined; + let externalDatabase: Awaited> | undefined; let gateway: Awaited> | undefined; const rawCleanupResults: Array<{ cleanup: string; synthetic: boolean; status: string }> = []; try { @@ -521,6 +523,11 @@ async function runAttempt(input: { "config.json", ); const port = await reserveRunnerE2EServerPort(); + if (process.env.PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL) { + externalDatabase = await createExternalControllerDatabase({ + adminUrl: process.env.PAPERCLIP_RUNNER_E2E_POSTGRES_ADMIN_URL, temporaryRoot, configPath, serverPort: port, + }); + } await Promise.all([ mkdir(paperclipHome, { recursive: true }), mkdir(workspace, { recursive: true }), @@ -891,7 +898,11 @@ async function runAttempt(input: { let cleanupError: unknown; const resourceAdmissionStarted = await access(path.join(privateDir, "resource-admission-started")) .then(() => true).catch((error: NodeJS.ErrnoException) => error.code !== "ENOENT"); - if (mustPreserveRecoveryState({ processCleanupFailed, resourceAdmissionStarted, results: rawCleanupResults })) { + const preserveRecovery = mustPreserveRecoveryState({ processCleanupFailed, resourceAdmissionStarted, results: rawCleanupResults }); + if (!preserveRecovery) { + try { await externalDatabase?.cleanup(); } catch (error) { cleanupError = error; } + } + if (preserveRecovery || cleanupError) { // Remote allocation cleanup is journaled in this instance's database. // Deleting it after the controller exits would strand uncertain creates. await chmod(temporaryRoot, 0o700); diff --git a/tests/runner-e2e/retained-controller-continuation.test.ts b/tests/runner-e2e/retained-controller-continuation.test.ts new file mode 100644 index 00000000000..a9636dfb06e --- /dev/null +++ b/tests/runner-e2e/retained-controller-continuation.test.ts @@ -0,0 +1,65 @@ +import { describe, expect, it, vi } from "vitest"; +import { gradeRetainedControllerContinuation, retainedControllerRun, retainedControllerFollowupTransport, submitRetainedControllerComment, onePersistedRetainedControllerComment } from "./retained-controller-continuation.js"; + +import { runnerMatrix } from "./catalog.js"; + +function fixture() { + const before = retainedControllerRun({ id: "a", companyId: "company", agentId: "agent", nativeIssueId: "task", + runtimeMode: "native", status: "succeeded", nativeSessionId: "private-native-session", sessionIdAfter: "private-provider-session", + finishedAt: "2026-10-30T00:00:00Z" }); + return { before, after: { ...before, runId: "b", finishedAt: "2026-10-30T00:01:00Z" }, + outage: { schema: "paperclip.e2e.controller-outage/v1" as const, requestId: "retained", signal: "SIGKILL" as const, + holdMs: 1000, startedAt: "2026-10-30T00:00:01Z", previousPid: 1, replacementPid: 2, + stoppedAfterMs: 0, replacementStartedAfterMs: 1000, readyAfterMs: 2000, cacheDiscarded: true }, + requestedAt: "2026-10-30T00:00:04Z", completedTool: true, usefulReply: true }; +} + +describe("retained controller continuation evidence", () => { + it("accepts a new useful turn after a completed source and cache retirement", () => { + expect(gradeRetainedControllerContinuation(fixture())[0]?.passed).toBe(true); + expect(JSON.stringify(fixture())).not.toContain("private-provider-session"); + }); + it.each(["provider", "native", "same-run", "before-completion", "no-cache-loss", "no-tool", "no-answer"])("rejects %s evidence", fault => { + const input = fixture(); + if (fault === "provider") input.after.providerSessionSha256 = "different"; + if (fault === "native") input.after.nativeSessionSha256 = null; + if (fault === "same-run") input.after.runId = input.before.runId; + if (fault === "before-completion") input.before.finishedAt = "2026-10-30T00:00:10Z"; + if (fault === "no-cache-loss") input.outage.cacheDiscarded = false; + if (fault === "no-tool") input.completedTool = false; + if (fault === "no-answer") input.usefulReply = false; + expect(gradeRetainedControllerContinuation(input)[0]?.passed).toBe(false); + }); +}); + +describe("post-controller API follow-up selection", () => { + const execution = runnerMatrix.find(cell => cell.id === "daytona-restart-continuity.runner-codex.daytona.recover-controller")!; + it("selects only the explicit dedicated definition and defaults to the browser", () => { + expect(retainedControllerFollowupTransport(execution)).toBe("task_comment_api"); + for (const changed of [ + { ...execution, suite: { ...execution.suite, id: "everyday-workflows" } }, + { ...execution, task: { ...execution.task, id: "another-case" } }, + { ...execution, suite: { ...execution.suite, definitionMetadata: { retainedControllerContinuation: true } } }, + { ...execution, suite: { ...execution.suite, definitionMetadata: { retainedControllerFollowupTransport: "task_comment_api" } } }, + ]) expect(retainedControllerFollowupTransport(changed)).toBe("browser"); + }); + it("posts one ordinary authenticated comment request with a stable client request ID", async () => { + const post = vi.fn(async () => ({ id: "saved-user-comment" })); + const api = { post } as unknown as Parameters[0]; + await expect(submitRetainedControllerComment(api, "task-id", "Exact original follow-up.", "request-id")) + .resolves.toEqual({ id: "saved-user-comment" }); + expect(post).toHaveBeenCalledExactlyOnceWith("/api/issues/task-id/comments", { + body: "Exact original follow-up.", reopen: true, clientRequestId: "request-id", + }); + }); + it("requires exactly the matching persisted user comment", () => { + const comment = { id: "comment", body: "Exact follow-up", authorType: "user" }; + const proof = { added: [comment], commentId: comment.id, body: comment.body }; + expect(onePersistedRetainedControllerComment(proof)).toBe(true); + expect(onePersistedRetainedControllerComment({ ...proof, commentId: "" })).toBe(false); + for (const added of [[], [comment, { ...comment, id: "duplicate" }], [{ ...comment, id: "other" }], + [{ ...comment, body: "changed" }], [{ ...comment, authorType: "system" }], [{ ...comment, authorAgentId: "agent" }]]) { + expect(onePersistedRetainedControllerComment({ ...proof, added })).toBe(false); + } + }); +}); diff --git a/tests/runner-e2e/retained-controller-continuation.ts b/tests/runner-e2e/retained-controller-continuation.ts new file mode 100644 index 00000000000..d0f5fa4bb31 --- /dev/null +++ b/tests/runner-e2e/retained-controller-continuation.ts @@ -0,0 +1,59 @@ +import type { RunnerApi } from "./api.js"; +import type { MatrixExecution } from "./types.js"; +import { createHash } from "node:crypto"; +import type { StoryCheck, StoryRun } from "./everyday-observations.js"; +import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js"; + +/** Only this explicit definition changes the post-replacement user entry point. */ +export function retainedControllerFollowupTransport(execution: Pick): "browser" | "task_comment_api" { + return execution.suite.id === "daytona-restart-continuity" && execution.task.id === "recover-controller" + && execution.suite.definitionMetadata?.retainedControllerContinuation === true + && execution.suite.definitionMetadata.retainedControllerFollowupTransport === "task_comment_api" + ? "task_comment_api" : "browser"; +} + +export async function submitRetainedControllerComment(api: Pick, issueId: string, body: string, clientRequestId: string) { + return api.post<{ id: string }>(`/api/issues/${issueId}/comments`, { body, reopen: true, clientRequestId }); +} + +export function onePersistedRetainedControllerComment(input: { + added: Array<{ id?: unknown; body?: unknown; authorAgentId?: unknown; authorType?: unknown }>; + commentId: string; body: string; +}) { + return typeof input.commentId === "string" && input.commentId.length > 0 + && input.added.length === 1 && input.added[0]?.id === input.commentId + && input.added[0].body === input.body && !input.added[0].authorAgentId && input.added[0].authorType === "user"; +} + +const hash = (value: string | null | undefined) => value ? createHash("sha256").update(value).digest("hex") : null; + +/** Public evidence keeps provider/session identifiers hashed. */ +export function retainedControllerRun(run: StoryRun) { + return { runId: run.id, companyId: run.companyId, agentId: run.agentId, issueId: run.nativeIssueId, + status: run.status, runtimeMode: run.runtimeMode, finishedAt: run.finishedAt, + nativeSessionSha256: hash(run.nativeSessionId), providerSessionSha256: hash(run.sessionIdAfter) }; +} + +export function gradeRetainedControllerContinuation(input: { + before: ReturnType; + after: ReturnType; + outage: ControllerOutageReceipt | undefined; + requestedAt: string; + completedTool: boolean; + usefulReply: boolean; +}): StoryCheck[] { + const { before, after, outage } = input; + const passed = validControllerOutageReceipt(outage) + && before.status === "succeeded" && after.status === "succeeded" + && before.runtimeMode === "native" && after.runtimeMode === "native" + && before.runId !== after.runId && before.companyId === after.companyId + && before.agentId === after.agentId && before.issueId === after.issueId + && Boolean(before.nativeSessionSha256) && before.nativeSessionSha256 === after.nativeSessionSha256 + && Boolean(before.providerSessionSha256) && before.providerSessionSha256 === after.providerSessionSha256 + && Date.parse(before.finishedAt ?? "") <= Date.parse(outage?.startedAt ?? "") + && Date.parse(input.requestedAt) >= Date.parse(outage?.startedAt ?? "") + (outage?.readyAfterMs ?? Infinity) + && Date.parse(after.finishedAt ?? "") >= Date.parse(input.requestedAt) + && input.completedTool && input.usefulReply; + return [{ id: "retained-controller-continuation", passed, + detail: "After completion and another empty-cache replacement, a new native turn must use the same retained provider session and execute the saved project for a useful reply." }]; +} diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 03739da45de..988ce11cb53 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1253,12 +1253,15 @@ for (const execution of executions) { const story = await runEverydayFlow({ page, api, fixtures, execution, nonce, workspacePath, privateDir, deadlineAt: startedAtMs + deadlineMs, - restartCount: upgradeManifest ? upgradeManifest.sequence.length - 1 : 1, + restartCount: upgradeManifest ? upgradeManifest.sequence.length - + (execution.suite.definitionMetadata?.retainedControllerContinuation === true ? 2 : 1) : 1, restart: async (recoveryBinding, controllerDowntimeMs) => { let receipt: ControllerOutageReceipt | undefined; restartIndex += 1; await restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}-${restartIndex}`, deadlineAt: startedAtMs + deadlineMs, - discardControllerCache: controllerOutage !== undefined, controllerOutage, controllerDowntimeMs, + discardControllerCache: controllerOutage !== undefined, + controllerOutage: controllerOutage && !recoveryBinding ? { ...controllerOutage, holdMs: 1000 } : controllerOutage, + controllerDowntimeMs, ...(controllerOutage ? { recoveryBinding } : {}), onControllerOutage: async value => { receipt = value;