PaperClipAI/doc/testing-native-remote-continuity.md

Native remote continuity regression

The opt-in case in server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts runs the real executor, PostgreSQL, TLS ingress, Rust runner, remote file transfer, and backup validation against a deterministic fake Codex process in Linux. It checks the same runner process identity and one provider process across three warm turns, provider history reuse, and verified suspension at idle close. Separate cases kill the actual controller process and recover the next turn. They reject a still-live controller and a replacement sandbox before reconnecting. Legacy checkpoints and changed process configuration require verified suspension and replacement, while preserving the conversation and starting each turn once. Recovery can restart a provider to refresh its tool contract; process reuse is required for ordinary healthy follow-ups. Each recovery case includes another healthy turn after the recovered turn and asserts that it does not start another runner or provider process. The same-server case also starts preparation lease renewal during recovery and holds instruction collection open past a renewal tick after coordinator release. The executor must acquire renewal before preparation releases its reference, then stop renewal before releasing the coordinator. Completion must not detach the retained runner or strand the next turn. Buffered transport messages and close notifications can arrive synchronously on subscription. The controller must register the connection before subscribing so completed message processing can drain and the suspended session can retire. It does not qualify a live model or a sandbox provider API.

Per-turn plugin continuation uses a stopped checkpoint on a replacement lease. The executor tests cover Claude and Codex, including reopening blocked work. The lease-authority integration test uses PostgreSQL to check that checkpoint discovery cannot launch work or bypass a termination receipt. A stale negative availability flag requires a fresh provider status before the checkpoint can be read for continuation.

The executor can restore one quarantined journal in the same session scope. It requires a terminal prior run, completed commands, a terminal event, and an identity-matched suspended plugin checkpoint with no pending delivery or cleanup. Multiple candidates, contradictory direct state, and unfinished work still fail closed. Recovery preserves the journal and the provider's existing state owner. Do not delete a quarantine or change database ownership to force continuation.

Use a disposable Linux container with Node.js, sh, tar, base64, ps, and setsid. Build paperclip-runnerd and fake-codex-app-server from the checkout's packages/paperclip-runner/runner/Cargo.toml for that container's architecture. Keep the Linux runner binary on the host as well: the product hashes and stages that artifact. Put the fake provider at /opt/continuity/fake-codex-app-server inside the container. Do not put credentials in this container.

Start the container with --init and publish container port 43127 on localhost. Read its assigned port with docker port CONTAINER 43127; an automatic host port can change after a container restart. The test opens a local TLS tunnel to this port. Create a test-only certificate and key:

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout /tmp/continuity-test.key -out /tmp/continuity-test.crt \
  -days 1 -subj '/CN=localhost' \
  -addext 'subjectAltName=IP:127.0.0.1,DNS:localhost'

Create an executable wrapper on the host at /tmp/continuity-fake-codex:

#!/bin/sh
if [ "$1" = '--version' ]; then
  echo 'codex-cli 0.999.0'
  exit 0
fi
exec /opt/continuity/fake-codex-app-server \
  --state-file-in-codex-home --durable-turn-ids --record-process-start \
  --call-log "$PWD/provider-calls.log" "$@"

Run the test with the container name, current host port, and Linux artifact path:

NODE_EXTRA_CA_CERTS=/tmp/continuity-test.crt \
PAPERCLIP_TEST_RUNNER_TLS_KEY=/tmp/continuity-test.key \
PAPERCLIP_TEST_RUNNER_CONTAINER=CONTAINER \
PAPERCLIP_TEST_RUNNER_PORT=HOST_PORT \
PAPERCLIP_TEST_LINUX_RUNNER_BINARY=/path/to/linux/paperclip-runnerd \
PAPERCLIP_TEST_FAKE_CODEX_WRAPPER=/tmp/continuity-fake-codex \
pnpm exec vitest run \
  server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts \
  -t 'preserves remote warm continuity' --maxWorkers=1

The caller owns the disposable container. Remove it after the run, including after failure; failed cases can retain remote processes and files for inspection. The test cleans up its database and host session directories.

Replacing the main server

Remote native runners admitted with durable controller persistence tolerate up to five minutes without the main server. The sandbox keeps its process, provider turn and event outbox. Server-dependent tools wait for reconnection; work already accepted into the queue resumes through normal admission. A single offline web server cannot accept new HTTP requests during that interval.

Cloud operators can read GET /api/instance/task-drain?upgradeReadiness=1 with existing instance-control authority. The version 1 upgradeReadiness report counts current remote executions with a bound current-format encrypted header and a live owner lease. The checkpoint owner authenticates/decrypts that header and delegates version and identity inspection to Runner; the identity must match the run, session and runner. Ordinary sandbox runs bind their workspace identity; plugin tasks bind their active dispatched environment lease and need no workspace-sync descriptor. Legacy/local executions, missing identities/checkpoints, foreign/expired owners, in-flight admission, and an incomplete census prevent a positive report. It returns aggregate counts only. The request does not stop admission, detach a runner, or authorize new work. Ordinary drain reads do not perform this SQL. The explicit probe uses a read-only transaction, a two-second statement timeout, and at most 1,001 run rows. Header reads are capped at 256 KiB each and 8 MiB total; oversized, malformed, legacy-format or local evidence falls back to the ordinary drain. This bounded header probe does not validate every journal record or prove a live sandbox. This is an observation; the existing shutdown and startup owners still perform the actual handoff and enforce Stop and fencing. A positive report alone does not prove hosted upgrade continuity.

Retain Postgres and the instance secret encryption key across deployments. The private native_runner_checkpoints journal holds encrypted authentication state, commands, and acknowledged event cursors. These records are company/run scoped and are not part of public run projections. The replacement reconstructs its local controller cache and authenticates the recorded running artifact, even when its own installed runner binary differs. It does not stage a new binary over the running process.

The runner package owns checkpoint encoding, validation, restoration, and command classification. Its persistence callback exposes opaque records and explicit ownership metadata. The server owns encryption and transactional lease checks; heartbeat discovers eligible runs through the native runtime. Tool recovery uses the current tool authority, including receipt decoding and read policy. Transport adapters retain ownership of protocol response envelopes.

Checkpoint v2 stores authority, command records, a small cursor, and recovery control events. It excludes ordinary item.delta output. usage.reported updates are coalesced: only the latest one is included when a control change triggers a checkpoint. Priority-zero events always retain their durability barrier. The writer receives changed records, exact removed keys, and an initial replacement flag and applies them under the owner lock. The format remains encrypted and company/run bound. The decoder reads v1 as well as v2 checkpoints.

There is no event batching timer. Streaming text/usage does not call the external checkpoint writer. Authentication, commands/results, semantic inputs, lifecycle and other control events still commit before their acknowledgement or effect is released. A failed control commit closes the connection, leaving the unacknowledged input in the runner outbox. Remote controller caches also skip per-chunk file writes; local-only controllers retain their full file journal. Existing application event/log callbacks remain; this does not remove accounting or business writes. Unknown event kinds retain the conservative persist-before-ack behavior.

Externally persisted controllers use Postgres as their durable store. Their local file becomes a marker that requires restoration from the external checkpoint before restart. Ordinary control changes do not rewrite a second local journal. The sandbox runner keeps its own journal.

Checkpoint transactions share a separate pool capped at two connections per server database client. They queue without occupying the application pool or the existing lease-renewal pool. This can delay a control acknowledgement during a burst; it does not release that acknowledgement before its commit. The cost is up to two additional active database connections per instance, not per agent. There is no new SQL per text chunk or while idle, and the number of control transactions is unchanged. SQL cost collection must include the checkpoint pool.

A sparse checkpoint is marked checkpointEventPolicy: "control-v1". Its peer must advertise recoveryCursorVersion: 1. During reconnection the controller echoes the runner's durable resume.ackedSourceSeq as recoveryAckedSourceSeq in the signed authentication challenge. The runner verifies that exact value against its state; the controller uses it only after client proof and current-owner admission pass. The controller can then advance over already-acknowledged streaming output absent from Postgres. It still requires contiguous new events and validates retained replays. Unacknowledged output and control events replay from the runner's existing disk outbox. No second remote journal or new polling loop is introduced.

The live event window keeps the latest identity, turn, result and usage anchors, as well as unsettled semantic inputs, through long output streams. The 4,096-event and 500-command limits remain. This checkpoint is intentionally not a complete transcript. With local-only logs, an abrupt crash can lose a display projection already acknowledged by the controller. The opt-in s3_segments log backend checkpoints its canonical native export cursor in S3 and replays missing records from the existing validated heartbeat_run_events source on takeover. This adds no payload writes or per-record checkpoint transactions. Generic process output and optional raw provider sidecars have separate durability boundaries. See run evidence storage. The same provider turn continues; recovery never reruns provider work to reconstruct display text. Tool outcomes and terminal state retain their durable barriers.

Rollback must use a controller that understands sparse checkpoints and the signed recovery cursor. A v2 decoder alone is insufficient. Runners admitted before this release do not acquire the new protocol capability or grace period retroactively.

Completed checkpoints are pruned opportunistically at server startup using the native harness backup enabled, dry-run, and retention-days settings (seven days by default). This is a minimum retention age, not a deletion deadline: a record that becomes eligible after startup remains until the next server restart. The shared delivery coordinator drains bounded startup pages, retries failed work, and preserves pending startup work across admission drain. It becomes quiet after completing the pass, including when protected history remains. The existing hourly filesystem sweep does not query database checkpoints. The database pass also works with empty controller storage. A partial header index keeps candidate discovery separate from event payloads and runs with no retained checkpoint. Each page checks at most 200 candidates and deletes at most 50 run checkpoints. It keeps checkpoints unless every run in the session is finished and old, all related issues are old and closed, finalization has settled, and no execution pointer, recovery owner, live environment lease, or saved task-session pointer refers to the session. Expiration locks finalizations and increments their controller generation before deletion to fence late writers, including warm predecessors. Business records and tool receipts remain. Closed sessions with saved resume pointers can therefore outlive the retention age; age alone never authorizes deletion.

Expired-controller recovery reuses the existing orphan sweep's active-run inventory. Checkpoint lookups occur only for native runs with an expired lease and no tracked execution. There is no additional idle checkpoint scan; startup recovery remains the first adoption path and the orphan cadence retries concrete unowned active runs while ownership is unresolved.

Opportunistic checkpoint retention stops after three failed passes in a startup, including candidate-query failures. It leaves retained rows intact, logs the deferred work and becomes quiet. The next startup can retry; a persistent storage or permission failure must not create an indefinite one-second database loop.

Plugin-owned workspace tasks use the same journal and recovery window. Their task owner verifies the saved identity and running status through the task API, then reconnects using the original active lease. It never fabricates a local PID or submits another task during adoption. Shared recovery dispatches the task claim; the plugin module retains task-protocol parsing and process ownership. Local, SSH, and sandbox drivers retain their existing process identity checks.

Remote execution leases expire after 60 seconds and renew every 20 seconds. Recovery grants that full lease from the database clock after locking the owner row, rather than from an earlier batch observation before cleanup or other claims. Fresh admission starts renewal immediately after its claim commits, before accounting history, retained-session cleanup or other preparation. The startup owner holds that same renewal through execution and drains it on preparation failure; recovery retains its existing shared preparation/execution owner. The local confirmed deadline starts immediately before the renewing UPDATE, after connection and row-lock acquisition. A monotonic clock accounts for the UPDATE/commit response delay without subtracting the earlier pool wait or depending on synchronization with the database clock. Late responses cannot restore a controller that has already lost local authority. Graceful shutdown detaches and releases the lease. Startup and the existing orphan sweep claim eligible expired owners; another container's PID is not ownership proof. The runner reconnect window is seven minutes, providing margin for the five-minute outage, lease expiry, and connection setup. Stop, company scope and normal admission gates retain precedence. A completed tool receipt can restore a lost response. Safe reads recheck current authority. An uncertain external mutation is not executed again without outcome evidence.

Title and feedback recovery use their existing receipt owners, including their normal input normalization and authorization. Semantic mutation recovery reads under the same receipt lock as ordinary retries. Creation and reassignment replay only their existing idempotent wake-up suffix after finding a matching receipt; they cannot enter the mutation or cancellation path when a receipt is absent. Monitor recovery uses the parsed input and preserves the saved deadline. tool-receipt-recovery.integration.test.ts exercises lost responses, failed post-commit dispatch, normalized keys, missing receipts, later title edits and Stop, checking the independent business rows and audit effects before and after.

Instruction update/restore tools recover their existing instruction:<callId> receipts through the instruction mutation owner. Recovery rechecks the current target permission, active run and exact arguments under the normal receipt lock; it cannot reserve or repeat a filesystem write. Missing receipts remain held, including a receipt removed between inspection and locked replay. The real Postgres/filesystem integration tests lose the commit acknowledgement for both tools and verify one write, one audit effect, current-permission revocation and Stop precedence. The optional PAPERCLIP_TEST_DATABASE_URL test host always uses a new owned database, never its existing data.

The first rollout does not extend grace already configured in an old runner. The contract starts with runners admitted by the updated server. Legacy stdio adapters, database loss and destroyed sandboxes are outside this guarantee.

The empty-controller-cache test uses a real Rust runner and scripted provider in separate directories, removes the controller directory, then restores solely from Postgres. It checks unchanged runner/provider process identities and one provider turn start. Build the host runner and fake provider first, then run:

PAPERCLIP_TEST_CONTROLLER_OUTAGE_MS=300000 pnpm exec vitest run \
  server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts \
  -t 'adopts the same runner from Postgres' --maxWorkers=1

This is protocol and persistence evidence. The separate everyday-workflows.runner-codex.daytona.recover-controller Product E2E cell qualifies the actual Daytona/provider path, including completion and workspace finalization. Neither test substitutes for the other's failure boundary.

Required native usage acceptance

Built-in native remote Runner execution accepts structured accounting before its PRP transport ACK. usage.reported, turn-start, and turn-terminal facts produce immutable run_usage_receipts rows. The receipt stores only structured usage and turn identity. It does not depend on output logs or raw trace capture. The source UUID derives from the run and Runner identity. Provider-turn IDs and turn-start source facts preserve aggregation when several provider turns share one PRP envelope turn. Source event IDs and sequences remain stable across receiver replacement. Repeated delivery checks the saved fact and does not create another receipt for the same report.

A failed receipt transaction rejects acceptance. The remote Runner keeps its outbox report and reconnects through its existing protocol lifecycle. A lost commit response can replay the same immutable receipt. A replacement receiver restores accepted per-turn aggregation from Postgres, then accepts pending reports from the same producer. The existing source fence prevents a retired producer from replacing the current source's usage. Missing or partial usage keeps accounting pending; it does not produce a zero-cost charge.

The native session authority epoch supplies the current run's accounting owner to reused warm transports. The usual controller checkpoint and generation checks still control transport acknowledgement and attachment. Ledger settlement, pricing, reservations, and budget enforcement retain their existing owners. No accounting receipt directory is created for this supported remote path.

Local native execution, legacy adapters, and external Dot/Muse execution retain their existing recorder behavior. This change does not establish disposable-web accounting durability for those paths. Server-originated model calls keep their existing admission and settlement records. No S3 accounting spool is introduced. The independent run-log/trace storage change can consume its own persisted sources; output acceptance is not accounting acceptance.

Run the focused PostgreSQL/Runner experiment with:

pnpm exec vitest run server/src/services/native-runtime/native-usage-delivery.integration.test.ts --maxWorkers=1

The provider fixture emits a real PRP usage report through the durable Rust Runner. A PostgreSQL trigger fails receipt insertion. The test removes the first receiver directory, replaces its accounting owner, reconnects the same Runner, and independently reads the receipt and ledger. The provider is controlled; no paid provider request is made. Additional cases cover lost commit responses, replay conflicts, per-turn aggregation, partial usage, source replacement, and budget holds. Optional log and trace callbacks are absent from the experiment.

The cold fixture build runs before opening PostgreSQL, with the existing native integration build budget. PR CI routes this suite through the existing cached native Runner lane. CI builds the release fixture; local execution uses debug. The behavior test has its own bounded receiver/process deadline after compilation.

Codex usage authority and release order

The qualified Codex producer keeps its current-run baseline and latest session totals in its existing durable provider state. It observes historical resume snapshots, preserves that baseline across process recovery, and resets the baseline to the latest totals when attaching the next run. Multiple request reports therefore emit the whole run delta, rather than the provider's last request. Regressing or incomplete live counters cannot certify completion.

Authoritative Codex reports carry runDeltaBasis: "run". This requires a Runner artifact built from the provider baseline implementation in this change. A version string alone does not establish support; use the qualified artifact digest. Existing unmarked Codex reports remain incomplete and unpriced in the new accounting path. Their budget debt remains pending for reconciliation. No deployed Runner artifact has been qualified by this source-only change.

PRP negotiates envelope versions and the admitted Runner digest. It does not negotiate this usage payload field. The existing event envelope validators leave usage payloads open, but an old standalone usage-schema consumer rejects this new field. Update receivers and strict usage-schema consumers first, then release the matching Runner artifact before claiming this durability guarantee. Finish or explicitly reconcile older admitted Codex runs during cutover. Do not claim that an old producer's final-request delta is a whole-run receipt.

Heartbeat completion reads the current immutable accepted receipt, including its saved exact rate-card amount and provenance. It does not reprice with current rates or replace that accepted estimate with an absent provider dollar amount. The terminal UPDATE preserves the current producer-owned accounting projection atomically. If another report arrived after completion read its checkpoint, the newer receipt, readiness, price, and budget exposure win; unrelated session and completion metadata can still update. A newer partial receipt stays pending. Non-accounting attachment control events keep their existing handoff acceptance; an old run's accounting report still cannot acquire a successor's authority.

The provider drains its bounded settled-turn usage tail before publishing the terminal event that closes the run. An authoritative active goal keeps its existing continuation path unless an uncorrected completion rejection closes the run. The same owner predicate selects both final usage drain and terminal outcome. A completed turn remains authoritative after a fully drained provider exit; ordinary session-exit handling still records an unavailable process independently. This keeps a late cumulative increase on the old run before the next run resets its baseline. Readiness and attachment retain accounting facts in the existing provider checkpoint instead of discarding them, including facts consumed before a later quiescence error. An increase that arrives after authoritative completion requires explicit reconciliation: PostgreSQL rejects changed finalized native usage or authoritative monetary evidence, so the existing Runner outbox retains it under the old source identity without another charge. Equal snapshots are harmless. No recurring scan or automatic adjustment is added.

If delayed usage arrives after attachment and cannot belong to the current provider turn, the provider retains its cumulative evidence and the matching saved prior identity, leaves the successor baseline unchanged, and emits a failed run terminal. The existing reconciliation lifecycle blocks further work, including after restart. Unknown older ownership stays explicit rather than being inferred from provider-supplied run identity. Manual reconciliation or a fresh session is required for this rare unsupported timing case.

Prior-turn increases are checked against the saved attachment baseline, even when a successor has already reported a higher session total. Required usage bypasses the optional diagnostic limit. An affected successor publishes an explicit incomplete usage report before failure; this preserves known budget exposure without settling ambiguous usage. That required report and its receipt retain the accounting measurements and matching saved prior identity, so an optional diagnostic acknowledgement cannot discard reconciliation evidence.