PaperClipAI/doc/testing-native-remote-continuity.md

Native remote continuity regression

The opt-in case in server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts runs the real executor, PostgreSQL, TLS ingress, Rust runner, remote file transfer, and backup validation against a deterministic fake Codex process in Linux. It checks the same runner process identity and one provider process across three warm turns, provider history reuse, and verified suspension at idle close. Separate cases kill the actual controller process and recover the next turn. They reject a live execution lease and a replacement sandbox before reconnecting. Legacy checkpoints and changed process configuration require verified suspension and replacement, while preserving the conversation and starting each turn once. Recovery can restart a provider to refresh its tool contract; process reuse is required for ordinary healthy follow-ups. Each recovery case includes another healthy turn after the recovered turn and asserts that it does not start another runner or provider process. The same-server case also starts preparation lease renewal during recovery and holds instruction collection open past a renewal tick after coordinator release. The executor must acquire renewal before preparation releases its reference, then stop renewal before releasing the coordinator. Completion must not detach the retained runner or strand the next turn. Buffered transport messages and close notifications can arrive synchronously on subscription. The controller must register the connection before subscribing so completed message processing can drain and the suspended session can retire. It does not qualify a live model or a sandbox provider API.

The diskless and diskless-per-turn cases delete both native controller directories after A completes, continue in a separate B process, delete those directories after B completes, and continue in C. They use the production source selector and snapshot rebinder for per-turn admission, followed by the real native executor. They do not inject PRP files or edit leases to produce recovery. Warm mode verifies compatible process reuse; per-turn mode verifies the existing suspension and history-resume policy. These component tests preserve workspace and remote sandbox storage, and use one source revision in three processes. They do not qualify distinct application releases or complete webserver disk loss.

native-controller-recovery.integration.test.ts separately verifies encrypted database discovery, one-source import, idempotency, live-owner waiting, corrupt bindings and paginated inventory. native-session-resume-projection.test.ts retains the 32-candidate and newer-progress barrier checks. Snapshot transaction counts and stale-generation rejection are exercised by the control-plane port tests; incomplete-publication retention is exercised by the expiration tests.

Per-turn plugin continuation uses a stopped checkpoint on a replacement lease. The executor tests cover Claude and Codex, including reopening blocked work. The lease-authority integration test uses PostgreSQL to check that checkpoint discovery cannot launch work or bypass a termination receipt. A stale negative availability flag requires a fresh provider status before the checkpoint can be read for continuation.

The executor can restore one quarantined journal in the same session scope. It requires a terminal prior run, completed commands, a terminal event, and an identity-matched suspended plugin checkpoint with no pending delivery or cleanup. Multiple candidates, contradictory direct state, and unfinished work still fail closed. Recovery preserves the journal and the provider's existing state owner. Do not delete a quarantine or change database ownership to force continuation.

Use a disposable Linux container with Node.js, sh, tar, base64, ps, and setsid. Build paperclip-runnerd and fake-codex-app-server from the checkout's packages/paperclip-runner/runner/Cargo.toml for that container's architecture. Keep the Linux runner binary on the host as well: the product hashes and stages that artifact. Put the fake provider at /opt/continuity/fake-codex-app-server inside the container. Do not put credentials in this container.

Start the container with --init and publish container port 43127 on localhost. Read its assigned port with docker port CONTAINER 43127; an automatic host port can change after a container restart. The test opens a local TLS tunnel to this port. Create a test-only certificate and key:

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout /tmp/continuity-test.key -out /tmp/continuity-test.crt \
  -days 1 -subj '/CN=localhost' \
  -addext 'subjectAltName=IP:127.0.0.1,DNS:localhost'

Create an executable wrapper on the host at /tmp/continuity-fake-codex:

#!/bin/sh
if [ "$1" = '--version' ]; then
  echo 'codex-cli 0.999.0'
  exit 0
fi
exec /opt/continuity/fake-codex-app-server \
  --state-file-in-codex-home --durable-turn-ids --record-process-start \
  --call-log "$PWD/provider-calls.log" "$@"

Run the test with the container name, current host port, and Linux artifact path:

NODE_EXTRA_CA_CERTS=/tmp/continuity-test.crt \
PAPERCLIP_TEST_RUNNER_TLS_KEY=/tmp/continuity-test.key \
PAPERCLIP_TEST_RUNNER_CONTAINER=CONTAINER \
PAPERCLIP_TEST_RUNNER_PORT=HOST_PORT \
PAPERCLIP_TEST_LINUX_RUNNER_BINARY=/path/to/linux/paperclip-runnerd \
PAPERCLIP_TEST_FAKE_CODEX_WRAPPER=/tmp/continuity-fake-codex \
pnpm exec vitest run \
  server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts \
  -t 'preserves remote warm continuity' --maxWorkers=1

The caller owns the disposable container. Remove it after the run, including after failure; failed cases can retain remote processes and files for inspection. The test cleans up its database and host session directories.

Replacing the main server

Remote native runners admitted with durable controller persistence tolerate up to five minutes without the main server. The sandbox keeps its process, provider turn and event outbox. Server-dependent tools wait for reconnection; work already accepted into the queue resumes through normal admission. A single offline web server cannot accept new HTTP requests during that interval.

Cloud operators can read GET /api/instance/task-drain?upgradeReadiness=1 with existing instance-control authority. The version 1 upgradeReadiness report counts current remote executions with a bound current-format encrypted header and a live owner lease. The checkpoint owner authenticates/decrypts that header and delegates version and identity inspection to Runner; the identity must match the run, session and runner. Ordinary sandbox runs bind their workspace identity; plugin tasks bind their active dispatched environment lease and need no workspace-sync descriptor. Legacy/local executions, missing identities/checkpoints, foreign/expired owners, in-flight admission, and an incomplete census prevent a positive report. It returns aggregate counts only. The request does not stop admission, detach a runner, or authorize new work. Ordinary drain reads do not perform this SQL. The explicit probe uses a read-only transaction, a two-second statement timeout, and at most 1,001 run rows. Header reads are capped at 256 KiB each and 8 MiB total; oversized, malformed, legacy-format or local evidence falls back to the ordinary drain. This bounded header probe does not validate every journal record or prove a live sandbox. This is an observation; the existing shutdown and startup owners still perform the actual handoff and enforce Stop and fencing. A positive report alone does not prove hosted upgrade continuity.

Retain Postgres and the instance secret encryption key across deployments. The private native_runner_checkpoints journal holds encrypted authentication state, commands, and acknowledged event cursors. These records are company/run scoped and are not part of public run projections. The replacement reconstructs its local controller cache and authenticates the recorded running artifact, even when its own installed runner binary differs. It does not stage a new binary over the running process.

The runner package owns checkpoint encoding, validation, restoration, and command classification. Its persistence callback exposes opaque records and explicit ownership metadata. The server owns encryption and transactional lease checks; heartbeat discovers eligible runs through the native runtime. Tool recovery uses the current tool authority, including receipt decoding and read policy. Transport adapters retain ownership of protocol response envelopes.

Checkpoint v2 stores authority, command records, a small cursor, and recovery control events. It excludes ordinary item.delta output. usage.reported updates are coalesced: only the latest one is included when a control change triggers a checkpoint. Priority-zero events always retain their durability barrier. The writer receives changed records, exact removed keys, and an initial replacement flag and applies them under the owner lock. The format remains encrypted and company/run bound. The decoder reads v1 as well as v2 checkpoints.

There is no event batching timer. Streaming text/usage does not call the external checkpoint writer. Authentication, commands/results, semantic inputs, lifecycle and other control events still commit before their acknowledgement or effect is released. A failed control commit closes the connection, leaving the unacknowledged input in the runner outbox. Remote controller caches also skip per-chunk file writes; local-only controllers retain their full file journal. Existing application event/log callbacks remain; this does not remove accounting or business writes. Unknown event kinds retain the conservative persist-before-ack behavior.

Externally persisted controllers use Postgres as their durable store. Their local file becomes a marker that requires restoration from the external checkpoint before restart. Ordinary control changes do not rewrite a second local journal. The sandbox runner keeps its own journal.

Checkpoint transactions share a separate pool capped at two connections per server database client. They queue without occupying the application pool or the existing lease-renewal pool. This can delay a control acknowledgement during a burst; it does not release that acknowledgement before its commit. The cost is up to two additional active database connections per instance, not per agent. There is no new SQL per text chunk or while idle, and the number of control transactions is unchanged. SQL cost collection must include the checkpoint pool.

A sparse checkpoint is marked checkpointEventPolicy: "control-v1". Its peer must advertise recoveryCursorVersion: 1. During reconnection the controller echoes the runner's durable resume.ackedSourceSeq as recoveryAckedSourceSeq in the signed authentication challenge. The runner verifies that exact value against its state; the controller uses it only after client proof and current-owner admission pass. The controller can then advance over already-acknowledged streaming output absent from Postgres. It still requires contiguous new events and validates retained replays. Unacknowledged output and control events replay from the runner's existing disk outbox. No second remote journal or new polling loop is introduced.

The live event window keeps the latest identity, turn, result and usage anchors, as well as unsettled semantic inputs, through long output streams. The 4,096-event and 500-command limits remain. This checkpoint is intentionally not a complete transcript: an abrupt crash can lose display output already acknowledged but not yet projected to the existing run log. The same provider turn continues; recovery never reruns work to reconstruct display text. Tool outcomes and terminal state retain their durable barriers.

Rollback must use a controller that understands sparse checkpoints and the signed recovery cursor. A v2 decoder alone is insufficient. Runners admitted before this release do not acquire the new protocol capability or grace period retroactively.

Completed checkpoints are pruned opportunistically at server startup using the native harness backup enabled, dry-run, and retention-days settings (seven days by default). This is a minimum retention age, not a deletion deadline: a record that becomes eligible after startup remains until the next server restart. The shared delivery coordinator drains bounded startup pages, retries failed work, and preserves pending startup work across admission drain. It becomes quiet after completing the pass, including when protected history remains. The existing hourly filesystem sweep does not query database checkpoints. The database pass also works with empty controller storage. A partial header index keeps candidate discovery separate from event payloads and runs with no retained checkpoint. Each page checks at most 200 candidates and deletes at most 50 run checkpoints. It keeps checkpoints unless every run in the session is finished and old, all related issues are old and closed, finalization has settled, and no execution pointer, recovery owner, live environment lease, or saved task-session pointer refers to the session. Expiration locks finalizations and increments their controller generation before deletion to fence late writers, including warm predecessors. Business records and tool receipts remain. Closed sessions with saved resume pointers can therefore outlive the retention age; age alone never authorizes deletion.

Expired-controller recovery reuses the existing orphan sweep's active-run inventory. Checkpoint lookups occur only for native runs with an expired lease and no tracked execution. There is no additional idle checkpoint scan; startup recovery remains the first adoption path and the orphan cadence retries concrete unowned active runs while ownership is unresolved.

Opportunistic checkpoint retention stops after three failed passes in a startup, including candidate-query failures. It leaves retained rows intact, logs the deferred work and becomes quiet. The next startup can retry; a persistent storage or permission failure must not create an indefinite one-second database loop.

Plugin-owned workspace tasks use the same journal and recovery window. Their task owner verifies the saved identity and running status through the task API, then reconnects using the original active lease. It never fabricates a local PID or submits another task during adoption. Shared recovery dispatches the task claim; the plugin module retains task-protocol parsing and process ownership. Local, SSH, and sandbox drivers retain their existing process identity checks.

Remote execution leases expire after 60 seconds and renew every 20 seconds. Recovery grants that full lease from the database clock after locking the owner row, rather than from an earlier batch observation before cleanup or other claims. Fresh admission starts renewal immediately after its claim commits, before accounting history, retained-session cleanup or other preparation. The startup owner holds that same renewal through execution and drains it on preparation failure; recovery retains its existing shared preparation/execution owner. The local confirmed deadline starts immediately before the renewing UPDATE, after connection and row-lock acquisition. A monotonic clock accounts for the UPDATE/commit response delay without subtracting the earlier pool wait or depending on synchronization with the database clock. Late responses cannot restore a controller that has already lost local authority. Graceful shutdown detaches and releases the lease. Startup and the existing orphan sweep claim eligible expired owners; another container's PID is not ownership proof. The runner reconnect window is seven minutes, providing margin for the five-minute outage, lease expiry, and connection setup. Stop, company scope and normal admission gates retain precedence. A completed tool receipt can restore a lost response. Safe reads recheck current authority. An uncertain external mutation is not executed again without outcome evidence.

Title and feedback recovery use their existing receipt owners, including their normal input normalization and authorization. Semantic mutation recovery reads under the same receipt lock as ordinary retries. Creation and reassignment replay only their existing idempotent wake-up suffix after finding a matching receipt; they cannot enter the mutation or cancellation path when a receipt is absent. Monitor recovery uses the parsed input and preserves the saved deadline. tool-receipt-recovery.integration.test.ts exercises lost responses, failed post-commit dispatch, normalized keys, missing receipts, later title edits and Stop, checking the independent business rows and audit effects before and after.

Instruction update/restore tools recover their existing instruction:<callId> receipts through the instruction mutation owner. Recovery rechecks the current target permission, active run and exact arguments under the normal receipt lock; it cannot reserve or repeat a filesystem write. Missing receipts remain held, including a receipt removed between inspection and locked replay. The real Postgres/filesystem integration tests lose the commit acknowledgement for both tools and verify one write, one audit effect, current-permission revocation and Stop precedence. The optional PAPERCLIP_TEST_DATABASE_URL test host always uses a new owned database, never its existing data.

The first rollout does not extend grace already configured in an old runner. The contract starts with runners admitted by the updated server. Legacy stdio adapters, database loss and destroyed sandboxes are outside this guarantee.

The empty-controller-cache test uses a real Rust runner and scripted provider in separate directories, removes the controller directory, then restores solely from Postgres. It waits for natural lease expiry and checks unchanged runner/provider process identities and one provider turn start. It explicitly restores the transport checkpoint, so it does not prove application startup discovery by itself. Build the host runner and fake provider first, then run:

PAPERCLIP_TEST_CONTROLLER_OUTAGE_MS=300000 pnpm exec vitest run \
  server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts \
  -t 'adopts the same runner from Postgres' --maxWorkers=1

This is protocol and persistence evidence. The separate everyday-workflows.runner-codex.daytona.recover-controller Product E2E cell qualifies the actual Daytona/provider path, including completion and workspace finalization. Neither test substitutes for the other's failure boundary.

The explicit daytona-restart-continuity.runner-codex.daytona.recover-controller cell fixes the outage at 300 seconds and adds a second replacement after completion. Its new turn executes the saved project through normal admission. The oracle compares hashed native/provider session identities and requires a new completed tool receipt and correct answer. An installed A/B/A artifact manifest uses the last replacement for this retained-session continuation/rollback check. The general everyday suite retains its 125-second default. These intervals are fault tests, not permitted HTTP deployment downtime. Both fixtures retain the workspace and remote sandbox; they do not prove whole-webserver disk loss.

Database cost qualification

scripts/benchmark-native-controller-recovery.ts compares a pinned source baseline with the candidate's real PRP writer, control-plane snapshot port and lease renewal. It uses identical fixtures with 300 controllers and 10,000 prior tasks, in baseline/candidate/candidate/baseline order. steady distributes control checkpoints; burst aligns them every five seconds. pilot checks instrumentation with ten controllers. No provider is called.

Run on an otherwise idle disposable Linux Postgres cluster with pg_stat_statements preloaded and its extension available. Give Postgres and the Node client two CPUs and 1536 MiB each. Set these explicit inputs:

BENCH_DISPOSABLE_POSTGRES=1 \
BENCH_DATABASE_URL=postgres://postgres:fixture-password@127.0.0.1:5432/postgres \
BENCH_BASELINE_ROOT=/absolute/path/to/pinned-baseline \
BENCH_REPORT_ROOT=/absolute/path/to/new-report-directory \
BENCH_POSTGRES_CGROUP=/absolute/path/to/postgres-cgroup \
node --import ./cli/node_modules/tsx/dist/loader.mjs \
  scripts/benchmark-native-controller-recovery.ts steady

The baseline needs the same installed dependency versions and built workspace packages. It supplies the baseline writer, snapshot port and PRP controller; the request fixtures, renewal path and pool policy are common controls. When running an exported tree without Git metadata, supply its exact BENCH_BASELINE_REVISION, BENCH_CANDIDATE_REVISION and BENCH_CANDIDATE_DIFF. Record the export source and verify its bytes separately. Use a fresh report directory for every attempt. Only new random databases are created; successful trial databases are dropped, and the owned template remains for inspection until the disposable cluster is removed.

The report counts SQL across all three pools, actual snapshot/PRP transactions, changed records/bytes, WAL, database CPU, sampled memory, pool admission, transaction latency, lock-waiter samples and database request latency. Pool admission ends at connection assignment/BEGIN encoding; BEGIN response time is separate. Postgres execution time includes lock waits. This does not yet provide a lock-wait latency distribution or HTTP latency. Report those limits; do not call this whole-application load qualification.

Correctness and absolute gates must pass for acceptance. Dropped operations invalidate paired comparisons. The report retains failures and refuses an overall pass if p99, drain, throughput, memory, CPU or the 10% relative regression gate fails. Do not change pool limits to conceal a failure.