PaperClipAI/doc/testing-native-remote-continuity.md

Native remote continuity regression

The opt-in case in server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts runs the real executor, PostgreSQL, TLS ingress, Rust runner, remote file transfer, and backup validation against a deterministic fake Codex process in Linux. It checks the same runner process identity and one provider process across three warm turns, provider history reuse, and verified suspension at idle close. Separate cases kill the actual controller process and recover the next turn. They reject a live execution lease and a replacement sandbox before reconnecting. Legacy checkpoints and changed process configuration require verified suspension and replacement, while preserving the conversation and starting each turn once. Recovery can restart a provider to refresh its tool contract; process reuse is required for ordinary healthy follow-ups. The stale-projection case leaves a full local control projection from just before the prior run's terminal while its database checkpoint is complete. It must refresh that same-run projection before the next turn, then preserve the same Runner and provider history through another healthy follow-up. This is a controlled cache fault, not a reconstruction of a historical incident. Each recovery case includes another healthy turn after the recovered turn and asserts that it does not start another runner or provider process. The warm-suspended case suspends the first turn, resumes the second turn, then keeps that controller warm for the third turn. It must retire the first turn's local suspension receipt so the third turn can reuse the resumed Runner and provider history without a second process or repeated turn. The finalizer-wait case gives the completed predecessor a live finalization lease. The next run must wait without starting a provider turn or consuming an attempt. After lease release, one durable resumption claim continues that same run and preserves provider history through the third turn. The same-server case also starts preparation lease renewal during recovery and holds instruction collection open past a renewal tick after coordinator release. The executor must acquire renewal before preparation releases its reference, then stop renewal before releasing the coordinator. Completion must not detach the retained runner or strand the next turn. Buffered transport messages and close notifications can arrive synchronously on subscription. The controller must register the connection before subscribing so completed message processing can drain and the suspended session can retire. It does not qualify a live model or a sandbox provider API.

The diskless and diskless-per-turn cases delete both native controller directories after A completes, continue in a separate B process, delete those directories after B completes, and continue in C. They use the production source selector and snapshot rebinder for per-turn admission, followed by the real native executor. They do not inject PRP files or edit leases to produce recovery. Warm mode verifies compatible process reuse; per-turn mode verifies the existing suspension and history-resume policy. These component tests preserve workspace and remote sandbox storage, and use one source revision in three processes. They do not qualify distinct application releases or complete webserver disk loss.

native-controller-recovery.integration.test.ts separately verifies encrypted database discovery, one-source import, idempotency, live-owner waiting, corrupt bindings and paginated inventory. native-recovered-owner-followup.test.ts admits a per-turn follow-up after recovery committed an owner that never saved a suspension receipt, and keeps unsettled or ambiguous owners held. native-session-resume-projection.test.ts retains the 32-candidate and newer-progress barrier checks. Snapshot transaction counts and stale-generation rejection are exercised by the control-plane port tests; incomplete-publication retention is exercised by the expiration tests.

The cold Codex transport regressions additionally retain a real provider event after completion, hold its persistence callback and require the original source tuple and Runner outbox to survive until acceptance. A separate case deletes the controller cache after its staged target is durably recorded and resumes through the same source before target attachment. Unknown semantic tails must stay held without target tool dispatch or a fabricated response. The staged restart helper and classifier tests cover target discovery before its first session snapshot, exact encrypted source/header/artifact bindings and the existing compatibility rebinder. Source accounting tests use real PostgreSQL to distinguish immutable receipt replay, equal finalized usage and late usage requiring reconciliation. Handoff crash tests must restore the exact queued command or authenticated receipt, reuse the verified physical Runner and preserve endpoint binding. Test both direct callbacks and the same guest listener when that ingress is enabled; changing an endpoint or losing physical proof must hold recovery. These component cases do not replace the normal three-controller Daytona gate.

Per-turn plugin continuation uses a stopped checkpoint on a replacement lease. The executor tests cover Claude and Codex, including reopening blocked work. The lease-authority integration test uses PostgreSQL to check that checkpoint discovery cannot launch work or bypass a termination receipt. A stale negative availability flag requires a fresh provider status before the checkpoint can be read for continuation.

The executor can restore one quarantined journal in the same session scope. It requires a terminal prior run, completed commands, a terminal event, and an identity-matched suspended plugin checkpoint with no pending delivery or cleanup. Multiple candidates, contradictory direct state, and unfinished work still fail closed. Recovery preserves the journal and the provider's existing state owner. Do not delete a quarantine or change database ownership to force continuation.

Use a disposable Linux container with Node.js, sh, tar, base64, ps, and setsid. Build paperclip-runnerd and fake-codex-app-server from the checkout's packages/paperclip-runner/runner/Cargo.toml for that container's architecture. Keep the Linux runner binary on the host as well: the product hashes and stages that artifact. Put the fake provider at /opt/continuity/fake-codex-app-server inside the container. Do not put credentials in this container.

Start the container with --init and publish container port 43127 on localhost. Read its assigned port with docker port CONTAINER 43127; an automatic host port can change after a container restart. The test opens a local TLS tunnel to this port. Create a test-only certificate and key:

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout /tmp/continuity-test.key -out /tmp/continuity-test.crt \
  -days 1 -subj '/CN=localhost' \
  -addext 'subjectAltName=IP:127.0.0.1,DNS:localhost'

Create an executable wrapper on the host at /tmp/continuity-fake-codex:

#!/bin/sh
if [ "$1" = '--version' ]; then
  echo 'codex-cli 0.999.0'
  exit 0
fi
exec /opt/continuity/fake-codex-app-server \
  --state-file-in-codex-home --durable-turn-ids --record-process-start \
  --call-log "$PWD/provider-calls.log" "$@"

Run the test with the container name, current host port, and Linux artifact path:

NODE_EXTRA_CA_CERTS=/tmp/continuity-test.crt \
PAPERCLIP_TEST_RUNNER_TLS_KEY=/tmp/continuity-test.key \
PAPERCLIP_TEST_RUNNER_CONTAINER=CONTAINER \
PAPERCLIP_TEST_RUNNER_PORT=HOST_PORT \
PAPERCLIP_TEST_LINUX_RUNNER_BINARY=/path/to/linux/paperclip-runnerd \
PAPERCLIP_TEST_FAKE_CODEX_WRAPPER=/tmp/continuity-fake-codex \
pnpm exec vitest run \
  server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts \
  -t 'preserves remote warm continuity' --maxWorkers=1

The caller owns the disposable container. Remove it after the run, including after failure; failed cases can retain remote processes and files for inspection. The test cleans up its database and host session directories.

Replacing the main server

Remote native runners admitted with durable controller persistence tolerate up to five minutes without the main server. The sandbox keeps its process, provider turn and event outbox. Server-dependent tools wait for reconnection; work already accepted into the queue resumes through normal admission. A single offline web server cannot accept new HTTP requests during that interval.

Cloud operators can read GET /api/instance/task-drain?upgradeReadiness=1 with existing instance-control authority. The version 1 upgradeReadiness report counts current remote executions with a bound current-format encrypted header and a live owner lease. The checkpoint owner authenticates/decrypts that header and delegates version and identity inspection to Runner; the identity must match the run, session and runner. Ordinary sandbox runs bind their workspace identity; plugin tasks bind their active dispatched environment lease and need no workspace-sync descriptor. Legacy/local executions, missing identities/checkpoints, foreign/expired owners, in-flight admission, and an incomplete census prevent a positive report. It returns aggregate counts only. The request does not stop admission, detach a runner, or authorize new work. Ordinary drain reads do not perform this SQL. The explicit probe uses a read-only transaction, a two-second statement timeout, and at most 1,001 run rows. Header reads are capped at 256 KiB each and 8 MiB total; oversized, malformed, legacy-format or local evidence falls back to the ordinary drain. This bounded header probe does not validate every journal record or prove a live sandbox. This is an observation; the existing shutdown and startup owners still perform the actual handoff and enforce Stop and fencing. A positive report alone does not prove hosted upgrade continuity.

Retain Postgres and the instance secret encryption key across deployments. The private native_runner_checkpoints journal holds encrypted authentication state, commands, and acknowledged event cursors. These records are company/run scoped and are not part of public run projections. The replacement reconstructs its local controller cache and authenticates the recorded running artifact, even when its own installed runner binary differs. It does not stage a new binary over the running process.

The runner package owns checkpoint encoding, validation, restoration, and command classification. Its persistence callback exposes opaque records and explicit ownership metadata. The server owns encryption and transactional lease checks; heartbeat discovers eligible runs through the native runtime. Tool recovery uses the current tool authority, including receipt decoding and read policy. Transport adapters retain ownership of protocol response envelopes.

Checkpoint v2 stores authority, command records, a small cursor, and recovery control events. It excludes ordinary item.delta output. usage.reported updates are coalesced: only the latest one is included when a control change triggers a checkpoint. Priority-zero events always retain their durability barrier. The writer receives changed records, exact removed keys, and an initial replacement flag and applies them under the owner lock. The format remains encrypted and company/run bound. The decoder reads v1 as well as v2 checkpoints.

There is no event batching timer. Streaming text/usage does not call the external checkpoint writer. Authentication, commands/results, semantic inputs, lifecycle and other control events still commit before their acknowledgement or effect is released. A failed control commit closes the connection, leaving the unacknowledged input in the runner outbox. Remote controller caches also skip per-chunk file writes; local-only controllers retain their full file journal. Existing application event/log callbacks remain; this does not remove accounting or business writes. Unknown event kinds retain the conservative persist-before-ack behavior.

Externally persisted controllers use Postgres as their durable store. Their local file becomes a marker that requires restoration from the external checkpoint before restart. Ordinary control changes do not rewrite a second local journal. The sandbox runner keeps its own journal.

Checkpoint transactions share a separate pool capped at three connections per server database client. They queue without occupying the application pool or the existing lease-renewal pool. This can delay a control acknowledgement during a burst; it does not release that acknowledgement before its commit. The cost is up to three additional active database connections per instance, not per agent. There is no new SQL per text chunk or while idle, and the number of control transactions is unchanged. SQL cost collection must include the checkpoint pool.

The cap follows a bounded 2 → 3 → 4 → 2 diagnostic: both cap-2 trials failed the existing burst-drain gate; 3 was the smallest tested passing cap. Retain those failures and the diagnostic separately. The subsequent native feature-cost comparison uses the common current DB factory and equal cap-3 policy for both variants, so its baseline is a controlled baseline rather than the unmodified historical cap-2 configuration. All existing benchmark gates remain unchanged.

A sparse checkpoint is marked checkpointEventPolicy: "control-v1". Its peer must advertise recoveryCursorVersion: 1. During reconnection the controller echoes the runner's durable resume.ackedSourceSeq as recoveryAckedSourceSeq in the signed authentication challenge. The runner verifies that exact value against its state; the controller uses it only after client proof and current-owner admission pass. The controller can then advance over already-acknowledged streaming output absent from Postgres. It still requires contiguous new events and validates retained replays. Unacknowledged output and control events replay from the runner's existing disk outbox. No second remote journal or new polling loop is introduced.

The live event window keeps the latest identity, turn, result and usage anchors, as well as unsettled semantic inputs, through long output streams. The 4,096-event and 500-command limits remain. This checkpoint is intentionally not a complete transcript. With local-only logs, an abrupt crash can lose a display projection already acknowledged by the controller. The opt-in s3_segments log backend checkpoints its canonical native export cursor in S3 and replays missing records from the existing validated heartbeat_run_events source on takeover. This adds no payload writes or per-record checkpoint transactions. Generic process output and optional raw provider sidecars have separate durability boundaries. See run evidence storage. The same provider turn continues; recovery never reruns provider work to reconstruct display text. Tool outcomes and terminal state retain their durable barriers.

Rollback must use a controller that understands sparse checkpoints and the signed recovery cursor. A v2 decoder alone is insufficient. Runners admitted before this release do not acquire the new protocol capability or grace period retroactively.

Completed checkpoints are pruned opportunistically at server startup using the native harness backup enabled, dry-run, and retention-days settings (seven days by default). This is a minimum retention age, not a deletion deadline: a record that becomes eligible after startup remains until the next server restart. The shared delivery coordinator drains bounded startup pages, retries failed work, and preserves pending startup work across admission drain. It becomes quiet after completing the pass, including when protected history remains. The existing hourly filesystem sweep does not query database checkpoints. The database pass also works with empty controller storage. A partial header index keeps candidate discovery separate from event payloads and runs with no retained checkpoint. Each page checks at most 200 candidates and deletes at most 50 run checkpoints. It keeps checkpoints unless every run in the session is finished and old, all related issues are old and closed, finalization has settled, and no execution pointer, recovery owner, live environment lease, or saved task-session pointer refers to the session. Expiration locks finalizations and increments their controller generation before deletion to fence late writers, including warm predecessors. Business records and tool receipts remain. Closed sessions with saved resume pointers can therefore outlive the retention age; age alone never authorizes deletion.

Expired-controller recovery reuses the existing orphan sweep's active-run inventory. Checkpoint lookups occur only for native runs with an expired lease and no tracked execution. There is no additional idle checkpoint scan; startup recovery remains the first adoption path and the orphan cadence retries concrete unowned active runs while ownership is unresolved.

Opportunistic checkpoint retention stops after three failed passes in a startup, including candidate-query failures. It leaves retained rows intact, logs the deferred work and becomes quiet. The next startup can retry; a persistent storage or permission failure must not create an indefinite one-second database loop.

Plugin-owned workspace tasks use the same journal and recovery window. Their task owner verifies the saved identity and running status through the task API, then reconnects using the original active lease. It never fabricates a local PID or submits another task during adoption. Shared recovery dispatches the task claim; the plugin module retains task-protocol parsing and process ownership. Local, SSH, and sandbox drivers retain their existing process identity checks.

Remote execution leases expire after 60 seconds and renew every 20 seconds. Recovery grants that full lease from the database clock after locking the owner row, rather than from an earlier batch observation before cleanup or other claims. Fresh admission starts renewal immediately after its claim commits, before accounting history, retained-session cleanup or other preparation. The startup owner holds that same renewal through execution and drains it on preparation failure; recovery retains its existing shared preparation/execution owner. The local confirmed deadline starts immediately before the renewing UPDATE, after connection and row-lock acquisition. A monotonic clock accounts for the UPDATE/commit response delay without subtracting the earlier pool wait or depending on synchronization with the database clock. Late responses cannot restore a controller that has already lost local authority. Graceful shutdown detaches and releases the lease. Startup and the existing orphan sweep claim eligible expired owners; another container's PID is not ownership proof. The runner reconnect window is seven minutes, providing margin for the five-minute outage, lease expiry, and connection setup. Stop, company scope and normal admission gates retain precedence. A completed tool receipt can restore a lost response. Safe reads recheck current authority. An uncertain external mutation is not executed again without outcome evidence.

Title and feedback recovery use their existing receipt owners, including their normal input normalization and authorization. Semantic mutation recovery reads under the same receipt lock as ordinary retries. Creation and reassignment replay only their existing idempotent wake-up suffix after finding a matching receipt; they cannot enter the mutation or cancellation path when a receipt is absent. Monitor recovery uses the parsed input and preserves the saved deadline. tool-receipt-recovery.integration.test.ts exercises lost responses, failed post-commit dispatch, normalized keys, missing receipts, later title edits and Stop, checking the independent business rows and audit effects before and after.

Instruction update/restore tools recover their existing instruction:<callId> receipts through the instruction mutation owner. Recovery rechecks the current target permission, active run and exact arguments under the normal receipt lock; it cannot reserve or repeat a filesystem write. Missing receipts remain held, including a receipt removed between inspection and locked replay. The real Postgres/filesystem integration tests lose the commit acknowledgement for both tools and verify one write, one audit effect, current-permission revocation and Stop precedence. The optional PAPERCLIP_TEST_DATABASE_URL test host always uses a new owned database, never its existing data.

The first rollout does not extend grace already configured in an old runner. The contract starts with runners admitted by the updated server. Legacy stdio adapters, database loss and destroyed sandboxes are outside this guarantee.

The empty-controller-cache test uses a real Rust runner and scripted provider in separate directories, removes the controller directory, then restores solely from Postgres. It waits for natural lease expiry and checks unchanged runner/provider process identities and one provider turn start. It explicitly restores the transport checkpoint, so it does not prove application startup discovery by itself. Build the host runner and fake provider first, then run:

PAPERCLIP_TEST_CONTROLLER_OUTAGE_MS=300000 pnpm exec vitest run \
  server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts \
  -t 'adopts the same runner from Postgres' --maxWorkers=1

This is protocol and persistence evidence. The separate everyday-workflows.runner-codex.daytona.recover-controller Product E2E cell qualifies the actual Daytona/provider path, including completion and workspace finalization. Neither test substitutes for the other's failure boundary.

The explicit daytona-restart-continuity.runner-codex.daytona.recover-controller cell fixes the outage at 300 seconds and adds a second replacement after completion. Its new turn executes the saved project through normal admission. The oracle compares hashed native/provider session identities and requires a new completed tool receipt and correct answer. An installed A/B/A artifact manifest uses the last replacement for this retained-session continuation/rollback check. The general everyday suite retains its 125-second default. These intervals are fault tests, not permitted HTTP deployment downtime. Both fixtures retain the workspace and remote sandbox; they do not prove whole-webserver disk loss.

The dedicated suite's version 6 uses the existing reusable warm environment, with a 15-minute provider idle window and unchanged 60/300/420-second ownership, outage and reconnect limits. Its prior per-turn/no-reuse measurement failed retained native/provider identity and remains failed. The test retires the whole runtime/paperclip-runner directory, including provider failover-backup bytes. Per-turn application qualification after sandbox disposal and this disk loss therefore awaits the provider-backup owner's durable storage contract; logical native checkpoints do not replace provider history or weaken the identity oracle.

The rollback reader must support both the recovery envelope and the settled PRP v2 idle goal suffix. Record its actual source commit and installed tree hash. An older reader rejecting that journal is an unsupported rollback, not a pass. A backported compatibility build is a development fixture until it is released; do not describe it as a previously published version. Distinct source-controller tests, compiled native-module tests, installed CLI startup and the full Product E2E journey provide different evidence and must retain those scope labels.

Database cost qualification

scripts/benchmark-native-controller-recovery.ts compares a pinned source baseline with the candidate's real PRP writer, control-plane snapshot port and lease renewal. It uses identical fixtures with 300 controllers and 10,000 prior tasks, in baseline/candidate/candidate/baseline order. steady distributes control checkpoints; burst aligns them every five seconds. pilot checks instrumentation with ten controllers. No provider is called.

Run on an otherwise idle disposable Linux Postgres cluster with pg_stat_statements preloaded and its extension available. Give Postgres and the Node client two CPUs and 1536 MiB each. Set these explicit inputs:

BENCH_DISPOSABLE_POSTGRES=1 \
BENCH_DATABASE_URL=postgres://postgres:fixture-password@127.0.0.1:5432/postgres \
BENCH_BASELINE_ROOT=/absolute/path/to/pinned-baseline \
BENCH_REPORT_ROOT=/absolute/path/to/new-report-directory \
BENCH_POSTGRES_CGROUP=/absolute/path/to/postgres-cgroup \
node --import ./cli/node_modules/tsx/dist/loader.mjs \
  scripts/benchmark-native-controller-recovery.ts steady

The baseline needs the same installed dependency versions and built workspace packages. It supplies the baseline writer, snapshot port and PRP controller; the request fixtures, renewal path and pool policy are common controls. When running an exported tree without Git metadata, supply its exact BENCH_BASELINE_REVISION, BENCH_CANDIDATE_REVISION and BENCH_CANDIDATE_DIFF. Record the export source and verify its bytes separately. Use a fresh report directory for every attempt. Only new random databases are created; successful trial databases are dropped, and the owned template remains for inspection until the disposable cluster is removed.

The report counts SQL across all three pools, actual snapshot/PRP transactions, changed records/bytes, WAL, database CPU, sampled memory, pool admission, transaction latency and database request latency. Pool admission ends at connection assignment/BEGIN encoding; BEGIN response time is separate. Postgres execution time includes lock waits. A 25 ms observer uses PostgreSQL's pg_locks.waitstart to report duration bounds for observed lock episodes by pool. The pilot calibrates it against an intentional 250 ms row-lock wait. Short waits can be missed, and delayed samples widen the estimated upper bound. The nominal 40 observer queries per second contribute to measured database CPU and statement statistics. Reports include actual sample counts, sampling gaps, CPU pressure, quota throttling and the measurement harness hash. They do not provide exact lock-exclusive SQL execution time or HTTP latency; this is component load qualification.

Correctness and absolute gates must pass for acceptance. Dropped operations invalidate paired comparisons. The report retains failures and refuses an overall pass if p99, drain, throughput, memory, CPU or the 10% relative regression gate fails. Do not change pool limits to conceal a failure.

Required native usage acceptance

Built-in native remote Runner execution accepts structured accounting before its PRP transport ACK. usage.reported, turn-start, and turn-terminal facts produce immutable run_usage_receipts rows. The receipt stores only structured usage and turn identity. It does not depend on output logs or raw trace capture. The source UUID derives from the run and Runner identity. Provider-turn IDs and turn-start source facts preserve aggregation when several provider turns share one PRP envelope turn. Source event IDs and sequences remain stable across receiver replacement. Repeated delivery checks the saved fact and does not create another receipt for the same report.

A failed receipt transaction rejects acceptance. The remote Runner keeps its outbox report and reconnects through its existing protocol lifecycle. A lost commit response can replay the same immutable receipt. A replacement receiver restores accepted per-turn aggregation from Postgres, then accepts pending reports from the same producer. The existing source fence prevents a retired producer from replacing the current source's usage. Missing or partial usage keeps accounting pending; it does not produce a zero-cost charge.

The native session authority epoch supplies the current run's accounting owner to reused warm transports. The usual controller checkpoint and generation checks still control transport acknowledgement and attachment. Ledger settlement, pricing, reservations, and budget enforcement retain their existing owners. No accounting receipt directory is created for this supported remote path.

Local native execution, legacy adapters, and external Dot/Muse execution retain their existing recorder behavior. This change does not establish disposable-web accounting durability for those paths. Server-originated model calls keep their existing admission and settlement records. No S3 accounting spool is introduced. The independent run-log/trace storage change can consume its own persisted sources; output acceptance is not accounting acceptance.

Run the focused PostgreSQL/Runner experiment with:

pnpm exec vitest run server/src/services/native-runtime/native-usage-delivery.integration.test.ts --maxWorkers=1

The provider fixture emits a real PRP usage report through the durable Rust Runner. A PostgreSQL trigger fails receipt insertion. The test removes the first receiver directory, replaces its accounting owner, reconnects the same Runner, and independently reads the receipt and ledger. The provider is controlled; no paid provider request is made. Additional cases cover lost commit responses, replay conflicts, per-turn aggregation, partial usage, source replacement, and budget holds. Optional log and trace callbacks are absent from the experiment.

The cold fixture build runs before opening PostgreSQL, with the existing native integration build budget. PR CI routes this suite through the existing cached native Runner lane. CI builds the release fixture; local execution uses debug. The behavior test has its own bounded receiver/process deadline after compilation.

Codex usage authority and release order

The qualified Codex producer keeps its current-run baseline and latest session totals in its existing durable provider state. It observes historical resume snapshots, preserves that baseline across process recovery, and resets the baseline to the latest totals when attaching the next run. Multiple request reports therefore emit the whole run delta, rather than the provider's last request. Regressing or incomplete live counters cannot certify completion.

Authoritative Codex reports carry runDeltaBasis: "run". This requires a Runner artifact built from the provider baseline implementation in this change. A version string alone does not establish support; use the qualified artifact digest. Existing unmarked Codex reports remain incomplete and unpriced in the new accounting path. Their budget debt remains pending for reconciliation. No deployed Runner artifact has been qualified by this source-only change.

PRP negotiates envelope versions and the admitted Runner digest. It does not negotiate this usage payload field. The existing event envelope validators leave usage payloads open, but an old standalone usage-schema consumer rejects this new field. Update receivers and strict usage-schema consumers first, then release the matching Runner artifact before claiming this durability guarantee. Finish or explicitly reconcile older admitted Codex runs during cutover. Do not claim that an old producer's final-request delta is a whole-run receipt.

Heartbeat completion reads the current immutable accepted receipt, including its saved exact rate-card amount and provenance. It does not reprice with current rates or replace that accepted estimate with an absent provider dollar amount. The terminal UPDATE preserves the current producer-owned accounting projection atomically. If another report arrived after completion read its checkpoint, the newer receipt, readiness, price, and budget exposure win; unrelated session and completion metadata can still update. A newer partial receipt stays pending. Non-accounting attachment control events keep their existing handoff acceptance; an old run's accounting report still cannot acquire a successor's authority.

The provider drains its bounded settled-turn usage tail before publishing the terminal event that closes the run. An authoritative active goal keeps its existing continuation path unless an uncorrected completion rejection closes the run. The same owner predicate selects both final usage drain and terminal outcome. A completed turn remains authoritative after a fully drained provider exit; ordinary session-exit handling still records an unavailable process independently. This keeps a late cumulative increase on the old run before the next run resets its baseline. Readiness and attachment retain accounting facts in the existing provider checkpoint instead of discarding them, including facts consumed before a later quiescence error. An increase that arrives after authoritative completion requires explicit reconciliation: PostgreSQL rejects changed finalized native usage or authoritative monetary evidence, so the existing Runner outbox retains it under the old source identity without another charge. Equal snapshots are harmless. No recurring scan or automatic adjustment is added.

If delayed usage arrives after attachment and cannot belong to the current provider turn, the provider retains its cumulative evidence and the matching saved prior identity, leaves the successor baseline unchanged, and emits a failed run terminal. The existing reconciliation lifecycle blocks further work, including after restart. Unknown older ownership stays explicit rather than being inferred from provider-supplied run identity. Manual reconciliation or a fresh session is required for this rare unsupported timing case.

Prior-turn increases are checked against the saved attachment baseline, even when a successor has already reported a higher session total. Required usage bypasses the optional diagnostic limit. An affected successor publishes an explicit incomplete usage report before failure; this preserves known budget exposure without settling ambiguous usage. That required report and its receipt retain the accounting measurements and matching saved prior identity, so an optional diagnostic acknowledgement cannot discard reconciliation evidence.