Recover native controllers from durable metadata without local markers

Co-Authored-By: Paperclip <noreply@paperclip.ing>

doc/DEVELOPING.md

@@ -2485,6 +2485,17 @@ and `packages/adapter-utils/src/sandbox-callback-bridge.test.ts` for these contr
 
 ### Native runner restart recovery
 
+For remote controller disk-loss work, use the existing control-plane snapshot
+transaction and `native-controller-recovery.ts`; do not add a second snapshot
+writer. The private versioned metadata is limited to 4 KiB. Its importer handles
+one verified retained source while the source disk is available. Workspace and
+provider-backup migrations remain with their storage owners. See
+[the continuity verification guide](testing-native-remote-continuity.md) for
+the direct/prepared transport and three-controller continuation fixtures.
+An upgrade readiness report contains preliminary native evidence, not permission
+to remove a deployment volume. Keep the checkpoint pool bounded and measure all
+pools when qualifying database cost; the current size of two is a default.
+
 Custom source-mode HTTP launchers should create their listener with
 `createPaperclipHttpServer` from `server/src/http/server.ts`, passing the selected
 API origin as `apiUrl`. This registers the native Runner WebSocket transport

doc/execution-semantics.md

@@ -2167,8 +2167,11 @@ digest. Earlier live qualification results keep their original profile identity.
 
 Healthy native warm turns retain their runner and provider processes. After a
 controller crash, continuation can recover a completed remote session from its
-saved checkpoint and exact sandbox process identity. The prior controller must
-be stopped, its result committed, and the successor must hold current task and
+saved checkpoint and exact sandbox process identity. For an externally persisted
+controller with validated recovery metadata, the prior lease must be released
+and its result committed. A PID in the replacement container does not establish
+whether the prior remote controller is alive. Legacy file-backed recovery retains
+its stopped-process check. The successor must hold current task and
 coordinator authority. Recovery reconnects through the prior run's route before
 the authenticated run attachment changes authority. Unavailable identity probes
 have a shared 30-second retry budget; an identity conflict never permits replay.
@@ -2647,3 +2650,44 @@ The existing prior-owner, suspended-state and backup checks still decide whether
 the next turn may start. Missing checkpoints and ambiguous identity remain
 blocked; restoration does not authorize a new provider session or change the
 configured lifecycle policy.
+
+Remote admission also handles the absence of both native controller projections.
+`native-controller-recovery.ts` selects the existing bounded task/session source
+and restores its encrypted PRP records before admission reads local identity.
+It reconstructs the supervisor envelope from `runnerProfileJson.sessionCheckpoint`
+and the private `nativeControllerRecovery` metadata. Missing, incompatible, or
+mixed evidence holds the continuation; it does not authorize fresh execution.
+The 32-candidate boundary and newer-progress barriers remain in force. Exhausted
+history and current-run progress cannot establish fresh-bootstrap proof.
+
+The version-one metadata contains a digest of the exact redacted stored snapshot,
+configuration digests and the PRP checkpoint binding. It is bounded to 4 KiB,
+excluded from public run projections, and saved by `checkpointSession` in its
+existing snapshot transaction. No second snapshot writer, per-text-chunk query,
+pool, or recurring scan is added. A null checkpoint binding records an incomplete
+publication and prevents expiration. Snapshot and result writes recheck the
+coordinator generation and lease after lock waits, including the existing limited
+post-result maintenance exception.
+
+For per-turn direct sandbox recovery, the executor reads the retained Runner's
+suspended lifecycle through the recorded sandbox lease and validates its complete
+identity and lack of pending delivery/cleanup. Only a lifecycle projection is
+cached locally; provider files remain on the execution host. Prepared tasks use
+their existing `read_state` contract. The normal snapshot rebinder, tool-contract
+checks and current admission authority still govern the next turn.
+
+An active remote owner causes startup recovery to retain the obligation at the
+actual lease deadline. The existing recovery dispatch timer is rearmed on drain
+resume; successful reconciliation leaves no recurring recovery work. Upgrade
+diagnostics retain their existing `ready` meaning and add bounded retained-pointer
+header evidence. That inventory is preliminary, reports partial coverage, and
+does not certify workspace durability or volume removal.
+
+`importNativeControllerRecovery` is an internal one-source operation for the
+migration owner while old controller files remain available. It requires the
+admitted configuration digest and exact supervisor snapshot, locks the issue,
+coordinator and run in order, rejects live or unsettled ownership, and publishes
+only missing metadata. Repeating the same import is a no-op. Normal lifecycle
+checkpoints publish the metadata without a separate migration write. Disk-free
+rollback requires a server that reads this envelope; earlier local-only readers
+remain outside that rollback contract.

doc/plans/2026-10-30-native-controller-checkpoints.md

@@ -361,12 +361,57 @@ volume dependencies and concurrent-server behavior are qualified. Sources checke
 Implementation is underway on `codex/native-controller-disk-independent-recovery`.
 The related Cloud policy work merged as paperclip-cloud PR #1048; it retains
 compatibility checks and drain fallback and does not implement this component.
-No new runtime or hosted qualification has passed yet.
+Verified locally: metadata/source-selection/lock-wait checks (21 tests), readiness
+and recovery scheduling (55 tests), durable discovery/import/inventory (7 tests),
+and the complete native executor suite. The combined executor/selection/retention
+run passed 1,063 tests and exposed two overbroad retention guards; those guards
+were corrected and all 75 retention cases then passed. The snapshot fence test
+also checks stale result acceptance and lease expiry during a run-row lock wait.
+
+Both warm and per-turn fixtures passed three separate controller processes,
+deleting both native controller directories before B and again before C. They
+check provider history, Runner identity, invocation counts, and the existing
+suspension policy. Per-turn uses the production admission selector and snapshot
+rebinder. These processes use the same candidate source, not distinct release
+artifacts. Workspace and remote sandbox storage remain available. This is not
+full-webserver filesystem-loss evidence.
+
+Both direct and prepared-task transport cases passed the 300-second outage with
+natural lease expiry. The 8/512/8,192-chunk SQL test passed: each sample used seven
+control transactions and twelve changed checkpoint records, with zero recovery
+SQL/transactions/rows during the text interval and zero idle SQL. The long sample
+included normal lease renewals. Its cluster-wide WAL values include simultaneous
+test migrations and are diagnostic only, not a load acceptance measurement.
+
+Workspace typecheck, build, and map structural checks passed. The full repository
+test command and additional warm compatibility cases are running. The paired
+300-controller/10,000-task workload now includes actual snapshot persistence.
+Its initial host pilot missed throughput gates; an isolated Linux client/database
+pilot is underway. Database cost acceptance, distinct-release application startup,
+compatible rollback, and native Daytona qualification remain open. Daytona needs
+an immutable qualification image; an asynchronous question requests that fixture.
+No production deployment or complete zero-downtime acceptance is claimed.
+
+Failures retained in the implementation record: sandbox listener denial and host
+embedded-Postgres startup limits; a corrected missing `phase` in a new fixture;
+a corrected unexecuted clock query in result persistence; Linux fixture builds
+first failed on shared build storage and then missing protocol source files,
+resolved by a container-local build with the complete inputs. The 8,192-chunk
+case exposed a test-provider cap of 4,096 (expanded in the test binary), then
+exceeded the old short-stream wait limit. Its longer measurement now uses the
+normal 60-second lease and renewal rather than an artificially long owner lease.
+The long-stream renewal fixture initially omitted the controller PID/start identity;
+renewal correctly rejected it. Complete fixture identity fixed the test. Per-turn
+characterization then exposed the missing remote suspended-lifecycle projection;
+that is now verified against the bound surviving sandbox. The fixture also now
+uses the existing admission snapshot rebinder, which its warm-only predecessor
+omitted. The first benchmark attempts lacked ES-module setup and the observer's
+pg_stat_statements extension; the isolated Linux pilot needed its native Sharp
+dependency. No database-load acceptance claim has been made from failed attempts.
 There is no Paperclip issue bound to this local chat; this versioned source plan
 is the handoff, rather than an upload to an unrelated company or issue.
 
-Next action: reconcile the latest uninterrupted-upgrades changes, then add the
-marker-free warm/per-turn characterization described above. Its result decides
-the minimum missing native envelope/discovery change. Full-home application
+Next action: complete the remaining native qualification and compatibility cells,
+retain reproducible measurements, and prepare the reviewed PR. Full-home application
 acceptance remains dependent on workspace/backup durability; no controller-only
 test can waive that gate.

doc/testing-native-remote-continuity.md

@@ -7,7 +7,7 @@ and backup validation against a deterministic fake Codex process in Linux. It
 checks the same runner process identity and one provider process across three warm
 turns, provider history reuse, and verified suspension at idle close. Separate
 cases kill the actual controller process and recover the next turn. They reject
-a still-live controller and a replacement sandbox before reconnecting. Legacy
+a live execution lease and a replacement sandbox before reconnecting. Legacy
 checkpoints and changed process configuration require verified suspension and
 replacement, while preserving the conversation and starting each turn once.
 Recovery can restart a provider to refresh its tool contract; process reuse is
@@ -24,6 +24,23 @@ subscription. The controller must register the connection before subscribing so
 completed message processing can drain and the suspended session can retire.
 It does not qualify a live model or a sandbox provider API.
 
+The `diskless` and `diskless-per-turn` cases delete both native controller
+directories after A completes, continue in a separate B process, delete those
+directories after B completes, and continue in C. They use the production source
+selector and snapshot rebinder for per-turn admission, followed by the real
+native executor. They do not inject PRP files or edit leases to produce recovery.
+Warm mode verifies compatible process reuse; per-turn mode verifies the existing
+suspension and history-resume policy. These component tests preserve workspace
+and remote sandbox storage, and use one source revision in three processes.
+They do not qualify distinct application releases or complete webserver disk loss.
+
+`native-controller-recovery.integration.test.ts` separately verifies encrypted
+database discovery, one-source import, idempotency, live-owner waiting, corrupt
+bindings and paginated inventory. `native-session-resume-projection.test.ts`
+retains the 32-candidate and newer-progress barrier checks. Snapshot transaction
+counts and stale-generation rejection are exercised by the control-plane port
+tests; incomplete-publication retention is exercised by the expiration tests.
+
 Per-turn plugin continuation uses a stopped checkpoint on a replacement lease.
 The executor tests cover Claude and Codex, including reopening blocked work.
 The lease-authority integration test uses PostgreSQL to check that checkpoint
@@ -265,8 +282,10 @@ adapters, database loss and destroyed sandboxes are outside this guarantee.
 
 The empty-controller-cache test uses a real Rust runner and scripted provider
 in separate directories, removes the controller directory, then restores solely
-from Postgres. It checks unchanged runner/provider process identities and one
-provider turn start. Build the host runner and fake provider first, then run:
+from Postgres. It waits for natural lease expiry and checks unchanged
+runner/provider process identities and one provider turn start. It explicitly
+restores the transport checkpoint, so it does not prove application startup
+discovery by itself. Build the host runner and fake provider first, then run:
 
 ```sh
 PAPERCLIP_TEST_CONTROLLER_OUTAGE_MS=300000 pnpm exec vitest run \