diff --git a/doc/plans/2026-10-30-native-controller-checkpoints.md b/doc/plans/2026-10-30-native-controller-checkpoints.md new file mode 100644 index 00000000000..d0755999b2d --- /dev/null +++ b/doc/plans/2026-10-30-native-controller-checkpoints.md @@ -0,0 +1,372 @@ +# Native-controller recovery during webserver upgrades + +Date: 2026-10-30. Status: implementation authorized; qualification pending. +Source inspected: `0c70cedce9a803bdc201324528abb35e423e7cd0` in the current task worktree. + +## Goal and finish line + +Make native controller state independent of the main Paperclip webserver's disk, +as one prerequisite for zero-downtime webserver deployments. A person starts a +task, the webserver is replaced, and the same remote execution continues. After +completion, another replacement can continue the retained session without the +previous webserver's files. Recovery must preserve Stop, current authorization, +accepted results, pending tool decisions, and single-owner execution. + +The implementation endpoint is a tested, reviewable PR for the native-controller +portion only. General HTTP readiness, traffic switching, request draining, +browser delivery, storage migration and Railway rollout belong to other chats. +Native shutdown integrates with the existing server lifecycle coordinator. + +Initial supported environment: native remote Runner execution, external Postgres, +a stable externally supplied instance encryption key, surviving remote sandbox +storage, and mutually compatible server/Runner releases. Qualify the existing +Daytona path, including its prepared-task transport. Preserve local/self-hosted +behavior. No new database, queue service, or shared filesystem is needed. + +Keep the existing outage contract: a 300-second controller outage, with recovery +completed within the Runner's existing 420-second reconnect grace. The remote +controller lease is currently 60 seconds. The 300-second fault-tolerance test is +not the permitted downtime of the webserver deployment. Do not shorten it or renew the Runner's +disconnected lifetime on each retry to make a test pass. This contract permits +server-dependent tools to wait; it does not promise uninterrupted HTTP service. + +## What already exists + +The earlier [Explore zero-downtime Railway deploy](codex://threads/01a1836e-8c8e-7b83-b09c-5db49a5e89de) +plan included remotely persisted native controller checkpoints. Much of that work +has since landed. Build on it instead of introducing another checkpoint store. + +| Existing capability | Owner / evidence inspected | +| --- | --- | +| Encrypted, company/task/run-bound PRP checkpoints in Postgres | [`native-runner-checkpoint.ts`](../../server/src/services/native-runtime/native-runner-checkpoint.ts), [`native_runner_checkpoints`](../../packages/db/src/schema/native_runner_checkpoints.ts); landed in `09ad9adce2` / PR #17439 | +| Opaque checkpoint codec, restore, authenticated replay cursor and persistence-before-effect barriers | [`durable-prp-control-plane.ts`](../../packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts) | +| Expiring ownership, controller generations, stale-writer fencing and isolated renewal/checkpoint pools | [`native-restart-recovery.ts`](../../server/src/services/native-runtime/native-restart-recovery.ts), [`native-execution-lease.ts`](../../server/src/services/native-runtime/native-execution-lease.ts) | +| Active remote reattachment from DB state and the admitted Runner artifact | [`native-session-executor.ts`](../../server/src/services/native-runtime/native-session-executor.ts), `executePaperclipNativeSessionWithinScope` | +| Bounded database discovery of prior task/session checkpoints | [`native-session-resume.ts`](../../server/src/services/native-runtime/native-session-resume.ts) | +| Empty-controller-directory, receipt-recovery, concurrency, retention and SQL-cost tests | [`native-runner-restart-recovery.integration.test.ts`](../../server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts), [continuity runbook](../testing-native-remote-continuity.md) | +| Read-only active-run upgrade eligibility | [`heartbeat/upgrade-readiness.ts`](../../server/src/services/heartbeat/upgrade-readiness.ts); landed in `567c043b17` / PR #18037 | + +The related [Enable uninterrupted agent upgrades](codex://threads/01a15e84-545f-7023-a854-9a642cbc76e1) +chat reports actual PAP staging replacements preserving active executions. It is +also working on broader default enablement. Those are reported results from that +chat, not a new qualification of this checkout or proof that the entire webserver +filesystem can be discarded. Reconcile that work before implementing overlapping +changes; do not duplicate its rollout or configuration work. + +## Remaining dependencies and evidence gaps + +1. **Ordinary continuation still discovers PRP state through local identity.** + `restoreWarmRunnerdController` and `restorePerTurnRunnerdController` first read + the local full state or external marker. They return when neither supplies a + matching identity. Active restart recovery instead starts from a DB run + binding. Extend that approach to ordinary continuation without skipping the + existing prior-owner, session, configuration, and suspended-state checks. +2. **Warm supervisor state has its own local persistence path.** + `persistWarmNativeCheckpoint` writes `paperclip.native-session-supervisor.v1` + with the snapshot, configuration digest, and process-configuration digest; + `loadWarmNativeCheckpoint` and warm process reuse read that file. The separate + Postgres PRP restore writes only `control-plane-state.json`. Existing saved + `runnerProfileJson.sessionCheckpoint` and admitted execution input provide + reusable data, but the complete warm reuse decision must become reconstructible. +3. **Current focused tests do not establish the whole application contract.** + The empty-directory transport fixture restores explicitly and sets the old + lease expired. It is valuable protocol evidence, but does not prove normal + admission with no markers, natural ownership expiry, HTTP routing during + overlap, or a second fresh-server continuation. Existing supervisor/Product + E2E coverage may prove some of these; reconcile it before adding new cells. +4. **Upgrade eligibility is narrower than disk independence.** + The current report checks active runs and bounded PRP headers. It does not + certify completed retained sessions or dereference workspace recovery files. + An empty active-run list must not imply that all controller history is safe + to discard. + +These are inspected dependencies and missing proof, not newly reproduced +production failures. Begin with characterization tests to determine which paths +already reconstruct sufficient state through their callers. + +## Design + +### One durable source for each kind of state + +| State | Durable owner | Local treatment | +| --- | --- | --- | +| PRP authentication, command receipts, control events and replay state | Existing encrypted `native_runner_checkpoints` | Reconstructed cache/marker | +| Run authority and finalization | Existing `native_run_finalizations`, run and issue locks | No local substitute | +| Provider/session snapshot and native reuse compatibility | Existing run profile plus a versioned native recovery envelope where fields are missing | Reconstructed supervisor projection | +| Session lookup | Existing task-session pointer and bounded prior-run selection | Never depends on directory enumeration or a marker | +| Provider process, provider history and unacknowledged outbox | Surviving remote Runner/sandbox | Never copied from an older controller cache over live state | +| Workspace descriptors, manifests, archives and provider backup blobs | Their existing workspace/backup owners, with remote durability supplied by that workstream | Logical references; independent eligibility requirement | + +Keep Runner's codec and opaque record contract in the Runner package. Native +runtime owns session selection, compatibility, and the server-side recovery +envelope. The database adapter owns encryption and transactional fencing. +Heartbeat only coordinates these contracts. Extract the affected persistence and +lookup responsibility from the large executor into a cohesive native-runtime +module rather than expanding the executor or creating a generic recovery system. + +### Reconstruct from durable identity + +Use the exact active run for restart adoption. For a subsequent turn, use the +existing task/session pointer and `native-session-resume.ts` selection policy to +locate the eligible prior run. Validate company, task, agent, normalized session, +workspace scope, admitted execution, provider configuration and current authority. +Retain the existing progress barriers: a newer incompatible or incomplete record +must not make recovery select an older convenient checkpoint. + +Load and validate the PRP checkpoint and native snapshot before materializing +local projections. The destination is computed from the replacement's local +root; absolute paths recorded by the old controller never identify durable data. +An existing local projection must agree with the selected committed state or be +replaced under the same authority. Missing, corrupt, undecodable or mismatched +state remains a specific recovery hold, without a fresh provider turn. + +Default persistence design: reuse `runnerProfileJson.sessionCheckpoint` and add +only a private, versioned native recovery envelope for missing compatibility and +checkpoint-binding metadata. Persist the snapshot and its metadata together +under the current run's fenced ownership. Do not duplicate the PRP journal or +store server-specific fields as arbitrary Runner checkpoint parts. If existing +profile metadata already supplies a field, reference it instead of copying it. +Use a schema migration only if characterization demonstrates that the existing +run/session records cannot express a required durable lookup or concurrency check. + +The private `nativeControllerRecovery` envelope is limited to 4 KiB serialized. +It contains a format version, digest of the exact redacted stored snapshot, +native configuration digest, process-configuration digest or explicit absence, +and PRP attachment/checkpoint binding. Run/company/task/agent/session identity is +validated against existing records and the snapshot, not duplicated execution +inputs. Keep credentials in the +existing encrypted authority; persist logical resource identities, not secret +values or host paths. Any newly sensitive recovery material uses the existing +encryption provider and remains excluded from public run projections. + +`PaperclipControlPlanePort.checkpointSession` already persists the snapshot before +the local supervisor callback. Add the envelope to that same transaction and +update; do not introduce a second snapshot writer. Compute stable compatibility +inputs once per admission/attachment, preserve unrelated profile fields, and +recheck durable ownership after lock waits with the established lock ordering. +Keep legitimate post-result suspension/cleanup checkpoints permitted. The generic +Runner persistence port must not learn Paperclip database fields. Remote local +files become projections; local-only consumers retain authoritative files. + +The recovery owner resolves, validates, materializes and attaches through existing +owners. Outcomes distinguish recoverable, waiting for an owner with its actual +deadline, missing/inconsistent evidence, unsupported format/capability, dependency +unavailable and proven no-prior-execution. Missing evidence is not fresh bootstrap. +Keep the 32-candidate selection window and newer-progress barriers; fetch the full +payload only for the selected source. PRP attachment identity and current business +run write authority differ during warm handoff and must remain explicit. + +Publish a resumable snapshot only after its referenced PRP boundary is durable. +Across separate commits, interruption may leave a pending obligation, never a +false ready state. Recovery reconciles that exact obligation without rerunning +the provider. Completion acceptance, workspace settlement, session reuse, and +cleanup remain separate facts. + +### Preserve ownership during replacement + +The new server may load recovery data while the old server serves, but it cannot +send commands, acknowledge events, or settle results until it owns the execution. +Reuse the existing boot identity, generation and lease-owner fences. Cross-host +takeover uses DB lease authority, never a local check of another container's PID. + +Graceful shutdown fences new local controller work, completes outstanding durable +barriers, detaches supported sessions, and releases only its own authority. A +crash uses natural expiry. The successor takes the existing fenced claim, +restores state, revalidates permissions and attaches to the exact remote process. +Lost authority detaches the stale controller without killing the surviving Runner. + +Retain wake/recovery obligations while the previous lease is live. Reconcile at +startup and the actual lease/retry deadline through the shared scheduler. Cover +a lost local release notification with durable state and the deadline. Add no +permanent checkpoint scans or independent polling timers. A future shared +cross-instance notification mechanism can accelerate this path without becoming +its source of truth. + +Exercise Stop and tool decisions arriving at either instance. Use existing durable +intents and receipts; fix a demonstrated missing native delivery path in its +owner. Never retransmit an uncertain external mutation just because its HTTP +response was lost. Current access, pause, cancellation and budget controls still +apply; restoring an admitted run is not a new grant of authority. + +### Compatibility, adoption and retention + +Use additive readers/writers. Before an ephemeral deployment is eligible, its +rollback target must understand the recovery envelope, sparse checkpoint policy +and authenticated cursor. A v2 decoder alone is insufficient. Preserve the +original Runner artifact identity; an old running binary does not gain new +capabilities merely because the webserver upgraded. + +For existing file-backed sessions, allow the current legitimate owner to publish +a validated durable snapshot at a normal checkpoint boundary. A bounded explicit +backfill may import retained supervisor state while the disk is still available, +after rechecking the same bindings and ownership. Never fabricate missing +credentials, checkpoints or remote process evidence. Unprovable sessions remain +ineligible until normal completion/retirement or explicit recovery. + +Keep local mode intact. Require the same externally supplied encryption key on +replacement; a recovery read must not generate a new key. Broad external-mode +startup validation belongs to the parent persistence workstream. + +Extend existing retention reference checks for any new envelope references. +Retain active owners, unresolved handoffs/finalization and selected continuation +sources. Cleanup must not race a new continuation claim or allow a late writer to +resurrect deleted authority. Preserve bounded startup cleanup and zero idle SQL. + +Expose an additive, versioned **native controller durability** diagnostic through +the existing internal upgrade report, with explicit unknown/ineligible reasons. +It must distinguish active adoption from retained-session continuation, and it +must never claim whole-instance volume removal readiness. Keep the existing +`ready` meaning compatible; the broader deploy coordinator composes this evidence +with workspace and other persistence checks. Derive eligibility from committed +records and supported readers, not a persisted boolean that can become stale. +Report active adoption, selected retained continuation, outstanding persistence or +handoff obligations and unsupported evidence separately. Include observation time, +bounded reason counts and coverage/truncation. Use bounded batched metadata/header +reads and pagination, and report incomplete inventory as unknown coverage. Add no +public endpoint. Establish the first envelope-capable release as the rollback +floor; an older local-only reader is not a valid empty-disk rollback target. + +## PostgreSQL load contract + +- Text streaming adds zero recovery SQL, transactions or checkpoint rows per chunk. +- Ordinary usage remains coalesced into control checkpoints. Preserve changed-part + PRP writes and exact removals; do not add a journal, transcript or archive copy. +- Snapshot metadata rides the existing snapshot transaction, including unchanged + metadata; no parallel writer, per-agent connection or new pool. +- Idle servers add no recurring SQL/network work once obligations settle. +- Scoped candidate selection fetches only the selected payload. Diagnostics use + bounded headers/metadata, never an automatic scan of historical payloads. + +Keep the existing dedicated checkpoint pool initially with its two-connection +default, lazy creation, idle release and shutdown fencing. Two is not an +architectural invariant. The invariant is bounded checkpoint traffic that meets +recovery deadlines without starving requests or renewals. Include both servers' +pools during overlap. Measure queue wait separately from SQL and lock wait. If +capacity fails acceptance, compare bounded sizes under the same workload and +choose the smallest passing size in a separately reviewable tuning change. Do not +increase the cap to conceal slow queries or lock contention. + +Run paired baseline/candidate workloads with identical fixtures and counts and +all database clients included. Record statement/transaction classes, changed rows, +persisted bytes, WAL, CPU, memory, connections, request latency, checkpoint queue/ +lock/execution latency, renewal latency/deadlines, throughput, retries and failures. +Use the existing 8/512-chunk test plus a stream beyond the retained-event window; +require exactly zero extra recovery SQL during text-only intervals. Repeat the +documented 300-controller/10,000-historical-task steady and aligned-burst workload, +alternating order and including actual session snapshot persistence. + +Gates: unchanged lifecycle sequence has no additional normal checkpoint +transactions; no new idle work, missed lease deadlines, lost owners, dropped +operations or premature ACKs; existing absolute latency/throughput gates pass. +Investigate any repeatable increase above 10% in steady CPU per completed operation, +WAL per completed operation or request p99 before handoff; report absolute values +too. Measure replacement bursts separately within the 420-second grace. Never +relax persistence-before-effect guarantees to meet a performance target. + +## Implementation slices + +| Slice | Owned surfaces and useful result | Dependency / review evidence | +| --- | --- | --- | +| 1. Complete native recovery state | Native checkpoint adapter, executor persistence/lookup, session-resume owner and focused tests. Normal active, warm and per-turn paths reconstruct with no local marker or supervisor file. | Reuse landed checkpoint work; first reproduce the exact missing path. Test current local mode and incompatible-session barriers. Add only demonstrated missing envelope fields. | +| 2. Replacement ownership and compatibility | Native lifecycle, shared recovery scheduling, shutdown wiring, retention and internal upgrade diagnostics. Competing servers hand off safely, and historical sessions have an honest eligibility path. | Slice 1; real Postgres ownership races, natural expiry, stale callbacks, Stop, migration/read compatibility and rollback tests. Keep heartbeat wiring small. | +| 3. Application and hosted qualification | Existing restart supervisor/Product E2E suites, continuity docs and acceptance evidence. Two replacements preserve execution and later continuation. | Slices 1–2; workspace durability dependency for full-home cases. Real immutable A/B releases and compatible rollback; no synthetic claims of Daytona qualification. | + +Tests and documentation travel with the behavior they verify. These are review +boundaries, not a fixed PR count. Combine tightly coupled implementation where +splitting would create an unusable intermediate state. Rebase onto the current +default branch and reconcile the active upgrades chat before writing source. + +## Acceptance matrix + +| Scenario | Required result and independently inspectable evidence | +| --- | --- | +| Fresh active remote run; graceful replacement | Same run, Runner process/start identity, provider session and active turn; one provider start, one final result | +| SIGKILL and absent controller storage | Normal startup claims after natural expiry; no manually edited lease, injected recovery marker or manual restore call | +| Output-only gap and pending tool/decision | Authenticated cursor recovers; pending input settles once; independent business receipt/effect count confirms no duplicate mutation | +| Completion while disconnected or during shutdown | Accepted result, accounting and finalization survive; incomplete persistence resumes without another provider turn | +| Completed warm and per-turn sessions | B can discover the correct source from DB; a later C continues it after B's controller storage is removed; no new provider session solely because a marker was missing | +| Warm config changes | Verified close/suspend and permitted saved-history resume; never incorrectly reuse changed credentials, tool contract or workspace | +| Old and new server overlap | B respects A's live lease; only one successor wins; delayed A cannot checkpoint, ACK, dispatch or finalize after losing ownership | +| Stop, revoked access, pause and cross-company attempts | Saved intent and current authorization prevail; no recovery bypass or delayed mutation | +| Corrupt/missing checkpoint, key or unsupported version | Precise ineligibility/hold; no silent fallback, replay, key generation or destruction of the last recoverable copy | +| Existing records and rollback | Valid local state can be imported under authority; unsupported state stays explicit; compatible prior release reads newly written records | +| Cleanup and idle | Selected checkpoints survive cleanup races; settled/idle states issue no new recurring SQL or network work; next legitimate producer reactivates recovery | +| Full application with empty homes | A → B → C uses the normal API/admission path and independently verifies the saved result; blocked until required workspace/backup dependencies are remotely durable | + +## Feedback path and verification + +First experiment: extend the existing real-Postgres remote continuation fixture +with a completed warm turn, remove both PRP marker and supervisor directories, +then submit the next turn through normal admission. Keep the remote Runner alive. +Assert the prior run/session chosen, process identity, one new turn and a useful +result. Also run the per-turn control. Do not prepopulate the desired recovery +state. This distinguishes a lookup/envelope gap from missing workspace material. + +Use the existing local Linux/container harness and scripted provider initially; +it avoids paid model calls while preserving database locks and process lifetime. +Measure setup separately from edit → run → inspected evidence. Start with a short +outage; run the real 300-second outage after the narrow behavior passes. Record +elapsed time and the exact revision in this plan when implementation begins. + +Then exercise separate compatible application revisions with distinct native roots +and shared Postgres, through normal startup/admission. Remove access to A's native +controller storage. Identify surviving non-native dependencies explicitly: this +is not evidence of whole-webserver filesystem loss. Test both SIGTERM and SIGKILL, B starting +while A still owns the run, decisions/Stop during handoff, and a second B → C +replacement. Retain an independent effects ledger and process identity evidence. +If this reaches a workspace/backup dependency, keep that failure as explicit +parent-workstream evidence; do not hide it with a copied controller directory. + +Qualify one disposable Daytona environment using the existing Product E2E path, +including completion and follow-up, plus a compatible rollback. Record server +revisions, Runner artifacts, run/session/sandbox identities, lease timestamps, +receipt counts, recovery latency and cleanup. Reuse an existing healthy campaign +only after checking its owner and exact revision. No fleet rollout is part of +this checkpoint feature. + +Required handoff checks for implementation: focused affected suites, then +`pnpm -r typecheck`, `pnpm test:run`, and `pnpm build`. If schema changes prove +necessary, use `pnpm db:generate` and the migration/snapshot checks. Update +`doc/execution-semantics.md`, `doc/testing-native-remote-continuity.md`, relevant +developer/database documentation and affected capability maps. Run +`node scripts/check-maps.mjs` after map edits. PRs use the complete repo template. +Report skipped or unavailable checks without promoting older evidence to a pass. + +## Parent-workstream dependencies and non-goals + +Native checkpoint success alone does not permit removing the webserver volume. +Workspace sync still resolves local descriptors/manifests in +[`native-workspace-sync.ts`](../../server/src/services/native-runtime/native-workspace-sync.ts). +That owner must provide durable logical recovery references and saved generations; +this plan consumes that contract rather than moving workspace archives into PRP. +Provider backup blobs and prepared resources needed after a process has suspended +must likewise resolve without the old server disk. + +Agent files/instructions, logs and accounting journals, imports, mutable packages, +global background-work claims, browser live-event delivery, HTTP readiness/drain, +and Cloud/Railway volume migration retain their own slices in the broader plan. +Do not turn native checkpoints into a second transcript: the existing sparse +checkpoint excludes display-only output, and this work does not strengthen the +run log's durability guarantee. Destroyed sandbox recovery, legacy local stdio +continuity and permanent horizontal scaling are outside this release. + +Railway still prevents overlapping deployments on a service with an attached +volume. Its separate overlap/drain controls become useful after all required +volume dependencies and concurrent-server behavior are qualified. Sources checked +2026-10-30: [volume caveats](https://docs.railway.com/volumes/reference#caveats), +[deployment teardown](https://docs.railway.com/deployments/deployment-teardown). + +## Current state and next action + +Implementation is underway on `codex/native-controller-disk-independent-recovery`. +The related Cloud policy work merged as paperclip-cloud PR #1048; it retains +compatibility checks and drain fallback and does not implement this component. +No new runtime or hosted qualification has passed yet. +There is no Paperclip issue bound to this local chat; this versioned source plan +is the handoff, rather than an upload to an unrelated company or issue. + +Next action: reconcile the latest uninterrupted-upgrades changes, then add the +marker-free warm/per-turn characterization described above. Its result decides +the minimum missing native envelope/discovery change. Full-home application +acceptance remains dependent on workspace/backup durability; no controller-only +test can waive that gate.