mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 15:35:12 +01:00
docs: record native-controller disk-independent recovery contract
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed
1 parent
ebd05e9f4c
commit
a86902833f
1 file changed
+372
@@ -0,0 +1,372 @@
|
||||
# Native-controller recovery during webserver upgrades
|
||||
|
||||
Date: 2026-10-30. Status: implementation authorized; qualification pending.
|
||||
Source inspected: `0c70cedce9a803bdc201324528abb35e423e7cd0` in the current task worktree.
|
||||
|
||||
## Goal and finish line
|
||||
|
||||
Make native controller state independent of the main Paperclip webserver's disk,
|
||||
as one prerequisite for zero-downtime webserver deployments. A person starts a
|
||||
task, the webserver is replaced, and the same remote execution continues. After
|
||||
completion, another replacement can continue the retained session without the
|
||||
previous webserver's files. Recovery must preserve Stop, current authorization,
|
||||
accepted results, pending tool decisions, and single-owner execution.
|
||||
|
||||
The implementation endpoint is a tested, reviewable PR for the native-controller
|
||||
portion only. General HTTP readiness, traffic switching, request draining,
|
||||
browser delivery, storage migration and Railway rollout belong to other chats.
|
||||
Native shutdown integrates with the existing server lifecycle coordinator.
|
||||
|
||||
Initial supported environment: native remote Runner execution, external Postgres,
|
||||
a stable externally supplied instance encryption key, surviving remote sandbox
|
||||
storage, and mutually compatible server/Runner releases. Qualify the existing
|
||||
Daytona path, including its prepared-task transport. Preserve local/self-hosted
|
||||
behavior. No new database, queue service, or shared filesystem is needed.
|
||||
|
||||
Keep the existing outage contract: a 300-second controller outage, with recovery
|
||||
completed within the Runner's existing 420-second reconnect grace. The remote
|
||||
controller lease is currently 60 seconds. The 300-second fault-tolerance test is
|
||||
not the permitted downtime of the webserver deployment. Do not shorten it or renew the Runner's
|
||||
disconnected lifetime on each retry to make a test pass. This contract permits
|
||||
server-dependent tools to wait; it does not promise uninterrupted HTTP service.
|
||||
|
||||
## What already exists
|
||||
|
||||
The earlier [Explore zero-downtime Railway deploy](codex://threads/01a1836e-8c8e-7b83-b09c-5db49a5e89de)
|
||||
plan included remotely persisted native controller checkpoints. Much of that work
|
||||
has since landed. Build on it instead of introducing another checkpoint store.
|
||||
|
||||
| Existing capability | Owner / evidence inspected |
|
||||
| --- | --- |
|
||||
| Encrypted, company/task/run-bound PRP checkpoints in Postgres | [`native-runner-checkpoint.ts`](../../server/src/services/native-runtime/native-runner-checkpoint.ts), [`native_runner_checkpoints`](../../packages/db/src/schema/native_runner_checkpoints.ts); landed in `09ad9adce2` / PR #17439 |
|
||||
| Opaque checkpoint codec, restore, authenticated replay cursor and persistence-before-effect barriers | [`durable-prp-control-plane.ts`](../../packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts) |
|
||||
| Expiring ownership, controller generations, stale-writer fencing and isolated renewal/checkpoint pools | [`native-restart-recovery.ts`](../../server/src/services/native-runtime/native-restart-recovery.ts), [`native-execution-lease.ts`](../../server/src/services/native-runtime/native-execution-lease.ts) |
|
||||
| Active remote reattachment from DB state and the admitted Runner artifact | [`native-session-executor.ts`](../../server/src/services/native-runtime/native-session-executor.ts), `executePaperclipNativeSessionWithinScope` |
|
||||
| Bounded database discovery of prior task/session checkpoints | [`native-session-resume.ts`](../../server/src/services/native-runtime/native-session-resume.ts) |
|
||||
| Empty-controller-directory, receipt-recovery, concurrency, retention and SQL-cost tests | [`native-runner-restart-recovery.integration.test.ts`](../../server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts), [continuity runbook](../testing-native-remote-continuity.md) |
|
||||
| Read-only active-run upgrade eligibility | [`heartbeat/upgrade-readiness.ts`](../../server/src/services/heartbeat/upgrade-readiness.ts); landed in `567c043b17` / PR #18037 |
|
||||
|
||||
The related [Enable uninterrupted agent upgrades](codex://threads/01a15e84-545f-7023-a854-9a642cbc76e1)
|
||||
chat reports actual PAP staging replacements preserving active executions. It is
|
||||
also working on broader default enablement. Those are reported results from that
|
||||
chat, not a new qualification of this checkout or proof that the entire webserver
|
||||
filesystem can be discarded. Reconcile that work before implementing overlapping
|
||||
changes; do not duplicate its rollout or configuration work.
|
||||
|
||||
## Remaining dependencies and evidence gaps
|
||||
|
||||
1. **Ordinary continuation still discovers PRP state through local identity.**
|
||||
`restoreWarmRunnerdController` and `restorePerTurnRunnerdController` first read
|
||||
the local full state or external marker. They return when neither supplies a
|
||||
matching identity. Active restart recovery instead starts from a DB run
|
||||
binding. Extend that approach to ordinary continuation without skipping the
|
||||
existing prior-owner, session, configuration, and suspended-state checks.
|
||||
2. **Warm supervisor state has its own local persistence path.**
|
||||
`persistWarmNativeCheckpoint` writes `paperclip.native-session-supervisor.v1`
|
||||
with the snapshot, configuration digest, and process-configuration digest;
|
||||
`loadWarmNativeCheckpoint` and warm process reuse read that file. The separate
|
||||
Postgres PRP restore writes only `control-plane-state.json`. Existing saved
|
||||
`runnerProfileJson.sessionCheckpoint` and admitted execution input provide
|
||||
reusable data, but the complete warm reuse decision must become reconstructible.
|
||||
3. **Current focused tests do not establish the whole application contract.**
|
||||
The empty-directory transport fixture restores explicitly and sets the old
|
||||
lease expired. It is valuable protocol evidence, but does not prove normal
|
||||
admission with no markers, natural ownership expiry, HTTP routing during
|
||||
overlap, or a second fresh-server continuation. Existing supervisor/Product
|
||||
E2E coverage may prove some of these; reconcile it before adding new cells.
|
||||
4. **Upgrade eligibility is narrower than disk independence.**
|
||||
The current report checks active runs and bounded PRP headers. It does not
|
||||
certify completed retained sessions or dereference workspace recovery files.
|
||||
An empty active-run list must not imply that all controller history is safe
|
||||
to discard.
|
||||
|
||||
These are inspected dependencies and missing proof, not newly reproduced
|
||||
production failures. Begin with characterization tests to determine which paths
|
||||
already reconstruct sufficient state through their callers.
|
||||
|
||||
## Design
|
||||
|
||||
### One durable source for each kind of state
|
||||
|
||||
| State | Durable owner | Local treatment |
|
||||
| --- | --- | --- |
|
||||
| PRP authentication, command receipts, control events and replay state | Existing encrypted `native_runner_checkpoints` | Reconstructed cache/marker |
|
||||
| Run authority and finalization | Existing `native_run_finalizations`, run and issue locks | No local substitute |
|
||||
| Provider/session snapshot and native reuse compatibility | Existing run profile plus a versioned native recovery envelope where fields are missing | Reconstructed supervisor projection |
|
||||
| Session lookup | Existing task-session pointer and bounded prior-run selection | Never depends on directory enumeration or a marker |
|
||||
| Provider process, provider history and unacknowledged outbox | Surviving remote Runner/sandbox | Never copied from an older controller cache over live state |
|
||||
| Workspace descriptors, manifests, archives and provider backup blobs | Their existing workspace/backup owners, with remote durability supplied by that workstream | Logical references; independent eligibility requirement |
|
||||
|
||||
Keep Runner's codec and opaque record contract in the Runner package. Native
|
||||
runtime owns session selection, compatibility, and the server-side recovery
|
||||
envelope. The database adapter owns encryption and transactional fencing.
|
||||
Heartbeat only coordinates these contracts. Extract the affected persistence and
|
||||
lookup responsibility from the large executor into a cohesive native-runtime
|
||||
module rather than expanding the executor or creating a generic recovery system.
|
||||
|
||||
### Reconstruct from durable identity
|
||||
|
||||
Use the exact active run for restart adoption. For a subsequent turn, use the
|
||||
existing task/session pointer and `native-session-resume.ts` selection policy to
|
||||
locate the eligible prior run. Validate company, task, agent, normalized session,
|
||||
workspace scope, admitted execution, provider configuration and current authority.
|
||||
Retain the existing progress barriers: a newer incompatible or incomplete record
|
||||
must not make recovery select an older convenient checkpoint.
|
||||
|
||||
Load and validate the PRP checkpoint and native snapshot before materializing
|
||||
local projections. The destination is computed from the replacement's local
|
||||
root; absolute paths recorded by the old controller never identify durable data.
|
||||
An existing local projection must agree with the selected committed state or be
|
||||
replaced under the same authority. Missing, corrupt, undecodable or mismatched
|
||||
state remains a specific recovery hold, without a fresh provider turn.
|
||||
|
||||
Default persistence design: reuse `runnerProfileJson.sessionCheckpoint` and add
|
||||
only a private, versioned native recovery envelope for missing compatibility and
|
||||
checkpoint-binding metadata. Persist the snapshot and its metadata together
|
||||
under the current run's fenced ownership. Do not duplicate the PRP journal or
|
||||
store server-specific fields as arbitrary Runner checkpoint parts. If existing
|
||||
profile metadata already supplies a field, reference it instead of copying it.
|
||||
Use a schema migration only if characterization demonstrates that the existing
|
||||
run/session records cannot express a required durable lookup or concurrency check.
|
||||
|
||||
The private `nativeControllerRecovery` envelope is limited to 4 KiB serialized.
|
||||
It contains a format version, digest of the exact redacted stored snapshot,
|
||||
native configuration digest, process-configuration digest or explicit absence,
|
||||
and PRP attachment/checkpoint binding. Run/company/task/agent/session identity is
|
||||
validated against existing records and the snapshot, not duplicated execution
|
||||
inputs. Keep credentials in the
|
||||
existing encrypted authority; persist logical resource identities, not secret
|
||||
values or host paths. Any newly sensitive recovery material uses the existing
|
||||
encryption provider and remains excluded from public run projections.
|
||||
|
||||
`PaperclipControlPlanePort.checkpointSession` already persists the snapshot before
|
||||
the local supervisor callback. Add the envelope to that same transaction and
|
||||
update; do not introduce a second snapshot writer. Compute stable compatibility
|
||||
inputs once per admission/attachment, preserve unrelated profile fields, and
|
||||
recheck durable ownership after lock waits with the established lock ordering.
|
||||
Keep legitimate post-result suspension/cleanup checkpoints permitted. The generic
|
||||
Runner persistence port must not learn Paperclip database fields. Remote local
|
||||
files become projections; local-only consumers retain authoritative files.
|
||||
|
||||
The recovery owner resolves, validates, materializes and attaches through existing
|
||||
owners. Outcomes distinguish recoverable, waiting for an owner with its actual
|
||||
deadline, missing/inconsistent evidence, unsupported format/capability, dependency
|
||||
unavailable and proven no-prior-execution. Missing evidence is not fresh bootstrap.
|
||||
Keep the 32-candidate selection window and newer-progress barriers; fetch the full
|
||||
payload only for the selected source. PRP attachment identity and current business
|
||||
run write authority differ during warm handoff and must remain explicit.
|
||||
|
||||
Publish a resumable snapshot only after its referenced PRP boundary is durable.
|
||||
Across separate commits, interruption may leave a pending obligation, never a
|
||||
false ready state. Recovery reconciles that exact obligation without rerunning
|
||||
the provider. Completion acceptance, workspace settlement, session reuse, and
|
||||
cleanup remain separate facts.
|
||||
|
||||
### Preserve ownership during replacement
|
||||
|
||||
The new server may load recovery data while the old server serves, but it cannot
|
||||
send commands, acknowledge events, or settle results until it owns the execution.
|
||||
Reuse the existing boot identity, generation and lease-owner fences. Cross-host
|
||||
takeover uses DB lease authority, never a local check of another container's PID.
|
||||
|
||||
Graceful shutdown fences new local controller work, completes outstanding durable
|
||||
barriers, detaches supported sessions, and releases only its own authority. A
|
||||
crash uses natural expiry. The successor takes the existing fenced claim,
|
||||
restores state, revalidates permissions and attaches to the exact remote process.
|
||||
Lost authority detaches the stale controller without killing the surviving Runner.
|
||||
|
||||
Retain wake/recovery obligations while the previous lease is live. Reconcile at
|
||||
startup and the actual lease/retry deadline through the shared scheduler. Cover
|
||||
a lost local release notification with durable state and the deadline. Add no
|
||||
permanent checkpoint scans or independent polling timers. A future shared
|
||||
cross-instance notification mechanism can accelerate this path without becoming
|
||||
its source of truth.
|
||||
|
||||
Exercise Stop and tool decisions arriving at either instance. Use existing durable
|
||||
intents and receipts; fix a demonstrated missing native delivery path in its
|
||||
owner. Never retransmit an uncertain external mutation just because its HTTP
|
||||
response was lost. Current access, pause, cancellation and budget controls still
|
||||
apply; restoring an admitted run is not a new grant of authority.
|
||||
|
||||
### Compatibility, adoption and retention
|
||||
|
||||
Use additive readers/writers. Before an ephemeral deployment is eligible, its
|
||||
rollback target must understand the recovery envelope, sparse checkpoint policy
|
||||
and authenticated cursor. A v2 decoder alone is insufficient. Preserve the
|
||||
original Runner artifact identity; an old running binary does not gain new
|
||||
capabilities merely because the webserver upgraded.
|
||||
|
||||
For existing file-backed sessions, allow the current legitimate owner to publish
|
||||
a validated durable snapshot at a normal checkpoint boundary. A bounded explicit
|
||||
backfill may import retained supervisor state while the disk is still available,
|
||||
after rechecking the same bindings and ownership. Never fabricate missing
|
||||
credentials, checkpoints or remote process evidence. Unprovable sessions remain
|
||||
ineligible until normal completion/retirement or explicit recovery.
|
||||
|
||||
Keep local mode intact. Require the same externally supplied encryption key on
|
||||
replacement; a recovery read must not generate a new key. Broad external-mode
|
||||
startup validation belongs to the parent persistence workstream.
|
||||
|
||||
Extend existing retention reference checks for any new envelope references.
|
||||
Retain active owners, unresolved handoffs/finalization and selected continuation
|
||||
sources. Cleanup must not race a new continuation claim or allow a late writer to
|
||||
resurrect deleted authority. Preserve bounded startup cleanup and zero idle SQL.
|
||||
|
||||
Expose an additive, versioned **native controller durability** diagnostic through
|
||||
the existing internal upgrade report, with explicit unknown/ineligible reasons.
|
||||
It must distinguish active adoption from retained-session continuation, and it
|
||||
must never claim whole-instance volume removal readiness. Keep the existing
|
||||
`ready` meaning compatible; the broader deploy coordinator composes this evidence
|
||||
with workspace and other persistence checks. Derive eligibility from committed
|
||||
records and supported readers, not a persisted boolean that can become stale.
|
||||
Report active adoption, selected retained continuation, outstanding persistence or
|
||||
handoff obligations and unsupported evidence separately. Include observation time,
|
||||
bounded reason counts and coverage/truncation. Use bounded batched metadata/header
|
||||
reads and pagination, and report incomplete inventory as unknown coverage. Add no
|
||||
public endpoint. Establish the first envelope-capable release as the rollback
|
||||
floor; an older local-only reader is not a valid empty-disk rollback target.
|
||||
|
||||
## PostgreSQL load contract
|
||||
|
||||
- Text streaming adds zero recovery SQL, transactions or checkpoint rows per chunk.
|
||||
- Ordinary usage remains coalesced into control checkpoints. Preserve changed-part
|
||||
PRP writes and exact removals; do not add a journal, transcript or archive copy.
|
||||
- Snapshot metadata rides the existing snapshot transaction, including unchanged
|
||||
metadata; no parallel writer, per-agent connection or new pool.
|
||||
- Idle servers add no recurring SQL/network work once obligations settle.
|
||||
- Scoped candidate selection fetches only the selected payload. Diagnostics use
|
||||
bounded headers/metadata, never an automatic scan of historical payloads.
|
||||
|
||||
Keep the existing dedicated checkpoint pool initially with its two-connection
|
||||
default, lazy creation, idle release and shutdown fencing. Two is not an
|
||||
architectural invariant. The invariant is bounded checkpoint traffic that meets
|
||||
recovery deadlines without starving requests or renewals. Include both servers'
|
||||
pools during overlap. Measure queue wait separately from SQL and lock wait. If
|
||||
capacity fails acceptance, compare bounded sizes under the same workload and
|
||||
choose the smallest passing size in a separately reviewable tuning change. Do not
|
||||
increase the cap to conceal slow queries or lock contention.
|
||||
|
||||
Run paired baseline/candidate workloads with identical fixtures and counts and
|
||||
all database clients included. Record statement/transaction classes, changed rows,
|
||||
persisted bytes, WAL, CPU, memory, connections, request latency, checkpoint queue/
|
||||
lock/execution latency, renewal latency/deadlines, throughput, retries and failures.
|
||||
Use the existing 8/512-chunk test plus a stream beyond the retained-event window;
|
||||
require exactly zero extra recovery SQL during text-only intervals. Repeat the
|
||||
documented 300-controller/10,000-historical-task steady and aligned-burst workload,
|
||||
alternating order and including actual session snapshot persistence.
|
||||
|
||||
Gates: unchanged lifecycle sequence has no additional normal checkpoint
|
||||
transactions; no new idle work, missed lease deadlines, lost owners, dropped
|
||||
operations or premature ACKs; existing absolute latency/throughput gates pass.
|
||||
Investigate any repeatable increase above 10% in steady CPU per completed operation,
|
||||
WAL per completed operation or request p99 before handoff; report absolute values
|
||||
too. Measure replacement bursts separately within the 420-second grace. Never
|
||||
relax persistence-before-effect guarantees to meet a performance target.
|
||||
|
||||
## Implementation slices
|
||||
|
||||
| Slice | Owned surfaces and useful result | Dependency / review evidence |
|
||||
| --- | --- | --- |
|
||||
| 1. Complete native recovery state | Native checkpoint adapter, executor persistence/lookup, session-resume owner and focused tests. Normal active, warm and per-turn paths reconstruct with no local marker or supervisor file. | Reuse landed checkpoint work; first reproduce the exact missing path. Test current local mode and incompatible-session barriers. Add only demonstrated missing envelope fields. |
|
||||
| 2. Replacement ownership and compatibility | Native lifecycle, shared recovery scheduling, shutdown wiring, retention and internal upgrade diagnostics. Competing servers hand off safely, and historical sessions have an honest eligibility path. | Slice 1; real Postgres ownership races, natural expiry, stale callbacks, Stop, migration/read compatibility and rollback tests. Keep heartbeat wiring small. |
|
||||
| 3. Application and hosted qualification | Existing restart supervisor/Product E2E suites, continuity docs and acceptance evidence. Two replacements preserve execution and later continuation. | Slices 1–2; workspace durability dependency for full-home cases. Real immutable A/B releases and compatible rollback; no synthetic claims of Daytona qualification. |
|
||||
|
||||
Tests and documentation travel with the behavior they verify. These are review
|
||||
boundaries, not a fixed PR count. Combine tightly coupled implementation where
|
||||
splitting would create an unusable intermediate state. Rebase onto the current
|
||||
default branch and reconcile the active upgrades chat before writing source.
|
||||
|
||||
## Acceptance matrix
|
||||
|
||||
| Scenario | Required result and independently inspectable evidence |
|
||||
| --- | --- |
|
||||
| Fresh active remote run; graceful replacement | Same run, Runner process/start identity, provider session and active turn; one provider start, one final result |
|
||||
| SIGKILL and absent controller storage | Normal startup claims after natural expiry; no manually edited lease, injected recovery marker or manual restore call |
|
||||
| Output-only gap and pending tool/decision | Authenticated cursor recovers; pending input settles once; independent business receipt/effect count confirms no duplicate mutation |
|
||||
| Completion while disconnected or during shutdown | Accepted result, accounting and finalization survive; incomplete persistence resumes without another provider turn |
|
||||
| Completed warm and per-turn sessions | B can discover the correct source from DB; a later C continues it after B's controller storage is removed; no new provider session solely because a marker was missing |
|
||||
| Warm config changes | Verified close/suspend and permitted saved-history resume; never incorrectly reuse changed credentials, tool contract or workspace |
|
||||
| Old and new server overlap | B respects A's live lease; only one successor wins; delayed A cannot checkpoint, ACK, dispatch or finalize after losing ownership |
|
||||
| Stop, revoked access, pause and cross-company attempts | Saved intent and current authorization prevail; no recovery bypass or delayed mutation |
|
||||
| Corrupt/missing checkpoint, key or unsupported version | Precise ineligibility/hold; no silent fallback, replay, key generation or destruction of the last recoverable copy |
|
||||
| Existing records and rollback | Valid local state can be imported under authority; unsupported state stays explicit; compatible prior release reads newly written records |
|
||||
| Cleanup and idle | Selected checkpoints survive cleanup races; settled/idle states issue no new recurring SQL or network work; next legitimate producer reactivates recovery |
|
||||
| Full application with empty homes | A → B → C uses the normal API/admission path and independently verifies the saved result; blocked until required workspace/backup dependencies are remotely durable |
|
||||
|
||||
## Feedback path and verification
|
||||
|
||||
First experiment: extend the existing real-Postgres remote continuation fixture
|
||||
with a completed warm turn, remove both PRP marker and supervisor directories,
|
||||
then submit the next turn through normal admission. Keep the remote Runner alive.
|
||||
Assert the prior run/session chosen, process identity, one new turn and a useful
|
||||
result. Also run the per-turn control. Do not prepopulate the desired recovery
|
||||
state. This distinguishes a lookup/envelope gap from missing workspace material.
|
||||
|
||||
Use the existing local Linux/container harness and scripted provider initially;
|
||||
it avoids paid model calls while preserving database locks and process lifetime.
|
||||
Measure setup separately from edit → run → inspected evidence. Start with a short
|
||||
outage; run the real 300-second outage after the narrow behavior passes. Record
|
||||
elapsed time and the exact revision in this plan when implementation begins.
|
||||
|
||||
Then exercise separate compatible application revisions with distinct native roots
|
||||
and shared Postgres, through normal startup/admission. Remove access to A's native
|
||||
controller storage. Identify surviving non-native dependencies explicitly: this
|
||||
is not evidence of whole-webserver filesystem loss. Test both SIGTERM and SIGKILL, B starting
|
||||
while A still owns the run, decisions/Stop during handoff, and a second B → C
|
||||
replacement. Retain an independent effects ledger and process identity evidence.
|
||||
If this reaches a workspace/backup dependency, keep that failure as explicit
|
||||
parent-workstream evidence; do not hide it with a copied controller directory.
|
||||
|
||||
Qualify one disposable Daytona environment using the existing Product E2E path,
|
||||
including completion and follow-up, plus a compatible rollback. Record server
|
||||
revisions, Runner artifacts, run/session/sandbox identities, lease timestamps,
|
||||
receipt counts, recovery latency and cleanup. Reuse an existing healthy campaign
|
||||
only after checking its owner and exact revision. No fleet rollout is part of
|
||||
this checkpoint feature.
|
||||
|
||||
Required handoff checks for implementation: focused affected suites, then
|
||||
`pnpm -r typecheck`, `pnpm test:run`, and `pnpm build`. If schema changes prove
|
||||
necessary, use `pnpm db:generate` and the migration/snapshot checks. Update
|
||||
`doc/execution-semantics.md`, `doc/testing-native-remote-continuity.md`, relevant
|
||||
developer/database documentation and affected capability maps. Run
|
||||
`node scripts/check-maps.mjs` after map edits. PRs use the complete repo template.
|
||||
Report skipped or unavailable checks without promoting older evidence to a pass.
|
||||
|
||||
## Parent-workstream dependencies and non-goals
|
||||
|
||||
Native checkpoint success alone does not permit removing the webserver volume.
|
||||
Workspace sync still resolves local descriptors/manifests in
|
||||
[`native-workspace-sync.ts`](../../server/src/services/native-runtime/native-workspace-sync.ts).
|
||||
That owner must provide durable logical recovery references and saved generations;
|
||||
this plan consumes that contract rather than moving workspace archives into PRP.
|
||||
Provider backup blobs and prepared resources needed after a process has suspended
|
||||
must likewise resolve without the old server disk.
|
||||
|
||||
Agent files/instructions, logs and accounting journals, imports, mutable packages,
|
||||
global background-work claims, browser live-event delivery, HTTP readiness/drain,
|
||||
and Cloud/Railway volume migration retain their own slices in the broader plan.
|
||||
Do not turn native checkpoints into a second transcript: the existing sparse
|
||||
checkpoint excludes display-only output, and this work does not strengthen the
|
||||
run log's durability guarantee. Destroyed sandbox recovery, legacy local stdio
|
||||
continuity and permanent horizontal scaling are outside this release.
|
||||
|
||||
Railway still prevents overlapping deployments on a service with an attached
|
||||
volume. Its separate overlap/drain controls become useful after all required
|
||||
volume dependencies and concurrent-server behavior are qualified. Sources checked
|
||||
2026-10-30: [volume caveats](https://docs.railway.com/volumes/reference#caveats),
|
||||
[deployment teardown](https://docs.railway.com/deployments/deployment-teardown).
|
||||
|
||||
## Current state and next action
|
||||
|
||||
Implementation is underway on `codex/native-controller-disk-independent-recovery`.
|
||||
The related Cloud policy work merged as paperclip-cloud PR #1048; it retains
|
||||
compatibility checks and drain fallback and does not implement this component.
|
||||
No new runtime or hosted qualification has passed yet.
|
||||
There is no Paperclip issue bound to this local chat; this versioned source plan
|
||||
is the handoff, rather than an upload to an unrelated company or issue.
|
||||
|
||||
Next action: reconcile the latest uninterrupted-upgrades changes, then add the
|
||||
marker-free warm/per-turn characterization described above. Its result decides
|
||||
the minimum missing native envelope/discovery change. Full-home application
|
||||
acceptance remains dependent on workspace/backup durability; no controller-only
|
||||
test can waive that gate.
|
||||
Reference in new issue
Block a user