## Thinking Path > - Paperclip must keep admitted remote work alive while its main server is replaced. > - Runner owns recovery mechanics and exposes opaque checkpoints through #17097. > - The server owns company scope, encryption, Stop, tool authority and execution ownership. > - A successor must prove current ownership before it adopts the original execution. > - Checkpoint bursts must also leave capacity for normal requests and lease renewal. > - This PR persists and adopts that authority through the existing runtime boundaries. ## Linked Issues or Issue Description **What existing behavior does this improve?** Remote agents retain the original execution during a bounded main-server outage. **Current behavior** A server restart can lose the controller's local state and connection. An unsafe replacement could repeat a turn or an uncertain mutation. Checkpoint writes in the request pool can also delay ordinary requests during an aligned burst. **Proposed behavior** Restore encrypted control records from Postgres and adopt the original runner under a fenced lease. Recover known tool receipts through current tool authority. Give control checkpoints a shared pool capped at two connections per database owner. Depends on #17097. Successors are #17455 and #17456. Qualification and presentation remain separate review slices. ## What Changed - Add company/run-bound encrypted checkpoint records and migration 0394. Keep the SQL idempotent after renumbering against current master. - Recover startup and expired ownership through native runtime and heartbeat. Preserve the sandbox lease, admitted runner artifact, harness configuration, native session and active turn. - Keep Stop, finalization, preparation ownership and stale-writer fencing transactional. Preserve graceful detach and natural crash takeover. - Recover committed tool receipts through the existing authority. Reauthorize reads and hold uncertain mutations. - Bound startup retention with the existing coordinator and preserve zero idle scans. - Share at most two checkpoint connections per database client. Keep application and liveness capacity separate. Close the checkpoint pool during shutdown; late writes cannot reopen it. - Count SQL across both application and checkpoint clients. Prove request/liveness progress, command recovery and failure after pool shutdown with real Postgres. ## Verification Project creation recovery now uses the existing project activity receipt owner. It retains authenticated input normalization, fingerprint checks and current run authority after the receipt lock. An absent receipt returns no result before mutation. The new lost-HTTP-response and no-effect regression uses real API routes and Postgres. The seven transport cases pass locally. This Mac skips all 20 database cases. On current source `5de803c2cd`, Linux [job 122593415890](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415890) executes all 20 project-tool cases successfully (9.480 seconds), including the committed-project lost-response regression. Its shard passes 1,627 cases with three unrelated skips. Server typecheck passes. The transport regression initially has two failures; a subsequent incorrect expectation about schema defaults was corrected before the final pass. Ordinary execution adds no checkpoint writes or recurring work. Current candidate `5de803c2cd` includes the project-creation receipt recovery fix and master `2cd747d1b6`. Migration 0394 retains the prior idempotent checkpoint DDL. Every master SQL file and journal entry is preserved. Snapshots 0390–0394, migration checks and server typecheck pass. The ordinary warm-turn regression fails on its second turn before `a219ab51ad`. After correction, three completed warm turns select the correct prior checkpoint and preserve the original session. All 288 provider-wiring cases pass. The control plane, external persistence callback, identity rotation and filesystem are real; the backend and database-load seam are scripted. The 13 warm admission/configuration variants also pass after final master integration. This does not qualify an underlying harness. At current source `5de803c2cd`, Linux [job 122593415907](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415907) passes all 940 native-session-executor cases (61.431 seconds), including the warm external-checkpoint behavior; the shard passes 2,358 cases with one skip. Current [Runner job 122593415475](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415475) passes the native recovery lane: 115 passes and five container-only skips, including the encrypted restore guard, fresh preparation (63.189 seconds), blocked claim lock (1.226 seconds) and recovery beyond 101 blocked owners (3.897 seconds). The first lane passes 2,215 cases with one skip. The SQL cost test reports zero idle and streaming SQL for both output volumes; this is component evidence, not a production capacity claim. At prior candidate `1ff7bb44f4`, Linux job 122549307215 passes all 938 native-executor cases, including the three-turn external-checkpoint regression. Job 122549306469 passes the native recovery lane with 115 passes and five container-only skips, including the encrypted-restore session guard, fresh preparation (63.664 seconds), claim-lock contention (3.961 seconds), and 101-owner recovery (3.824 seconds). This Mac skips the real-Postgres suite. The earlier full local executor attempt has 921 passes and 16 failures during embedded-Postgres initialization before assertions. The original fixture collisions and their corrections remain recorded. At earlier candidate `8fbe62b41d`, server shard job 122535673841 has 3,876 passes, one failure and one skip in an unchanged expiry-worker retry/drain timing test. The same server/database tree on qualification source `b64cac6951` passes all 20 expiry cases and all 3,877 shard cases. The single diagnostic retry, job 122541891915, passes all 20 expiry cases and all 3,877 shard cases (one skip). No assertion was weakened. The original failure remains recorded and its exact timing race is not proven. All Actions checks and legacy status passed on `8fbe62b41d`, with completed Greptile 5/5 and no open feedback. The next candidate `1ff7bb44f4` also passed all Actions checks and Greptile 5/5. Its Paperclip Review remained queued before the new master conflict required integration. The new candidate needs fresh exact-head CI and reviews; old checks do not qualify it. Migration generation is checkpoint-only, all master SQL and journal entries remain unchanged, and migration checks, server typecheck and all 13 warm-owner variants pass after integration. The first migration check was blocked by TSX sandbox IPC; its permitted repeat passes. The first warm-test command used an incorrect path and ran no tests; the corrected command supplies the 13-case receipt. All 16 database pool-lifetime/options/teardown cases pass after granting loopback permission; the first batch had two EPERM failures and 14 passes. Earlier semantic-tool/HTTP recovery, adoption, encryption-batch, zero-streaming-write and idle-SQL receipts remain source-attributed in the [qualification record](https://github.com/paperclipai/paperclip/blob/2ff66f57da0da35bac1c19a2366ea4b84d87a986/doc/plans/2026-10-22-hot-upgrade-qualification.md). That record retains earlier assertion, setup and cleanup failures and their corrections. No failed gate is waived. The historical 300-session component campaign passes four after phases: steady read p99 is 107–108 ms at 500 reads/s and 100 writes/s; aligned-burst read p99 is 103–105 ms after the bounded pool, versus 584–689 ms before. This comparison isolates the persistence corrections. It does not measure the final stack's hosted active-agent capacity. The hosted zero-agent API pair leaves SQL and CPU essentially unchanged, but both arms fail absolute latency gates and candidate peak cgroup memory fails its gate. The active-agent/viewer matrix, different-artifact upgrade and rollback remain open. Run `PAPERCLIP_TEST_DATABASE_URL=<owned-disposable-postgres> pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`. Previous head `f2db8a6568` had one failed workspace setup gate: [job 122555672441](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122555672441) passes all 4,169 UI tests and 285 CLI cases (one skip), but the company import/export suite reaches its 30-second health deadline during doctor startup. Its output does not reach the server-start step. The CLI, server, database, shared, adapter and plugin trees are identical on qualification source `a866619d7b`, whose [job 122555886663](https://github.com/paperclipai/paperclip/actions/runs/40609057937/job/122555886663) passes that fixture and all 286 CLI cases. The one diagnostic retry, [job 122561057225](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122561057225), passes the formerly failing fixture and all 286 CLI cases. All Actions checks and legacy status on `f2db8a6568` pass. Its exact-head Greptile completed at 5/5 with no open feedback. The current integrated head requires fresh checks and a completed Paperclip Review before merge. The original failure remains recorded; its exact timing cause is unproven. A repeated failure requires investigation and repair rather than another retry. All current-head CI, legacy status, exact-head completed Greptile 5/5, resolved feedback and ordinary GitHub merge eligibility remain required before merge. The latest integration preserves master’s published Copilot v70 profiles and every published migration. Checkpoint migration `0394_conscious_whizzer.sql` was generated with `pnpm db:generate` and retains the prior idempotent DDL. The Runner vendor rebuild and replay parity checks pass. Server typecheck and all 13 warm-owner variants pass after that rebuild. The imported report-lock patch is byte-identical to master; its intentional patch-context whitespace is retained, and the feature delta passes whitespace checks. No hosted experiment was run. ## Risks The pool adds up to two active connections per instance, shared across agents. Checkpoint delivery can wait behind that bound; it still requires its commit. Steady component-test Postgres CPU rises by about 1.7 percentage points of a two-CPU allocation. These corrections do not remove the stack's control and renewal writes. The migration and key must survive replacement. Fresh-database tests and local load do not establish different-artifact hosted upgrade or rollback continuity. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
13 KiB
Plugin Runner execution
Core can run paperclip_runner tasks in a plugin environment.
The plugin owns the execution host and its files.
Core owns run identity, selected policy, assigned tools, and the PRP journal.
Required hooks
Set supportsTasks: true on the environment driver.
Declare the environment.drivers.register capability.
Implement these hooks:
onEnvironmentAcquireLeaseonEnvironmentTaskonEnvironmentReleaseLeaseonEnvironmentDestroyLease
Task execution does not require onEnvironmentExecute or
onEnvironmentRealizeWorkspace. The task API has no shell command or file installation
operation. Use a separate workflow to install agent files. Providers that
advertise workspaceFileReadVersion: 1 can return task output files through the
bounded read_file operation for attachments and deliverables.
Core supports per-turn workspace sessions with Codex, OpenCode, and ACPX.
OpenCode and ACPX require the existing qualified provider profiles.
Workspace-free providers are not supported here. Drivers opt into warm retention through warmTaskVersion: 1. Core prefers warm execution for new task runs on capable drivers, including ACPX; existing provider admission still determines which harnesses the driver supports.
The plugin owns project workspace preparation. Core does not prepare or reuse
a Core Git worktree for a task-only environment.
Selecting a plain task directory does not add redundant default configuration.
Retaining that directory does not itself change the run configuration.
For plugin-backed project storage, Core uses the workspace provider contract.
The storage plugin must also provide the selected execution environment.
Its declared workspace modes and strategies remain in effect.
Core passes the pinned workspace binding with each task operation.
Core rejects configured workspace services for task-only environments.
Fresh retries that import retained files are not supported for plugin storage.
An agent-owned workspace cannot move to another agent or environment.
Core also rejects reuse of a local workspace in a task-only environment.
Keep its original owner, or use the workspace storage provider contract.
Prepare and launch
- Return an opaque task ID from lease acquisition as
providerLeaseId. - Return the installed Runner version and SHA-256 digest from
describe. - Return
launchPreparationVersion: 1fromdescribe. - Handle
preparewith the launch preparation contract. - Save the result under its configuration ID.
- Handle
submitwith that ID inpreparedLaunchId. - Return an authenticated WSS endpoint from
connection.
Preparation must not start a process. It must preserve Core's provider, model,
policy, instructions, and resource assignment identities. Resolve assigned
resources to execution-host paths. Use bindRunnerLaunchResources and
createRunnerProviderLaunch when the built-in descriptor builder is suitable.
Core checks the returned requirements digest and descriptor before it registers PRP or submits the task. A failed check does not enable a fallback launch. Recovery within the same run must return the exact saved configuration.
The plugin supplies executable paths and the provider workspace in the descriptor. It owns the process environment and account installation. For example, a persistent computer can use these locations:
HOME=/home/agent
AGENT_HOME=/home/agent/.agents
CODEX_HOME=/home/agent/.codex
instructions=/home/agent/.agents/AGENTS.md
skills=/home/agent/.agents/skills/<name>/SKILL.md
These are execution-host locations. Core does not copy the home for each run.
The plugin must declare projectAuthCheckVersion: 1 for all native tasks.
Core requires an allowed check_project_auth result, or a prepared result with
projectAuthCheckAtSubmit: true. The latter delegates the check to the trusted
plugin's submit handler before credential installation or task start. The plugin
must reject conflicting project or parent authentication settings. It must keep
the checked configuration fixed until submission. See
launch authentication validation.
Core sends authorized project IDs, the PRP identity, launch limits, and the
bootstrap ticket through submit. It does not send a process command.
The plugin must submit each task at most once. A lost reply requires status
reconciliation for the same task.
Credentials and tools
Core can transfer a selected Codex account or a direct Claude credential. Other managed routes fail closed until they have a typed credential contract. Do not put credentials in task metadata, journals, logs, or errors.
Assigned MCP tools run through the Core gateway over PRP. The gateway credentials stay on Core. The plugin must preserve the MCP binding identity in the prepared context.
Core records credential writeback authority before dispatch. After process termination, Core saves refreshed credentials before it completes the task. A failed save retains the task grants for cleanup.
Stop and recovery
A terminal phase does not prove process termination.
Return executionStopped: true only after the task and its descendants stop.
Return sessionStateAvailable: true only when the suspended state remains usable.
Keep retained state when Core completes the task.
Core checks managed plugin workspaces through their storage provider before
completion, including after a restart. Core does not inspect these folders locally.
If this check fails, Core defers the run and continues recovery for other runs.
Core keeps known branch and owner error codes with the repair instructions.
The task contract exposes two checkpoint operations:
read_statereturns the exact Runner durable state for the supplied identity.archive_statemoves a suspended Runner authority checkpoint under the supplied archive key. It returns the same state on retry. It keeps provider session files.
These operations do not accept paths. Core checks the active lease and retained source task before it calls the plugin. The existing Runner transport validates state identity and suspension before it rotates authority.
For controller replacement, Core restores its encrypted PRP journal from Postgres
and reuses the original active task lease. The task owner checks running status
and exact Runner identity, then reacquires the connection. It does not call
prepare, submit, or stop. The running artifact remains pinned even if
describe reports a newer installed version. Launch preparation files on the
old controller are not required to adopt an already running task.
The five-minute outage contract and its prerequisites are described in the
continuity recipe.
Core checks company, agent, environment, workspace, driver, and termination
receipts before it supplies resumeFrom. Historical controller recovery without
the durable journal first stops and releases the old task. A failed stop prevents replacement.
A new run prepares its current assignments. Provider compatibility checks remain
in the existing backends. Missing or incompatible retained state fails closed.
Setup verification
Core creates a private probe lease with an expiry. It runs the same describe, prepare, submit, and WSS/PRP path. The probe checks provider authentication and the selected model with tools denied. Core bounds Runner lifetime to the remaining probe time. It then requires suspension, confirmed termination, and task cleanup. For a new agent-owned computer, the saved-agent lifecycle runs verification after it saves the agent. Core does not require a computer before this save.
Verification
- Lease and SDK RPC integration
- Real Runner setup and recovery
- Task ownership and cancellation
- Credential transfer
The Runner fixture uses a synthetic Codex provider and local sockets. It does not qualify a deployed plugin, live account, or production WSS proxy.
Workspace completion and maintenance
A storage provider can declare workspaceProvider.finalizationVersion: 1.
Core then sends workspaceOperation with operation.kind: "finalize", the retained
binding, lifecycle generation, and originating runId and agentId. The operation
must verify workspace identity and make the run's retained writes durable before
returning { kind: "finalized", headSha, branchName }. Git workspaces must still
report the assigned branch. Non-Git workspaces return null Git fields.
Completion permits dirty, untracked, ignored, and unpublished work. It does not prove remote recoverability or authorize deletion. The provider chooses how to meet this contract; Core does not prescribe a VM, filesystem, or Git layout. Core gives completion a 30-second deadline. An unavailable execution host or an unconfirmed durability barrier must fail retryably, never return success. Recovery receives the original run and agent context.
The existing agent-independent inspect and remove operations continue to
protect storage maintenance. Providers that do not declare finalization version 1
retain the inspection-based completion path. No stored binding migration is required.
Warm task retention
A driver declares warmTaskVersion: 1, implements onEnvironmentResumeLease,
and returns warmTaskVersion: 1 from describe only when its execution host
supports authorized run attachment. Newly hired agents targeting such a driver
are configured for warm execution. Existing agents keep their explicit policy;
plugin runs without an explicit policy prefer warm execution. There is no
provider-specific warm gate or additional setting. Existing runs retain their
admitted lifecycle policy during restart recovery. The existing
runnerIdleTimeoutMs controls bounded retention. Fresh warm admission requires
a Core-managed credential identity before lease acquisition. Plugin-owned installed
accounts continue per-turn until such an identity is available; the saved warm
preference does not bypass account ownership checks. Drivers without warm
capability and setup probes remain ephemeral.
Plugin task execution does not create a Core-managed GitHub broker. Warm reuse and restart recovery use the same broker-ownership rule as initial startup. Project authorization is still checked for each new run.
A Launcher task owns one Runner process, independently of Core heartbeat runs.
Successful warm run completion retains the task and its mounts. The reuse
fingerprint includes the authorized project set; changing it retires the previous
task before checkpoint recovery. Adoption and each warm handoff require a current,
configuration-bound project-auth check against the exact retained mount scope. Core atomically
hands its retained provider identity to one new run lease, revalidates current
project authentication, prepares the new run's resources, and uses PRP
run.attach. Preparation includes the exact priorIdentity and stable
accountIdentity. The provider journals that transition idempotently before
allowing the new authority. The original ingress address remains unchanged.
Runner accepts connection: { mode: "external" } on warm attachment to preserve
the process owner's listener and TLS settings while rotating the run identity.
Idle retirement suspends and stops Runner, saves its exact checkpoint, writes credentials through the durable lease handoff, then completes the provider task. Core permits read-only suspension proof for the exact saved identity of a successful warm run after its reuse lease is retained or superseded. This does not authorize new work or file access through that lease. Temporary credential homes are not needed after the originating run ends. Termination receipts cover all run leases that owned the retired physical task. A stopped task gets a new provider task ID for checkpoint resumption. Inspection precedes in-memory session selection so a replacement cannot inherit the stopped session. Core saves the fresh provider expiry and forwards and enforces any caller-imposed deadline. A missing or unreachable computer never authorizes a replacement. Controller recovery can adopt a verified surviving idle process through its authenticated PRP connection; ambiguous transitions remain fenced by the existing durable recovery checks.
Changes to account, credential generation, workspace, runtime or process policy retire the old owner before recovery. A new authority epoch archives the stopped checkpoint under its stable archive key and preserves provider session files. Warm status monitoring relies on PRP after startup and does not poll an idle Launcher. ACPX uses this task-provider contract; other execution transports retain their provider-specific lifecycle policies.