fix(server): resume suspended sessions after controller replacement (#18554)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A web server replacement must preserve a completed conversation as
well as active execution.
> - A warm remote Runner can suspend before the replacement admits
another turn.
> - The replacement restores its controller checkpoint from PostgreSQL,
but lacks the Runner suspension receipt.
> - It therefore tries to adopt a live process and rejects the safely
suspended source.
> - This change restores the verified receipt and preserves retained
launch files while the old authority is reconciled.
> - Later turns can keep the same provider conversation after the old
web server filesystem is deleted.

## Linked Issues or Issue Description

Refs #18552, #18543, #18366.

**What happened?**

The real rolling-deployment journey passed both HTTP handoffs, with 765
reads and 78 verified saves. The first three agent turns kept their
native and provider sessions. After the second replacement, the fourth
continuation failed with `remote_warm_session_not_idle`. The remote
Runner was suspended with an empty outbox and no pending terminal. A
focused regression reproduced that exact failure. After restoring its
suspension receipt, the same test exposed a stale launch-file read
during source recovery.

**Expected behavior**

A replacement can resume the same provider conversation from a settled
suspended Runner. It must validate the physical sandbox and all
persisted identity fields. A live source must still pass the existing
adoption checks.

**Steps to reproduce**

Enable the Linux Runner fixture for
`native-runner-restart-recovery.integration.test.ts`. Run the
`diskless-warm-suspended` cases. The fixture completes and gracefully
suspends each source, deletes the controller projections, and admits a
later turn from another process. It does not write a synthetic
suspension receipt.

## What Changed

- Reuse the existing per-turn suspended-receipt reader before warm
admission chooses live adoption or suspended recovery.
- Keep the receipt read retryable when a failed remote read leaves only
the controller projection.
- Leave every non-suspended warm state to the existing live-adoption
checks. A preliminary backpressure observation cannot reject admission
before that owner inspects the current Runner state.
- Preserve current sandbox binding, settled-outbox, exact-identity,
local-evidence and admission checks.
- Pass the structured launch specification to remote asset preparation.
Source reconciliation keeps the old authority's retained launch files; a
current-authority launch still stages its current files.
- Add two Linux Runner/PostgreSQL regressions for two controller-cache
deletions and three turns on one provider thread, including an
interrupted receipt read.
- Update the deployment qualification record, including the failed live
evidence and the successful staging database-idle observation.

## Verification

- Before the fix, the focused regression fails with the exact live
`remote_warm_session_not_idle` error.
- Both new regressions pass with real PostgreSQL and Linux Runner. The
provider is deterministic; this is not a live model qualification.
- The first receipt-only fix failed on the deleted controller skills
cache. The completed fix passes that same scenario without recreating
fixture launch assets.
- All 1,067 surrounding launch supervision, warm recovery, launch-asset
and controller recovery tests pass. The exact `cc23019691` server build
passes. All ten Linux continuity cases pass, including live adoption,
graceful suspension, per-turn resume, changed configuration and
failed-read retry. Each case checks provider-thread identity and exact
turn counts. The live Codex/Daytona journey passes on the same clean
source and exact build: 518 reads, 53 saves and 53 verified saves with
zero probe failures; both predecessor homes deleted; the active
execution completes once; all four turns preserve the provider
conversation and native receipts; saved files and the final command
output are independently checked. No automatic retries. Child credential
and object-storage bucket cleanup pass. Browser reconnection remains a
separate hosted acceptance check.
- The review refinement has a failing-before regression: only the
preliminary read reports transient backpressure; the subsequent adoption
reads the real Runner state. It and both suspension cases pass on the
final source (three cases), with server typecheck. The existing idle
check remains unchanged. An initial retry hit a fixture port left by the
intentional pre-fix failure; a fresh container resolved that
infrastructure error.
- Independent cleanup verification found two archived setup sandboxes
left by the live harness. Both were deleted and confirmed absent. The
failed automatic-cleanup check remains recorded.
- No claim of the ten-deploy Railway campaign or production canary yet.

## Risks

- A cold warm-session admission can now make one bounded remote state
read when local lifecycle evidence is absent. Its failure stops
admission and remains retryable.
- This path cannot replace contradictory local evidence, transfer
authority, or treat a suspended process as a live one.
- Source recovery must keep the previous authority's launch files until
reconciliation completes. The launch identity selects that behavior;
ordinary launches still prepare current assets.
- No schema migration or public API change.

## Model Used

OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution
and integration-test tools. The exact runtime model variant and
context-window size are not exposed in this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

CI qualification note: first final-head run `41322376054` passed the
native/runtime and remaining suites but failed the unrelated
`automatic_message_on_reactivation` browser fixture at its
pre-reactivation deferred-queue count (0 rather than 1). The unchanged
test passed locally against isolated PostgreSQL (1 test, 43.6 s);
inspection found an asynchronous company-pause/fixture-seeding timing
window. Original CI logs and local evidence are retained. The single
failed-job retry passed; all final-head CI gates are green. Both reviews
are 5/5 and no review threads remain unresolved. The first local
embedded-Postgres launch failed before tests during cluster
initialization; the isolated external-Postgres test passed, and its
disposable database was removed after correcting the private launcher
cleanup call.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/plans/2026-10-31-zero-downtime-web-deploys.md

@@ -1,9 +1,10 @@
 # Zero-downtime web deployments
 
-Date: 2026-10-31. Status (2026-11-01 14:54 UTC): existing-volume staging enrollment
+Date: 2026-10-31. Status (2026-11-01 15:39 UTC): existing-volume staging enrollment
 and manual sleep/wake preserve saved files and read/save behavior. The full local
-active-agent journey, natural database idling and ten-deploy staging campaign
-remain unqualified. No production canary yet.
+active-agent journey now passes on `cc23019691`; hosted browser continuity and
+the ten-deploy staging campaign remain unqualified. Natural database autosuspension
+passes after abandoned test-resource cleanup. No production canary yet.
 
 Core #18543 merged as `7751fc3107` after 55 successful CI checks, Greptile 5/5
 and no unresolved threads. It supplies runtime admission, immutable-cache
@@ -56,7 +57,51 @@ readiness. All 33 readiness, managed-skill and durable-file tests pass, with ser
 typecheck. The failed attempt remains failed evidence; its child credential,
 bucket and all three provider sandboxes were cleaned and checked. Review also
 replaced the whole agent-file manifest read with a scalar SQL digest projection.
-The next live attempt must exercise both replacements and later continuation.
+PR #18552 merged as `c0550e7010` with all required CI green, Greptile 5/5,
+and its scalar-query review findings resolved. The real `a15a12651c` attempt
+passed both web replacements: 765 reads, 78 saves and 78 verified saves with
+zero HTTP probe failures. The active execution survived once; the first three
+turns retained the native/provider session and accepted terminal contract.
+The fourth continuation FAILED: the retained Runner was safely suspended with
+an empty outbox and no pending terminal, but warm recovery attempted live-idle
+reattachment. Its provider state and diagnostics are preserved; all three owned
+sandboxes were independently confirmed absent. The next bounded fix reuses the
+existing per-turn suspended-receipt restoration for warm admission after cache
+loss, preserving physical sandbox binding and ordinary admission fences.
+A real PostgreSQL/Linux Runner regression reproduces the exact live error before
+the fix. Recovery then exposed a second stale-cache read: source reconciliation
+tried to stage new launch assets that the transport deliberately had not created.
+The existing remote process launcher now passes its structured launch identity
+to asset preparation, so reconciliation preserves the previous authority's
+retained launch files. Both new regressions pass: three turns preserve one
+provider thread after two cache deletions, including a failed remote state read
+and retry. All ten Linux continuity scenarios and 1,067 surrounding boundary tests pass.
+The exact `cc23019691` server build passes. PR #18554 has Greptile 5/5; CI and
+its second review identified a narrower dispatch boundary: warm receipt
+inspection must leave every non-suspended state to the existing live-adoption
+checks. A new preliminary-read regression fails before that refinement. The
+refinement plus both suspension regressions pass in a fresh Linux fixture, and
+server typecheck passes. The first retry reused a container whose deliberately
+failed pre-fix case left its live Runner on the ingress port; that infrastructure
+failure is preserved. The local wrapper now creates and cleans one container per
+invocation. This does not weaken the existing idle check or claim support for
+adopting a persistently backpressured Runner.
+
+The live Codex/Daytona attempt at that clean source PASSED on its first attempt,
+with no automatic retries: 518 reads, 53 saves and 53 verified saves across both
+replacements, zero probe failures, both complete predecessor homes deleted.
+The active execution resumes under the replacement and completes once. All four
+turns use the same native/provider session with accepted native terminal records.
+The final turn runs the saved project and returns the independently checked
+output. The original Git commit, untracked workspace reference, prior downloads
+and revised files survive. Browser reconnection remains explicitly unqualified
+in this API-led continuation case and must pass in the hosted campaign.
+Child credential, bucket and local test-home cleanup passed. Independent provider
+verification found two archived setup sandboxes left by the harness; those exact
+owned residuals were deleted separately and independently confirmed absent,
+preserving the failed automatic-cleanup verification. The product journey pass does not prove automatic fixture cleanup. The attempt's
+reported-plus-estimated model/runtime cost is about $0.243. Its immutable evidence
+is `zdt-live-2026-11-01T15-24-43-429Z` under the private live-evidence directory.
 
 Review also identified save quotas incorrectly applied during cache recovery.
 Recovery uses the existing inbound remote projection and preserves path/link
@@ -77,7 +122,13 @@ this is migration continuity evidence, not uninterrupted old-tab asset loading.
 
 Three eight-minute natural database-idle observations failed. A separate bounded SQL
 attribution window saw no other app-database sessions; it is diagnostic stimulus,
-not idle proof, and does not establish the source of provider activity. The third window, bracketed by closed SQL counter snapshots, recorded 36 transactions and small read deltas on environment setup, leases, environments and plugins; no app-row writes. The source is not yet attributed. Automatic
+not idle proof, and does not establish the source of provider activity. The third window, bracketed by closed SQL counter snapshots, recorded 36 transactions and small read deltas on environment setup, leases, environments and plugins; no app-row writes. A later scoped diagnostic found an abandoned ephemeral setup sandbox still
+archived at the provider despite a historical cleanup-success row. The normal
+verification worker revisited it every five minutes. After deleting that exact
+owned sandbox, the product independently stamped its destroyed-resource proof;
+no historical row or cleanup evidence was manually rewritten. The fresh provider-metadata-only observation passed: the same single volume-free
+web deployment remained serving while Neon automatically suspended after its
+five-minute idle timeout. No app HTTP or SQL was used during the observation. Automatic
 idle sleep was correctly refused because this development fixture is not eligible
 for the free/demo policy. Manual sleep with force disabled passed: the registry
 reports sleeping, Railway has no active web deployment or attached volume, and
@@ -151,9 +202,9 @@ Measure one-time setup separately from repeated attempts. Keep failed evidence.
 | Item | State | Next action |
 | --- | --- | --- |
 | Prior prerequisites | #18403 and all five storage workstreams merged | Retain the ephemeral filesystem contract |
-| Core implementation | #18518, #18526, #18529, #18530, #18531 and #18533 merged; local packaged migration, browser overlap, cold restoration and durable command proofs recorded below | Complete live remote execution, accounting and later continuation qualification |
+| Core implementation | #18518, #18526, #18529, #18530, #18531, #18533, #18543 and #18552 merged; #18554 reviewed at 5/5 with CI pending; local packaged migration, browser overlap, cold restoration and durable command proofs recorded below | Publish and qualify the exact fixed release after CI/review merge |
 | Cloud implementation | #1096, #1098, #1099, #1100, #1101 and #1102 merged; #1102 serving on staging | Verify repeated Railway overlap behavior after Core recovery qualification |
-| Staging | Original enrollment complete; migrated data and signing key preserved; volume detached and retained | Diagnose idle behavior, verify sleep/wake and repair, then qualify ten app-only replacements |
+| Staging | Original enrollment complete; migrated data and signing key preserved; volume detached and retained | Sleep/wake and natural database idle pass; verify repair and ten app-only replacements with browser continuity |
 | Production | No canary mutation performed | Proceed only after the staging and active-agent acceptance gates pass |
 
 Keep PRs, tested revisions, artifact identities, evidence paths, deployment targets

server/src/__tests__/helpers/remote-warm-controller.test-worker.ts

@@ -1,5 +1,5 @@
 import { createDb } from "@paperclipai/db";
-import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js";
+import { closeIdleWarmNativeSessionsForRestart, executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js";
 import { dockerCommandRunner } from "./docker-command-runner.js";
 
 // Separate TypeScript fixture import time from actual controller admission.
@@ -9,17 +9,34 @@ process.once("message", async (input: {
   connectionString: string;
   container: string;
   ingressUrl: string;
+  suspendAfterTurn?: boolean;
+  transientBackpressureInspection?: boolean;
   options: Parameters<typeof executePaperclipNativeSession>[0];
 }) => {
   const target = input.options.runnerExecutionTarget;
   if (target?.kind !== "remote") throw new Error("Expected remote worker target");
   try {
+    const runner = dockerCommandRunner(input.container);
+    let inspectionOverridden = false;
+    const execute = runner.execute.bind(runner);
+    runner.execute = async request => {
+      const result = await execute(request);
+      if (input.transientBackpressureInspection && !inspectionOverridden && result.exitCode === 0
+        && request.args?.some(arg => arg.includes("base64 <") && arg.includes("runner-state.json"))) {
+        // Only the preliminary read reports transient pressure. The subsequent
+        // live-adoption owner must still inspect the actual Runner state.
+        const state = JSON.parse(Buffer.from(result.stdout.replace(/\s+/g, ""), "base64").toString("utf8"));
+        inspectionOverridden = true;
+        return { ...result, stdout: Buffer.from(JSON.stringify({ ...state, lifecycle: "backpressure" })).toString("base64") };
+      }
+      return result;
+    };
     const result = await executePaperclipNativeSession({
       ...input.options,
       db: createDb(input.connectionString),
       runnerExecutionTarget: {
         ...target,
-        runner: dockerCommandRunner(input.container),
+        runner,
         getRunnerIngressEndpoint: async ({ path }) => {
           const endpoint = {
             kind: "authenticated_websocket" as const,
@@ -31,7 +48,11 @@ process.once("message", async (input: {
       },
       onLog: async (stream, text) => { process.send?.({ log: { stream, text } }); },
     });
-    process.send?.({ result });
+    if (input.suspendAfterTurn) {
+      const closed = await closeIdleWarmNativeSessionsForRestart();
+      if (closed.closed !== 1 || closed.busy || closed.failed) throw new Error("Warm fixture failed to suspend");
+    }
+    process.send?.({ result: { ...result, inspectionOverridden } });
   } catch (error) {
     process.send?.({ error: error instanceof Error ? error.stack : String(error) });
   }

server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts

@@ -358,10 +358,12 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
     expect(next.kind).toBe("bootstrap_incomplete");
   });
 
-  it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "legacy", "changed", "same-server", "diskless", "diskless-per-turn", "diskless-per-turn-read-retry"])("preserves remote warm continuity across completed turns: %s", async (scenario) => {
+  it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "legacy", "changed", "same-server", "diskless", "diskless-per-turn", "diskless-per-turn-read-retry", "diskless-warm-suspended", "diskless-warm-suspended-read-retry", "diskless-live-inspection"])("preserves remote warm continuity across completed turns: %s", async (scenario) => {
     const hardRestart = !["warm", "same-server"].includes(scenario);
     const diskless = scenario.startsWith("diskless");
     const perTurn = scenario.startsWith("diskless-per-turn");
+    const warmSuspended = scenario.startsWith("diskless-warm-suspended");
+    const resumedAfterSuspension = perTurn || warmSuspended;
     const container = process.env.PAPERCLIP_TEST_RUNNER_CONTAINER!;
     const tunnel = await createRunnerTestTlsTunnel({ port: Number(process.env.PAPERCLIP_TEST_RUNNER_PORT),
       key: process.env.PAPERCLIP_TEST_RUNNER_TLS_KEY!, cert: process.env.NODE_EXTRA_CA_CERTS! });
@@ -425,7 +427,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
       onLog: async (_stream: "stdout" | "stderr", text: string) => { logs.push(text); },
     };
     let controller: ReturnType<typeof spawn> | undefined;
-    const executeInController = (executionOptions: Parameters<typeof executePaperclipNativeSession>[0]) => new Promise<{ exitCode: number | null }>((resolveResult, reject) => {
+    const executeInController = (executionOptions: Parameters<typeof executePaperclipNativeSession>[0]) => new Promise<{ exitCode: number | null; inspectionOverridden?: boolean }>((resolveResult, reject) => {
         controller = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), resolve("server/src/__tests__/helpers/remote-warm-controller.test-worker.ts")],
           { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env });
         let timer = setTimeout(() => { controller?.kill("SIGKILL"); reject(new Error("Controller fixture imports exceeded 120 seconds")); }, 120_000);
@@ -437,7 +439,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
           if (message.ready) {
             clearTimeout(timer);
             timer = setTimeout(() => { controller?.kill("SIGKILL"); reject(new Error(`Warm controller admission exceeded 60 seconds: ${logs.join("").slice(-3_000)}`)); }, 60_000);
-            controller!.send?.({ connectionString: temporary.connectionString, container, ingressUrl: `wss://127.0.0.1:${tunnel.port}`,
+            controller!.send?.({ connectionString: temporary.connectionString, container, suspendAfterTurn: warmSuspended, transientBackpressureInspection: scenario === "diskless-live-inspection" && executionOptions.execution.binding.runId !== fixture.runId, ingressUrl: `wss://127.0.0.1:${tunnel.port}`,
               options: { ...executionOptions, db: undefined, onLog: undefined, runnerExecutionTarget: { ...executionOptions.runnerExecutionTarget, runner: undefined } } });
           }
           if (message.log) logs.push(message.log.text);
@@ -520,7 +522,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
       const snapshot = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] });
       expect(snapshot.exitCode).toBe(0);
       const saved = JSON.parse(snapshot.stdout);
-      expect(saved.lifecycle).toBe(perTurn ? "suspended" : "ready");
+      expect(saved.lifecycle).toBe(resumedAfterSuspension ? "suspended" : "ready");
       const processBefore = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] });
       expect(processBefore.exitCode).toBe(0);
       const historyBefore = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/filesystem/codex-home/fake-codex-state.json`] });
@@ -551,7 +553,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
         (controller as ReturnType<typeof spawn>).kill("SIGKILL");
         await exited;
         controller = undefined;
-        if (!perTurn) await expect(executePaperclipNativeSession({ ...nextOptions,
+        if (!resumedAfterSuspension) await expect(executePaperclipNativeSession({ ...nextOptions,
           runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget,
             sandboxLeaseAcquisition: { outcome: "replacement", providerLeaseId: "unrelated-sandbox" } },
         })).rejects.toThrow("prior_owner_terminal_state_indeterminate");
@@ -589,7 +591,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
           ...(perTurn ? { sessionCheckpoint: admission.checkpoint } : {}) } }).where(eq(heartbeatRuns.id, row.id));
       };
       await admitContinuation(nextOptions.execution);
-      if (scenario === "diskless-per-turn-read-retry") {
+      if (scenario.endsWith("-read-retry")) {
         let rejectedReads = 0;
         await expect(executePaperclipNativeSession({ ...nextOptions,
           runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget, runner: { ...runner,
@@ -608,6 +610,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
       }
       const second = diskless ? await executeInController(nextOptions) : await executePaperclipNativeSession(nextOptions);
       expect(second.exitCode).toBe(0);
+      if (scenario === "diskless-live-inspection") expect("inspectionOverridden" in second && second.inspectionOverridden).toBe(true);
       const historyAfter = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/filesystem/codex-home/fake-codex-state.json`] });
       const providerAfter = JSON.parse(historyAfter.stdout);
       expect(providerAfter.threadId).toBe(providerBefore.threadId);
@@ -617,10 +620,10 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
       expect(calls.stdout.split("\n").filter(line => line === "thread/start")).toHaveLength(1);
       const providerStarts = calls.stdout.split("\n").filter(line => line === "process-start").length;
       expect(providerStarts).toBeGreaterThanOrEqual(1);
-      expect(providerStarts).toBeLessThanOrEqual(perTurn || ["legacy", "changed"].includes(scenario) ? 3 : hardRestart ? 2 : 1);
+      expect(providerStarts).toBeLessThanOrEqual(resumedAfterSuspension || ["legacy", "changed"].includes(scenario) ? 3 : hardRestart ? 2 : 1);
       const processAfter = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] });
       expect(processAfter.exitCode).toBe(0);
-      if (perTurn || ["legacy", "changed"].includes(scenario)) expect(processAfter.stdout).not.toBe(processBefore.stdout);
+      if (resumedAfterSuspension || ["legacy", "changed"].includes(scenario)) expect(processAfter.stdout).not.toBe(processBefore.stdout);
       else expect(processAfter.stdout).toBe(processBefore.stdout);
       await finalizeNativeRun({ db: fixture.db, runId: nextRunId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
       if (diskless && controller) {
@@ -659,9 +662,9 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
       const finalCalls = await runner.execute({ command: "cat", args: [`${remoteCwd}/provider-calls.log`] });
       expect(finalCalls.stdout.split("\n").filter(line => line === "turn/start")).toHaveLength(3);
       expect(finalCalls.stdout.split("\n").filter(line => line === "thread/start")).toHaveLength(1);
-      expect(finalCalls.stdout.split("\n").filter(line => line === "process-start")).toHaveLength(providerStarts + (perTurn ? 1 : 0));
+      expect(finalCalls.stdout.split("\n").filter(line => line === "process-start")).toHaveLength(providerStarts + (resumedAfterSuspension ? 1 : 0));
       const finalProcess = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] });
-      if (perTurn) expect(finalProcess.stdout).not.toBe(processAfter.stdout);
+      if (resumedAfterSuspension) expect(finalProcess.stdout).not.toBe(processAfter.stdout);
       else expect(finalProcess.stdout).toBe(processAfter.stdout);
       expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: perTurn ? 0 : 1, busy: 0, failed: 0 });
       const suspended = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] });

server/src/services/native-runtime/native-session-executor.ts

@@ -5881,12 +5881,21 @@ async function restorePerTurnRunnerdController(input: {
       root, nativeSessionKey(input.execution));
     if (!restored && !identity) throw new Error("native_runner_checkpoint_missing");
   }
+  await restoreRemoteSuspendedRunnerCheckpoint(input, root);
+}
+
+/** The controller checkpoint and Runner lifecycle have separate authorities.
+ * Rehydrate a missing suspension receipt before choosing live warm adoption. */
+async function restoreRemoteSuspendedRunnerCheckpoint(input: {
+  db: Db; execution: NativeExecutionInput;
+  runnerExecutionTarget?: AdapterExecutionTarget | null;
+}, root: string): Promise<void> {
   const target = input.runnerExecutionTarget;
   const prior = readRunnerdDurableIdentity(root);
   // PRP and lifecycle evidence have different owners. A failed remote read can
   // leave a valid PRP projection, so its presence must not suppress this retry.
   // Never overwrite contradictory local Runner evidence with a remote receipt.
-  if (target.transport !== "sandbox" || target.sandboxLeaseAcquisition?.outcome !== "resumed"
+  if (target?.kind !== "remote" || target.transport !== "sandbox" || target.sandboxLeaseAcquisition?.outcome !== "resumed"
     || !durableIdentityMatchesSession(prior, input.execution) || prior.runId === input.execution.binding.runId
     || runnerdAuthorityLifecycle(root, prior) !== "absent") return;
   let state: Record<string, unknown>;
@@ -5903,6 +5912,9 @@ async function restorePerTurnRunnerdController(input: {
       createHash("sha256").update(nativeSessionKey(input.execution)).digest("hex"), "runner");
     state = await readRemoteRunnerState({ runner: target.runner, stateDirectory });
   }
+  // A running warm source keeps the existing live-adoption identity, idle,
+  // controller-fencing and physical-process checks. Only suspension is cached.
+  if (input.execution.session.lifecyclePolicy.mode === "warm" && state.lifecycle !== "suspended") return;
   if (state.lifecycle !== "suspended" || !Array.isArray(state.outbox) || state.outbox.length !== 0
     || state.pendingTerminalDelivery != null || state.pendingProviderCleanup != null || state.warmTransition != null) {
     throw new Error("native_controller_recovery_missing_evidence:runner_not_settled");
@@ -5924,7 +5936,10 @@ async function restoreWarmRunnerdController(input: {
   const identity = readRunnerdDurableIdentity(root);
   const reference = identity ?? readRunnerdDurableIdentity(root, true);
   if (!idleWarmNativeSessionOwnerRunId(input) && (!reference || !existsSync(nativeSessionCheckpointPath(input.execution)))) {
-    if (await restoreNativeControllerProjections(input.db, input.execution, root)) return;
+    if (await restoreNativeControllerProjections(input.db, input.execution, root)) {
+      await restoreRemoteSuspendedRunnerCheckpoint(input, root);
+      return;
+    }
   }
   if (!durableIdentityMatchesSession(reference, input.execution)) return;
   const retainedRunId = idleWarmNativeSessionOwnerRunId(input);
@@ -5945,11 +5960,13 @@ async function restoreWarmRunnerdController(input: {
       issueId = prior.issueId;
     }
   }
-  if (identity?.runId === runId) return;
-  const restored = await restoreNativeRunnerCheckpoint(input.db,
-    { companyId: input.execution.binding.companyId, issueId, runId }, root, nativeSessionKey(input.execution));
-  // A legacy file-backed root still follows its existing prior-owner checks.
-  if (!restored && !identity) throw new Error("native_runner_checkpoint_missing");
+  if (identity?.runId !== runId) {
+    const restored = await restoreNativeRunnerCheckpoint(input.db,
+      { companyId: input.execution.binding.companyId, issueId, runId }, root, nativeSessionKey(input.execution));
+    // A legacy file-backed root still follows its existing prior-owner checks.
+    if (!restored && !identity) throw new Error("native_runner_checkpoint_missing");
+  }
+  if (!retainedRunId) await restoreRemoteSuspendedRunnerCheckpoint(input, root);
 }
 
 async function restoreNativeControllerProjections(db: Db, execution: NativeExecutionInput, root: string): Promise<boolean> {
@@ -11884,7 +11901,7 @@ export function createRemoteRunnerProcessLauncher(input: {
   stateDirectory: string;
   diagnosticsDirectory: string;
   runnerInstanceId: string;
-  ensureArtifact?: () => Promise<void>;
+  ensureArtifact?: (spec: RunnerProcessLaunchSpec) => Promise<void>;
   requireProcessStartFingerprint?: boolean;
   onSpawn?: (meta: {
     pid: number;
@@ -11970,11 +11987,11 @@ export function createRemoteRunnerProcessLauncher(input: {
         if (input.trace) {
           await input.trace.measure(
             "runner.runtime.stage",
-            input.ensureArtifact,
+            () => input.ensureArtifact!(spec),
             { parentName: "runner.session.startup" },
           );
         } else {
-          await input.ensureArtifact();
+          await input.ensureArtifact(spec);
         }
       }
       if (pendingSignal) return cancelledBeforeDispatch();
@@ -13804,7 +13821,11 @@ async function createRunnerdBackendWithinSessionClaim(
           stateDirectory: remoteStateDirectory,
           diagnosticsDirectory: posix.join(remoteSessionRoot!, "diagnostics"),
           runnerInstanceId: input.runnerInstanceId,
-          ensureArtifact: ensureRemoteRunner,
+          // Source recovery relaunches the previous authority against its
+          // retained provider home. The transport intentionally has not created
+          // this turn's launch assets until that source is reconciled.
+          ensureArtifact: spec => ensureRemoteRunner(
+            !spec.nativeRunner || spec.nativeRunner.identity.runId === input.execution.binding.runId),
           onSpawn: input.onSpawn,
           onLog: input.onLog,
           trace: input.trace,