fix(server): complete volume-free controller admission and recovery (#18543)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its volume-free web server must admit remote warm execution without depending on a local disk. > - Fresh run rows retain the initial legacy label during workspace preparation. > - The native runtime selector commits the execution mode later. > - The earlier guard rejected valid remote runs before that selection. > - This PR checks storage during preparation and the resolved lifecycle during runtime selection. > - Agent-file recovery also has to reconstruct disposable cache paths from the retained remote directory. > - The deployment test removes the complete predecessor home, including immutable caches, to enforce that contract. ## Linked Issues or Issue Description Refs #18518, #18529, #18533. **What happened?** A real Codex/Daytona deployment test failed during setup because the volume-free workspace guard checked the initial run label. After fixing admission, controller retirement first failed on an immutable skill cache. Removing that cache exposed a production recovery failure: B rejected the saved agent-directory path from A, even though the remote execution identity was unchanged. The next live attempt recovered that directory and transferred the same runner, but adoption then tried to recreate provider launch assets from the deleted controller home. **Expected behavior** Allow remote workspace preparation before native selection. Reject actual legacy and per-turn execution at the runtime boundary. Delete the entire stopped predecessor home, including immutable caches, without following links to shared dependencies. B must reconstruct its local agent-file projection from the exact registered remote lease while preserving the save baseline and unsaved remote edits. Reattaching an already-running provider must not prepare a new launch or require the old launch files. The full backend must recover the admitted instructions from the matching durable run checkpoint. **Steps to reproduce** Run the explicit `rolling-web-deployment.runner-codex.daytona.recover-controller` Product E2E cell with a complete server/UI build, disposable PostgreSQL/S3, and an owned Daytona fixture. The new boundary regressions also reproduce the premature guard and missing runtime rejection without a model call. **Paperclip version or commit** The failures were observed on `247552fe0719dcc92f7b42c50b658be6f624986d`. This branch is based on `e122478368`. ## What Changed - Validate Runner and remote workspace authority before runtime selection without trusting the initial run label. - Reject resolved legacy execution and local-source fallback, preserving the native warm-policy check before admission. - Rebind remote agent-file caches on ephemeral controllers and retrieve their validated contents without uploading over the retained execution. Preserve pending saves when the remote collector is unavailable. - Restore over-quota remote directories without applying save quotas to run admission; ordinary saves still enforce quotas. - Skip provider launch asset materialization when adopting an existing Runner. A real-process regression deletes the old skill/instruction sources before reattachment. - Recover admitted system instructions through Runner-owned checkpoint validation for the same run, session and context digest. Fresh launches still compose current instructions. - Make the attention-retention test advance past its database-assigned deadline, fixing a clean-master clock mismatch. - Serialize remote collection and cleanup across controllers with an operation-scoped Postgres advisory lock; fence stale cache-path updates. - Keep synthetic deployment probe tasks in a separate disposable company so they cannot block the business completion oracle. - Share the launcher's directory cleanup with rolling retirement. Reject a symlink root and skip linked descendants. - Test admission with real PostgreSQL transactions and cleanup with real read-only directories and external symlinks. - Update developer documentation and the canonical qualification record. Broader deployment acceptance remains in progress. ## Verification - `pnpm -r typecheck` and `pnpm build` passed. - All 156 workspace preparation, runtime selection and ephemeral contract tests passed. Two new regressions failed before the source fix. - Seven focused Product E2E cleanup/replacement tests and `pnpm test:e2e:runner:typecheck` passed. - `pnpm test:run` stopped at an unrelated 15-second chat integration timeout after 338 passes. That exact test passed in isolation without increasing its timeout. The later full-suite shards did not run. - `pnpm test:e2e:runner:unit` reported 5,898 passes and 13 failures. Clean master reproduced 11 failures in unchanged onboarding expectations, operating receipts, Pi catalog assertions, workflow bootstrap counts and embedded PostgreSQL startup. The other two workflow assertions were fixed by the intervening master commit. These failures are retained, not counted as passes. - All 122 workspace-preparation tests pass after the reviewed host-fallback fix. The first broader attempt exposed a test environment leak; restoring the ephemeral flag between cases resolved it. - Agent-directory and instruction regressions passed 179 tests. Five focused cache-loss cases pass: different home, same path on an empty filesystem, foreign lease, concurrent controller, and missing collector. - The live fixture at `a18f6b85a3` gracefully stopped A and deleted its entire home, then failed the 420-second remote reattachment deadline on the saved local agent-directory path. That failure is retained; exact owned sandbox cleanup was independently confirmed. The next live attempt at `e14ca5f3e9` restored the cache and transferred the same runner, but failed while rematerializing launch assets from A. Both failures are retained and their owned sandboxes confirmed absent. No active-overlap, ten-deploy, or production-canary acceptance is claimed. - All 71 related Runner adoption/recovery cases pass, including the new removed-context regression (failed before the source fix). - The revised full agent-directory suite passed 110 cases and failed three cleanup timing cases. All 20 cleanup/cache-loss cases then passed in isolation with unchanged timeouts, including over-quota recovery. - Server and Runner TypeScript checks pass after the latest corrections; the live retry remains pending. - The live test at `67714af9ff` adopted the same remote execution after A was deleted, then failed while the full backend read the retired instruction bundle. All owned sandboxes were independently confirmed absent. Its regression now passes with actual backend recovery and no new thread or turn. All 59 backend factory tests and focused checkpoint identity tests pass. Runner and server TypeScript checks pass; the exact committed server build passes. A fresh paid-model retry followed. - The attention-retention CI timeout reproduced on clean master. The deadline correction passes all 53 tests in that file with unchanged timeouts. - The real `cd690de4d3` run transferred the same runner/session after deleting A, completed, and completed the next revision. During handoff, 372 reads and 38 saves had zero failures; all 38 saves were verified. The full attempt was interrupted because the probe's open task contaminated the business company oracle. This remains incomplete evidence. The scope regression fails before the fixture fix, then passes; all nine deployment support tests and E2E typecheck pass. All three owned sandboxes were confirmed absent and temporary credentials/storage were cleaned up. A clean corrected attempt is running. ## Risks - The runtime guard must still reject legacy and per-turn execution before dispatch. Tests cover these refusals and a valid resolved warm policy. - Directory permission changes apply only during cleanup of stopped, test-owned homes. Traversal does not follow symlinks. This changes no production cache permissions. - Cache reconstruction must not change remote execution authority or rebase the original save baseline. Tests verify foreign-lease rejection, preserved receipts, and absence of local-to-remote upload. - Postgres locking is scoped to active remote file work and adds no idle polling; local/durable-server locking remains unchanged. - No schema or public API changes. External conversation exemptions remain unchanged. ## Model Used OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution and live integration-test tools. The exact runtime model variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest real qualification at f5915fdc33: both empty-home replacements completed with 523 reads, 53 saves and 53 verified saves, zero probe failures. The active remote execution survived and completed once; saved project bytes and the later useful answer passed. The full workflow remains FAILED: the later continuation reset its native/provider session and the public run row lacked the native terminal contract. These newly exposed compatibility/projection gaps are tracked in the next implementation slice; this PR is not a claim of full Railway qualification. All owned Daytona sandboxes were independently confirmed absent after cleanup. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 08:28:48 +01:00 · 2026-11-01 08:24:14 -06:00
1 parent e122478368
commit 7751fc3107
@@ -14,8 +14,8 @@ import os from "node:os";
 import path from "node:path";
 import { createHash, randomUUID } from "node:crypto";
 import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
-import { eq } from "drizzle-orm";
-import { agentFileStore, fileHash, inspectAgentFile, snapshotAgentFiles, MAX_AGENT_FILE_BYTES, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENTRIES } from "../services/agent-file-store.js";
+import { eq, sql } from "drizzle-orm";
+import { agentFileStore, fileHash, inspectAgentFile, inspectAgentDirectory, snapshotAgentFiles, MAX_AGENT_FILE_BYTES, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENTRIES } from "../services/agent-file-store.js";
 import { agents, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, environmentLeases, environments, agentInstructionWorkingCopies, agentInstructionRevisions, agentInstructionHeads, createDb, fileStorageDb } from "@paperclipai/db";
 import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
 import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js";
@@ -902,6 +902,87 @@ describe("persistent agent directories", () => {
    expect((await fs.stat(active.localRoot)).isDirectory()).toBe(true);
  });

+  it.each(["new-home", "same-path-empty", "foreign-lease", "other-controller", "missing-collector", "over-quota"] as const)("recovers remote agent files after controller cache loss: %s", async scenario => {
+    const copy = await run({ warm: true });
+    const environmentId = randomUUID(), leaseId = randomUUID(), remoteCwd = "/fixture/retained-task";
+    const executionRoot = path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", agentId, copy.runId);
+    await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" });
+    await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, heartbeatRunId: copy.runId,
+      provider: "daytona", providerLeaseId: "original-sandbox", status: "active", metadata: { remoteCwd } });
+    await db.update(agentInstructionWorkingCopies).set({ location: `remote:${environmentId}`, executionRoot,
+      receipt: { ...copy.receipt, cleanup: { leaseId, remoteCwd } } }).where(eq(agentInstructionWorkingCopies.runId, copy.runId));
+    const saved = (await copies.get(companyId, copy.runId))!;
+    const replacementHome = path.join(home, `replacement-${randomUUID()}`);
+    const remoteFiles = await fs.mkdtemp(path.join(home, "remote-checkpoint-"));
+    await fs.mkdir(path.join(remoteFiles, "files", "policy"), { recursive: true });
+    await fs.writeFile(path.join(remoteFiles, "files", entryFile), "Remote instructions changed during the turn");
+    await fs.writeFile(path.join(remoteFiles, "files", "unsaved.txt"), "Only the running sandbox has this");
+    await fs.rm(path.dirname(copy.localRoot), { recursive: true, force: true });
+    if (scenario === "over-quota") {
+      const handle = await fs.open(path.join(remoteFiles, "files", "too-large.bin"), "w");
+      try { await handle.truncate(MAX_AGENT_FILE_BYTES + 1); } finally { await handle.close(); }
+    }
+    const transport = vi.spyOn(executionTargetTools, "prepareAdapterExecutionTargetRuntime").mockImplementation(async options => ({
+      target: options.target!, workspaceRemoteDir: options.workspaceRemoteDir!, runtimeRootDir: options.runtimeRootDir!,
+      assetDirs: {}, additionalSourceDirs: {}, additionalSourceFailures: [], workspaceSyncSnapshot: null,
+      restoreWorkspace: async () => { await fs.cp(path.join(remoteFiles, "files"), options.workspaceLocalDir, { recursive: true }); },
+    }));
+    const ephemeralBefore = process.env.PAPERCLIP_EPHEMERAL_SERVER;
+    process.env.PAPERCLIP_EPHEMERAL_SERVER = "true";
+    if (scenario === "new-home") process.env.PAPERCLIP_HOME = replacementHome;
+    const remote = { kind: "remote" as const, transport: "sandbox" as const, environmentId,
+      leaseId: scenario === "foreign-lease" ? randomUUID() : leaseId, remoteCwd, runner: { execute: vi.fn() } };
+    try {
+      if (scenario === "missing-collector") {
+        const retained = await copies.collectStopped({ companyId, runId: copy.runId });
+        expect(retained).toMatchObject({ state: "pending_collection", errorCode: "INSTRUCTION_REMOTE_COLLECTOR_UNAVAILABLE", baseHash: saved.baseHash });
+        expect(retained?.receipt?.baseline).toEqual(saved.receipt?.baseline);
+        expect(retained?.nextAttemptAt).toBeInstanceOf(Date);
+        expect(transport).not.toHaveBeenCalled();
+        return;
+      }
+      if (scenario === "other-controller") {
+        await db.transaction(async tx => {
+          await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`agent-directory:${companyId}:${agentId}`}, 0))`);
+          await expect(copies.prepare({ ...target(), runId: copy.runId, cwd: remoteCwd, target: remote, warm: true }))
+            .rejects.toThrow("Agent directory operation is already active");
+          expect(transport).not.toHaveBeenCalled();
+        });
+        return;
+      }
+      const preparing = copies.prepare({ ...target(), runId: copy.runId, cwd: remoteCwd, target: remote, warm: true });
+      if (scenario === "foreign-lease") {
+        await expect(preparing).rejects.toThrow("Agent directory recovery lease changed");
+        expect(transport).not.toHaveBeenCalled();
+        return;
+      }
+      const recovered = (await preparing)!;
+      expect(recovered.executionRoot).toBe(executionRoot);
+      expect(recovered.baseHash).toBe(saved.baseHash);
+      expect(recovered.receipt).toEqual(saved.receipt);
+      if (scenario === "new-home") {
+        expect(recovered.localRoot.startsWith(replacementHome)).toBe(true);
+        await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
+      } else expect(recovered.localRoot).toBe(copy.localRoot);
+      expect(await fs.readFile(path.join(recovered.localRoot, "unsaved.txt"), "utf8")).toBe("Only the running sandbox has this");
+      expect(await fs.readFile(path.join(recovered.localRoot, entryFile), "utf8")).toBe("Remote instructions changed during the turn");
+      expect(transport).toHaveBeenNthCalledWith(1, expect.objectContaining({ adapterKey: "agent-files-recovery", workspaceRemoteDir: executionRoot,
+        workspaceBaseline: { exclude: [".paperclip-runtime"], entries: new Map() }, workspaceInboundMode: "adopt_remote" }));
+      if (scenario === "over-quota") {
+        expect((await fs.stat(path.join(recovered.localRoot, "too-large.bin"))).size).toBe(MAX_AGENT_FILE_BYTES + 1);
+        await expect(inspectAgentDirectory(recovered.localRoot)).rejects.toThrow();
+      }
+      expect(transport).toHaveBeenCalledWith(expect.objectContaining({ workspaceLocalDir: recovered.localRoot, workspaceInboundMode: "adopt_remote" }));
+      await copies.prepare({ ...target(), runId: copy.runId, cwd: remoteCwd, target: remote, warm: true });
+      expect(transport).toHaveBeenCalledTimes(2);
+    } finally {
+      process.env.PAPERCLIP_HOME = home;
+      if (ephemeralBefore === undefined) delete process.env.PAPERCLIP_EPHEMERAL_SERVER;
+      else process.env.PAPERCLIP_EPHEMERAL_SERVER = ephemeralBefore;
+      transport.mockRestore();
+    }
+  });
+
  it("retains remote cleanup across restart and failed retries until the original lease is cleaned", async () => {
    const copy = await run();
    const environmentId = randomUUID(), leaseId = randomUUID(), remoteCwd = "/fixture/task";
@@ -161,7 +161,10 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000;
    expect(await db.select().from(issueRelations).where(and(eq(issueRelations.issueId, child.id),
      eq(issueRelations.relatedIssueId, parent.id)))).toHaveLength(1);
    await settle(); expect(coordinator!.nextWakeAt()).not.toBeNull();
-    await vi.advanceTimersByTimeAsync(delay + 2000); await archived("review", child.id); await quiet();
+    // SQL assigns updatedAt using the database clock, independently of the fake
+    // scheduler clock. Cross the saved deadline rather than a fixed duration.
+    await vi.advanceTimersByTimeAsync(Math.max(0, child.updatedAt.getTime() + delay - Date.now()) + 2000);
+    await archived("review", child.id); await quiet();
    expect(await db.select().from(decisionArchiveNotificationOutbox).where(eq(decisionArchiveNotificationOutbox.sourceId, child.id))).toHaveLength(1);
  });

@@ -26,14 +26,15 @@ it("never connects the listener with migration privileges, another tenant, or th
 });
 it("rejects disk-dependent execution and instructions while retaining remote warm execution", () => {
  vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
-  const remote = { externalConversation: false, native: true, adapterType: "paperclip_runner", pluginTaskWorkspace: false, remoteAuthority: true };
+  const remote = { externalConversation: false, adapterType: "paperclip_runner", pluginTaskWorkspace: false, remoteAuthority: true };
  expect(() => assertEphemeralWorkspaceExecution(remote)).not.toThrow();
  expect(() => assertEphemeralWorkspaceExecution({ ...remote, remoteAuthority: false, pluginTaskWorkspace: true })).not.toThrow();
-  for (const patch of [{ native: false }, { remoteAuthority: false }, { adapterType: "process" }]) {
+  for (const patch of [{ remoteAuthority: false }, { adapterType: "process" }]) {
    expect(() => assertEphemeralWorkspaceExecution({ ...remote, ...patch })).toThrow("warm Paperclip Runner");
  }
  expect(() => assertEphemeralNativeLifecycle("warm")).not.toThrow();
  expect(() => assertEphemeralNativeLifecycle("per_turn")).toThrow("warm Paperclip Runner");
+  expect(() => assertEphemeralNativeLifecycle(undefined)).toThrow("warm Paperclip Runner");
  const agent = { id: "agent", companyId: "company", name: "Agent", adapterConfig: {} };
  expect(() => assertEphemeralAgentInstructions(agent)).not.toThrow();
  expect(() => assertEphemeralAgentInstructions({ ...agent, adapterConfig: { instructionsBundleMode: "external", instructionsRootPath: "/old-volume" } })).toThrow("managed agent instructions");
@@ -3,7 +3,7 @@ import fs from "node:fs/promises";
 import path from "node:path";
 import { isDeepStrictEqual } from "node:util";
 import { and, eq, inArray, isNull, sql } from "drizzle-orm";
-import { agents, environmentLeases, environments, agentInstructionWorkingCopies as copies, type Db } from "@paperclipai/db";
+import { agents, environmentLeases, environments, agentInstructionWorkingCopies as copies, withDedicatedDbConnection, type Db } from "@paperclipai/db";
 import { syncDirectoryToSsh, restoreWorkspaceFromSshExecution } from "@paperclipai/adapter-utils/ssh";
 import { prepareAdapterExecutionTargetRuntime, runAdapterExecutionTargetShellCommand, type AdapterExecutionTarget, type PreparedAdapterExecutionTargetRuntime } from "@paperclipai/adapter-utils/execution-target";
 import { withDirectoryMergeLock, directorySnapshotSha256, parseDirectorySnapshot, serializeDirectorySnapshot, type DirectoryMergeLockOperation } from "@paperclipai/adapter-utils/workspace-restore-merge";
@@ -18,6 +18,7 @@ import type { Environment, EnvironmentLease } from "@paperclipai/shared";
 import { agentDirectoryBaselineDigest, agentDirectoryProbeProgram, observeLocalAgentDirectory } from "./agent-directory-probe.js";
 import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
 import { cachedAgentFileManifest, captureAgentFileCheckpoint, checkpointBaseline, checkpointSnapshot, type AgentFileManifest } from "./agent-file-checkpoints.js";
+import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js";
 import { logger } from "../middleware/logger.js";

 type Copy = typeof copies.$inferSelect;
@@ -54,7 +55,16 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
    });
  }
  const ownerFile = (row: Copy) => path.join(path.dirname(row.localRoot), "owner.json");
+  const disposableRemoteCopy = (row: Copy) => process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && row.location.startsWith("remote:");
+  function projectionRoot(row: Copy): string {
+    const origin = String(row.receipt?.materializationRunId ?? row.receipt?.directoryRunId ?? row.runId);
+    if (!/^[a-zA-Z0-9_-]{1,128}$/.test(origin)) throw conflict("Invalid agent directory owner");
+    return path.join(path.dirname(resolveManagedInstructionsRoot({ id: row.agentId, companyId: row.companyId, name: "", adapterConfig: {} })), "file-sync", "runs", origin, "live");
+  }
  async function owns(row: Copy) {
+    // Remote ownership is durable. A marker on a discarded controller cannot
+    // establish ownership of the retained sandbox on its replacement.
+    if (disposableRemoteCopy(row)) return row.state !== "superseded" && !row.receipt?.retainedByRunId;
    if (row.receipt?.warm !== true) return true;
    return await fs.readFile(ownerFile(row), "utf8").then(s => JSON.parse(s).runId === row.runId).catch(error => error.code === "ENOENT" && row.processStoppedAt !== null);
  }
@@ -159,7 +169,7 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
      ? path.posix.join(input.target.remoteCwd, ".paperclip-runtime", "agent-files", input.agentId, origin)
      : localRoot;
    const location = input.target?.kind === "remote" ? `remote:${input.target.environmentId ?? ""}` : "local";
-    if (row && (row.localRoot !== localRoot || row.executionRoot !== executionRoot || row.agentId !== input.agentId || row.location !== location)) throw conflict("Agent directory belongs to a different execution environment");
+    if (row && ((!disposableRemoteCopy(row) && row.localRoot !== localRoot) || row.executionRoot !== executionRoot || row.agentId !== input.agentId || row.location !== location)) throw conflict("Agent directory belongs to a different execution environment");
    if (row && !serialHeld) {
      // Restart preparation can replace a completed copy under the same run ID.
      // Join cleanup's lock before resetting its receipt or creating new bytes.
@@ -168,7 +178,10 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
      return serial(row, () => prepareCopy(input, true), "agent_directory_prepare");
    }
    if (row && !completed.has(row.state) && row.state !== "preparing") {
-      if (input.target?.kind === "remote" && !transports.has(key(row))) transports.set(key(row), await transport(row, input.target, true));
+      if (input.target?.kind === "remote") {
+        if (disposableRemoteCopy(row)) await restoreProjection(row, input.target);
+        if (!transports.has(key(row))) transports.set(key(row), await transport(row, input.target, true));
+      }
      return row;
    }
    // Register ownership before creating any bytes, so a crash during the copy
@@ -241,6 +254,39 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
    }
  }

+  async function restoreProjection(row: Copy, target: AdapterExecutionTarget) {
+    const exists = await fs.lstat(row.localRoot).then(stat => {
+      if (!stat.isDirectory() || stat.isSymbolicLink()) throw conflict("Agent directory projection is not a directory");
+      return true;
+    }, error => {
+      if (error.code === "ENOENT") return false;
+      throw error;
+    });
+    if (exists) return;
+    const lease = await ownedRemoteLease(row);
+    const cleanup = row.receipt?.cleanup as { leaseId?: string; remoteCwd?: string } | undefined;
+    if (target.kind !== "remote" || !lease || target.leaseId !== lease.id || lease.status !== "active"
+      || lease.releasedAt || hasRemoteTerminationReceipt(lease) || cleanup?.remoteCwd !== target.remoteCwd
+      || row.location !== `remote:${target.environmentId}`) throw conflict("Agent directory recovery lease changed");
+    // Reconstruct a private mirror without applying save quotas to admission.
+    // An empty merge baseline downloads all remote bytes without uploading any;
+    // the original save baseline remains unchanged in Postgres.
+    await fs.mkdir(row.localRoot, { recursive: true, mode: 0o700 });
+    let projection: PreparedAdapterExecutionTargetRuntime | undefined;
+    try {
+      projection = await prepareAdapterExecutionTargetRuntime({ target, runId: row.runId, adapterKey: "agent-files-recovery",
+        workspaceLocalDir: row.localRoot, workspaceRemoteDir: row.executionRoot,
+        runtimeRootDir: sandboxTransferRoot(row, target.remoteCwd), syncWorkspace: true,
+        workspaceInboundMode: "adopt_remote", workspaceBaseline: { exclude: [".paperclip-runtime"], entries: new Map() },
+        workspaceGitSnapshot: null, workspaceRepositories: [], workspaceFileMode: "all", workspaceExclude: [".paperclip-runtime", ".paperclip-runtime/**"] });
+      await projection.restoreWorkspace();
+      await inspectAgentDirectory(row.localRoot, false);
+      if (row.receipt?.warm === true) await fs.writeFile(ownerFile(row), JSON.stringify({ runId: row.runId }));
+    } catch (error) {
+      await fs.rm(row.localRoot, { recursive: true, force: true });
+      throw error;
+    } finally { await projection?.cleanupWorkspaceSnapshot?.(); }
+  }
  async function retrieve(row: Copy, target?: AdapterExecutionTarget | null) {
    if (row.location !== "local") {
      let runtime = transports.get(key(row));
@@ -356,6 +402,14 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
  }
  async function checkpoint(row: Copy, target?: AdapterExecutionTarget | null, stopped = false): Promise<Copy> {
    if (!await owns(row) || row.state === "superseded") return row;
+    target ??= transports.get(key(row))?.target;
+    if (row.location !== "local" && target?.kind !== "remote") {
+      // A missing controller cache is not the remote directory. Retain the
+      // pending save until its exact remote collector or destruction proof exists.
+      return patch(row, { ...(stopped ? { state: "pending_collection", processStoppedAt: row.processStoppedAt ?? new Date() } : {}),
+        errorCode: "INSTRUCTION_REMOTE_COLLECTOR_UNAVAILABLE", errorMessage: "The remote agent directory is retained; its collector is unavailable. No save or discard is claimed.",
+        nextAttemptAt: stopped ? new Date(Date.now() + 30_000) : null });
+    }
    let failure: unknown;
    for (let attempt = 0; attempt < 3; attempt++) {
      let captured: Awaited<ReturnType<typeof captureAgentFileCheckpoint>> | undefined;
@@ -575,13 +629,29 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
    return result.exitCode === 0 && !result.timedOut;
  }
  async function serial<T>(row: Copy, fn: (current: Copy) => Promise<T>, operation: DirectoryMergeLockOperation): Promise<T> {
-    // Duplicate stop callbacks and restart recovery must not race while moving
-    // the stopped working copy. This lock is outside the writable tree.
-    return withDirectoryMergeLock(path.resolve(row.localRoot, "../../.."), async () => {
-      const current = await get(row.companyId, row.runId);
-      if (!current) throw notFound("Agent directory copy not found");
-      return fn(current);
-    }, process.env, operation);
+    const run = async () => {
+      const lockRoot = path.resolve(disposableRemoteCopy(row) ? projectionRoot(row) : row.localRoot, "../../..");
+      if (disposableRemoteCopy(row)) await fs.mkdir(lockRoot, { recursive: true, mode: 0o700 });
+      return withDirectoryMergeLock(lockRoot, async () => {
+        let current = await get(row.companyId, row.runId);
+        if (!current) throw notFound("Agent directory copy not found");
+        if (disposableRemoteCopy(current) && current.localRoot !== projectionRoot(current)) {
+          current = await patch(current, { localRoot: projectionRoot(current) });
+          if (current.localRoot !== projectionRoot(current)) throw conflict("Agent directory projection changed");
+          transports.delete(key(current));
+        }
+        return fn(current);
+      }, process.env, operation);
+    };
+    if (!disposableRemoteCopy(row)) return run();
+    // Different controllers have different cache roots. Serialize their active
+    // collection/cleanup operations in Postgres, without holding row locks over I/O.
+    return withDedicatedDbConnection(db, dedicated => dedicated.transaction(async tx => {
+      const [lock] = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`agent-directory:${row.companyId}:${row.agentId}`}, 0)) as acquired,
+        (select setting::bigint from pg_settings where name = 'idle_in_transaction_session_timeout') as idle_timeout_ms`);
+      if (lock?.acquired !== true) throw conflict("Agent directory operation is already active");
+      return withWorkspaceLockKeepalive(tx as unknown as Db, Number(lock.idle_timeout_ms), run);
+    }));
  }
  return { prepare: (input: Parameters<typeof prepareCopy>[0]) => prepareCopy(input), hasChanges, adopt, canReuse, recoverUnavailable,
    recoverStoppedRemote: (row: Copy) => serial(row, current => current.receipt?.retainedByRunId ? Promise.resolve(current) : recoverStoppedRemote(current), "agent_directory_collect"),
@@ -121,7 +121,7 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
      if (signalsCleanup) await notifyDirectoryCleanup(tx, { companyId: row.companyId, runId: row.runId });
      if (signalsCollection) await notifyInstructionCollection(tx, { companyId: row.companyId, runId: row.runId });
      const [updated] = await tx.update(copies).set({ ...values, updatedAt: new Date() }).where(and(
-        scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash),
+        scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash), eq(copies.localRoot, row.localRoot),
        sql`${copies.receipt}->>'retainedByRunId' IS NOT DISTINCT FROM ${typeof row.receipt?.retainedByRunId === "string" ? row.receipt.retainedByRunId : null}`,
        requireDue ? or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())) : undefined, condition,
      )).returning();
@@ -328,6 +328,17 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
    expect(await coordinators(run.id)).toHaveLength(0);
  });

+  it("rejects resolved legacy execution on an ephemeral server before runtime admission", async () => {
+    const { input, run } = await fixture("process");
+    vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
+    expect(input.nativeRuntimeResolution.kind).toBe("legacy");
+    await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("warm Paperclip Runner");
+    expect((await read(run.id)).runtimeMode).toBe("legacy");
+    expect(await coordinators(run.id)).toHaveLength(0);
+    expect(input.onNativeLifecycleSelected).not.toHaveBeenCalled();
+    expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
+  });
+
  it.each(["warm", "per_turn"] as const)("checks resolved %s policy for ephemeral admission", async mode => {
    const { input, run } = await fixture();
    vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
@@ -267,6 +267,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
  const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config;
  const runOptions = input.recovery;
  const { providerTraceRequested, providerTraceCapture } = input.trace;
+  if (nativeRuntimeResolution.kind !== "native" && !isDotRun) assertEphemeralNativeLifecycle(undefined);
  let nativeExecution: NativeExecutionInput | null = null;
  let getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
  let nativeRunnerInstanceId: string | null = null;
@@ -52,7 +52,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
  let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
  let db: Db;
  let home: string;
-  const persistenceKeys = ["DATABASE_URL", "PAPERCLIP_STORAGE_PROVIDER", "PAPERCLIP_SECRETS_MASTER_KEY", "PAPERCLIP_SECRETS_MASTER_KEY_FILE"];
+  const persistenceKeys = ["PAPERCLIP_EPHEMERAL_SERVER", "DATABASE_URL", "PAPERCLIP_STORAGE_PROVIDER", "PAPERCLIP_SECRETS_MASTER_KEY", "PAPERCLIP_SECRETS_MASTER_KEY_FILE"];
  let persistenceBefore = new Map<string, string | undefined>();
  const configureRemotePersistence = () => Object.assign(process.env, { DATABASE_URL: database.connectionString,
    PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture" });
@@ -170,14 +170,15 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
  });

  it.each([
-    { kind: "empty", repoUrl: null, branchMode: null, fresh: false },
+    { kind: "empty", repoUrl: null, branchMode: null, fresh: false, ephemeral: true },
    { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: null, fresh: false },
    { kind: "git", repoUrl: "ssh://git@github.com/example/remote.git", branchMode: null, fresh: false },
    { kind: "git", repoUrl: "git@github.com:example/remote.git", branchMode: null, fresh: false },
    { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "existing", fresh: false },
    { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: false },
    { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true },
-  ] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh) without materializing a controller checkout", async ({ kind, repoUrl, branchMode, fresh }) => {
+  ] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh) without materializing a controller checkout", async (scenario) => {
+    const { kind, repoUrl, branchMode, fresh } = scenario;
    const { input, source, issue, envOrchestrator } = await fixture();
    const [savedEnvironment] = await db.insert(environments).values({ name: `Remote ${randomUUID()}`,
      driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning();
@@ -227,10 +228,13 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
      throw new Error("reached remote acquisition");
    });
    const configured = { DATABASE_URL: database.connectionString, PAPERCLIP_STORAGE_PROVIDER: "s3",
-      PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture" };
+      PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture",
+      PAPERCLIP_EPHEMERAL_SERVER: "ephemeral" in scenario ? "true" : "false" };
    const previous = new Map(Object.keys(configured).map(key => [key, process.env[key]]));
    Object.assign(process.env, configured);
    try {
+      // Fresh runs are still marked legacy until the runtime selector owns promotion.
+      expect(input.run.runtimeMode).toBe("legacy");
      await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached remote acquisition");
      expect(resolve).not.toHaveBeenCalled();
      expect(repositoryPreparation).not.toHaveBeenCalled();
@@ -268,7 +272,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
    expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
  });

-  it("auto-enrollment leaves a configured local folder on the host authority", async () => {
+  it.each([false, true])("configured local folders retain host authority only on durable servers (ephemeral=%s)", async ephemeral => {
    const { input, envOrchestrator } = await fixture();
    input.agent.adapterType = "paperclip_runner";
    input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
@@ -277,8 +281,11 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
    const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
    configureRemotePersistence();
    const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("local folder retained"));
-    await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("local folder retained");
-    expect(resolve).toHaveBeenCalledOnce();
+    process.env.PAPERCLIP_EPHEMERAL_SERVER = String(ephemeral);
+    await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(ephemeral
+      ? "Volume-free servers require warm Paperclip Runner execution with remote workspace storage." : "local folder retained");
+    if (ephemeral) expect(resolve).not.toHaveBeenCalled();
+    else expect(resolve).toHaveBeenCalledOnce();
    expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
    expect(acquire).not.toHaveBeenCalled();
  });
@@ -241,7 +241,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
      && !projectExecutionWorkspacePolicy?.storageProviderPluginId,
    existingWorkspace: reusableExistingExecutionWorkspace ?? existingExecutionWorkspace,
  });
-  assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, native: run.runtimeMode === "native",
+  assertEphemeralWorkspaceExecution({ externalConversation: isDotRun,
    adapterType: agent.adapterType,
    pluginTaskWorkspace, remoteAuthority });
  const assertRemotePreparationOwned = run.runtimeMode === "native" ? () => assertNativeSessionPreparationAuthority({
@@ -322,7 +322,13 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
                  issue: issueRef, agent, projectId: issueRef!.projectId, repoRef: null,
                })) : null),
        });
-        if (sourceState.authority === "host") remoteAuthority = false;
+        if (sourceState.authority === "host") {
+          remoteAuthority = false;
+          // Source resolution can retain a configured host folder. Check the
+          // resolved authority before materializing it or acquiring a sandbox.
+          assertEphemeralWorkspaceExecution({ externalConversation: isDotRun,
+            adapterType: agent.adapterType, pluginTaskWorkspace, remoteAuthority });
+        }
        else {
          remoteSourceState.value = sourceState;
          if (!enrolledRemoteAuthority) await assertRemotePreparationOwned();
@@ -145,6 +145,7 @@ import {
  executeNativeSession,
  applyNativeSessionGoalControl,
  inspectStagedRunAttachment,
+  inspectRetainedRunInstructions,
  inspectWarmRunTransition,
  parseNativeExecutionInput,
  parsePaperclipQuestionSet,
@@ -12545,6 +12546,19 @@ async function createRunnerdBackendWithinSessionClaim(
      : null;
  const stagedRecovery = input.restartRecovery?.kind === "reattach_remote_runner"
    ? await verifyStagedAttachmentRestart(input) : null;
+  const retainedSystemInstructions = input.restartRecovery?.kind === "reattach_remote_runner"
+    && "runtimeContext" in input.execution
+    && input.execution.schema !== "paperclip.native-execution-input.v6"
+    && input.execution.schema !== "paperclip.native-execution-input.v8"
+    ? inspectRetainedRunInstructions(JSON.parse(readBoundedNativeFile(
+        resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
+        "native_runner_recovery_instructions_unavailable").toString("utf8")), {
+        runId: input.execution.binding.runId,
+        normalizedSessionId: nativeSessionKey(input.execution),
+        runtimeContextDigest: input.execution.runtimeContext.aggregateDigest,
+      })
+    : undefined;
+  if (retainedSystemInstructions === null) throw new Error("native_runner_recovery_instructions_unavailable");
  let stagedRecoveryPending = stagedRecovery !== null;
  const recoveryUsageIdentity = stagedRecovery?.sourceIdentity ?? input.remoteWarmRecovery?.identity ?? durableIdentity;
  const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity
@@ -12647,6 +12661,7 @@ async function createRunnerdBackendWithinSessionClaim(
      return current.execute(call);
    };
    const backend = createNativeSessionBackend(input.execution, {
+      retainedSystemInstructions,
      // The plugin resolves the VM directory while preparing the launch. Core keeps
      // its host environment out of the plugin and records the prepared directory.
      runnerInstanceId: prior?.runnerInstanceId ?? effectiveRunnerInstanceId, environment: {},
@@ -13981,6 +13996,7 @@ async function createRunnerdBackendWithinSessionClaim(
      runId: input.execution.binding.runId, prepareRemote: prepareRemoteRunner });
  }
  const backend = createNativeSessionBackend(runnerExecution, {
+    retainedSystemInstructions,
    runnerInstanceId:
      retainedTransition?.runnerInstanceId ?? input.runnerInstanceId,
    environment: effectiveRunnerEnvironment,
@@ -76,11 +76,13 @@ export function shouldUseRemoteWorkspaces(input: {
 * remote controller may replace its local projection; host-owned execution
 * cannot make that promise. External conversation providers own no task disk. */
 export function assertEphemeralWorkspaceExecution(input: {
-  externalConversation: boolean; native: boolean; adapterType: string;
+  externalConversation: boolean; adapterType: string;
  pluginTaskWorkspace: boolean; remoteAuthority: boolean;
 }) {
  if (process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true" || input.externalConversation) return;
-  if (!input.native || input.adapterType !== "paperclip_runner"
+  // Runtime selection has not promoted a fresh run yet. Validate storage here;
+  // the runtime selector separately admits only its resolved warm native policy.
+  if (input.adapterType !== "paperclip_runner"
    || (!input.pluginTaskWorkspace && !input.remoteAuthority)) {
    throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.",
      { code: "ephemeral_execution_configuration_required" });
@@ -89,7 +91,7 @@ export function assertEphemeralWorkspaceExecution(input: {

 /** The runtime selector calls this after environment, credentials and saved
 * execution policy have resolved. Raw agent settings are not lifecycle authority. */
-export function assertEphemeralNativeLifecycle(mode: "warm" | "per_turn") {
+export function assertEphemeralNativeLifecycle(mode: "warm" | "per_turn" | undefined) {
  if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && mode !== "warm") {
    throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.",
      { code: "ephemeral_execution_configuration_required" });
@@ -95,6 +95,7 @@ export const RUNNER_PRP_PROTOCOL_MAX = runner.RUNNER_PRP_PROTOCOL_MAX;
 export const runnerCodexDynamicToolsFit = runner.runnerCodexDynamicToolsFit;
 export const inspectWarmRunTransition = runner.inspectWarmRunTransition;
 export const inspectStagedRunAttachment = runner.inspectStagedRunAttachment;
+export const inspectRetainedRunInstructions = runner.inspectRetainedRunInstructions;
 export const inspectStagedRunAttachmentHeader = runner.inspectStagedRunAttachmentHeader;
 export type StagedRunAttachment = import("@paperclipai/paperclip-runner").StagedRunAttachment;
 export const readRunnerdArtifactBinding = runner.readRunnerdArtifactBinding;