fix(server): complete volume-free controller admission and recovery (#18543)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its volume-free web server must admit remote warm execution without
depending on a local disk.
> - Fresh run rows retain the initial legacy label during workspace
preparation.
> - The native runtime selector commits the execution mode later.
> - The earlier guard rejected valid remote runs before that selection.
> - This PR checks storage during preparation and the resolved lifecycle
during runtime selection.
> - Agent-file recovery also has to reconstruct disposable cache paths
from the retained remote directory.
> - The deployment test removes the complete predecessor home, including
immutable caches, to enforce that contract.

## Linked Issues or Issue Description

Refs #18518, #18529, #18533.

**What happened?**

A real Codex/Daytona deployment test failed during setup because the
volume-free workspace guard checked the initial run label. After fixing
admission, controller retirement first failed on an immutable skill
cache. Removing that cache exposed a production recovery failure: B
rejected the saved agent-directory path from A, even though the remote
execution identity was unchanged. The next live attempt recovered that
directory and transferred the same runner, but adoption then tried to
recreate provider launch assets from the deleted controller home.

**Expected behavior**

Allow remote workspace preparation before native selection. Reject
actual legacy and per-turn execution at the runtime boundary. Delete the
entire stopped predecessor home, including immutable caches, without
following links to shared dependencies. B must reconstruct its local
agent-file projection from the exact registered remote lease while
preserving the save baseline and unsaved remote edits. Reattaching an
already-running provider must not prepare a new launch or require the
old launch files. The full backend must recover the admitted
instructions from the matching durable run checkpoint.

**Steps to reproduce**

Run the explicit
`rolling-web-deployment.runner-codex.daytona.recover-controller` Product
E2E cell with a complete server/UI build, disposable PostgreSQL/S3, and
an owned Daytona fixture. The new boundary regressions also reproduce
the premature guard and missing runtime rejection without a model call.

**Paperclip version or commit**

The failures were observed on
`247552fe0719dcc92f7b42c50b658be6f624986d`. This branch is based on
`e122478368`.

## What Changed

- Validate Runner and remote workspace authority before runtime
selection without trusting the initial run label.
- Reject resolved legacy execution and local-source fallback, preserving
the native warm-policy check before admission.
- Rebind remote agent-file caches on ephemeral controllers and retrieve
their validated contents without uploading over the retained execution.
Preserve pending saves when the remote collector is unavailable.
- Restore over-quota remote directories without applying save quotas to
run admission; ordinary saves still enforce quotas.
- Skip provider launch asset materialization when adopting an existing
Runner. A real-process regression deletes the old skill/instruction
sources before reattachment.
- Recover admitted system instructions through Runner-owned checkpoint
validation for the same run, session and context digest. Fresh launches
still compose current instructions.
- Make the attention-retention test advance past its database-assigned
deadline, fixing a clean-master clock mismatch.
- Serialize remote collection and cleanup across controllers with an
operation-scoped Postgres advisory lock; fence stale cache-path updates.
- Keep synthetic deployment probe tasks in a separate disposable company
so they cannot block the business completion oracle.
- Share the launcher's directory cleanup with rolling retirement. Reject
a symlink root and skip linked descendants.
- Test admission with real PostgreSQL transactions and cleanup with real
read-only directories and external symlinks.
- Update developer documentation and the canonical qualification record.
Broader deployment acceptance remains in progress.

## Verification

- `pnpm -r typecheck` and `pnpm build` passed.
- All 156 workspace preparation, runtime selection and ephemeral
contract tests passed. Two new regressions failed before the source fix.
- Seven focused Product E2E cleanup/replacement tests and `pnpm
test:e2e:runner:typecheck` passed.
- `pnpm test:run` stopped at an unrelated 15-second chat integration
timeout after 338 passes. That exact test passed in isolation without
increasing its timeout. The later full-suite shards did not run.
- `pnpm test:e2e:runner:unit` reported 5,898 passes and 13 failures.
Clean master reproduced 11 failures in unchanged onboarding
expectations, operating receipts, Pi catalog assertions, workflow
bootstrap counts and embedded PostgreSQL startup. The other two workflow
assertions were fixed by the intervening master commit. These failures
are retained, not counted as passes.
- All 122 workspace-preparation tests pass after the reviewed
host-fallback fix. The first broader attempt exposed a test environment
leak; restoring the ephemeral flag between cases resolved it.
- Agent-directory and instruction regressions passed 179 tests. Five
focused cache-loss cases pass: different home, same path on an empty
filesystem, foreign lease, concurrent controller, and missing collector.
- The live fixture at `a18f6b85a3` gracefully stopped A and deleted its
entire home, then failed the 420-second remote reattachment deadline on
the saved local agent-directory path. That failure is retained; exact
owned sandbox cleanup was independently confirmed. The next live attempt
at `e14ca5f3e9` restored the cache and transferred the same runner, but
failed while rematerializing launch assets from A. Both failures are
retained and their owned sandboxes confirmed absent. No active-overlap,
ten-deploy, or production-canary acceptance is claimed.
- All 71 related Runner adoption/recovery cases pass, including the new
removed-context regression (failed before the source fix).
- The revised full agent-directory suite passed 110 cases and failed
three cleanup timing cases. All 20 cleanup/cache-loss cases then passed
in isolation with unchanged timeouts, including over-quota recovery.
- Server and Runner TypeScript checks pass after the latest corrections;
the live retry remains pending.

- The live test at `67714af9ff` adopted the same remote execution after
A was deleted, then failed while the full backend read the retired
instruction bundle. All owned sandboxes were independently confirmed
absent. Its regression now passes with actual backend recovery and no
new thread or turn. All 59 backend factory tests and focused checkpoint
identity tests pass. Runner and server TypeScript checks pass; the exact
committed server build passes. A fresh paid-model retry followed.
- The attention-retention CI timeout reproduced on clean master. The
deadline correction passes all 53 tests in that file with unchanged
timeouts.

- The real `cd690de4d3` run transferred the same runner/session after
deleting A, completed, and completed the next revision. During handoff,
372 reads and 38 saves had zero failures; all 38 saves were verified.
The full attempt was interrupted because the probe's open task
contaminated the business company oracle. This remains incomplete
evidence. The scope regression fails before the fixture fix, then
passes; all nine deployment support tests and E2E typecheck pass. All
three owned sandboxes were confirmed absent and temporary
credentials/storage were cleaned up. A clean corrected attempt is
running.

## Risks

- The runtime guard must still reject legacy and per-turn execution
before dispatch. Tests cover these refusals and a valid resolved warm
policy.
- Directory permission changes apply only during cleanup of stopped,
test-owned homes. Traversal does not follow symlinks. This changes no
production cache permissions.
- Cache reconstruction must not change remote execution authority or
rebase the original save baseline. Tests verify foreign-lease rejection,
preserved receipts, and absence of local-to-remote upload.
- Postgres locking is scoped to active remote file work and adds no idle
polling; local/durable-server locking remains unchanged.
- No schema or public API changes. External conversation exemptions
remain unchanged.

## Model Used

OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution
and live integration-test tools. The exact runtime model variant and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Latest real qualification at f5915fdc33: both empty-home replacements
completed with 523 reads, 53 saves and 53 verified saves, zero probe
failures. The active remote execution survived and completed once; saved
project bytes and the later useful answer passed. The full workflow
remains FAILED: the later continuation reset its native/provider session
and the public run row lacked the native terminal contract. These newly
exposed compatibility/projection gaps are tracked in the next
implementation slice; this PR is not a claim of full Railway
qualification. All owned Daytona sandboxes were independently confirmed
absent after cleanup.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/services/native-runtime/native-session-executor.ts

@@ -145,6 +145,7 @@ import {
   executeNativeSession,
   applyNativeSessionGoalControl,
   inspectStagedRunAttachment,
+  inspectRetainedRunInstructions,
   inspectWarmRunTransition,
   parseNativeExecutionInput,
   parsePaperclipQuestionSet,
@@ -12545,6 +12546,19 @@ async function createRunnerdBackendWithinSessionClaim(
       : null;
   const stagedRecovery = input.restartRecovery?.kind === "reattach_remote_runner"
     ? await verifyStagedAttachmentRestart(input) : null;
+  const retainedSystemInstructions = input.restartRecovery?.kind === "reattach_remote_runner"
+    && "runtimeContext" in input.execution
+    && input.execution.schema !== "paperclip.native-execution-input.v6"
+    && input.execution.schema !== "paperclip.native-execution-input.v8"
+    ? inspectRetainedRunInstructions(JSON.parse(readBoundedNativeFile(
+        resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
+        "native_runner_recovery_instructions_unavailable").toString("utf8")), {
+        runId: input.execution.binding.runId,
+        normalizedSessionId: nativeSessionKey(input.execution),
+        runtimeContextDigest: input.execution.runtimeContext.aggregateDigest,
+      })
+    : undefined;
+  if (retainedSystemInstructions === null) throw new Error("native_runner_recovery_instructions_unavailable");
   let stagedRecoveryPending = stagedRecovery !== null;
   const recoveryUsageIdentity = stagedRecovery?.sourceIdentity ?? input.remoteWarmRecovery?.identity ?? durableIdentity;
   const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity
@@ -12647,6 +12661,7 @@ async function createRunnerdBackendWithinSessionClaim(
       return current.execute(call);
     };
     const backend = createNativeSessionBackend(input.execution, {
+      retainedSystemInstructions,
       // The plugin resolves the VM directory while preparing the launch. Core keeps
       // its host environment out of the plugin and records the prepared directory.
       runnerInstanceId: prior?.runnerInstanceId ?? effectiveRunnerInstanceId, environment: {},
@@ -13981,6 +13996,7 @@ async function createRunnerdBackendWithinSessionClaim(
       runId: input.execution.binding.runId, prepareRemote: prepareRemoteRunner });
   }
   const backend = createNativeSessionBackend(runnerExecution, {
+    retainedSystemInstructions,
     runnerInstanceId:
       retainedTransition?.runnerInstanceId ?? input.runnerInstanceId,
     environment: effectiveRunnerEnvironment,