fix(remote-workspaces): keep retained runner state when replacing a retired root (#19251)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces save a task's files as snapshots. Runtime state
under `.paperclip-runtime` is never in a snapshot, because it belongs to
the sandbox
> - #19231 lets a parent task on a retained sandbox restore newer child
work over its own retired workspace root
> - That replace removed the whole retired root, including the parent's
retained runner session state under
`.paperclip-runtime/paperclip-runner/`
> - The parent's native continuation then reads that runner state on the
resumed sandbox, cannot find it, and fails with
`runner_remote_state_unavailable`
> - This pull request moves the retired root's runtime state into the
restored root before it removes the retired root
> - The benefit is that delegating parents on retained sandboxes
continue their session with the children's saved work

## Linked Issues or Issue Description

Refs #19231

**What happened?**

After #19231, a parent task on an agent with a retained sandbox
delegates two subtasks. Both subtasks finish. The parent's wake run now
restores the newer saved work successfully, but the run fails right
after with `runner_remote_state_unavailable: exit=1 timedOut=false`. The
run error code is `native_continuation_requires_reconciliation` and the
task becomes `blocked`. In one staging wave, 4 of 4 such parents failed.

Root cause: the native continuation on a resumed sandbox reads
`<remoteCwd>/.paperclip-runtime/paperclip-runner/sessions/<session>/runner/runner-state.json`.
Snapshots exclude `.paperclip-runtime`, so the restored root has only
the new restore receipt. #19231 parked the retired root and then removed
it, so the runner state was removed too.

**Expected behavior**

The restored root has the newest saved files and the sandbox's retained
runtime state. The parent's continuation reads its runner state and
continues.

**Steps to reproduce**

1. Prepare a remote workspace on sandbox A. Write files and runner state
under `.paperclip-runtime/paperclip-runner/`. Save.
2. Make a second run the writer on sandbox B. It restores, writes new
files, and saves.
3. Make a third run the writer on sandbox A again and prepare. The
restore replaces the retired root.
4. Read the runner state with `readRemoteRunnerState`. Before this
change, it fails with `runner_remote_state_unavailable: exit=1
timedOut=false`.

**Paperclip version or commit**

`master` at `26391707fb`, which includes #19231.

## What Changed

- `server/src/services/remote-workspace-transport.ts`: before it removes
the retired root, the restore script moves each entry of the retired
`.paperclip-runtime` into the restored root's `.paperclip-runtime`. It
does not move `workspace-ready.json`, `workspace-restored.json`, or
`workspace-snapshots`, and it does not replace an entry that already
exists. The move is a rename, so it keeps sockets and large session
state. The script also finishes an interrupted carry when the same
restore repeats. Only an authorized replace with that restore identity
creates the `<root>.retired-<identity>` path.
- `server/src/services/remote-workspaces.integration.test.ts`: the
retained-sandbox regression now writes parent runner state, restores
newer child work, and reads the state with the same
`readRemoteRunnerState` call that the continuation uses. It also covers
an interrupted carry on a repeated restore.
- `doc/execution-semantics.md`: document that runtime state moves with
the restored root.

## Verification

- Red before the fix: `npx vitest run
src/services/remote-workspaces.integration.test.ts -t "retired root"`
failed in the `own retired root`, `own restored root`, and `read-only
folder` cases with `Error: runner_remote_state_unavailable: exit=1
timedOut=false`. This is the same error as the staging failure.
- Green after the fix: all nine cases pass. The continuation read
returns the parent's runner state. The old `workspace-ready.json` is
gone and the new restore receipt is present. A repeated restore finishes
an interrupted carry and removes the retired root and its stale files.
- `npx vitest run src/services/remote-workspaces.integration.test.ts
src/services/remote-workspace-transport.test.ts
src/services/heartbeat/run-completion.test.ts`: 250 passed.
- Staging evidence (read-only): the failed parent wake run recorded a
succeeded `remote_workspace_restore` at the parent's own provider lease,
and then failed in the runner-state read.
- I did not run a live staging run. This PR does not deploy.

## Risks

- Runtime state from the retired root now persists into the restored
root. The state is the sandbox's own state for this workspace path,
which is the same state that a same-allocation reuse keeps.
- If the restored root already has an entry with the same name, the old
entry is not moved and is removed with the retired root.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. It used tool use for
code search, shell, test runs, and read-only staging inspection in the
browser.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/execution-semantics.md

@@ -1066,7 +1066,11 @@ recorded use of that root on this sandbox is an earlier epoch that ended in a
 quiesced, committed final save, and the folder carries a Paperclip preparation or
 restore receipt. Its files are then already part of the saved lineage being
 restored. Any other occupant (no receipt, a later unsaved use, or a sandbox that
-never saved that root) is still refused with `reason=occupied_root`.
+never saved that root) is still refused with `reason=occupied_root`. The replaced
+root's `.paperclip-runtime` state, such as a retained runner session, is never in
+a snapshot and belongs to the sandbox, so it moves into the restored root. Only the
+old preparation or restore receipt and export staging are dropped. A retained
+native continuation then reads its runner state at the same path.
 
 The provider's own idle auto-stop must not coincide with the warm runner idle
 window. The host forwards the run's effective warm idle window (environment

server/src/services/remote-workspace-transport.ts

@@ -500,10 +500,20 @@ export async function restoreRemoteWorkspaceDirectory(input: {
     return false;
    };
    const parked=root+'.allocation-'+identity,retired=root+'.retired-'+identity;
+   // Runtime state beside the files (for example a retained runner session) is
+   // never saved; it belongs to this sandbox. Move it into the restored root
+   // before removing the retired one. Its old receipts and export staging stay behind.
+   // Only an authorized replace with this identity creates that path, so a repeat
+   // of the same restore finishes an interrupted carry without a new proof.
    // Saved directory modes can be read-only. Make the parked tree's own directories
    // owner-writable without following symlinks, then remove it.
    const removeRetired=async()=>{if(!await stat(retired))return;
-    const open=async d=>{const s=await fs.lstat(d);if(!s.isDirectory()||s.isSymbolicLink())return;
+    stage='carry_runtime';const from=path.join(retired,'.paperclip-runtime'),to=path.join(root,'.paperclip-runtime');
+    if(await stat(from)){await directory(from);await directory(to);
+     for(const name of await fs.readdir(from)){
+      if(['workspace-ready.json','workspace-restored.json','workspace-snapshots'].includes(name)||await stat(path.join(to,name)))continue;
+      await fs.rename(path.join(from,name),path.join(to,name));}}
+    stage='cleanup_parked';const open=async d=>{const s=await fs.lstat(d);if(!s.isDirectory()||s.isSymbolicLink())return;
      await fs.chmod(d,(s.mode&0o7777)|0o700);for(const name of await fs.readdir(d))await open(path.join(d,name));};
     await open(retired);await fs.rm(retired,{recursive:true,force:true});};
    stage='allocation';if(await stat(parked)){if(allowEmpty!=='true')throw Error('unexpected_allocation');await allocation(parked);if(!await stat(root))await fs.rename(parked,root);}
@@ -515,7 +525,7 @@ export async function restoreRemoteWorkspaceDirectory(input: {
     const receipt=JSON.parse(await fs.readFile(marker,'utf8'));
     if(receipt.identity===identity&&receipt.manifestDigest===digest){
      if(await stat(parked)){await allocation(parked);await fs.rm(parked,{recursive:true});}
-     if(replaceRetired==='true')await removeRetired();return;}
+     await removeRetired();return;}
     if(replaceRetired!=='true')throw Error('workspace_owner_changed');
    }
    stage='inspect_root';
@@ -537,7 +547,7 @@ export async function restoreRemoteWorkspaceDirectory(input: {
      if(await stat(holding))throw Error('occupied_parking');await fs.rename(root,holding);}
     try{await fs.rename(staged,root);}catch(e){if(prior&&!await stat(root))await fs.rename(holding,root);throw e;}
     stage='cleanup_parked';if(prior&&!retire){await allocation(parked);await fs.rm(parked,{recursive:true});}
-    if(replaceRetired==='true')await removeRetired();
+    await removeRetired();
    }finally{await fs.rm(staged,{recursive:true,force:true});}
   })().catch(error=>{process.stderr.write(setupFailure('remote_workspace_restore_failed',stage,error));process.exitCode=1});`;
   await command(input.target, "restore", ["-e", restore, staged.helper, staged.archive, input.remoteCwd,

server/src/services/remote-workspaces.integration.test.ts

@@ -27,6 +27,7 @@ import type { createGitRemoteAuthProvider } from "./git-credentials.js";
 import { reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js";
 import { remoteTerminationReceipt } from "./remote-execution-termination.js";
 import { RemoteWorkspaceRestoreError } from "./remote-workspace-transport.js";
+import { readRemoteRunnerState } from "./native-runtime/native-session-executor.js";
 import { notFound } from "../errors.js";
 import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
 import { createRemoteWorkspaceFiles } from "./remote-workspace-files.js";
@@ -479,6 +480,13 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
     const reserved = await reserve({ kind: "empty" }), parentSandbox = await target("sandbox-A");
     const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: parentSandbox });
     await fs.writeFile(path.join(prepared.remoteRoot, "plan.md"), "parent plan\n");
+    // The parent's retained native session keeps its runner state beside the
+    // workspace files. Snapshots exclude it; it belongs to this sandbox.
+    const runnerStateDirectory = path.join(prepared.remoteRoot, ".paperclip-runtime", "paperclip-runner", "sessions",
+      createHash("sha256").update("parent-session").digest("hex"), "runner");
+    const runnerState = { lifecycle: "suspended", outbox: [], pendingTerminalDelivery: null, runId: ids.run };
+    await fs.mkdir(runnerStateDirectory, { recursive: true });
+    await fs.writeFile(path.join(runnerStateDirectory, "runner-state.json"), JSON.stringify(runnerState));
     await saveRemoteWorkspace({ db, storage, workspace: prepared.workspace, runId: ids.run,
       lease: prepared.lease, target: parentSandbox, assertOwned: async () => undefined });
     const writer = async (providerLeaseId: string) => {
@@ -557,11 +565,27 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
       .toEqual([path.basename(restored.remoteRoot)]);
     expect(readRemoteWorkspaceAuthority(restored.workspace.metadata)).toMatchObject({ snapshotId: childSaved.snapshotId, dirty: true,
       writerRunId: parent.runId, location: { providerLeaseId: prepared.lease.providerLeaseId, remoteCwd: prepared.remoteRoot } });
+    // The parent's continuation reads its retained runner state from the restored root.
+    expect(await readRemoteRunnerState({ runner: parentSandbox.runner!, stateDirectory: runnerStateDirectory })).toEqual(runnerState);
+    // Only the new restore receipt identifies the root; the retired preparation receipt is gone.
+    expect(await fs.lstat(path.join(restored.remoteRoot, ".paperclip-runtime", "workspace-ready.json")).catch(() => null)).toBeNull();
+    expect(JSON.parse(await fs.readFile(path.join(restored.remoteRoot, ".paperclip-runtime", "workspace-restored.json"), "utf8")))
+      .toMatchObject({ identity: expect.stringMatching(/^[a-f0-9]{64}$/) });
     // A lost reply repeats the same preparation without replacing the restored root again.
     await fs.writeFile(path.join(restored.remoteRoot, "parent-continued.md"), "unsaved parent edits\n");
+    // An attempt interrupted after publish left its retired root; the repeat keeps its runtime state.
+    const { identity } = JSON.parse(await fs.readFile(path.join(restored.remoteRoot, ".paperclip-runtime", "workspace-restored.json"), "utf8"));
+    const interrupted = `${restored.remoteRoot}.retired-${identity}`;
+    await fs.mkdir(path.join(interrupted, ".paperclip-runtime", "other-session"), { recursive: true });
+    await fs.writeFile(path.join(interrupted, ".paperclip-runtime", "other-session", "state.json"), "{}");
+    await fs.writeFile(path.join(interrupted, "stale.md"), "old\n");
     const repeated = await prepareRemoteWorkspace({ db, storage, workspace: parent.workspace, runId: parent.runId, lease: parent.lease, target: reattach() });
     expect(repeated.remoteRoot).toBe(prepared.remoteRoot);
     expect(await fs.readFile(path.join(restored.remoteRoot, "parent-continued.md"), "utf8")).toBe("unsaved parent edits\n");
+    expect(await fs.readFile(path.join(restored.remoteRoot, ".paperclip-runtime", "other-session", "state.json"), "utf8")).toBe("{}");
+    expect(await fs.lstat(interrupted).catch(() => null)).toBeNull();
+    expect(await fs.lstat(path.join(restored.remoteRoot, "stale.md")).catch(() => null)).toBeNull();
+    expect(await readRemoteRunnerState({ runner: parentSandbox.runner!, stateDirectory: runnerStateDirectory })).toEqual(runnerState);
   }, 60_000);
 
   it.each(["per_turn", "warm", "occupied", "foreign-target", "foreign-allocation", "forged-receipt"])("preserves retained native workspace identity through migration: %s", async scenario => {