fix(native): open a remote replacement as a new runner after archiving the reattached one (#20242)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, a remote (Daytona) run can resume by reattaching to
its surviving sandbox runner. If provider recovery fails and the
checkpoint allows it, the run opens a replacement provider session
instead.
> - Before a replacement, the executor archives the local controller
state and the remote session root as a continuity break, then opens a
new session.
> - On a reattach backend, the replacement still adopted the archived
runner and asked for its old authority, so it failed. The archive had
already moved the remote state, so the next retry could not verify the
runner and the issue waited 17 minutes on an unconfirmed stop.
> - Review showed that a reattach backend fixes many launch inputs once
at creation (runner binary, provider pack and companions, artifacts,
live-recovery mode, authority, spawn barrier), so resetting them after
the archive does not scale.
> - This pull request builds the replacement on a new backend for the
same run, created under a `bootstrap_incomplete` claim after the
archive, so every launch input resolves through the ordinary fresh path.
> - The benefit is that a failed reattach after a restart falls back to
a working replacement instead of a stuck issue.

## Linked Issues or Issue Description

**What happened?**
In restart-chaos Daytona K13 storm wave seed 9503, task t36 graded R1
(no rest within budget). A K2 drain suspended run a53dd948's sandbox
runner (pid 589) during startup. On restart the run claimed
`reattach_remote_runner`. The adopted runner authenticated and restored,
then its connection closed, and provider recovery failed. The runtime
opened a replacement: `provider.session.archive_before_replacement`
archived the local and remote state, then the replacement failed with
"PRP provider resume state is unavailable". The same-run retry at
10:25:50 then failed with `runner_remote_state_unavailable: exit=1` (the
remote `runner-state.json` had moved), and file preservation deferred on
`agent_stop_unconfirmed` until the wave ended.

**Expected behavior**
After the continuity archive, the replacement opens a new runner for
this run, as it does for every other claim kind.

**Steps to reproduce**
1. Run a remote (Daytona) native run and restart the server so the run
claims `reattach_remote_runner`.
2. Make provider recovery fail on the adopted runner with a checkpoint
that allows replacement.
3. See the replacement open on the adopted runner's authority and fail
after the archive.

**Paperclip version or commit**
master at 465da4227a

## What Changed

- `native-session-executor.ts` `openReplacementSession`, on a
`reattach_remote_runner` backend: archives the continuity state (local
and remote, as before), then calls
`createRunnerdBackendWithinSessionClaim` for the same run with the claim
re-kinded to `bootstrap_incomplete` (minus `remote` and
`stagedAttachment`) and without warm recovery, and opens the replacement
on that backend. Root migration then quarantines the empty archived root
(the #20241 rule) and every launch input resolves fresh.
- A backend that adopts only through warm recovery (no claim to re-kind)
throws `native_runner_replacement_requires_fresh_launch` **before**
archiving, so its saved state stays intact.
- Tests: a classic Daytona reattach backend for `codex` and `acpx`
(Claude). The replacement comes from a second backend: the original
backend's `openSession` is never called, and the new transport has no
adopted runner, uses `turn-<runId>`/`item-<runId>`, has a new
still-pending spawn barrier, and (for `acpx`) carries the packaged
provider paths. The bundled-image replacement test now expects a new
backend and its own pack admission, not the recovered session's cold
admission.

## Verification

- Red before the fix: both new replacement tests fail on master. They
pass with the fix.
- `native-session-executor.test.ts`,
`native-recovered-owner-followup.test.ts`,
`native-harness-filesystem-independence-continuation.integration.test.ts`:
1,398 passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server`: passes.

## Risks

- Medium. The replacement now constructs a second backend inside the
same session claim. The new backend's tool-authority epoch supersedes
the reattach backend's, which is being abandoned anyway.
- A warm-recovery-only replacement now fails with a named error instead
of archiving and then failing. Its saved state stays intact for the next
recovery.
- Previous-release story: no saved state or schema changes.
- Not covered: why provider recovery on the suspended, adopted runner
failed in t36 in the first place. This PR makes the fallback work.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-09 08:12:24 -06:00
1 parent a3a25ce6c9
commit 6459488984
2 files changed
+265 -11

No files matched your search

@@ -651,7 +651,7 @@ describe("remote controller restart adoption", () => {
it("signals nothing when the sandbox runner belongs to another identity", async () => {
const { execute, target } = fixture({ runnerInstanceId: "different" });
await expect(stopStaleRemoteRunnerBeforeBootstrap({ execution, runnerExecutionTarget: target as never, identity },
{ waitMs: 20, pollMs: 1 })).resolves.toBe("no_prior_runner");
{ waitMs: 20, pollMs: 1 })).resolves.toBe("stop_unproven");
expect(signalled(execute)).toEqual([]);
});
it("signals nothing when no prior runner is alive", async () => {
@@ -663,7 +663,7 @@ describe("remote controller restart adoption", () => {
{ waitMs: 20, pollMs: 1 })).resolves.toBe("absent_confirmed");
expect(signalled(execute)).toEqual([]);
});
it.each([["absent_confirmed", true], ["stopped", true], ["killed", true], ["stop_failed", false], ["no_prior_runner", false]] as const)(
it.each([["absent_confirmed", true], ["stopped", true], ["killed", true], ["stop_failed", false], ["stop_unproven", false], ["no_prior_runner", false]] as const)(
"records a bootstrap restart proof only for a confirmed stop: %s", (stopped, proven) => {
noteRemoteBootstrapRestartProven("run", "stopped");
expect(recordRemoteBootstrapRestartProof("run", stopped)).toBe(proven);
@@ -674,7 +674,7 @@ describe("remote controller restart adoption", () => {
const { execute, target } = fixture();
const verify = async () => { throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "probe_timeout" } as never); };
await expect(stopStaleRemoteRunnerBeforeBootstrap({ execution, runnerExecutionTarget: target as never, identity },
{ waitMs: 20, pollMs: 1, verify: verify as never })).resolves.toBe("no_prior_runner");
{ waitMs: 20, pollMs: 1, verify: verify as never })).resolves.toBe("stop_unproven");
expect(signalled(execute)).toEqual([]);
});
});
@@ -2608,7 +2608,16 @@ describe("remote provider pack manifest", () => {
next.digest = "sha256:" + createHash("sha256").update(canonical(next.payload)).digest("hex");
await writeFile(state.bundledPackManifestPath!, JSON.stringify(next));
await writeFile(state.bundledRunnerBinary!, Buffer.concat([daemon, Buffer.from("new release")]));
let recoveredRunnerStopped = false;
const recoveryExecute = vi.fn(async (request: { args: string[] }) => {
if (request.args[2] === "paperclip-runner-recovery-check" && request.args[1]?.includes('kill -"$6"')) {
const exitCode = recoveredRunnerStopped ? 45 : 0;
recoveredRunnerStopped = true;
return { exitCode, timedOut: false, stderr: "", stdout: "" };
}
if (request.args[2] === "paperclip-runner-recovery-check" && recoveredRunnerStopped) {
return { exitCode: 45, timedOut: false, stderr: "", stdout: "" };
}
if (request.args[0] === "--build-metadata") return { exitCode: 0, timedOut: false, stderr: "", stdout: JSON.stringify({
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
@@ -2625,7 +2634,8 @@ describe("remote provider pack manifest", () => {
runnerInstanceId: identity.runnerInstanceId, runnerIngressAuthorized: true,
...(opencode ? { runnerEnvironment: configuredEnvironmentProjection({ PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS: "1" }) } : {}),
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace",
environmentId: "environment", leaseId: "lease", providerKey: "daytona", effectiveCapabilities: { runnerWebSocketIngress: true }, runner: { execute: recoveryExecute },
environmentId: "environment", leaseId: "lease", providerKey: "daytona", effectiveCapabilities: { runnerWebSocketIngress: true },
runner: { execute: recoveryExecute, syncIn: vi.fn(async () => undefined), syncOut: vi.fn(async () => undefined) },
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never,
restartRecovery: { kind: "reattach_remote_runner", runId: remoteExecution.binding.runId,
leaseOwner: "controller", controllerGeneration: 2, providerAttempt: 1, restartKind: "graceful", recoveryRequestId: null,
@@ -2684,9 +2694,14 @@ describe("remote provider pack manifest", () => {
expect(admittedNativeProviderPack(knownResult.session!, nativeIdentity)).toEqual(coldAdmission);
if (pinned) {
const replacement = { identity: () => nativeIdentity } as never;
Object.assign(state.createBackend.mock.results.at(-1)!.value, { openSession: async () => replacement });
const recoveredBackendMock = state.createBackend.mock.results.at(-1)!.value;
state.createBackend.mockReturnValueOnce({ kind: "test", openSession: async () => replacement } as never);
const freshPreparation = vi.spyOn(nativeExecutionDriver, "prepareNativeRemoteRunner").mockResolvedValue(undefined as never);
onTestFinished(() => freshPreparation.mockRestore());
const replaced = await knownBackend.openReplacementSession!({ identity: nativeIdentity }, checkpoint);
expect(freshPreparation).toHaveBeenCalled();
expect(replaced).toBe(replacement);
expect(state.createBackend.mock.results.at(-1)!.value).not.toBe(recoveredBackendMock);
expect(admittedNativeProviderPack(replaced, nativeIdentity)).toBeNull();
}
}
@@ -16225,6 +16240,218 @@ describe("runnerd provider runtime wiring", () => {
|| request.args.includes("paperclip-runner-claim-unstarted-session"))).toBe(false);
});
it.each(["codex", "acpx"] as const)("opens a remote %s replacement as a new runner, not the surviving runner it archived", async provider => {
const base = scopedNativeExecution(`replacement-after-reattach-${provider}`);
const current = { ...base, ...(provider === "acpx" ? { provider: { kind: "acpx", agent: "claude", model: null },
session: { ...base.session, driverKind: "acpx_runtime" } } : {}) } as NativeExecutionInputV1;
const driver = vi.spyOn(nativeExecutionDriver, "createNativeRunnerExecutionDriver");
onTestFinished(() => driver.mockRestore());
const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!,
runnerInstanceId: "admitted-codex", environmentLeaseId: "admitted-lease",
turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` };
const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId);
await mkdir(join(root, "control-plane"), { recursive: true });
await writeFile(join(root, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity)));
let installed: unknown = null;
if (provider === "acpx") {
const packRoot = await mkdtemp(join(tmpdir(), "paperclip-replacement-pack-"));
onTestFinished(() => rm(packRoot, { recursive: true, force: true }));
const digest = "sha256:" + "a".repeat(64);
const canonical = (value: unknown): string => Array.isArray(value) ? `[${value.map(canonical).join(",")}]`
: value && typeof value === "object" ? `{${Object.keys(value).sort().map(key =>
`${JSON.stringify(key)}:${canonical((value as Record<string, unknown>)[key])}`).join(",")}}` : JSON.stringify(value);
const artifacts = Object.fromEntries(Object.entries({
grokLauncher: "dist/providers/grok/launcher.cjs", nodeCommand: "node_modules/node/bin/node",
productionLock: "pnpm-lock.yaml", opencodeCommand: "node_modules/.bin/opencode",
opencodeExecutable: "node_modules/opencode-ai/bin/opencode.exe",
opencodeProxy: "dist/cli/opencode-app-server-proxy.cjs", acpxSidecar: "dist/cli/acpx-runtime-sidecar.cjs",
}).map(([key, path]) => [key, { path, sha256: digest }]));
const payload = { compatibility: { opencode: 1, acpx: 1 }, contractDigest: digest,
target: { platform: "linux", architecture: "x64" }, runnerSourceRevision: "a".repeat(40),
pins: { nodeMinimum: "24.11.0", codex: QUALIFIED_ACPX_PROFILES.codex.agentRuntimeVersion,
opencode: "1.18.34", acpx: "0.13.1", grok: QUALIFIED_ACPX_PROFILES.grok.agentRuntimeVersion,
claudeAcp: QUALIFIED_ACPX_PROFILES.claude.agentServerVersion, codexAcp: QUALIFIED_ACPX_PROFILES.codex.agentServerVersion },
acpxProfileDigests: Object.fromEntries(["grok", "claude", "codex"].map(agent =>
[agent, QUALIFIED_ACPX_PROFILES[agent as "grok" | "claude" | "codex"].commandDigest])),
distDigest: digest, artifacts,
bridgeDigest: "sha256:" + createHash("sha256").update(digest + "\n" + digest + "\n" + digest).digest("hex") };
installed = { schema: "paperclip-runner/remote-provider-pack/v1", payload,
digest: "sha256:" + createHash("sha256").update(canonical(payload)).digest("hex") };
state.bundledPackManifestPath = join(packRoot, "provider-pack.json");
await writeFile(state.bundledPackManifestPath, JSON.stringify(installed));
state.bundledRunnerBinary = join(packRoot, "linux-paperclip-runnerd");
const daemon = Buffer.alloc(64); daemon.set([0x7f, 0x45, 0x4c, 0x46, 2, 1]); daemon.writeUInt16LE(62, 18);
await writeFile(state.bundledRunnerBinary, daemon);
state.resolveRemoteRunnerBinary.mockResolvedValue(state.bundledRunnerBinary);
state.resolveRunnerBinary.mockReturnValue(state.bundledRunnerBinary);
}
let runnerStopped = false, sessionArchived = false;
const execute = vi.fn(async (request: { args: string[] }) => {
const script = request.args[1] ?? "";
if (script.includes(".continuity-break-")) sessionArchived = true;
if (request.args[2] === "paperclip-runner-recovery-check") {
const signalled = script.includes('kill -"$6"');
const exitCode = runnerStopped ? 45 : 0;
if (signalled) runnerStopped = true;
return { exitCode, timedOut: false, stderr: "", stdout: "" };
}
if (sessionArchived && script.includes("runner-state.json")) return { exitCode: 1, timedOut: false, stderr: "", stdout: "" };
return { exitCode: 0, timedOut: false, stderr: "",
stdout: request.args[0] === "--build-metadata" ? JSON.stringify({
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
providerPacks: installed ? [{ root: "/opt/paperclip-runner/provider-pack", manifest: installed, immutable: true }] : [],
}) : script.includes("base64") ? Buffer.from(JSON.stringify({ ...identity, lifecycle: "suspended" })).toString("base64")
: script.includes("cat --") ? `nonce\n123\n2026-09-19T10:00:00.000Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : "" };
});
state.createBackend.mockClear(); state.createTransport.mockClear();
const nativeIdentity = { ...current.binding, sessionId: current.session.normalizedSessionId! };
const openSession = vi.fn(async () => ({ close: async () => undefined, identity: () => nativeIdentity }));
const freshOpenSession = vi.fn(async () => ({ close: async () => undefined, identity: () => nativeIdentity }));
state.createBackend.mockReturnValueOnce({ kind: "test", openSession, descriptor: () => ({ name: "reattached" }) } as never)
.mockReturnValueOnce({ kind: "test", openSession: freshOpenSession, descriptor: () => ({ name: "replacement" }) } as never);
const backend = await createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
runnerIngressAuthorized: true,
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-version", digest: `sha256:${"a".repeat(64)}` } },
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace",
environmentId: "environment", leaseId: identity.environmentLeaseId, providerKey: "daytona", runner: { execute },
effectiveCapabilities: { runnerWebSocketIngress: true },
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never,
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller",
controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null,
remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } },
});
const factory = state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!;
factory();
expect(state.createTransport.mock.calls.at(-1)![0]).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: 123 } });
const spawnedBefore = driver.mock.calls.at(-1)![0].processStarted;
expect(backend.descriptor()).toEqual({ name: "reattached" });
const replaced = await backend.openReplacementSession!({ identity: nativeIdentity } as never, {} as never);
expect(runnerStopped).toBe(true);
const signalIndex = execute.mock.calls.findIndex(([request]) => request.args[1]?.includes('kill -"$6"'));
const archiveIndex = execute.mock.calls.findIndex(([request]) => request.args[1]?.includes(".continuity-break-"));
expect(signalIndex).toBeGreaterThanOrEqual(0);
expect(signalIndex).toBeLessThan(archiveIndex);
expect(backend.descriptor()).toEqual({ name: "replacement" });
expect(backend.bindManagedSession(replaced)).toBe(replaced);
expect(openSession).not.toHaveBeenCalled();
expect(freshOpenSession).toHaveBeenCalledOnce();
expect(state.createBackend).toHaveBeenCalledTimes(2);
expect(execute.mock.calls.some(([request]) => request.args[1]?.includes(".continuity-break-"))).toBe(true);
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
const replacement = state.createTransport.mock.calls.at(-1)![0];
expect(replacement.adoptExistingRunner).toBeUndefined();
expect(replacement.prpIdentity).toMatchObject({ runId: current.binding.runId, turnId: `turn-${current.binding.runId}`,
itemId: `item-${current.binding.runId}` });
const spawnedAfter = driver.mock.calls.at(-1)![0].processStarted!;
expect(spawnedAfter).not.toBe(spawnedBefore);
expect(await Promise.race([spawnedAfter.then(() => "spawned"), Promise.resolve("pending")])).toBe("pending");
if (provider === "acpx") {
expect(replacement).toMatchObject({ providerNodeCommand: expect.any(String), acpxSidecarPath: expect.any(String) });
expect(admittedNativeProviderPack(replaced, nativeIdentity)).toMatchObject({
authority: "core_verified_selection", providerPackDigest: (installed as { digest: string }).digest });
}
});
it("refuses a remote replacement and keeps the saved state when the adopted runner's stop is unproven", async () => {
const provider = "codex" as "codex" | "acpx";
const base = scopedNativeExecution(`replacement-unproven-stop`);
const current = { ...base, ...(provider === "acpx" ? { provider: { kind: "acpx", agent: "claude", model: null },
session: { ...base.session, driverKind: "acpx_runtime" } } : {}) } as NativeExecutionInputV1;
const driver = vi.spyOn(nativeExecutionDriver, "createNativeRunnerExecutionDriver");
onTestFinished(() => driver.mockRestore());
const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!,
runnerInstanceId: "admitted-codex", environmentLeaseId: "admitted-lease",
turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` };
const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId);
await mkdir(join(root, "control-plane"), { recursive: true });
await writeFile(join(root, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity)));
let installed: unknown = null;
if (provider === "acpx") {
const packRoot = await mkdtemp(join(tmpdir(), "paperclip-replacement-pack-"));
onTestFinished(() => rm(packRoot, { recursive: true, force: true }));
const digest = "sha256:" + "a".repeat(64);
const canonical = (value: unknown): string => Array.isArray(value) ? `[${value.map(canonical).join(",")}]`
: value && typeof value === "object" ? `{${Object.keys(value).sort().map(key =>
`${JSON.stringify(key)}:${canonical((value as Record<string, unknown>)[key])}`).join(",")}}` : JSON.stringify(value);
const artifacts = Object.fromEntries(Object.entries({
grokLauncher: "dist/providers/grok/launcher.cjs", nodeCommand: "node_modules/node/bin/node",
productionLock: "pnpm-lock.yaml", opencodeCommand: "node_modules/.bin/opencode",
opencodeExecutable: "node_modules/opencode-ai/bin/opencode.exe",
opencodeProxy: "dist/cli/opencode-app-server-proxy.cjs", acpxSidecar: "dist/cli/acpx-runtime-sidecar.cjs",
}).map(([key, path]) => [key, { path, sha256: digest }]));
const payload = { compatibility: { opencode: 1, acpx: 1 }, contractDigest: digest,
target: { platform: "linux", architecture: "x64" }, runnerSourceRevision: "a".repeat(40),
pins: { nodeMinimum: "24.11.0", codex: QUALIFIED_ACPX_PROFILES.codex.agentRuntimeVersion,
opencode: "1.18.34", acpx: "0.13.1", grok: QUALIFIED_ACPX_PROFILES.grok.agentRuntimeVersion,
claudeAcp: QUALIFIED_ACPX_PROFILES.claude.agentServerVersion, codexAcp: QUALIFIED_ACPX_PROFILES.codex.agentServerVersion },
acpxProfileDigests: Object.fromEntries(["grok", "claude", "codex"].map(agent =>
[agent, QUALIFIED_ACPX_PROFILES[agent as "grok" | "claude" | "codex"].commandDigest])),
distDigest: digest, artifacts,
bridgeDigest: "sha256:" + createHash("sha256").update(digest + "\n" + digest + "\n" + digest).digest("hex") };
installed = { schema: "paperclip-runner/remote-provider-pack/v1", payload,
digest: "sha256:" + createHash("sha256").update(canonical(payload)).digest("hex") };
state.bundledPackManifestPath = join(packRoot, "provider-pack.json");
await writeFile(state.bundledPackManifestPath, JSON.stringify(installed));
state.bundledRunnerBinary = join(packRoot, "linux-paperclip-runnerd");
const daemon = Buffer.alloc(64); daemon.set([0x7f, 0x45, 0x4c, 0x46, 2, 1]); daemon.writeUInt16LE(62, 18);
await writeFile(state.bundledRunnerBinary, daemon);
state.resolveRemoteRunnerBinary.mockResolvedValue(state.bundledRunnerBinary);
state.resolveRunnerBinary.mockReturnValue(state.bundledRunnerBinary);
}
let runnerStopped = false, sessionArchived = false;
const execute = vi.fn(async (request: { args: string[] }) => {
const script = request.args[1] ?? "";
if (script.includes(".continuity-break-")) sessionArchived = true;
if (request.args[2] === "paperclip-runner-recovery-check") {
const signalled = script.includes('kill -"$6"');
const exitCode = runnerStopped ? 45 : 0;
if (signalled) runnerStopped = true;
return { exitCode, timedOut: false, stderr: "", stdout: "" };
}
if (sessionArchived && script.includes("runner-state.json")) return { exitCode: 1, timedOut: false, stderr: "", stdout: "" };
return { exitCode: 0, timedOut: false, stderr: "",
stdout: request.args[0] === "--build-metadata" ? JSON.stringify({
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
providerPacks: installed ? [{ root: "/opt/paperclip-runner/provider-pack", manifest: installed, immutable: true }] : [],
}) : script.includes("base64") ? Buffer.from(JSON.stringify({ ...identity, lifecycle: "suspended" })).toString("base64")
: script.includes("cat --") ? `nonce\n123\n2026-09-19T10:00:00.000Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : "" };
});
state.createBackend.mockClear(); state.createTransport.mockClear();
const nativeIdentity = { ...current.binding, sessionId: current.session.normalizedSessionId! };
const openSession = vi.fn(async () => ({ close: async () => undefined, identity: () => nativeIdentity }));
const freshOpenSession = vi.fn(async () => ({ close: async () => undefined, identity: () => nativeIdentity }));
state.createBackend.mockReturnValueOnce({ kind: "test", openSession, descriptor: () => ({ name: "reattached" }) } as never);
const backend = await createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
runnerIngressAuthorized: true,
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-version", digest: `sha256:${"a".repeat(64)}` } },
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace",
environmentId: "environment", leaseId: identity.environmentLeaseId, providerKey: "daytona", runner: { execute },
effectiveCapabilities: { runnerWebSocketIngress: true },
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never,
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller",
controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null,
remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } },
});
const factory = state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!;
factory();
expect(state.createTransport.mock.calls.at(-1)![0]).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: 123 } });
const spawnedBefore = driver.mock.calls.at(-1)![0].processStarted;
execute.mockImplementation(async (request: { args: string[] }) => request.args[2] === "paperclip-runner-recovery-check"
? { exitCode: 0, timedOut: true, stderr: "", stdout: "" }
: { exitCode: 0, timedOut: false, stderr: "", stdout: "" });
await expect(backend.openReplacementSession!({ identity: nativeIdentity } as never, {} as never))
.rejects.toThrow("native_runner_replacement_stop_unproven");
expect(execute.mock.calls.some(([request]) => request.args[1]?.includes(".continuity-break-"))).toBe(false);
expect(execute.mock.calls.some(([request]) => request.args[1]?.includes('kill -"$6"'))).toBe(false);
expect(freshOpenSession).not.toHaveBeenCalled();
expect(state.createBackend).toHaveBeenCalledTimes(1);
expect(existsSync(join(root, "control-plane", "control-plane-state.json"))).toBe(true);
});
it.each(["failed", "timeout", "succeeded"] as const)("retains the local controller checkpoint until remote continuity archiving succeeds: %s", async mode => {
const current = scopedNativeExecution("failed-remote-continuity-archive");
state.createBackend.mockClear(); state.createTransport.mockClear();
@@ -6438,7 +6438,7 @@ export async function stopStaleRemoteRunnerBeforeBootstrap(input: {
verify?: typeof verifyRemoteRunnerReattachment;
waitMs?: number;
pollMs?: number;
} = {}): Promise<"no_prior_runner" | "absent_confirmed" | "stopped" | "killed" | "stop_failed"> {
} = {}): Promise<"no_prior_runner" | "absent_confirmed" | "stopped" | "killed" | "stop_failed" | "stop_unproven"> {
const target = input.runnerExecutionTarget;
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner) return "no_prior_runner";
const providerLeaseId = target.sandboxLeaseAcquisition?.providerLeaseId;
@@ -6454,7 +6454,7 @@ export async function stopStaleRemoteRunnerBeforeBootstrap(input: {
} catch (error) {
// No runner with this run's exact identity and generation is alive here.
return error instanceof RemoteRunnerRecoveryError && error.remoteRunnerRecovery.reason === "process_not_running"
? "absent_confirmed" : "no_prior_runner";
? "absent_confirmed" : "stop_unproven";
}
try {
let outcome: "stopped" | "killed" = "stopped";
@@ -16801,11 +16801,15 @@ async function createRunnerdBackendWithinSessionClaim(
await recordAdmittedRemoteProviderPack(providerPackAuthority, message => input.onLog?.("stderr", message));
}
};
let replacementBackend: ManagedRunnerdBackend | null = null;
return {
bindManagedSession: wrapManagedSession,
providerAdmissionRequired: () => Boolean(artifactStorage
&& (providerArtifacts || !legacyProviderContinuity && !liveProviderRecovery)),
bindManagedSession: (session, options) => replacementBackend
? replacementBackend.bindManagedSession(session, options) : wrapManagedSession(session),
providerAdmissionRequired: () => replacementBackend?.providerAdmissionRequired
? replacementBackend.providerAdmissionRequired()
: Boolean(artifactStorage && (providerArtifacts || !legacyProviderContinuity && !liveProviderRecovery)),
providerAdmissionPersisted: artifactStorage ? async () => {
if (replacementBackend) return replacementBackend.providerAdmissionPersisted?.();
const backupOwner = sessionHarnessBackupOwners.get(authorityEpoch);
if ((liveProviderRecovery || legacyProviderContinuity) && !providerArtifacts) return;
if (!backupOwner || !expectedProviderPackManifest || !remoteCommandRunner) {
@@ -16823,6 +16827,7 @@ async function createRunnerdBackendWithinSessionClaim(
source => retainNativeProviderArtifacts({ db: input.db, storage: artifactStorage, owner: backupOwner, source, signal }));
} : undefined,
bindDiagnosticSession: session => {
if (replacementBackend) return replacementBackend.bindDiagnosticSession?.(session);
const retained = nativeSessionDiagnostics(session);
if (retained && retained !== diagnostics) {
const lease = retained.activate(diagnosticOwner, input.onLog);
@@ -16830,6 +16835,7 @@ async function createRunnerdBackendWithinSessionClaim(
} else diagnostics.bind(session);
},
sessionCheckpointPersisted: snapshot => {
if (replacementBackend) return replacementBackend.sessionCheckpointPersisted?.(snapshot);
const staged = runnerControlPlane?.store.state.stagedRunAttachment;
if (staged && snapshot.identity.runId === staged.targetIdentity.runId &&
snapshot.identity.sessionId === staged.targetIdentity.normalizedSessionId) {
@@ -16838,7 +16844,7 @@ async function createRunnerdBackendWithinSessionClaim(
}
}
},
descriptor: () => backend.descriptor(),
descriptor: () => (replacementBackend ?? backend).descriptor(),
openSession: async (sessionInput) => {
cursorCloudRuntime?.register();
try {
@@ -16859,12 +16865,33 @@ async function createRunnerdBackendWithinSessionClaim(
} catch (error) { cursorCloudRuntime?.unregister(); throw error; }
},
openReplacementSession: async (sessionInput, previous) => {
const reattachClaim = input.restartRecovery?.kind === "reattach_remote_runner" ? input.restartRecovery : null;
if (!reattachClaim && input.remoteWarmRecovery) {
throw new Error("native_runner_replacement_requires_fresh_launch");
}
if (reattachClaim) {
const stopped = await stopStaleRemoteRunnerBeforeBootstrap({ execution: input.execution,
runnerExecutionTarget: input.runnerExecutionTarget, identity: durableIdentity ?? {}, onLog: input.onLog });
if (stopped === "stop_failed" || stopped === "stop_unproven") {
throw new Error(`native_runner_replacement_${stopped}`);
}
}
await measureNativeRunnerSpan(
input.trace,
"provider.session.archive_before_replacement",
() => archiveContinuityState(previous),
{ parentName: "native.session.execute" },
);
if (reattachClaim) {
const { remote: _remote, stagedAttachment: _staged, ...claim } = reattachClaim;
const fresh = await createRunnerdBackendWithinSessionClaim({
...input,
remoteWarmRecovery: undefined,
restartRecovery: { ...claim, kind: "bootstrap_incomplete" },
}, sessionScopeId);
replacementBackend = fresh;
return fresh.openSession(sessionInput);
}
cursorCloudRuntime?.register();
try {
const session = await backend.openSession(sessionInput);