From 6459488984557d175553c51f3990c3d2f470ab70 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 9 Nov 2026 08:12:24 -0600 Subject: [PATCH] fix(native): open a remote replacement as a new runner after archiving the reattached one (#20242) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, a remote (Daytona) run can resume by reattaching to its surviving sandbox runner. If provider recovery fails and the checkpoint allows it, the run opens a replacement provider session instead. > - Before a replacement, the executor archives the local controller state and the remote session root as a continuity break, then opens a new session. > - On a reattach backend, the replacement still adopted the archived runner and asked for its old authority, so it failed. The archive had already moved the remote state, so the next retry could not verify the runner and the issue waited 17 minutes on an unconfirmed stop. > - Review showed that a reattach backend fixes many launch inputs once at creation (runner binary, provider pack and companions, artifacts, live-recovery mode, authority, spawn barrier), so resetting them after the archive does not scale. > - This pull request builds the replacement on a new backend for the same run, created under a `bootstrap_incomplete` claim after the archive, so every launch input resolves through the ordinary fresh path. > - The benefit is that a failed reattach after a restart falls back to a working replacement instead of a stuck issue. ## Linked Issues or Issue Description **What happened?** In restart-chaos Daytona K13 storm wave seed 9503, task t36 graded R1 (no rest within budget). A K2 drain suspended run a53dd948's sandbox runner (pid 589) during startup. On restart the run claimed `reattach_remote_runner`. The adopted runner authenticated and restored, then its connection closed, and provider recovery failed. The runtime opened a replacement: `provider.session.archive_before_replacement` archived the local and remote state, then the replacement failed with "PRP provider resume state is unavailable". The same-run retry at 10:25:50 then failed with `runner_remote_state_unavailable: exit=1` (the remote `runner-state.json` had moved), and file preservation deferred on `agent_stop_unconfirmed` until the wave ended. **Expected behavior** After the continuity archive, the replacement opens a new runner for this run, as it does for every other claim kind. **Steps to reproduce** 1. Run a remote (Daytona) native run and restart the server so the run claims `reattach_remote_runner`. 2. Make provider recovery fail on the adopted runner with a checkpoint that allows replacement. 3. See the replacement open on the adopted runner's authority and fail after the archive. **Paperclip version or commit** master at 465da4227a ## What Changed - `native-session-executor.ts` `openReplacementSession`, on a `reattach_remote_runner` backend: archives the continuity state (local and remote, as before), then calls `createRunnerdBackendWithinSessionClaim` for the same run with the claim re-kinded to `bootstrap_incomplete` (minus `remote` and `stagedAttachment`) and without warm recovery, and opens the replacement on that backend. Root migration then quarantines the empty archived root (the #20241 rule) and every launch input resolves fresh. - A backend that adopts only through warm recovery (no claim to re-kind) throws `native_runner_replacement_requires_fresh_launch` **before** archiving, so its saved state stays intact. - Tests: a classic Daytona reattach backend for `codex` and `acpx` (Claude). The replacement comes from a second backend: the original backend's `openSession` is never called, and the new transport has no adopted runner, uses `turn-`/`item-`, has a new still-pending spawn barrier, and (for `acpx`) carries the packaged provider paths. The bundled-image replacement test now expects a new backend and its own pack admission, not the recovered session's cold admission. ## Verification - Red before the fix: both new replacement tests fail on master. They pass with the fix. - `native-session-executor.test.ts`, `native-recovered-owner-followup.test.ts`, `native-harness-filesystem-independence-continuation.integration.test.ts`: 1,398 passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server`: passes. ## Risks - Medium. The replacement now constructs a second backend inside the same session claim. The new backend's tool-authority epoch supersedes the reattach backend's, which is being abandoned anyway. - A warm-recovery-only replacement now fails with a named error instead of archiving and then failing. Its saved state stays intact for the next recovery. - Previous-release story: no saved state or schema changes. - Not covered: why provider recovery on the suspended, adopted runner failed in t36 in the first place. This PR makes the fallback work. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-session-executor.test.ts | 237 +++++++++++++++++- .../native-runtime/native-session-executor.ts | 39 ++- 2 files changed, 265 insertions(+), 11 deletions(-) diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index c08d42ab0c9..b23ef186ef2 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -651,7 +651,7 @@ describe("remote controller restart adoption", () => { it("signals nothing when the sandbox runner belongs to another identity", async () => { const { execute, target } = fixture({ runnerInstanceId: "different" }); await expect(stopStaleRemoteRunnerBeforeBootstrap({ execution, runnerExecutionTarget: target as never, identity }, - { waitMs: 20, pollMs: 1 })).resolves.toBe("no_prior_runner"); + { waitMs: 20, pollMs: 1 })).resolves.toBe("stop_unproven"); expect(signalled(execute)).toEqual([]); }); it("signals nothing when no prior runner is alive", async () => { @@ -663,7 +663,7 @@ describe("remote controller restart adoption", () => { { waitMs: 20, pollMs: 1 })).resolves.toBe("absent_confirmed"); expect(signalled(execute)).toEqual([]); }); - it.each([["absent_confirmed", true], ["stopped", true], ["killed", true], ["stop_failed", false], ["no_prior_runner", false]] as const)( + it.each([["absent_confirmed", true], ["stopped", true], ["killed", true], ["stop_failed", false], ["stop_unproven", false], ["no_prior_runner", false]] as const)( "records a bootstrap restart proof only for a confirmed stop: %s", (stopped, proven) => { noteRemoteBootstrapRestartProven("run", "stopped"); expect(recordRemoteBootstrapRestartProof("run", stopped)).toBe(proven); @@ -674,7 +674,7 @@ describe("remote controller restart adoption", () => { const { execute, target } = fixture(); const verify = async () => { throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "probe_timeout" } as never); }; await expect(stopStaleRemoteRunnerBeforeBootstrap({ execution, runnerExecutionTarget: target as never, identity }, - { waitMs: 20, pollMs: 1, verify: verify as never })).resolves.toBe("no_prior_runner"); + { waitMs: 20, pollMs: 1, verify: verify as never })).resolves.toBe("stop_unproven"); expect(signalled(execute)).toEqual([]); }); }); @@ -2608,7 +2608,16 @@ describe("remote provider pack manifest", () => { next.digest = "sha256:" + createHash("sha256").update(canonical(next.payload)).digest("hex"); await writeFile(state.bundledPackManifestPath!, JSON.stringify(next)); await writeFile(state.bundledRunnerBinary!, Buffer.concat([daemon, Buffer.from("new release")])); + let recoveredRunnerStopped = false; const recoveryExecute = vi.fn(async (request: { args: string[] }) => { + if (request.args[2] === "paperclip-runner-recovery-check" && request.args[1]?.includes('kill -"$6"')) { + const exitCode = recoveredRunnerStopped ? 45 : 0; + recoveredRunnerStopped = true; + return { exitCode, timedOut: false, stderr: "", stdout: "" }; + } + if (request.args[2] === "paperclip-runner-recovery-check" && recoveredRunnerStopped) { + return { exitCode: 45, timedOut: false, stderr: "", stdout: "" }; + } if (request.args[0] === "--build-metadata") return { exitCode: 0, timedOut: false, stderr: "", stdout: JSON.stringify({ schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd", packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2, @@ -2625,7 +2634,8 @@ describe("remote provider pack manifest", () => { runnerInstanceId: identity.runnerInstanceId, runnerIngressAuthorized: true, ...(opencode ? { runnerEnvironment: configuredEnvironmentProjection({ PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS: "1" }) } : {}), runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", - environmentId: "environment", leaseId: "lease", providerKey: "daytona", effectiveCapabilities: { runnerWebSocketIngress: true }, runner: { execute: recoveryExecute }, + environmentId: "environment", leaseId: "lease", providerKey: "daytona", effectiveCapabilities: { runnerWebSocketIngress: true }, + runner: { execute: recoveryExecute, syncIn: vi.fn(async () => undefined), syncOut: vi.fn(async () => undefined) }, sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never, restartRecovery: { kind: "reattach_remote_runner", runId: remoteExecution.binding.runId, leaseOwner: "controller", controllerGeneration: 2, providerAttempt: 1, restartKind: "graceful", recoveryRequestId: null, @@ -2684,9 +2694,14 @@ describe("remote provider pack manifest", () => { expect(admittedNativeProviderPack(knownResult.session!, nativeIdentity)).toEqual(coldAdmission); if (pinned) { const replacement = { identity: () => nativeIdentity } as never; - Object.assign(state.createBackend.mock.results.at(-1)!.value, { openSession: async () => replacement }); + const recoveredBackendMock = state.createBackend.mock.results.at(-1)!.value; + state.createBackend.mockReturnValueOnce({ kind: "test", openSession: async () => replacement } as never); + const freshPreparation = vi.spyOn(nativeExecutionDriver, "prepareNativeRemoteRunner").mockResolvedValue(undefined as never); + onTestFinished(() => freshPreparation.mockRestore()); const replaced = await knownBackend.openReplacementSession!({ identity: nativeIdentity }, checkpoint); + expect(freshPreparation).toHaveBeenCalled(); expect(replaced).toBe(replacement); + expect(state.createBackend.mock.results.at(-1)!.value).not.toBe(recoveredBackendMock); expect(admittedNativeProviderPack(replaced, nativeIdentity)).toBeNull(); } } @@ -16225,6 +16240,218 @@ describe("runnerd provider runtime wiring", () => { || request.args.includes("paperclip-runner-claim-unstarted-session"))).toBe(false); }); + it.each(["codex", "acpx"] as const)("opens a remote %s replacement as a new runner, not the surviving runner it archived", async provider => { + const base = scopedNativeExecution(`replacement-after-reattach-${provider}`); + const current = { ...base, ...(provider === "acpx" ? { provider: { kind: "acpx", agent: "claude", model: null }, + session: { ...base.session, driverKind: "acpx_runtime" } } : {}) } as NativeExecutionInputV1; + const driver = vi.spyOn(nativeExecutionDriver, "createNativeRunnerExecutionDriver"); + onTestFinished(() => driver.mockRestore()); + const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!, + runnerInstanceId: "admitted-codex", environmentLeaseId: "admitted-lease", + turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` }; + const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId); + await mkdir(join(root, "control-plane"), { recursive: true }); + await writeFile(join(root, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity))); + let installed: unknown = null; + if (provider === "acpx") { + const packRoot = await mkdtemp(join(tmpdir(), "paperclip-replacement-pack-")); + onTestFinished(() => rm(packRoot, { recursive: true, force: true })); + const digest = "sha256:" + "a".repeat(64); + const canonical = (value: unknown): string => Array.isArray(value) ? `[${value.map(canonical).join(",")}]` + : value && typeof value === "object" ? `{${Object.keys(value).sort().map(key => + `${JSON.stringify(key)}:${canonical((value as Record)[key])}`).join(",")}}` : JSON.stringify(value); + const artifacts = Object.fromEntries(Object.entries({ + grokLauncher: "dist/providers/grok/launcher.cjs", nodeCommand: "node_modules/node/bin/node", + productionLock: "pnpm-lock.yaml", opencodeCommand: "node_modules/.bin/opencode", + opencodeExecutable: "node_modules/opencode-ai/bin/opencode.exe", + opencodeProxy: "dist/cli/opencode-app-server-proxy.cjs", acpxSidecar: "dist/cli/acpx-runtime-sidecar.cjs", + }).map(([key, path]) => [key, { path, sha256: digest }])); + const payload = { compatibility: { opencode: 1, acpx: 1 }, contractDigest: digest, + target: { platform: "linux", architecture: "x64" }, runnerSourceRevision: "a".repeat(40), + pins: { nodeMinimum: "24.11.0", codex: QUALIFIED_ACPX_PROFILES.codex.agentRuntimeVersion, + opencode: "1.18.34", acpx: "0.13.1", grok: QUALIFIED_ACPX_PROFILES.grok.agentRuntimeVersion, + claudeAcp: QUALIFIED_ACPX_PROFILES.claude.agentServerVersion, codexAcp: QUALIFIED_ACPX_PROFILES.codex.agentServerVersion }, + acpxProfileDigests: Object.fromEntries(["grok", "claude", "codex"].map(agent => + [agent, QUALIFIED_ACPX_PROFILES[agent as "grok" | "claude" | "codex"].commandDigest])), + distDigest: digest, artifacts, + bridgeDigest: "sha256:" + createHash("sha256").update(digest + "\n" + digest + "\n" + digest).digest("hex") }; + installed = { schema: "paperclip-runner/remote-provider-pack/v1", payload, + digest: "sha256:" + createHash("sha256").update(canonical(payload)).digest("hex") }; + state.bundledPackManifestPath = join(packRoot, "provider-pack.json"); + await writeFile(state.bundledPackManifestPath, JSON.stringify(installed)); + state.bundledRunnerBinary = join(packRoot, "linux-paperclip-runnerd"); + const daemon = Buffer.alloc(64); daemon.set([0x7f, 0x45, 0x4c, 0x46, 2, 1]); daemon.writeUInt16LE(62, 18); + await writeFile(state.bundledRunnerBinary, daemon); + state.resolveRemoteRunnerBinary.mockResolvedValue(state.bundledRunnerBinary); + state.resolveRunnerBinary.mockReturnValue(state.bundledRunnerBinary); + } + let runnerStopped = false, sessionArchived = false; + const execute = vi.fn(async (request: { args: string[] }) => { + const script = request.args[1] ?? ""; + if (script.includes(".continuity-break-")) sessionArchived = true; + if (request.args[2] === "paperclip-runner-recovery-check") { + const signalled = script.includes('kill -"$6"'); + const exitCode = runnerStopped ? 45 : 0; + if (signalled) runnerStopped = true; + return { exitCode, timedOut: false, stderr: "", stdout: "" }; + } + if (sessionArchived && script.includes("runner-state.json")) return { exitCode: 1, timedOut: false, stderr: "", stdout: "" }; + return { exitCode: 0, timedOut: false, stderr: "", + stdout: request.args[0] === "--build-metadata" ? JSON.stringify({ + schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd", + packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2, + durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"], + providerPacks: installed ? [{ root: "/opt/paperclip-runner/provider-pack", manifest: installed, immutable: true }] : [], + }) : script.includes("base64") ? Buffer.from(JSON.stringify({ ...identity, lifecycle: "suspended" })).toString("base64") + : script.includes("cat --") ? `nonce\n123\n2026-09-19T10:00:00.000Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : "" }; + }); + state.createBackend.mockClear(); state.createTransport.mockClear(); + const nativeIdentity = { ...current.binding, sessionId: current.session.normalizedSessionId! }; + const openSession = vi.fn(async () => ({ close: async () => undefined, identity: () => nativeIdentity })); + const freshOpenSession = vi.fn(async () => ({ close: async () => undefined, identity: () => nativeIdentity })); + state.createBackend.mockReturnValueOnce({ kind: "test", openSession, descriptor: () => ({ name: "reattached" }) } as never) + .mockReturnValueOnce({ kind: "test", openSession: freshOpenSession, descriptor: () => ({ name: "replacement" }) } as never); + const backend = await createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId, + runnerIngressAuthorized: true, + controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-version", digest: `sha256:${"a".repeat(64)}` } }, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", + environmentId: "environment", leaseId: identity.environmentLeaseId, providerKey: "daytona", runner: { execute }, + effectiveCapabilities: { runnerWebSocketIngress: true }, + sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never, + restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller", + controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, + remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } }, + }); + const factory = state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!; + factory(); + expect(state.createTransport.mock.calls.at(-1)![0]).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: 123 } }); + const spawnedBefore = driver.mock.calls.at(-1)![0].processStarted; + expect(backend.descriptor()).toEqual({ name: "reattached" }); + const replaced = await backend.openReplacementSession!({ identity: nativeIdentity } as never, {} as never); + expect(runnerStopped).toBe(true); + const signalIndex = execute.mock.calls.findIndex(([request]) => request.args[1]?.includes('kill -"$6"')); + const archiveIndex = execute.mock.calls.findIndex(([request]) => request.args[1]?.includes(".continuity-break-")); + expect(signalIndex).toBeGreaterThanOrEqual(0); + expect(signalIndex).toBeLessThan(archiveIndex); + expect(backend.descriptor()).toEqual({ name: "replacement" }); + expect(backend.bindManagedSession(replaced)).toBe(replaced); + expect(openSession).not.toHaveBeenCalled(); + expect(freshOpenSession).toHaveBeenCalledOnce(); + expect(state.createBackend).toHaveBeenCalledTimes(2); + expect(execute.mock.calls.some(([request]) => request.args[1]?.includes(".continuity-break-"))).toBe(true); + state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); + const replacement = state.createTransport.mock.calls.at(-1)![0]; + expect(replacement.adoptExistingRunner).toBeUndefined(); + expect(replacement.prpIdentity).toMatchObject({ runId: current.binding.runId, turnId: `turn-${current.binding.runId}`, + itemId: `item-${current.binding.runId}` }); + const spawnedAfter = driver.mock.calls.at(-1)![0].processStarted!; + expect(spawnedAfter).not.toBe(spawnedBefore); + expect(await Promise.race([spawnedAfter.then(() => "spawned"), Promise.resolve("pending")])).toBe("pending"); + if (provider === "acpx") { + expect(replacement).toMatchObject({ providerNodeCommand: expect.any(String), acpxSidecarPath: expect.any(String) }); + expect(admittedNativeProviderPack(replaced, nativeIdentity)).toMatchObject({ + authority: "core_verified_selection", providerPackDigest: (installed as { digest: string }).digest }); + } + }); + + it("refuses a remote replacement and keeps the saved state when the adopted runner's stop is unproven", async () => { + const provider = "codex" as "codex" | "acpx"; + const base = scopedNativeExecution(`replacement-unproven-stop`); + const current = { ...base, ...(provider === "acpx" ? { provider: { kind: "acpx", agent: "claude", model: null }, + session: { ...base.session, driverKind: "acpx_runtime" } } : {}) } as NativeExecutionInputV1; + const driver = vi.spyOn(nativeExecutionDriver, "createNativeRunnerExecutionDriver"); + onTestFinished(() => driver.mockRestore()); + const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!, + runnerInstanceId: "admitted-codex", environmentLeaseId: "admitted-lease", + turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` }; + const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId); + await mkdir(join(root, "control-plane"), { recursive: true }); + await writeFile(join(root, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity))); + let installed: unknown = null; + if (provider === "acpx") { + const packRoot = await mkdtemp(join(tmpdir(), "paperclip-replacement-pack-")); + onTestFinished(() => rm(packRoot, { recursive: true, force: true })); + const digest = "sha256:" + "a".repeat(64); + const canonical = (value: unknown): string => Array.isArray(value) ? `[${value.map(canonical).join(",")}]` + : value && typeof value === "object" ? `{${Object.keys(value).sort().map(key => + `${JSON.stringify(key)}:${canonical((value as Record)[key])}`).join(",")}}` : JSON.stringify(value); + const artifacts = Object.fromEntries(Object.entries({ + grokLauncher: "dist/providers/grok/launcher.cjs", nodeCommand: "node_modules/node/bin/node", + productionLock: "pnpm-lock.yaml", opencodeCommand: "node_modules/.bin/opencode", + opencodeExecutable: "node_modules/opencode-ai/bin/opencode.exe", + opencodeProxy: "dist/cli/opencode-app-server-proxy.cjs", acpxSidecar: "dist/cli/acpx-runtime-sidecar.cjs", + }).map(([key, path]) => [key, { path, sha256: digest }])); + const payload = { compatibility: { opencode: 1, acpx: 1 }, contractDigest: digest, + target: { platform: "linux", architecture: "x64" }, runnerSourceRevision: "a".repeat(40), + pins: { nodeMinimum: "24.11.0", codex: QUALIFIED_ACPX_PROFILES.codex.agentRuntimeVersion, + opencode: "1.18.34", acpx: "0.13.1", grok: QUALIFIED_ACPX_PROFILES.grok.agentRuntimeVersion, + claudeAcp: QUALIFIED_ACPX_PROFILES.claude.agentServerVersion, codexAcp: QUALIFIED_ACPX_PROFILES.codex.agentServerVersion }, + acpxProfileDigests: Object.fromEntries(["grok", "claude", "codex"].map(agent => + [agent, QUALIFIED_ACPX_PROFILES[agent as "grok" | "claude" | "codex"].commandDigest])), + distDigest: digest, artifacts, + bridgeDigest: "sha256:" + createHash("sha256").update(digest + "\n" + digest + "\n" + digest).digest("hex") }; + installed = { schema: "paperclip-runner/remote-provider-pack/v1", payload, + digest: "sha256:" + createHash("sha256").update(canonical(payload)).digest("hex") }; + state.bundledPackManifestPath = join(packRoot, "provider-pack.json"); + await writeFile(state.bundledPackManifestPath, JSON.stringify(installed)); + state.bundledRunnerBinary = join(packRoot, "linux-paperclip-runnerd"); + const daemon = Buffer.alloc(64); daemon.set([0x7f, 0x45, 0x4c, 0x46, 2, 1]); daemon.writeUInt16LE(62, 18); + await writeFile(state.bundledRunnerBinary, daemon); + state.resolveRemoteRunnerBinary.mockResolvedValue(state.bundledRunnerBinary); + state.resolveRunnerBinary.mockReturnValue(state.bundledRunnerBinary); + } + let runnerStopped = false, sessionArchived = false; + const execute = vi.fn(async (request: { args: string[] }) => { + const script = request.args[1] ?? ""; + if (script.includes(".continuity-break-")) sessionArchived = true; + if (request.args[2] === "paperclip-runner-recovery-check") { + const signalled = script.includes('kill -"$6"'); + const exitCode = runnerStopped ? 45 : 0; + if (signalled) runnerStopped = true; + return { exitCode, timedOut: false, stderr: "", stdout: "" }; + } + if (sessionArchived && script.includes("runner-state.json")) return { exitCode: 1, timedOut: false, stderr: "", stdout: "" }; + return { exitCode: 0, timedOut: false, stderr: "", + stdout: request.args[0] === "--build-metadata" ? JSON.stringify({ + schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd", + packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2, + durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"], + providerPacks: installed ? [{ root: "/opt/paperclip-runner/provider-pack", manifest: installed, immutable: true }] : [], + }) : script.includes("base64") ? Buffer.from(JSON.stringify({ ...identity, lifecycle: "suspended" })).toString("base64") + : script.includes("cat --") ? `nonce\n123\n2026-09-19T10:00:00.000Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : "" }; + }); + state.createBackend.mockClear(); state.createTransport.mockClear(); + const nativeIdentity = { ...current.binding, sessionId: current.session.normalizedSessionId! }; + const openSession = vi.fn(async () => ({ close: async () => undefined, identity: () => nativeIdentity })); + const freshOpenSession = vi.fn(async () => ({ close: async () => undefined, identity: () => nativeIdentity })); + state.createBackend.mockReturnValueOnce({ kind: "test", openSession, descriptor: () => ({ name: "reattached" }) } as never); + const backend = await createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId, + runnerIngressAuthorized: true, + controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-version", digest: `sha256:${"a".repeat(64)}` } }, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", + environmentId: "environment", leaseId: identity.environmentLeaseId, providerKey: "daytona", runner: { execute }, + effectiveCapabilities: { runnerWebSocketIngress: true }, + sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never, + restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller", + controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, + remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } }, + }); + const factory = state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!; + factory(); + expect(state.createTransport.mock.calls.at(-1)![0]).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: 123 } }); + const spawnedBefore = driver.mock.calls.at(-1)![0].processStarted; + execute.mockImplementation(async (request: { args: string[] }) => request.args[2] === "paperclip-runner-recovery-check" + ? { exitCode: 0, timedOut: true, stderr: "", stdout: "" } + : { exitCode: 0, timedOut: false, stderr: "", stdout: "" }); + await expect(backend.openReplacementSession!({ identity: nativeIdentity } as never, {} as never)) + .rejects.toThrow("native_runner_replacement_stop_unproven"); + expect(execute.mock.calls.some(([request]) => request.args[1]?.includes(".continuity-break-"))).toBe(false); + expect(execute.mock.calls.some(([request]) => request.args[1]?.includes('kill -"$6"'))).toBe(false); + expect(freshOpenSession).not.toHaveBeenCalled(); + expect(state.createBackend).toHaveBeenCalledTimes(1); + expect(existsSync(join(root, "control-plane", "control-plane-state.json"))).toBe(true); + }); + it.each(["failed", "timeout", "succeeded"] as const)("retains the local controller checkpoint until remote continuity archiving succeeds: %s", async mode => { const current = scopedNativeExecution("failed-remote-continuity-archive"); state.createBackend.mockClear(); state.createTransport.mockClear(); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index aec710c4219..452ee43d4ce 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -6438,7 +6438,7 @@ export async function stopStaleRemoteRunnerBeforeBootstrap(input: { verify?: typeof verifyRemoteRunnerReattachment; waitMs?: number; pollMs?: number; -} = {}): Promise<"no_prior_runner" | "absent_confirmed" | "stopped" | "killed" | "stop_failed"> { +} = {}): Promise<"no_prior_runner" | "absent_confirmed" | "stopped" | "killed" | "stop_failed" | "stop_unproven"> { const target = input.runnerExecutionTarget; if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner) return "no_prior_runner"; const providerLeaseId = target.sandboxLeaseAcquisition?.providerLeaseId; @@ -6454,7 +6454,7 @@ export async function stopStaleRemoteRunnerBeforeBootstrap(input: { } catch (error) { // No runner with this run's exact identity and generation is alive here. return error instanceof RemoteRunnerRecoveryError && error.remoteRunnerRecovery.reason === "process_not_running" - ? "absent_confirmed" : "no_prior_runner"; + ? "absent_confirmed" : "stop_unproven"; } try { let outcome: "stopped" | "killed" = "stopped"; @@ -16801,11 +16801,15 @@ async function createRunnerdBackendWithinSessionClaim( await recordAdmittedRemoteProviderPack(providerPackAuthority, message => input.onLog?.("stderr", message)); } }; + let replacementBackend: ManagedRunnerdBackend | null = null; return { - bindManagedSession: wrapManagedSession, - providerAdmissionRequired: () => Boolean(artifactStorage - && (providerArtifacts || !legacyProviderContinuity && !liveProviderRecovery)), + bindManagedSession: (session, options) => replacementBackend + ? replacementBackend.bindManagedSession(session, options) : wrapManagedSession(session), + providerAdmissionRequired: () => replacementBackend?.providerAdmissionRequired + ? replacementBackend.providerAdmissionRequired() + : Boolean(artifactStorage && (providerArtifacts || !legacyProviderContinuity && !liveProviderRecovery)), providerAdmissionPersisted: artifactStorage ? async () => { + if (replacementBackend) return replacementBackend.providerAdmissionPersisted?.(); const backupOwner = sessionHarnessBackupOwners.get(authorityEpoch); if ((liveProviderRecovery || legacyProviderContinuity) && !providerArtifacts) return; if (!backupOwner || !expectedProviderPackManifest || !remoteCommandRunner) { @@ -16823,6 +16827,7 @@ async function createRunnerdBackendWithinSessionClaim( source => retainNativeProviderArtifacts({ db: input.db, storage: artifactStorage, owner: backupOwner, source, signal })); } : undefined, bindDiagnosticSession: session => { + if (replacementBackend) return replacementBackend.bindDiagnosticSession?.(session); const retained = nativeSessionDiagnostics(session); if (retained && retained !== diagnostics) { const lease = retained.activate(diagnosticOwner, input.onLog); @@ -16830,6 +16835,7 @@ async function createRunnerdBackendWithinSessionClaim( } else diagnostics.bind(session); }, sessionCheckpointPersisted: snapshot => { + if (replacementBackend) return replacementBackend.sessionCheckpointPersisted?.(snapshot); const staged = runnerControlPlane?.store.state.stagedRunAttachment; if (staged && snapshot.identity.runId === staged.targetIdentity.runId && snapshot.identity.sessionId === staged.targetIdentity.normalizedSessionId) { @@ -16838,7 +16844,7 @@ async function createRunnerdBackendWithinSessionClaim( } } }, - descriptor: () => backend.descriptor(), + descriptor: () => (replacementBackend ?? backend).descriptor(), openSession: async (sessionInput) => { cursorCloudRuntime?.register(); try { @@ -16859,12 +16865,33 @@ async function createRunnerdBackendWithinSessionClaim( } catch (error) { cursorCloudRuntime?.unregister(); throw error; } }, openReplacementSession: async (sessionInput, previous) => { + const reattachClaim = input.restartRecovery?.kind === "reattach_remote_runner" ? input.restartRecovery : null; + if (!reattachClaim && input.remoteWarmRecovery) { + throw new Error("native_runner_replacement_requires_fresh_launch"); + } + if (reattachClaim) { + const stopped = await stopStaleRemoteRunnerBeforeBootstrap({ execution: input.execution, + runnerExecutionTarget: input.runnerExecutionTarget, identity: durableIdentity ?? {}, onLog: input.onLog }); + if (stopped === "stop_failed" || stopped === "stop_unproven") { + throw new Error(`native_runner_replacement_${stopped}`); + } + } await measureNativeRunnerSpan( input.trace, "provider.session.archive_before_replacement", () => archiveContinuityState(previous), { parentName: "native.session.execute" }, ); + if (reattachClaim) { + const { remote: _remote, stagedAttachment: _staged, ...claim } = reattachClaim; + const fresh = await createRunnerdBackendWithinSessionClaim({ + ...input, + remoteWarmRecovery: undefined, + restartRecovery: { ...claim, kind: "bootstrap_incomplete" }, + }, sessionScopeId); + replacementBackend = fresh; + return fresh.openSession(sessionInput); + } cursorCloudRuntime?.register(); try { const session = await backend.openSession(sessionInput);