From 2eede8b3183f877f9afbf06655aa1eee07590d28 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 10 Nov 2026 19:14:43 -0600 Subject: [PATCH] fix(native): give a relaunched settled runner its admitted provider pack (#20683) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - After a server restart or a same-run retry, a remote native run reattaches to its runner in the sandbox (`reattach_remote_runner`). > - When the controller saved the runner artifact, the executor prepares that run as an adoption. It skips provider-pack selection, because a live adopted runner keeps its launched provider. > - The reattach probe can instead find the runner settled and exited (`verifyRemoteRunnerOrSettledExit`). Then nothing is adopted, and the transport relaunches the runner from its durable state. That relaunch must start the provider again, so it needs the provider-pack launch inputs. > - The executor did not give them. OpenCode failed with "OpenCode executable is missing from the provider pack". ACPX failed with "ACPX launch profile omitted its provider-pack authority or verified artifact digests". Both ended the run. > - This pull request selects the saved (admitted) provider-pack authority on that settled-exit relaunch and verifies it as an exact installed identity, as an ordinary reattach does. > - The benefit is that a restart or a same-run retry can resume an exited runner with the same provider pack, instead of failing. ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, two OpenCode runs and one ACPX (Claude) run were interrupted by a deploy (SIGTERM) or ended a turn without a result. The server scheduled a same-run retry that reattached to the existing remote runner. The runner had exited cleanly (`runner_remote_process_exited lifecycle=suspended`). The retry then failed within three seconds with `runner_remote_provider_artifact_incompatible`. **Steps to reproduce** 1. Prepare a remote OpenCode or ACPX run with a persisted runner artifact (`controllerPersistence.runnerArtifact`). 2. Reattach with `restartRecovery.kind = "reattach_remote_runner"`, where the runner process is not running and its state is `suspended` and settled. 3. Observe that the transport options have no `adoptExistingRunner`, no `providerPackAuthorityDigest` and no `opencodeCommand` / `acpxSidecarSha256`. **Expected behavior** The relaunch gets the admitted provider pack's launch inputs, or it fails with a clear pack-identity error. ## What Changed - `createRunnerdBackend` (native session executor): when a persisted-runner reattachment finds no live runner to adopt, select the saved provider-pack authority (`reattach: true`), so `expectedProviderPackManifest` is the admitted pack. - `prepareRemoteRunnerAssets` no longer returns early for that relaunch. It verifies the installed pack as an exact identity and sets the active pack root, the same path as a reattach without a persisted artifact. - A live adopted runner is unchanged: it still skips pack selection. - The test `uses the bundled Linux image authority for remote opencode|claude-x64` now also covers the settled-exit relaunch for OpenCode and ACPX. ## Verification - RED on `origin/master` (`c444da7d05`): the extended test fails for `opencode` and `claude-x64` (the relaunch options have no provider-pack inputs). - GREEN with this change: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "uses the bundled Linux image authority"` (14 passed). - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify`: see the result below. ## Risks - Upgrade safety: the relaunch reads the saved authority, so a newer controller does not select a newer pack for an older session. It requires an exact installed identity. If the sandbox no longer has that pack, the relaunch fails with a pack-identity error, and the existing Retry path applies, as before. - No change to a live adoption, to fresh runs or to the normal path before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude --- .../native-session-executor.test.ts | 24 +++++++++++++++++++ .../native-runtime/native-session-executor.ts | 13 +++++++++- 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 928353e57be..b67247d57a4 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -2650,6 +2650,30 @@ describe("remote provider pack manifest", () => { leaseOwner: "controller", controllerGeneration: 2, providerAttempt: 1, restartKind: "graceful", recoveryRequestId: null, remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } }, }; + // Staging failure (OpenCode and ACPX): a same-run + // retry under a persisted runner artifact found the runner settled and + // exited. Nothing is adopted, so the transport relaunches it from its + // durable state and needs the admitted provider pack's launch inputs. + const settledExecute = vi.fn(async (request: { args: string[] }) => { + if (request.args[2] === "paperclip-runner-recovery-check") return { exitCode: 45, timedOut: false, stderr: "", stdout: "" }; + if (request.args[1]?.includes("base64")) return { exitCode: 0, timedOut: false, stderr: "", stdout: Buffer.from(JSON.stringify({ + ...identity, lifecycle: "suspended", outbox: [], pendingTerminalDelivery: null, pendingProviderCleanup: null, warmTransition: null, + })).toString("base64") }; + return recoveryExecute(request); + }); + const settledTarget = { ...(recoveryInput.runnerExecutionTarget as unknown as Record), + runner: { execute: settledExecute, syncIn: vi.fn(async () => undefined), syncOut: vi.fn(async () => undefined) } }; + await createRunnerdBackend({ ...recoveryInput, runnerExecutionTarget: settledTarget as never, + controllerPersistence: { bootId: "replacement-controller", runnerArtifact: artifact } }); + state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); + const relaunched = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions; + expect(relaunched.adoptExistingRunner).toBeUndefined(); + expect(relaunched).toMatchObject({ providerPackAuthorityDigest: admitted.digest, runnerArtifact: artifact, + providerNodeCommandSha256: admitted.payload.artifacts.nodeCommand.sha256, + ...(opencode + ? { opencodeCommand: `/opt/paperclip-runner/provider-pack/${admitted.payload.artifacts.opencodeExecutable.path}`, + opencodeCommandSha256: admitted.payload.artifacts.opencodeExecutable.sha256 } + : { acpxSidecarSha256: (admitted.payload.artifacts as Record).acpxSidecar.sha256 }) }); const recoveredBackend = await createRunnerdBackend(recoveryInput); state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); const recovered = state.createTransport.mock.calls.at(-1)![0]; diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 37a7a8631d2..fb5c533b658 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -15435,6 +15435,10 @@ async function createRunnerdBackendWithinSessionClaim( let expectedProviderPackManifest: RemoteProviderPackManifest | null = providerPackAuthority?.manifest ?? providerArtifacts?.manifest as RemoteProviderPackManifest ?? null; let providerPackVerified = false; + // Set when a persisted-runner reattachment finds that runner settled and + // exited. The transport then relaunches it from durable state, which needs + // the admitted provider pack's launch inputs that adoption does not. + let relaunchesSettledRemoteRunner = false; const explicitRemoteCodex = input.runnerRemoteCodexPath?.trim() || null; const remoteCodexNpmSpec = input.runnerRemoteCodexNpmSpec?.trim() || null; if (explicitRemoteCodex && remoteCodexNpmSpec) { @@ -15595,7 +15599,7 @@ async function createRunnerdBackendWithinSessionClaim( return; selectedRemoteMode = requiredMode; if (input.restartRecovery?.kind === "reattach_remote_runner" || input.remoteWarmRecovery) { - if (input.controllerPersistence?.runnerArtifact) { + if (input.controllerPersistence?.runnerArtifact && !relaunchesSettledRemoteRunner) { remotePrepared = true; return; } @@ -16690,6 +16694,13 @@ async function createRunnerdBackendWithinSessionClaim( signalVerifiedRecoveryProcess(localRecoveryProcess, signal), } : undefined; + if (adoptingPersistedRemoteRunner && !adoptedProcess && requiresRemoteProviderPack) { + // Bind the relaunch to the pack this session was admitted with (the saved + // authority, verified as an exact installed identity), never a newer one. + relaunchesSettledRemoteRunner = true; + providerPackAuthority = selectProviderPackAuthority(true); + expectedProviderPackManifest = providerPackAuthority?.manifest ?? expectedProviderPackManifest; + } if (adoptedProcess && remoteTarget) { const markSpawned = resolveRemoteRunnerProcessSpawned as (() => void) | null; markSpawned?.();