Persist portable Runner ownership through controller recovery (#21066)

## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed 2026-11-12 10:43:48 -08:00
1 parent a4a0668a5c
commit a924d2e2c7
40 files changed
+1585 -223

No files matched your search

@@ -9,8 +9,10 @@ import * as stagedAttachmentRecovery from "./native-staged-attachment-recovery.j
import * as controllerRecovery from "./native-controller-recovery.js";
import { isSettledRemoteWarmControlState } from "./remote-warm-session-recovery.js";
import * as remoteWarmRecoveryRuntime from "./remote-warm-session-recovery.js";
import * as pluginTaskOwnership from "../plugin-task-ownership.js";
import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js";
import * as environmentTaskRuntime from "../environment-task-runtime.js";
import * as pluginCredentialWriteback from "../plugin-task-credential-writeback.js";
import * as pluginTaskRuntime from "./plugin-runner-task.js";
import * as completionFeedback from "./native-completion-feedback.js";
import { environmentTaskOperationSchema } from "@paperclipai/plugin-sdk";
@@ -19,7 +21,7 @@ import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@open
import { configuredEnvironment, configuredEnvironmentProjection } from "../../vendor/paperclip-runner/index.js";
import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from "vitest";
import { createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../system-recovery-state.js";
import { readFileSync } from "node:fs";
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { once } from "node:events";
import { PgDialect } from "drizzle-orm/pg-core";
import type { SQL } from "drizzle-orm";
@@ -14176,15 +14178,20 @@ describe("native warm session supervision", () => {
{ name: "rotated provider key", firstEnvironment: { OPENAI_API_KEY: "fixture-key" }, secondEnvironment: { OPENAI_API_KEY: "fixture-rotated-key" }, reuse: false },
{ name: "rotated Claude token", firstEnvironment: { CLAUDE_CODE_OAUTH_TOKEN: "fixture-token" }, secondEnvironment: { CLAUDE_CODE_OAUTH_TOKEN: "fixture-rotated-token" }, reuse: false },
{ name: "tool refresh", refreshTools: true, reuse: false },
{ name: "unsupported tool refresh", refreshTools: true, unsupportedRefresh: true, reuse: false },
{ name: "network access change", firstEnvironment: { PAPERCLIP_RUNNER_NETWORK_ACCESS: "disabled" }, secondEnvironment: { PAPERCLIP_RUNNER_NETWORK_ACCESS: "enabled" }, reuse: false },
{ name: "run-scoped environment broker", firstEnvironment: { PAPERCLIP_GITHUB_BROKER_TOKEN: "first-run-capability" }, secondEnvironment: { PAPERCLIP_GITHUB_BROKER_TOKEN: "second-run-capability" }, reuse: false },
{ name: "Core-hosted GitHub access", managedGitHub: true, reuse: true, adoptionReuse: false },
] as Array<{ name: string; firstEnvironment?: Record<string, string>; secondEnvironment?: Record<string, string>;
refreshTools?: boolean; managedGitHub?: boolean; reuse: boolean; adoptionReuse?: boolean }>)(
refreshTools?: boolean; unsupportedRefresh?: boolean; managedGitHub?: boolean; reuse: boolean; adoptionReuse?: boolean }>)(
"decides warm and adopted process reuse from one key: $name",
async ({ name, firstEnvironment = {}, secondEnvironment = {}, refreshTools, managedGitHub = false, reuse, adoptionReuse = reuse }) => {
async ({ name, firstEnvironment = {}, secondEnvironment = {}, refreshTools, unsupportedRefresh = false,
managedGitHub = false, reuse, adoptionReuse = reuse }) => {
const decisions: Record<string, boolean> = {};
for (const path of ["warm", "adoption"] as const) {
for (const path of ["warm", "adoption", ...(refreshTools ? ["portable", "portable-bootstrap", "legacy-plugin"] as const : [])] as const) {
const portable = path === "portable" || path === "portable-bootstrap";
const bootstrap = path === "portable-bootstrap";
const pluginRecovery = portable || path === "legacy-plugin";
githubAccess.create.mockReset().mockImplementation(async () => ({ env: {}, ready: true, duplexRequested: false,
activate: githubAccess.activate.mockReset().mockImplementation(() => vi.fn()), stop: githubAccess.stop.mockReset().mockResolvedValue(undefined) }));
const stateBase = await mkdtemp(join(tmpdir(), "paperclip-runnerd-reuse-key-"));
@@ -14195,15 +14202,24 @@ describe("native warm session supervision", () => {
const scope = `${path}-${name.replaceAll(" ", "-")}`;
const first = {
...execution,
...(pluginRecovery ? { schema: "paperclip.native-execution-input.v5", executionMode: "default",
provider: { ...execution.provider, approvalPolicy: "never" }, planningContext: null, runtimeContext: nativeRuntimeContextFixture() } as const : {}),
binding: { ...execution.binding, runId: `run-reuse-key-first-${scope}`, executionWorkspaceId: "workspace-reuse-key" },
workspace: { cwd: "/tmp/runnerd-reuse-key", repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: `session-reuse-key-${scope}`, driverKind: "codex_app_server" as const,
protocolVersion: 1 as const, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
} as NativeExecutionInputV1;
const second = { ...first, binding: { ...first.binding, runId: `run-reuse-key-second-${scope}` } } as NativeExecutionInputV1;
} as NativeExecutionInput;
if (pluginRecovery) parseNativeExecutionInput(first);
const second = { ...first, binding: { ...first.binding, runId: `run-reuse-key-second-${scope}` } };
if ("runtimeContext" in second) {
second.runtimeContext = structuredClone(second.runtimeContext);
second.runtimeContext.mcp.digest = "1".repeat(64);
second.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(second.runtimeContext);
}
const target = { kind: "remote" as const, transport: "sandbox" as const, environmentId: "environment-reuse-key",
remoteCwd: "/home/daytona/paperclip-workspace", runner: { execute: vi.fn() } } as never;
remoteCwd: "/home/daytona/paperclip-workspace", runner: { execute: vi.fn() } };
const firstSession = { close: vi.fn(async () => undefined) };
const recoveredSession = { close: vi.fn(async () => undefined), snapshot: vi.fn() };
const result = { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" }, turnId: "turn",
normalizedSessionId: first.session.normalizedSessionId, providerSessionId: "provider-reuse-key", driverKind: "test",
driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
@@ -14219,36 +14235,138 @@ describe("native warm session supervision", () => {
return result;
})
.mockImplementationOnce(async (options) => {
decisions[path] = options.existingSession === firstSession;
decisions[path] = options.existingSession === (portable ? recoveredSession : firstSession);
return result;
});
const inspected = vi.spyOn(remoteWarmRecoveryRuntime, "inspectRemoteWarmSessionRecovery").mockImplementation(async request => {
decisions[path] = request.canReuseProcess(first);
const reuseProcess = request.canReuseProcess(first);
if (pluginRecovery) {
expect(reuseProcess).toBe(false);
expect(request.matchesExecution(first)).toBe(!unsupportedRefresh);
if (unsupportedRefresh) throw new Error("fixture unsupported tool refresh refused");
expect(request.matchesExecution({ ...first, provider: { ...first.provider, model: "another-model" } } as NativeExecutionInput)).toBe(false);
if ("runtimeContext" in first) {
const changed = structuredClone(first);
changed.runtimeContext.instructions.bundle.digest = "2".repeat(64);
changed.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(changed.runtimeContext);
expect(request.matchesExecution(changed)).toBe(false);
}
const checkpoint = request.loadCheckpoint(first);
expect(checkpoint?.identity.runId).toBe(first.binding.runId);
recoveredSession.snapshot.mockResolvedValue(checkpoint);
return { identity: request.identity as Record<string, string>, checkpoint: checkpoint!,
previousExecution: first, reuseProcess, ...(portable ? { portableSourceTaskId: "source-task" } : {}),
remote: { providerLeaseId: "target-task", remoteCwd: target.remoteCwd }, revalidate: async () => {} };
}
decisions[path] = reuseProcess;
throw new Error("fixture adoption decision recorded");
});
const ownership = portable
? vi.spyOn(pluginTaskOwnership, "readyPortableTaskSource").mockImplementation(async (_db, _company, _lease, identity, scope) => {
expect(scope?.accountIdentity).toBe("fixture-account");
return { taskId: "source-task", identity: identity as NonNullable<Awaited<ReturnType<typeof pluginTaskOwnership.readyPortableTaskSource>>>["identity"] };
}) : undefined;
const projection = vi.spyOn(controllerRecovery, "resolveNativeControllerRecovery").mockResolvedValue({ kind: "no_prior_execution" });
const restore = path === "legacy-plugin" ? vi.spyOn(runnerCheckpoint, "restoreNativeRunnerCheckpoint")
.mockImplementation(async (_db, binding) => binding.runId === first.binding.runId && recoveredSession.close.mock.calls.length ? {} as never : null) : undefined;
const issueProjection = path === "legacy-plugin" ? vi.spyOn(issueServiceModule, "issueService")
.mockReturnValue({ update: vi.fn(async () => null) } as never) : undefined;
try {
await executePaperclipNativeSession({ db: leaseDb(first), execution: first, runnerInstanceId: "runner-reuse-key",
useRunnerd: true, managedGitHub, runnerExecutionTarget: target,
useRunnerd: true, managedGitHub, runnerExecutionTarget: target as never,
...(portable ? { managedAiAccountIdentity: "fixture-account", managedAiCredentialIdentity: "old-revision" } : {}),
runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...firstEnvironment } });
const durableRoot = (await readdir(stateBase, { withFileTypes: true }))
.filter(entry => entry.isDirectory() && /^[a-f0-9]{64}$/.test(entry.name)).map(entry => join(stateBase, entry.name))[0]!;
const durableIdentity = { runId: first.binding.runId, normalizedSessionId: first.session.normalizedSessionId,
runnerInstanceId: "runner-reuse-key", environmentLeaseId: first.binding.executionWorkspaceId };
runnerInstanceId: "runner-reuse-key", environmentLeaseId: first.binding.executionWorkspaceId,
turnId: "prior-turn", itemId: "prior-item" };
await mkdir(join(durableRoot, "control-plane"), { recursive: true });
await writeFile(join(durableRoot, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(durableIdentity)));
await mkdir(join(durableRoot, "runner"), { recursive: true });
await writeFile(join(durableRoot, "runner", "runner-state.json"), JSON.stringify(durableRunnerState(durableIdentity, "ready")));
if (path === "adoption") expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
if (path !== "warm") expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
if (portable) {
const directory = join(resolvePaperclipInstanceRoot(), "runtime/paperclip-runner/sessions");
const path = join(directory, (await readdir(directory))[0]!);
const { snapshot, configDigest, processConfigDigest } = JSON.parse(await readFile(path, "utf8"));
await rm(path);
await rm(durableRoot, { recursive: true });
projection.mockResolvedValueOnce({
kind: "recoverable", snapshot,
metadata: controllerRecovery.createNativeControllerRecovery(snapshot, { configDigest, processConfigDigest }, null),
checkpoint: { restoreSessionDirectory(root: string) {
mkdirSync(join(root, "control-plane"), { recursive: true });
writeFileSync(join(root, "control-plane/control-plane-state.json"), JSON.stringify(durableControlPlaneState(durableIdentity)));
} } as never,
});
}
if (pluginRecovery) state.createBackend.mockReturnValueOnce({
kind: "test", recoverSession: async () => ({ recovered: true, session: recoveredSession }),
} as never);
const pluginTaskDescription = { kind: "description", taskId: "target-task", warmTaskVersion: 1,
launchPreparationVersion: 1, projectAuthCheckVersion: 1, releaseResumeVersion: 1,
runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } } as const;
const portableTarget = { ...target, runner: undefined, leaseId: "target-lease",
sandboxLeaseAcquisition: { outcome: path === "legacy-plugin" ? "resumed" : "created", providerLeaseId: "target-task" },
runnerTask: vi.fn(async (operation: { kind: string }) => {
if (path === "legacy-plugin") {
expect(recoveredSession.close).toHaveBeenCalledOnce();
expect(operation.kind).toBe("describe");
throw new Error("fixture replacement inspection reached");
}
return pluginTaskDescription;
}),
};
if (pluginRecovery) await writeFile(join(stateBase, "auth.json"),
JSON.stringify({ OPENAI_API_KEY: "synthetic-fixture" }), { mode: 0o600 });
const leaseOwner = bootstrap ? "bootstrap-recovery-owner" : undefined;
const controller = bootstrap ? await import("./native-restart-recovery.js").then(module => module.currentNativeControllerIdentity()) : undefined;
const db = leaseDb(second, controller ? { leaseOwner, controllerBootId: controller.bootId,
controllerPid: controller.pid, controllerGeneration: 1, leaseExpiresAt: new Date(Date.now() + 60_000) } : {});
const select = db.select.bind(db);
db.select = ((fields?: Record<string, unknown>) => {
if (fields?.status === heartbeatRuns.status && fields?.runnerProfileJson === heartbeatRuns.runnerProfileJson) {
return { from: () => ({ where: () => ({ limit: async () =>
[{ status: "succeeded", runnerProfileJson: { nativeExecutionInput: first } }] }) }) };
}
if (pluginRecovery && fields?.metadata === environmentLeases.metadata) {
return { from: () => ({ where: async () => [{ providerLeaseId: pluginTaskDescription.taskId, unlaunchedPortable: bootstrap,
metadata: ownership ? { portableTaskPreviousLeaseId: "source-lease", portableTaskOwnershipReady: true } : {} }] }) };
}
return select(fields as never);
}) as Db["select"];
const continuation = executePaperclipNativeSession({
db: { ...leaseDb(second), select: () => ({ from: () => ({ where: () => ({ limit: () =>
Promise.resolve([{ status: "succeeded", runnerProfileJson: { nativeExecutionInput: first } }]) }) }) }) } as unknown as Db,
execution: second, runnerInstanceId: "runner-reuse-key", useRunnerd: true, managedGitHub, refreshTools,
runnerExecutionTarget: target, runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...secondEnvironment } });
db, execution: second, runnerInstanceId: "runner-reuse-key", useRunnerd: true, managedGitHub, refreshTools, toolRefreshOnResume: !unsupportedRefresh,
runnerExecutionTarget: (pluginRecovery ? portableTarget : target) as never,
...(leaseOwner ? { leaseOwner, restartRecovery: { kind: "bootstrap_incomplete" as const, runId: second.binding.runId,
leaseOwner, controllerGeneration: 1, providerAttempt: 0, restartKind: "hard" as const, recoveryRequestId: null,
portablePreparationLeaseId: portableTarget.leaseId } } : {}),
...(pluginRecovery ? { pluginTaskDescription, runnerIngressAuthorized: true, managedAiCredentialHome: stateBase,
managedAiAccountIdentity: "fixture-account", managedAiCredentialIdentity: "current-revision" } : {}),
runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...secondEnvironment } });
if (pluginRecovery && unsupportedRefresh) {
await expect(continuation).rejects.toThrow("fixture unsupported tool refresh refused");
continue;
}
if (path === "legacy-plugin") {
await expect(continuation).rejects.toThrow("fixture replacement inspection reached");
expect(portableTarget.runnerTask).toHaveBeenCalledOnce();
continue;
}
if (path === "adoption") await expect(continuation).rejects.toThrow("fixture adoption decision recorded");
else await continuation;
expect(decisions[path]).toBe(path === "adoption" ? adoptionReuse : reuse);
expect(decisions[path]).toBe(portable ? true : path === "adoption" ? adoptionReuse : reuse);
if (portable) {
expect(portableTarget.runnerTask).not.toHaveBeenCalled();
expect(recoveredSession.close).not.toHaveBeenCalled();
}
if (path === "warm") expect(firstSession.close).toHaveBeenCalledTimes(reuse ? 0 : 1);
} finally {
projection.mockRestore();
restore?.mockRestore();
issueProjection?.mockRestore();
ownership?.mockRestore();
inspected.mockRestore();
await closeIdleWarmNativeSessionsForRestart();
if (previousStateDirectory === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
@@ -14258,7 +14376,7 @@ describe("native warm session supervision", () => {
await rm(stateBase, { recursive: true, force: true });
}
}
expect(decisions).toEqual({ warm: reuse, adoption: adoptionReuse });
expect(decisions).toEqual({ warm: reuse, adoption: adoptionReuse, ...(refreshTools && !unsupportedRefresh ? { portable: true, "portable-bootstrap": true } : {}) });
},
);
@@ -17963,8 +18081,9 @@ describe("runnerd provider runtime wiring", () => {
} finally { resume.mockRestore(); }
});
it.each(["verified", "wrong-state", "wrong-lease"] as const)("reattaches a surviving plugin task through its owner: %s", async mode => {
it.each(["verified", "portable", "wrong-claim", "wrong-state", "wrong-lease"] as const)("reattaches a surviving plugin task through its owner: %s", async mode => {
const current = scopedNativeExecution(`plugin-live-${mode}`);
const moved = mode === "portable" || mode === "wrong-claim";
const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!,
runnerInstanceId: "plugin-runner", environmentLeaseId: "same-task-lease", turnId: "admitted-turn", itemId: "admitted-item" };
const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId);
@@ -17980,24 +18099,35 @@ describe("runnerd provider runtime wiring", () => {
runner: { version: "new-artifact", digest: `sha256:${"b".repeat(64)}` } };
});
state.createBackend.mockClear(); state.createTransport.mockClear();
const backend = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
runnerIngressAuthorized: true, runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".",
leaseId: mode === "wrong-lease" ? "replacement-task" : identity.environmentLeaseId, runnerTask: task },
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-artifact", digest: `sha256:${"a".repeat(64)}` } },
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller", controllerGeneration: 2,
providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, remote: { kind: "plugin_task", leaseId: identity.environmentLeaseId } } });
if (mode !== "verified") {
await expect(backend).rejects.toThrow(mode === "wrong-state" ? "adoption identity mismatch" : "recovery_lease_mismatch");
expect(state.createBackend).not.toHaveBeenCalled();
return;
}
await backend;
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
const options = state.createTransport.mock.calls.at(-1)![0];
expect(options).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: null, startedAt: null },
runnerArtifact: { version: "admitted-artifact" } });
expect(options).toMatchObject({ runnerReconnectGraceMs: 420_000 });
expect(task.mock.calls.some(([op]) => ["prepare", "submit", "stop"].includes(op.kind))).toBe(false);
const portable = moved ? vi.spyOn(pluginTaskOwnership, "assertAttachedPortableTask")
.mockImplementation(async () => {
if (mode === "wrong-claim") throw new Error("native_portable_task_ownership_unverified");
}) : undefined;
const leaseId = moved ? "moved-task-lease" : identity.environmentLeaseId;
try {
const backend = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
runnerIngressAuthorized: true, runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".",
leaseId: mode === "wrong-lease" ? "replacement-task" : leaseId, runnerTask: task,
...(moved ? { sandboxLeaseAcquisition: { outcome: "resumed" as const, providerLeaseId: "same-task" } } : {}) },
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-artifact", digest: `sha256:${"a".repeat(64)}` } },
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller", controllerGeneration: 2,
providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, remote: { kind: "plugin_task", leaseId } } });
if (mode !== "verified" && mode !== "portable") {
await expect(backend).rejects.toThrow(mode === "wrong-state" ? "adoption identity mismatch" : mode === "wrong-claim" ? "ownership_unverified" : "recovery_lease_mismatch");
expect(state.createBackend).not.toHaveBeenCalled();
return;
}
await backend;
if (moved) expect(portable).toHaveBeenCalledWith(expect.anything(), {
companyId: current.binding.companyId, leaseId, taskId: "same-task", runId: current.binding.runId, identity,
});
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
const options = state.createTransport.mock.calls.at(-1)![0];
expect(options).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: null, startedAt: null },
runnerArtifact: { version: "admitted-artifact" } });
expect(options).toMatchObject({ runnerReconnectGraceMs: 420_000 });
expect(task.mock.calls.some(([op]) => ["prepare", "submit", "stop"].includes(op.kind))).toBe(false);
} finally { portable?.mockRestore(); }
});
it.each(["codex", "acpx-codex", "grok"] as const)("writes managed %s events through the current warm exporter without per-turn reads", async provider => {
@@ -18175,8 +18305,11 @@ describe("runnerd provider runtime wiring", () => {
} finally {owner.mockRestore();await rm(home,{recursive:true,force:true});}
});
it.each((["codex", "claude"] as const).flatMap(provider => [true, false].map(reuseProcess => [provider, reuseProcess] as const)))("reconnects task-only %s warm recovery under the saved identity before reuse or retirement: %s", async (provider, reuseProcess) => {
const current=scopedNativeExecution(`plugin-restart-${provider}-${reuseProcess}`);
it.each([
...(["codex", "claude"] as const).flatMap(provider => [true, false].map(reuseProcess => [provider, reuseProcess, false] as const)),
["codex", true, true] as const,
])("reconnects task-only %s warm recovery under the saved identity before reuse or retirement: %s (portable=%s)", async (provider, reuseProcess, portable) => {
const current=scopedNativeExecution(`plugin-restart-${provider}-${reuseProcess}-${portable}`);
if(provider === "claude") current.provider={kind:"acpx",agent:"claude",model:"fixture",permissionMode:"deny-all"} as never;
current.session.lifecyclePolicy={mode:"warm",idleTimeoutMs:300000};
const identity={runnerInstanceId:"old-runner",environmentLeaseId:"old-lease",runId:"old-run",
@@ -18184,22 +18317,29 @@ describe("runnerd provider runtime wiring", () => {
const home=await mkdtemp(join(tmpdir(),"plugin-restart-account-"));
await writeFile(join(home,"auth.json"),JSON.stringify({OPENAI_API_KEY:"synthetic-fixture"}),{mode:0o600});
const owner=vi.spyOn(pluginTaskRuntime,"createPluginRunnerTaskOwner");
const register = vi.spyOn(pluginCredentialWriteback, "registerPluginTaskCredentialHandoff").mockResolvedValue(undefined);
const binding = { grantId: randomUUID(), secretId: randomUUID(), sessionEpoch: 1, versionSelector: "latest" as const };
const task=vi.fn(async()=>({kind:"description",taskId:"saved-task",launchPreparationVersion:1,projectAuthCheckVersion:1,
warmTaskVersion:1,runner:{version:"fixture",digest:`sha256:${"a".repeat(64)}`}}));
state.createBackend.mockClear();state.createTransport.mockClear();
try {
await createRunnerdBackend({db:leaseDb(current),execution:current,runnerInstanceId:"new-runner",runnerIngressAuthorized:true,
runnerEnvironment:{ANTHROPIC_API_KEY:"synthetic-fixture"},managedAiCredentialHome:home,managedAiCredentialIdentity:reuseProcess ? "old-revision" : "new-revision",managedAiAccountIdentity:"account",
managedAiCredentialWriteback:{} as never,
managedAiCredentialWriteback: binding,
remoteWarmRecovery:{identity,reuseProcess,previousExecution:current,checkpoint:{} as never,
remote:{providerLeaseId:"saved-task",remoteCwd:"."},revalidate:async()=>{}},
remote:{providerLeaseId:"saved-task",remoteCwd:"."},revalidate:async()=>{},
...(portable ? { portableSourceTaskId: "old-task" } : {})},
runnerExecutionTarget:{kind:"remote",transport:"sandbox",leaseId:"new-lease",remoteCwd:".",runnerTask:task,
sandboxLeaseAcquisition:{outcome:"resumed",providerLeaseId:"saved-task"}}});
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
expect(state.createTransport.mock.calls.at(-1)![0].prpIdentity).toEqual(identity);
expect(owner.mock.calls.at(-1)![0].adoptIdentity).toEqual(identity);
expect(task.mock.calls).toEqual([[{kind:"describe"}]]);
} finally {owner.mockRestore();await rm(home,{recursive:true,force:true});}
if (portable) expect(register).toHaveBeenCalledExactlyOnceWith(expect.anything(), {
companyId: current.binding.companyId, leaseId: "new-lease",
}, "account", binding);
else expect(register).not.toHaveBeenCalled();
} finally {register.mockRestore(); owner.mockRestore();await rm(home,{recursive:true,force:true});}
});
it.each((["codex", "claude"] as const).flatMap(provider => [