Persist portable Runner ownership through controller recovery (#21066)

## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed 2026-11-12 10:43:48 -08:00
1 parent a4a0668a5c
commit a924d2e2c7
40 files changed
+1585 -223

No files matched your search

@@ -54,7 +54,8 @@ import { writeExecutionControl } from "../execution-control-notifications.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import { createRunUsageRecorder } from "../usage-receipts.js";
import { syncPluginTaskCredentialHandoff, registerPluginTaskCredentialHandoff } from "../plugin-task-credential-writeback.js";
import { resolvePluginTaskResume } from "../environment-task-runtime.js";
import { resolvePluginTaskResume, type PluginTaskDescription } from "../environment-task-runtime.js";
import { assertAttachedPortableTask, readyPortableTaskSource, unlaunchedPortableTaskLease } from "../plugin-task-ownership.js";
import { environmentTaskOperationSchema, environmentTaskResultSchema } from "@paperclipai/plugin-sdk";
import { preparePluginRunnerCredentials } from "./plugin-runner-credentials.js";
import { createPluginRunnerTaskOwner, verifyPluginRunnerTaskReattachment } from "./plugin-runner-task.js";
@@ -278,7 +279,7 @@ import {
renderNativeRunnerStagedAttachmentPrompt,
stageNativeRunnerWakeAttachments,
} from "./native-runner-file-handoff.js";
import { NATIVE_PROVIDER_SIDE_EFFECT_PENDING, NATIVE_PROVIDER_STATE_INCOMPATIBLE, nativeRuntimeContractForProvider, nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
import { NATIVE_PROVIDER_SIDE_EFFECT_PENDING, NATIVE_PROVIDER_STATE_INCOMPATIBLE, nativeRuntimeContextsMatch, nativeRuntimeContractForProvider, nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
import { clearRunProviderStateUnsaved, markAcceptedRunProviderStateUnsaved } from "./native-provider-state-unsaved.js";
import { verifyRetainedMaintenanceNoLaunch } from "./native-maintenance-no-launch.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
@@ -6247,6 +6248,29 @@ async function recordPriorOwnerFenceRefusal(input: { db: Db; execution: NativeEx
}).catch(() => undefined);
}
async function assertPluginTaskReattachmentBinding(input: {
db: Db;
execution: NativeExecutionInput;
restartRecovery?: NativeRestartRecoveryClaim;
}, target: Extract<AdapterExecutionTarget, { kind: "remote" }>, identity: Record<string, unknown> | null | undefined) {
const claim = input.restartRecovery;
const leaseId = target.leaseId;
if (claim?.kind !== "reattach_remote_runner" || claim.remote.kind !== "plugin_task" ||
claim.runId !== input.execution.binding.runId || !leaseId || claim.remote.leaseId !== leaseId ||
!identity || !durableIdentityMatchesExecution(identity, input.execution) ||
typeof identity.turnId !== "string" || !identity.turnId || typeof identity.itemId !== "string" || !identity.itemId) {
throw new Error("native_plugin_recovery_lease_mismatch");
}
const taskId = target.transport === "sandbox" ? target.sandboxLeaseAcquisition?.providerLeaseId : undefined;
if (identity.environmentLeaseId !== leaseId) {
if (!taskId) throw new Error("native_plugin_recovery_lease_mismatch");
await assertAttachedPortableTask(input.db, {
companyId: input.execution.binding.companyId, leaseId, taskId, runId: claim.runId, identity,
});
}
return taskId;
}
async function migrateRunnerdStateRootForExecution(input: {
db: Db;
execution: NativeExecutionInput;
@@ -6370,12 +6394,12 @@ async function migrateRunnerdStateRootForExecution(input: {
if (input.restartRecovery?.kind === "reattach_remote_runner") {
if (input.restartRecovery.remote.kind === "plugin_task") {
const target = input.runnerExecutionTarget;
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runnerTask
|| target.leaseId !== input.restartRecovery.remote.leaseId || identity.environmentLeaseId !== target.leaseId
|| !durableIdentityMatchesExecution(identity, input.execution)) {
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runnerTask) {
throw new Error("native_plugin_recovery_lease_mismatch");
}
await verifyPluginRunnerTaskReattachment(target.runnerTask, identity);
const taskId = await assertPluginTaskReattachmentBinding(input, target, identity);
await verifyPluginRunnerTaskReattachment(operation => target.runnerTask!(operation,
taskId ? { expectedTaskId: taskId } : undefined), identity);
return;
}
try {
@@ -7281,6 +7305,19 @@ function nativeSessionConfigDigest(
.digest("hex")}`;
}
function matchesRemoteWarmExecution(
previous: NativeExecutionInput,
current: NativeExecutionInput,
toolRefreshOnResume?: boolean,
): boolean {
if (nativeSessionScopeKey(previous) !== nativeSessionScopeKey(current)) return false;
const currentDigest = nativeSessionConfigDigest(current, "remote");
if (nativeSessionConfigDigest(previous, "remote") === currentDigest) return true;
return toolRefreshOnResume === true && "runtimeContext" in previous && "runtimeContext" in current &&
nativeRuntimeContextsMatch(previous.runtimeContext, current.runtimeContext, toolRefreshOnResume) &&
nativeSessionConfigDigest({ ...previous, runtimeContext: current.runtimeContext }, "remote") === currentDigest;
}
export function nativeControllerMigrationDescriptor(execution: NativeExecutionInput, recordedToolContractFingerprint: unknown) {
if (typeof recordedToolContractFingerprint !== "string" || !/^sha256:[a-f0-9]{64}$/.test(recordedToolContractFingerprint)) {
throw new Error("native_controller_migration_tool_contract_missing");
@@ -10203,6 +10240,7 @@ export async function executePaperclipNativeSession(input: {
getFreshSessionHandoff?: () => Promise<string | null>;
/** Retire a retained transport so recovery can replace provider tool declarations. */
refreshTools?: boolean;
toolRefreshOnResume?: boolean;
db: Db;
execution: NativeExecutionInput;
/** Private server preparation proof; never persisted or accepted from a tool. */
@@ -10259,6 +10297,7 @@ export async function executePaperclipNativeSession(input: {
managedAiCredentialWriteback?: ManagedAiCredentialWritebackBinding;
billingIdentity?: Pick<AdapterExecutionResult, "provider" | "biller" | "billingType">;
onTaskDispatched?: () => Promise<void> | void;
pluginTaskDescription?: PluginTaskDescription;
runnerExecutionTarget?: AdapterExecutionTarget | null;
/** Resolved per-run authorization; not an independent instance setting. */
runnerIngressAuthorized?: boolean;
@@ -10657,7 +10696,22 @@ async function executePaperclipNativeSessionAttempt(
const stagedRestart = input.restartRecovery?.kind === "reattach_remote_runner"
? await verifyStagedAttachmentRestart(input) : null;
const timeController = createRunPreparationTiming(input.execution.binding.runId, "native.controller");
if (input.useRunnerd) await timeController("warm_restore", () => restoreWarmRunnerdController(input));
const target = input.runnerExecutionTarget;
const portableTarget = target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask ? target : undefined;
const companyId = input.execution.binding.companyId;
const portablePreparationLeaseId = input.restartRecovery?.kind === "bootstrap_incomplete"
? input.restartRecovery.portablePreparationLeaseId : undefined;
const targetLease = portableTarget?.leaseId ? (await input.db.select({
metadata: environmentLeases.metadata, providerLeaseId: environmentLeases.providerLeaseId,
...(portablePreparationLeaseId ? { unlaunchedPortable: unlaunchedPortableTaskLease() } : {}),
}).from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId), eq(environmentLeases.id, portableTarget.leaseId))))[0] : undefined;
if (portablePreparationLeaseId && (portableTarget?.leaseId !== portablePreparationLeaseId || targetLease?.unlaunchedPortable !== true)) {
throw remoteRecoveryLeaseMismatch("runner_verification", "plugin_task");
}
const replayPortablePreparation = portablePreparationLeaseId !== undefined;
const canRecoverWarmOwner = !input.restartRecovery || replayPortablePreparation;
if (input.useRunnerd) await timeController("warm_restore", () => restoreWarmRunnerdController(
replayPortablePreparation ? { ...input, restartRecovery: undefined } : input));
const identityRedactor = createAgentIdentityRedactor(input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY);
const redactIdentityText = (text: string) => redactSensitiveText(
identityRedactor.chunk("diagnostic", text) + identityRedactor.finish("diagnostic"),
@@ -10757,29 +10811,51 @@ async function executePaperclipNativeSessionAttempt(
});
let retainedTransition: VerifiedWarmTransitionBinding | undefined;
let remoteWarmRecovery: RemoteWarmSessionRecovery | undefined;
let pluginTaskDescription: Extract<ReturnType<typeof environmentTaskResultSchema.parse>,{kind:"description"}> | undefined;
let pluginTaskDescription: PluginTaskDescription | undefined;
let portableSource: Awaited<ReturnType<typeof readyPortableTaskSource>>;
if (portableTarget?.runnerTask && portableTarget.leaseId) {
if (input.pluginTaskDescription && input.pluginTaskDescription.taskId === targetLease?.providerLeaseId) {
pluginTaskDescription = input.pluginTaskDescription;
}
if (targetLease?.metadata?.portableTaskPreviousLeaseId || targetLease?.metadata?.portableTaskPredecessor) {
const description = targetLease.metadata.portableTaskOwnershipReady === true && pluginTaskDescription
? pluginTaskDescription : environmentTaskResultSchema.parse(await portableTarget.runnerTask({ kind: "describe" }));
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
pluginTaskDescription = description;
portableSource = await readyPortableTaskSource(input.db, companyId, portableTarget.leaseId,
readRunnerdDurableIdentity(scopedRunnerdStateRoot(input.execution)), {
attachedRunId: input.execution.binding.runId,
selectedSessionId: nativeSessionKey(input.execution),
accountIdentity: input.managedAiAccountIdentity ?? input.managedAiCredentialIdentity,
});
}
}
const cachedReleasedOwner = warmNativeSessions.get(nativeSessionScopeKey(input.execution));
if (cachedReleasedOwner?.released) {
const proof = releasedWarmRunner(readWarmNativeEnvelope(input.execution));
const proof = portableSource ? undefined : releasedWarmRunner(readWarmNativeEnvelope(input.execution));
const identity = readRunnerdDurableIdentity(scopedRunnerdStateRoot(input.execution));
const target = input.runnerExecutionTarget;
if (proof && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId
&& target.sandboxLeaseAcquisition?.outcome === "created" && proof.taskId !== target.sandboxLeaseAcquisition.providerLeaseId) {
const source = portableSource ?? (proof && { taskId: proof.taskId, identity: proof.source });
if (source && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId
&& (portableSource || target.sandboxLeaseAcquisition?.outcome === "created")
&& source.taskId !== target.sandboxLeaseAcquisition?.providerLeaseId) {
const ownInstructionPreparation = cachedReleasedOwner.instructionPreparationRunId === input.execution.binding.runId
&& input.instructionWorkingCopy?.runId === input.execution.binding.runId;
if ((cachedReleasedOwner.busy && !ownInstructionPreparation)
|| cachedReleasedOwner.preparingRunId && cachedReleasedOwner.preparingRunId !== input.execution.binding.runId
|| cachedReleasedOwner.instructionPreparationRunId && !ownInstructionPreparation
|| cachedReleasedOwner.closeOnReleaseReason) throw new Error("native_session_supervisor_busy");
if (proof.taskId !== cachedReleasedOwner.pluginTaskId || proof.scopeDigest !== releasedWarmScope(input)
|| canonicalJson(proof.source) !== canonicalJson(identity)) throw new Error("native_session_released_owner_scope_mismatch");
if (source.taskId !== cachedReleasedOwner.pluginTaskId || cachedReleasedOwner.runId !== identity?.runId ||
(proof && (proof.scopeDigest !== releasedWarmScope(input) || canonicalJson(proof.source) !== canonicalJson(identity)))) {
throw new Error("native_session_released_owner_scope_mismatch");
}
const wasBusy = cachedReleasedOwner.busy;
cachedReleasedOwner.busy = true;
try {
// Only physical stop releases the cached controller; ordinary checkpoint admission still proves continuation.
const resumeFrom = await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
leaseId: target.leaseId, priorIdentity: proof.source, inspectCheckpoint: true });
if (resumeFrom !== proof.taskId) throw new Error("native_session_released_owner_scope_mismatch");
// Ordinary replacement still needs a stop receipt for the old task.
const resumeFrom = portableSource?.taskId ?? await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
leaseId: target.leaseId, priorIdentity: source.identity, inspectCheckpoint: true });
if (resumeFrom !== source.taskId) throw new Error("native_session_released_owner_scope_mismatch");
try {
await detachReleasedWarmNativeSession(cachedReleasedOwner);
} catch (error) {
@@ -10790,7 +10866,7 @@ async function executePaperclipNativeSessionAttempt(
} finally { cachedReleasedOwner.busy = wasBusy; }
} else requireReleasedWarmScope(input, proof, identity);
}
if (input.useRunnerd && !input.restartRecovery && !input.backend &&
if (input.useRunnerd && canRecoverWarmOwner && !input.backend &&
input.runnerExecutionTarget?.kind === "remote" &&
input.execution.session.lifecyclePolicy.mode === "warm" &&
!idleWarmNativeSessionOwnerRunId(input)) {
@@ -10800,10 +10876,12 @@ async function executePaperclipNativeSessionAttempt(
if (taskTarget.transport === "sandbox" && taskTarget.runnerTask && taskTarget.leaseId &&
durableIdentityMatchesSession(identity,input.execution) && identity.runId !== input.execution.binding.runId) {
// Inspection can replace a stopped retained task before deciding between adoption and checkpoint recovery.
const description = environmentTaskResultSchema.parse(await taskTarget.runnerTask({kind:"describe"}));
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
pluginTaskDescription = description;
if (taskTarget.sandboxLeaseAcquisition?.outcome !== "resumed") {
if (!pluginTaskDescription || pluginTaskDescription.taskId !== taskTarget.sandboxLeaseAcquisition?.providerLeaseId) {
const description = environmentTaskResultSchema.parse(await taskTarget.runnerTask({ kind: "describe" }));
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
pluginTaskDescription = description;
}
if (taskTarget.sandboxLeaseAcquisition?.outcome !== "resumed" && !portableSource) {
await recoverPluginRunnerCheckpoint(input, root);
}
}
@@ -10817,22 +10895,23 @@ async function executePaperclipNativeSessionAttempt(
}
if (durableIdentityMatchesSession(identity, input.execution) && identity.runId !== input.execution.binding.runId &&
runnerdAuthorityLifecycleWithVerifiedBackup({ root, identity, execution: input.execution, allowVerifiedBackup: true }) !== "suspended") {
let releasedSource: DurableRecoveryIdentity | undefined;
let releasedSource = portableSource?.identity;
remoteWarmRecovery = await inspectRemoteWarmSessionRecovery({
db: input.db, execution: input.execution, target: input.runnerExecutionTarget, identity,
portableSourceTaskId: portableSource?.taskId,
readControlState: () => record(JSON.parse(readBoundedNativeFile(
resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
"runner_durable_identity_too_large").toString("utf8"))),
matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(input.execution, "remote"),
matchesExecution: previous => matchesRemoteWarmExecution(previous, input.execution, input.toolRefreshOnResume),
loadCheckpoint: previous => {
const proof = releasedWarmRunner(readWarmNativeEnvelope(previous));
if (proof) {
releasedSource = requireReleasedWarmScope(input, proof, identity);
if (!portableSource) {
const proof = releasedWarmRunner(readWarmNativeEnvelope(previous));
if (proof) releasedSource = requireReleasedWarmScope(input, proof, identity);
}
return loadWarmNativeCheckpoint(previous, nativeSessionConfigDigest(previous, "remote"), "remote");
},
canReuseProcess: previous => !needsCoreGitHubAccess(input) && canReuseNativeProcess(input,
canReuseProcess: previous => !portableSource && !needsCoreGitHubAccess(input) &&
nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(input.execution, "remote") && canReuseNativeProcess(input,
record(JSON.parse(readBoundedNativeFile(nativeSessionCheckpointPath(previous),
NATIVE_WARM_CHECKPOINT_MAX_BYTES, "native_session_supervisor_checkpoint_too_large").toString("utf8"))).processConfigDigest),
}) ?? undefined;
@@ -11061,7 +11140,7 @@ async function executePaperclipNativeSessionAttempt(
: coordinator.controllerGeneration + 1;
const target = input.runnerExecutionTarget;
const retained = warmNativeSessions.get(nativeSessionScopeKey(input.execution));
if (!input.restartRecovery && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId &&
if (canRecoverWarmOwner && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId &&
((retained?.pluginTaskId && idleWarmNativeSessionOwnerRunId(input) !== undefined) || remoteWarmRecovery?.releasedSource) &&
(input.execution.provider.kind === "codex" || (input.execution.provider.kind === "acpx" && input.execution.provider.agent === "claude"))) {
claimedStartupContext = new NativeExecutionContext(input.execution, Math.min(executionLeaseTtlMs,
@@ -11801,7 +11880,7 @@ async function executePaperclipNativeSessionAttempt(
executionContext = undefined;
startupForAdmission!.executionContext = undefined;
};
if (warmSessionId !== null && warmConfigDigest !== null && !input.restartRecovery) {
if (warmSessionId !== null && warmConfigDigest !== null && canRecoverWarmOwner) {
const registered = warmNativeSessions.get(warmSessionId);
const entry = registered && forgetUnretiredOwnerOnOtherSandbox(warmSessionId, registered, input.runnerExecutionTarget)
? undefined : registered;
@@ -12088,11 +12167,15 @@ async function executePaperclipNativeSessionAttempt(
signal: executionContext?.signal ?? new AbortController().signal,
});
if (!recovered?.recovered || !recovered.session) throw new Error("runner_state_identity_mismatch: remote_warm_session_unproven");
if (remoteWarmRecovery.reuseProcess) {
const movedSession = remoteWarmRecovery.portableSourceTaskId !== undefined;
// A portable resume already starts a new Runner. Its run attachment
// prepares the current tools and runtime on the verified target.
if (movedSession || remoteWarmRecovery.reuseProcess) {
existingWarmSession = recovered.session;
await emitStartupReuse(input.onEvent, "runner", { state: "reused", reason: "verified_reuse" });
await emitStartupReuse(input.onEvent, "runner", movedSession
? { state: "fresh", reason: "replacement" } : { state: "reused", reason: "verified_reuse" });
await emitStartupReuse(input.onEvent, "provider_session", { state: "reused", reason: "verified_reuse" });
trace.setBudgetStart("warm");
if (!movedSession) trace.setBudgetStart("warm");
persistedWarmSession = null;
} else {
await recovered.session.close({ reason: "recovered warm session requires current process configuration" });
@@ -12104,12 +12187,22 @@ async function executePaperclipNativeSessionAttempt(
remoteWarmRecovery.previousExecution.binding, scopedRunnerdStateRoot(input.execution),
nativeSessionKey(input.execution));
if (!suspendedController) throw new Error("native_runner_checkpoint_missing_after_suspension");
persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution,
nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote"), await recovered.session.snapshot());
persistedWarmSession = loadWarmNativeCheckpoint(input.execution, warmConfigDigest!, "remote");
runnerdBackend = await createRunnerdBackend({ ...input, onDiagnosticLease: ownDiagnosticLease, providerPackAdmissionSource, recoveryPurpose: claimedStopCleanup ? "stop_cleanup" : "execution", onAccountingEvent: producerUsage?.capture, runnerInstanceId: effectiveRunnerInstanceId,
const verifiedConfigDigest = nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote");
persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution, verifiedConfigDigest, await recovered.session.snapshot());
persistedWarmSession = loadWarmNativeCheckpoint(input.execution, verifiedConfigDigest, "remote");
runnerdBackend = await createRunnerdBackend({
...input,
pluginTaskDescription: undefined,
onDiagnosticLease: ownDiagnosticLease,
providerPackAdmissionSource,
recoveryPurpose: claimedStopCleanup ? "stop_cleanup" : "execution",
onAccountingEvent: producerUsage?.capture,
runnerInstanceId: effectiveRunnerInstanceId,
controllerPersistence: input.runnerExecutionTarget?.kind === "remote" ? { bootId: controller.bootId } : undefined,
durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId, trace, toolTrace });
durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId,
trace,
toolTrace,
});
}
}
const readRemoteCleanupLease = async () => input.runnerExecutionTarget?.kind === "remote" &&
@@ -15710,6 +15803,7 @@ export function remoteCheckpointIncompleteFailure(
export async function createRunnerdBackend(input: {
executionContext?: NativeExecutionContext;
casesEnabled?: boolean;
toolRefreshOnResume?: boolean;
db: Db;
providerPackAdmissionSource?: NativeProviderPackAdmissionSource;
controllerPersistence?: { bootId: string; runnerArtifact?: { version: string; digest: string } };
@@ -15724,7 +15818,7 @@ export async function createRunnerdBackend(input: {
remoteWarmRecovery?: Omit<RemoteWarmSessionRecovery, "revalidate"> & { revalidate: () => Promise<void> };
retainedPluginTaskOwner?: ReturnType<typeof createPluginRunnerTaskOwner>;
beginRetainedRunAttachment?: () => void;
pluginTaskDescription?: Extract<ReturnType<typeof environmentTaskResultSchema.parse>,{kind:"description"}>;
pluginTaskDescription?: PluginTaskDescription;
onSpawn?: (meta: {
pid: number;
processGroupId: number | null;
@@ -16128,8 +16222,10 @@ async function createRunnerdBackendWithinSessionClaim(
const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity
? createNativeRecoveryUsageDelivery({
db: input.db, execution: input.execution, sourceIdentity: { ...recoveryUsageIdentity },
matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind),
matchesExecution: previous => input.remoteWarmRecovery
? matchesRemoteWarmExecution(previous, input.execution, input.toolRefreshOnResume)
: nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind),
})
: undefined;
const accountingTransportOptions = {
@@ -16185,15 +16281,19 @@ async function createRunnerdBackendWithinSessionClaim(
const { root, durableIdentity, durableBinding } = durableState ??= readDurableState();
const provider = input.execution.provider;
if (provider.kind !== "codex" && provider.kind !== "opencode" && provider.kind !== "acpx") throw new Error("Plugin Runner tasks require a workspace harness");
if (input.restartRecovery?.kind === "reattach_remote_runner") {
await assertPluginTaskReattachmentBinding(input, remoteTarget, durableBinding ? durableIdentity : null);
}
let priorDescription = input.pluginTaskDescription?.taskId === remoteTarget.sandboxLeaseAcquisition?.providerLeaseId
? input.pluginTaskDescription : undefined;
if (remoteTarget.sandboxLeaseAcquisition?.outcome === "resumed" && !durableBinding && !input.retainedPluginTaskOwner && !input.remoteWarmRecovery) {
// A new logical session (including fresh retries) cannot inherit the old process authority.
await pluginTask({kind:"stop"});
const stopped=environmentTaskResultSchema.parse(await pluginTask({kind:"status"}));
if(stopped.kind!=="status" || stopped.executionStopped!==true) throw new Error("Plugin Runner termination is unconfirmed; reconcile the saved task");
await pluginTask({kind:"complete"});
priorDescription = undefined;
}
const priorDescription = input.pluginTaskDescription?.taskId === remoteTarget.sandboxLeaseAcquisition?.providerLeaseId
? input.pluginTaskDescription : undefined;
const description = priorDescription ?? environmentTaskResultSchema.parse(await pluginTask({ kind: "describe" }));
if (description.kind !== "description" || description.launchPreparationVersion !== 1 || description.projectAuthCheckVersion !== 1) throw new Error("Plugin Runner launch preparation v1 unavailable");
const saved = durableBinding ? durableIdentity as RunnerdDurableIdentity : undefined;
@@ -16212,9 +16312,7 @@ async function createRunnerdBackendWithinSessionClaim(
normalizedSessionId: input.remoteWarmRecovery.identity.normalizedSessionId,
turnId: input.remoteWarmRecovery.identity.turnId, itemId: input.remoteWarmRecovery.identity.itemId,
} : undefined;
if (input.restartRecovery?.kind === "reattach_remote_runner" && (!prior || !prior.turnId || !prior.itemId || prior.environmentLeaseId !== remoteTarget.leaseId)) {
throw new Error("native_plugin_recovery_lease_mismatch");
}
const resumeFrom = !retainedOwner && !adopting && prior && prior.environmentLeaseId !== remoteTarget.leaseId
? await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
leaseId: remoteTarget.leaseId!, priorIdentity: prior }) : undefined;
@@ -16224,7 +16322,8 @@ async function createRunnerdBackendWithinSessionClaim(
home: input.managedAiCredentialHome, environment: input.runnerEnvironment ?? {},
});
if (credentials.needsWriteback && !input.managedAiCredentialWriteback) throw new Error("Plugin Runner credential writeback authority unavailable");
if (!adopting && input.managedAiCredentialWriteback && credentials.accountIdentity) {
// Same-lease recovery already has a writeback binding. A portable move creates a new lease.
if ((!adopting || input.remoteWarmRecovery?.portableSourceTaskId) && input.managedAiCredentialWriteback && credentials.accountIdentity) {
if (!remoteTarget.leaseId) throw new Error("Plugin Runner credential lease unavailable");
await registerPluginTaskCredentialHandoff(input.db, { companyId: input.execution.binding.companyId, leaseId: remoteTarget.leaseId },
credentials.accountIdentity, input.managedAiCredentialWriteback);
@@ -16251,7 +16350,8 @@ async function createRunnerdBackendWithinSessionClaim(
releaseResumeVersion: description.releaseResumeVersion,
authorityRunId: description.releaseResumeVersion === 1 ? input.execution.binding.runId : undefined,
releasedSource: input.remoteWarmRecovery?.releasedSource,
onResumed: async source => updateReleasedWarmRunner(input.execution, source, description.taskId, null),
sessionHandoff: !!input.remoteWarmRecovery?.portableSourceTaskId,
onResumed: async source => updateReleasedWarmRunner(input.execution, source, input.remoteWarmRecovery?.portableSourceTaskId ?? description.taskId, null),
onRetired: async source => updateReleasedWarmRunner(input.execution, source, description.taskId, null),
adoptIdentity: adoptionIdentity ?? (adopting && prior ? {
runnerInstanceId: prior.runnerInstanceId, environmentLeaseId: prior.environmentLeaseId,