mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
a4a0668a5c
commit
a924d2e2c7
40 files changed
+1585
-223
No files matched your search
@@ -54,7 +54,8 @@ import { writeExecutionControl } from "../execution-control-notifications.js";
|
||||
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
|
||||
import { createRunUsageRecorder } from "../usage-receipts.js";
|
||||
import { syncPluginTaskCredentialHandoff, registerPluginTaskCredentialHandoff } from "../plugin-task-credential-writeback.js";
|
||||
import { resolvePluginTaskResume } from "../environment-task-runtime.js";
|
||||
import { resolvePluginTaskResume, type PluginTaskDescription } from "../environment-task-runtime.js";
|
||||
import { assertAttachedPortableTask, readyPortableTaskSource, unlaunchedPortableTaskLease } from "../plugin-task-ownership.js";
|
||||
import { environmentTaskOperationSchema, environmentTaskResultSchema } from "@paperclipai/plugin-sdk";
|
||||
import { preparePluginRunnerCredentials } from "./plugin-runner-credentials.js";
|
||||
import { createPluginRunnerTaskOwner, verifyPluginRunnerTaskReattachment } from "./plugin-runner-task.js";
|
||||
@@ -278,7 +279,7 @@ import {
|
||||
renderNativeRunnerStagedAttachmentPrompt,
|
||||
stageNativeRunnerWakeAttachments,
|
||||
} from "./native-runner-file-handoff.js";
|
||||
import { NATIVE_PROVIDER_SIDE_EFFECT_PENDING, NATIVE_PROVIDER_STATE_INCOMPATIBLE, nativeRuntimeContractForProvider, nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
|
||||
import { NATIVE_PROVIDER_SIDE_EFFECT_PENDING, NATIVE_PROVIDER_STATE_INCOMPATIBLE, nativeRuntimeContextsMatch, nativeRuntimeContractForProvider, nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
|
||||
import { clearRunProviderStateUnsaved, markAcceptedRunProviderStateUnsaved } from "./native-provider-state-unsaved.js";
|
||||
import { verifyRetainedMaintenanceNoLaunch } from "./native-maintenance-no-launch.js";
|
||||
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
||||
@@ -6247,6 +6248,29 @@ async function recordPriorOwnerFenceRefusal(input: { db: Db; execution: NativeEx
|
||||
}).catch(() => undefined);
|
||||
}
|
||||
|
||||
async function assertPluginTaskReattachmentBinding(input: {
|
||||
db: Db;
|
||||
execution: NativeExecutionInput;
|
||||
restartRecovery?: NativeRestartRecoveryClaim;
|
||||
}, target: Extract<AdapterExecutionTarget, { kind: "remote" }>, identity: Record<string, unknown> | null | undefined) {
|
||||
const claim = input.restartRecovery;
|
||||
const leaseId = target.leaseId;
|
||||
if (claim?.kind !== "reattach_remote_runner" || claim.remote.kind !== "plugin_task" ||
|
||||
claim.runId !== input.execution.binding.runId || !leaseId || claim.remote.leaseId !== leaseId ||
|
||||
!identity || !durableIdentityMatchesExecution(identity, input.execution) ||
|
||||
typeof identity.turnId !== "string" || !identity.turnId || typeof identity.itemId !== "string" || !identity.itemId) {
|
||||
throw new Error("native_plugin_recovery_lease_mismatch");
|
||||
}
|
||||
const taskId = target.transport === "sandbox" ? target.sandboxLeaseAcquisition?.providerLeaseId : undefined;
|
||||
if (identity.environmentLeaseId !== leaseId) {
|
||||
if (!taskId) throw new Error("native_plugin_recovery_lease_mismatch");
|
||||
await assertAttachedPortableTask(input.db, {
|
||||
companyId: input.execution.binding.companyId, leaseId, taskId, runId: claim.runId, identity,
|
||||
});
|
||||
}
|
||||
return taskId;
|
||||
}
|
||||
|
||||
async function migrateRunnerdStateRootForExecution(input: {
|
||||
db: Db;
|
||||
execution: NativeExecutionInput;
|
||||
@@ -6370,12 +6394,12 @@ async function migrateRunnerdStateRootForExecution(input: {
|
||||
if (input.restartRecovery?.kind === "reattach_remote_runner") {
|
||||
if (input.restartRecovery.remote.kind === "plugin_task") {
|
||||
const target = input.runnerExecutionTarget;
|
||||
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runnerTask
|
||||
|| target.leaseId !== input.restartRecovery.remote.leaseId || identity.environmentLeaseId !== target.leaseId
|
||||
|| !durableIdentityMatchesExecution(identity, input.execution)) {
|
||||
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runnerTask) {
|
||||
throw new Error("native_plugin_recovery_lease_mismatch");
|
||||
}
|
||||
await verifyPluginRunnerTaskReattachment(target.runnerTask, identity);
|
||||
const taskId = await assertPluginTaskReattachmentBinding(input, target, identity);
|
||||
await verifyPluginRunnerTaskReattachment(operation => target.runnerTask!(operation,
|
||||
taskId ? { expectedTaskId: taskId } : undefined), identity);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
@@ -7281,6 +7305,19 @@ function nativeSessionConfigDigest(
|
||||
.digest("hex")}`;
|
||||
}
|
||||
|
||||
function matchesRemoteWarmExecution(
|
||||
previous: NativeExecutionInput,
|
||||
current: NativeExecutionInput,
|
||||
toolRefreshOnResume?: boolean,
|
||||
): boolean {
|
||||
if (nativeSessionScopeKey(previous) !== nativeSessionScopeKey(current)) return false;
|
||||
const currentDigest = nativeSessionConfigDigest(current, "remote");
|
||||
if (nativeSessionConfigDigest(previous, "remote") === currentDigest) return true;
|
||||
return toolRefreshOnResume === true && "runtimeContext" in previous && "runtimeContext" in current &&
|
||||
nativeRuntimeContextsMatch(previous.runtimeContext, current.runtimeContext, toolRefreshOnResume) &&
|
||||
nativeSessionConfigDigest({ ...previous, runtimeContext: current.runtimeContext }, "remote") === currentDigest;
|
||||
}
|
||||
|
||||
export function nativeControllerMigrationDescriptor(execution: NativeExecutionInput, recordedToolContractFingerprint: unknown) {
|
||||
if (typeof recordedToolContractFingerprint !== "string" || !/^sha256:[a-f0-9]{64}$/.test(recordedToolContractFingerprint)) {
|
||||
throw new Error("native_controller_migration_tool_contract_missing");
|
||||
@@ -10203,6 +10240,7 @@ export async function executePaperclipNativeSession(input: {
|
||||
getFreshSessionHandoff?: () => Promise<string | null>;
|
||||
/** Retire a retained transport so recovery can replace provider tool declarations. */
|
||||
refreshTools?: boolean;
|
||||
toolRefreshOnResume?: boolean;
|
||||
db: Db;
|
||||
execution: NativeExecutionInput;
|
||||
/** Private server preparation proof; never persisted or accepted from a tool. */
|
||||
@@ -10259,6 +10297,7 @@ export async function executePaperclipNativeSession(input: {
|
||||
managedAiCredentialWriteback?: ManagedAiCredentialWritebackBinding;
|
||||
billingIdentity?: Pick<AdapterExecutionResult, "provider" | "biller" | "billingType">;
|
||||
onTaskDispatched?: () => Promise<void> | void;
|
||||
pluginTaskDescription?: PluginTaskDescription;
|
||||
runnerExecutionTarget?: AdapterExecutionTarget | null;
|
||||
/** Resolved per-run authorization; not an independent instance setting. */
|
||||
runnerIngressAuthorized?: boolean;
|
||||
@@ -10657,7 +10696,22 @@ async function executePaperclipNativeSessionAttempt(
|
||||
const stagedRestart = input.restartRecovery?.kind === "reattach_remote_runner"
|
||||
? await verifyStagedAttachmentRestart(input) : null;
|
||||
const timeController = createRunPreparationTiming(input.execution.binding.runId, "native.controller");
|
||||
if (input.useRunnerd) await timeController("warm_restore", () => restoreWarmRunnerdController(input));
|
||||
const target = input.runnerExecutionTarget;
|
||||
const portableTarget = target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask ? target : undefined;
|
||||
const companyId = input.execution.binding.companyId;
|
||||
const portablePreparationLeaseId = input.restartRecovery?.kind === "bootstrap_incomplete"
|
||||
? input.restartRecovery.portablePreparationLeaseId : undefined;
|
||||
const targetLease = portableTarget?.leaseId ? (await input.db.select({
|
||||
metadata: environmentLeases.metadata, providerLeaseId: environmentLeases.providerLeaseId,
|
||||
...(portablePreparationLeaseId ? { unlaunchedPortable: unlaunchedPortableTaskLease() } : {}),
|
||||
}).from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId), eq(environmentLeases.id, portableTarget.leaseId))))[0] : undefined;
|
||||
if (portablePreparationLeaseId && (portableTarget?.leaseId !== portablePreparationLeaseId || targetLease?.unlaunchedPortable !== true)) {
|
||||
throw remoteRecoveryLeaseMismatch("runner_verification", "plugin_task");
|
||||
}
|
||||
const replayPortablePreparation = portablePreparationLeaseId !== undefined;
|
||||
const canRecoverWarmOwner = !input.restartRecovery || replayPortablePreparation;
|
||||
if (input.useRunnerd) await timeController("warm_restore", () => restoreWarmRunnerdController(
|
||||
replayPortablePreparation ? { ...input, restartRecovery: undefined } : input));
|
||||
const identityRedactor = createAgentIdentityRedactor(input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY);
|
||||
const redactIdentityText = (text: string) => redactSensitiveText(
|
||||
identityRedactor.chunk("diagnostic", text) + identityRedactor.finish("diagnostic"),
|
||||
@@ -10757,29 +10811,51 @@ async function executePaperclipNativeSessionAttempt(
|
||||
});
|
||||
let retainedTransition: VerifiedWarmTransitionBinding | undefined;
|
||||
let remoteWarmRecovery: RemoteWarmSessionRecovery | undefined;
|
||||
let pluginTaskDescription: Extract<ReturnType<typeof environmentTaskResultSchema.parse>,{kind:"description"}> | undefined;
|
||||
let pluginTaskDescription: PluginTaskDescription | undefined;
|
||||
let portableSource: Awaited<ReturnType<typeof readyPortableTaskSource>>;
|
||||
if (portableTarget?.runnerTask && portableTarget.leaseId) {
|
||||
if (input.pluginTaskDescription && input.pluginTaskDescription.taskId === targetLease?.providerLeaseId) {
|
||||
pluginTaskDescription = input.pluginTaskDescription;
|
||||
}
|
||||
if (targetLease?.metadata?.portableTaskPreviousLeaseId || targetLease?.metadata?.portableTaskPredecessor) {
|
||||
const description = targetLease.metadata.portableTaskOwnershipReady === true && pluginTaskDescription
|
||||
? pluginTaskDescription : environmentTaskResultSchema.parse(await portableTarget.runnerTask({ kind: "describe" }));
|
||||
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
|
||||
pluginTaskDescription = description;
|
||||
portableSource = await readyPortableTaskSource(input.db, companyId, portableTarget.leaseId,
|
||||
readRunnerdDurableIdentity(scopedRunnerdStateRoot(input.execution)), {
|
||||
attachedRunId: input.execution.binding.runId,
|
||||
selectedSessionId: nativeSessionKey(input.execution),
|
||||
accountIdentity: input.managedAiAccountIdentity ?? input.managedAiCredentialIdentity,
|
||||
});
|
||||
}
|
||||
}
|
||||
const cachedReleasedOwner = warmNativeSessions.get(nativeSessionScopeKey(input.execution));
|
||||
if (cachedReleasedOwner?.released) {
|
||||
const proof = releasedWarmRunner(readWarmNativeEnvelope(input.execution));
|
||||
const proof = portableSource ? undefined : releasedWarmRunner(readWarmNativeEnvelope(input.execution));
|
||||
const identity = readRunnerdDurableIdentity(scopedRunnerdStateRoot(input.execution));
|
||||
const target = input.runnerExecutionTarget;
|
||||
if (proof && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId
|
||||
&& target.sandboxLeaseAcquisition?.outcome === "created" && proof.taskId !== target.sandboxLeaseAcquisition.providerLeaseId) {
|
||||
const source = portableSource ?? (proof && { taskId: proof.taskId, identity: proof.source });
|
||||
if (source && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId
|
||||
&& (portableSource || target.sandboxLeaseAcquisition?.outcome === "created")
|
||||
&& source.taskId !== target.sandboxLeaseAcquisition?.providerLeaseId) {
|
||||
const ownInstructionPreparation = cachedReleasedOwner.instructionPreparationRunId === input.execution.binding.runId
|
||||
&& input.instructionWorkingCopy?.runId === input.execution.binding.runId;
|
||||
if ((cachedReleasedOwner.busy && !ownInstructionPreparation)
|
||||
|| cachedReleasedOwner.preparingRunId && cachedReleasedOwner.preparingRunId !== input.execution.binding.runId
|
||||
|| cachedReleasedOwner.instructionPreparationRunId && !ownInstructionPreparation
|
||||
|| cachedReleasedOwner.closeOnReleaseReason) throw new Error("native_session_supervisor_busy");
|
||||
if (proof.taskId !== cachedReleasedOwner.pluginTaskId || proof.scopeDigest !== releasedWarmScope(input)
|
||||
|| canonicalJson(proof.source) !== canonicalJson(identity)) throw new Error("native_session_released_owner_scope_mismatch");
|
||||
if (source.taskId !== cachedReleasedOwner.pluginTaskId || cachedReleasedOwner.runId !== identity?.runId ||
|
||||
(proof && (proof.scopeDigest !== releasedWarmScope(input) || canonicalJson(proof.source) !== canonicalJson(identity)))) {
|
||||
throw new Error("native_session_released_owner_scope_mismatch");
|
||||
}
|
||||
const wasBusy = cachedReleasedOwner.busy;
|
||||
cachedReleasedOwner.busy = true;
|
||||
try {
|
||||
// Only physical stop releases the cached controller; ordinary checkpoint admission still proves continuation.
|
||||
const resumeFrom = await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
|
||||
leaseId: target.leaseId, priorIdentity: proof.source, inspectCheckpoint: true });
|
||||
if (resumeFrom !== proof.taskId) throw new Error("native_session_released_owner_scope_mismatch");
|
||||
// Ordinary replacement still needs a stop receipt for the old task.
|
||||
const resumeFrom = portableSource?.taskId ?? await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
|
||||
leaseId: target.leaseId, priorIdentity: source.identity, inspectCheckpoint: true });
|
||||
if (resumeFrom !== source.taskId) throw new Error("native_session_released_owner_scope_mismatch");
|
||||
try {
|
||||
await detachReleasedWarmNativeSession(cachedReleasedOwner);
|
||||
} catch (error) {
|
||||
@@ -10790,7 +10866,7 @@ async function executePaperclipNativeSessionAttempt(
|
||||
} finally { cachedReleasedOwner.busy = wasBusy; }
|
||||
} else requireReleasedWarmScope(input, proof, identity);
|
||||
}
|
||||
if (input.useRunnerd && !input.restartRecovery && !input.backend &&
|
||||
if (input.useRunnerd && canRecoverWarmOwner && !input.backend &&
|
||||
input.runnerExecutionTarget?.kind === "remote" &&
|
||||
input.execution.session.lifecyclePolicy.mode === "warm" &&
|
||||
!idleWarmNativeSessionOwnerRunId(input)) {
|
||||
@@ -10800,10 +10876,12 @@ async function executePaperclipNativeSessionAttempt(
|
||||
if (taskTarget.transport === "sandbox" && taskTarget.runnerTask && taskTarget.leaseId &&
|
||||
durableIdentityMatchesSession(identity,input.execution) && identity.runId !== input.execution.binding.runId) {
|
||||
// Inspection can replace a stopped retained task before deciding between adoption and checkpoint recovery.
|
||||
const description = environmentTaskResultSchema.parse(await taskTarget.runnerTask({kind:"describe"}));
|
||||
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
|
||||
pluginTaskDescription = description;
|
||||
if (taskTarget.sandboxLeaseAcquisition?.outcome !== "resumed") {
|
||||
if (!pluginTaskDescription || pluginTaskDescription.taskId !== taskTarget.sandboxLeaseAcquisition?.providerLeaseId) {
|
||||
const description = environmentTaskResultSchema.parse(await taskTarget.runnerTask({ kind: "describe" }));
|
||||
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
|
||||
pluginTaskDescription = description;
|
||||
}
|
||||
if (taskTarget.sandboxLeaseAcquisition?.outcome !== "resumed" && !portableSource) {
|
||||
await recoverPluginRunnerCheckpoint(input, root);
|
||||
}
|
||||
}
|
||||
@@ -10817,22 +10895,23 @@ async function executePaperclipNativeSessionAttempt(
|
||||
}
|
||||
if (durableIdentityMatchesSession(identity, input.execution) && identity.runId !== input.execution.binding.runId &&
|
||||
runnerdAuthorityLifecycleWithVerifiedBackup({ root, identity, execution: input.execution, allowVerifiedBackup: true }) !== "suspended") {
|
||||
let releasedSource: DurableRecoveryIdentity | undefined;
|
||||
let releasedSource = portableSource?.identity;
|
||||
remoteWarmRecovery = await inspectRemoteWarmSessionRecovery({
|
||||
db: input.db, execution: input.execution, target: input.runnerExecutionTarget, identity,
|
||||
portableSourceTaskId: portableSource?.taskId,
|
||||
readControlState: () => record(JSON.parse(readBoundedNativeFile(
|
||||
resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
|
||||
"runner_durable_identity_too_large").toString("utf8"))),
|
||||
matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
|
||||
nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(input.execution, "remote"),
|
||||
matchesExecution: previous => matchesRemoteWarmExecution(previous, input.execution, input.toolRefreshOnResume),
|
||||
loadCheckpoint: previous => {
|
||||
const proof = releasedWarmRunner(readWarmNativeEnvelope(previous));
|
||||
if (proof) {
|
||||
releasedSource = requireReleasedWarmScope(input, proof, identity);
|
||||
if (!portableSource) {
|
||||
const proof = releasedWarmRunner(readWarmNativeEnvelope(previous));
|
||||
if (proof) releasedSource = requireReleasedWarmScope(input, proof, identity);
|
||||
}
|
||||
return loadWarmNativeCheckpoint(previous, nativeSessionConfigDigest(previous, "remote"), "remote");
|
||||
},
|
||||
canReuseProcess: previous => !needsCoreGitHubAccess(input) && canReuseNativeProcess(input,
|
||||
canReuseProcess: previous => !portableSource && !needsCoreGitHubAccess(input) &&
|
||||
nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(input.execution, "remote") && canReuseNativeProcess(input,
|
||||
record(JSON.parse(readBoundedNativeFile(nativeSessionCheckpointPath(previous),
|
||||
NATIVE_WARM_CHECKPOINT_MAX_BYTES, "native_session_supervisor_checkpoint_too_large").toString("utf8"))).processConfigDigest),
|
||||
}) ?? undefined;
|
||||
@@ -11061,7 +11140,7 @@ async function executePaperclipNativeSessionAttempt(
|
||||
: coordinator.controllerGeneration + 1;
|
||||
const target = input.runnerExecutionTarget;
|
||||
const retained = warmNativeSessions.get(nativeSessionScopeKey(input.execution));
|
||||
if (!input.restartRecovery && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId &&
|
||||
if (canRecoverWarmOwner && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId &&
|
||||
((retained?.pluginTaskId && idleWarmNativeSessionOwnerRunId(input) !== undefined) || remoteWarmRecovery?.releasedSource) &&
|
||||
(input.execution.provider.kind === "codex" || (input.execution.provider.kind === "acpx" && input.execution.provider.agent === "claude"))) {
|
||||
claimedStartupContext = new NativeExecutionContext(input.execution, Math.min(executionLeaseTtlMs,
|
||||
@@ -11801,7 +11880,7 @@ async function executePaperclipNativeSessionAttempt(
|
||||
executionContext = undefined;
|
||||
startupForAdmission!.executionContext = undefined;
|
||||
};
|
||||
if (warmSessionId !== null && warmConfigDigest !== null && !input.restartRecovery) {
|
||||
if (warmSessionId !== null && warmConfigDigest !== null && canRecoverWarmOwner) {
|
||||
const registered = warmNativeSessions.get(warmSessionId);
|
||||
const entry = registered && forgetUnretiredOwnerOnOtherSandbox(warmSessionId, registered, input.runnerExecutionTarget)
|
||||
? undefined : registered;
|
||||
@@ -12088,11 +12167,15 @@ async function executePaperclipNativeSessionAttempt(
|
||||
signal: executionContext?.signal ?? new AbortController().signal,
|
||||
});
|
||||
if (!recovered?.recovered || !recovered.session) throw new Error("runner_state_identity_mismatch: remote_warm_session_unproven");
|
||||
if (remoteWarmRecovery.reuseProcess) {
|
||||
const movedSession = remoteWarmRecovery.portableSourceTaskId !== undefined;
|
||||
// A portable resume already starts a new Runner. Its run attachment
|
||||
// prepares the current tools and runtime on the verified target.
|
||||
if (movedSession || remoteWarmRecovery.reuseProcess) {
|
||||
existingWarmSession = recovered.session;
|
||||
await emitStartupReuse(input.onEvent, "runner", { state: "reused", reason: "verified_reuse" });
|
||||
await emitStartupReuse(input.onEvent, "runner", movedSession
|
||||
? { state: "fresh", reason: "replacement" } : { state: "reused", reason: "verified_reuse" });
|
||||
await emitStartupReuse(input.onEvent, "provider_session", { state: "reused", reason: "verified_reuse" });
|
||||
trace.setBudgetStart("warm");
|
||||
if (!movedSession) trace.setBudgetStart("warm");
|
||||
persistedWarmSession = null;
|
||||
} else {
|
||||
await recovered.session.close({ reason: "recovered warm session requires current process configuration" });
|
||||
@@ -12104,12 +12187,22 @@ async function executePaperclipNativeSessionAttempt(
|
||||
remoteWarmRecovery.previousExecution.binding, scopedRunnerdStateRoot(input.execution),
|
||||
nativeSessionKey(input.execution));
|
||||
if (!suspendedController) throw new Error("native_runner_checkpoint_missing_after_suspension");
|
||||
persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution,
|
||||
nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote"), await recovered.session.snapshot());
|
||||
persistedWarmSession = loadWarmNativeCheckpoint(input.execution, warmConfigDigest!, "remote");
|
||||
runnerdBackend = await createRunnerdBackend({ ...input, onDiagnosticLease: ownDiagnosticLease, providerPackAdmissionSource, recoveryPurpose: claimedStopCleanup ? "stop_cleanup" : "execution", onAccountingEvent: producerUsage?.capture, runnerInstanceId: effectiveRunnerInstanceId,
|
||||
const verifiedConfigDigest = nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote");
|
||||
persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution, verifiedConfigDigest, await recovered.session.snapshot());
|
||||
persistedWarmSession = loadWarmNativeCheckpoint(input.execution, verifiedConfigDigest, "remote");
|
||||
runnerdBackend = await createRunnerdBackend({
|
||||
...input,
|
||||
pluginTaskDescription: undefined,
|
||||
onDiagnosticLease: ownDiagnosticLease,
|
||||
providerPackAdmissionSource,
|
||||
recoveryPurpose: claimedStopCleanup ? "stop_cleanup" : "execution",
|
||||
onAccountingEvent: producerUsage?.capture,
|
||||
runnerInstanceId: effectiveRunnerInstanceId,
|
||||
controllerPersistence: input.runnerExecutionTarget?.kind === "remote" ? { bootId: controller.bootId } : undefined,
|
||||
durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId, trace, toolTrace });
|
||||
durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId,
|
||||
trace,
|
||||
toolTrace,
|
||||
});
|
||||
}
|
||||
}
|
||||
const readRemoteCleanupLease = async () => input.runnerExecutionTarget?.kind === "remote" &&
|
||||
@@ -15710,6 +15803,7 @@ export function remoteCheckpointIncompleteFailure(
|
||||
export async function createRunnerdBackend(input: {
|
||||
executionContext?: NativeExecutionContext;
|
||||
casesEnabled?: boolean;
|
||||
toolRefreshOnResume?: boolean;
|
||||
db: Db;
|
||||
providerPackAdmissionSource?: NativeProviderPackAdmissionSource;
|
||||
controllerPersistence?: { bootId: string; runnerArtifact?: { version: string; digest: string } };
|
||||
@@ -15724,7 +15818,7 @@ export async function createRunnerdBackend(input: {
|
||||
remoteWarmRecovery?: Omit<RemoteWarmSessionRecovery, "revalidate"> & { revalidate: () => Promise<void> };
|
||||
retainedPluginTaskOwner?: ReturnType<typeof createPluginRunnerTaskOwner>;
|
||||
beginRetainedRunAttachment?: () => void;
|
||||
pluginTaskDescription?: Extract<ReturnType<typeof environmentTaskResultSchema.parse>,{kind:"description"}>;
|
||||
pluginTaskDescription?: PluginTaskDescription;
|
||||
onSpawn?: (meta: {
|
||||
pid: number;
|
||||
processGroupId: number | null;
|
||||
@@ -16128,8 +16222,10 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity
|
||||
? createNativeRecoveryUsageDelivery({
|
||||
db: input.db, execution: input.execution, sourceIdentity: { ...recoveryUsageIdentity },
|
||||
matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
|
||||
nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind),
|
||||
matchesExecution: previous => input.remoteWarmRecovery
|
||||
? matchesRemoteWarmExecution(previous, input.execution, input.toolRefreshOnResume)
|
||||
: nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
|
||||
nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind),
|
||||
})
|
||||
: undefined;
|
||||
const accountingTransportOptions = {
|
||||
@@ -16185,15 +16281,19 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
const { root, durableIdentity, durableBinding } = durableState ??= readDurableState();
|
||||
const provider = input.execution.provider;
|
||||
if (provider.kind !== "codex" && provider.kind !== "opencode" && provider.kind !== "acpx") throw new Error("Plugin Runner tasks require a workspace harness");
|
||||
if (input.restartRecovery?.kind === "reattach_remote_runner") {
|
||||
await assertPluginTaskReattachmentBinding(input, remoteTarget, durableBinding ? durableIdentity : null);
|
||||
}
|
||||
let priorDescription = input.pluginTaskDescription?.taskId === remoteTarget.sandboxLeaseAcquisition?.providerLeaseId
|
||||
? input.pluginTaskDescription : undefined;
|
||||
if (remoteTarget.sandboxLeaseAcquisition?.outcome === "resumed" && !durableBinding && !input.retainedPluginTaskOwner && !input.remoteWarmRecovery) {
|
||||
// A new logical session (including fresh retries) cannot inherit the old process authority.
|
||||
await pluginTask({kind:"stop"});
|
||||
const stopped=environmentTaskResultSchema.parse(await pluginTask({kind:"status"}));
|
||||
if(stopped.kind!=="status" || stopped.executionStopped!==true) throw new Error("Plugin Runner termination is unconfirmed; reconcile the saved task");
|
||||
await pluginTask({kind:"complete"});
|
||||
priorDescription = undefined;
|
||||
}
|
||||
const priorDescription = input.pluginTaskDescription?.taskId === remoteTarget.sandboxLeaseAcquisition?.providerLeaseId
|
||||
? input.pluginTaskDescription : undefined;
|
||||
const description = priorDescription ?? environmentTaskResultSchema.parse(await pluginTask({ kind: "describe" }));
|
||||
if (description.kind !== "description" || description.launchPreparationVersion !== 1 || description.projectAuthCheckVersion !== 1) throw new Error("Plugin Runner launch preparation v1 unavailable");
|
||||
const saved = durableBinding ? durableIdentity as RunnerdDurableIdentity : undefined;
|
||||
@@ -16212,9 +16312,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
normalizedSessionId: input.remoteWarmRecovery.identity.normalizedSessionId,
|
||||
turnId: input.remoteWarmRecovery.identity.turnId, itemId: input.remoteWarmRecovery.identity.itemId,
|
||||
} : undefined;
|
||||
if (input.restartRecovery?.kind === "reattach_remote_runner" && (!prior || !prior.turnId || !prior.itemId || prior.environmentLeaseId !== remoteTarget.leaseId)) {
|
||||
throw new Error("native_plugin_recovery_lease_mismatch");
|
||||
}
|
||||
|
||||
const resumeFrom = !retainedOwner && !adopting && prior && prior.environmentLeaseId !== remoteTarget.leaseId
|
||||
? await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
|
||||
leaseId: remoteTarget.leaseId!, priorIdentity: prior }) : undefined;
|
||||
@@ -16224,7 +16322,8 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
home: input.managedAiCredentialHome, environment: input.runnerEnvironment ?? {},
|
||||
});
|
||||
if (credentials.needsWriteback && !input.managedAiCredentialWriteback) throw new Error("Plugin Runner credential writeback authority unavailable");
|
||||
if (!adopting && input.managedAiCredentialWriteback && credentials.accountIdentity) {
|
||||
// Same-lease recovery already has a writeback binding. A portable move creates a new lease.
|
||||
if ((!adopting || input.remoteWarmRecovery?.portableSourceTaskId) && input.managedAiCredentialWriteback && credentials.accountIdentity) {
|
||||
if (!remoteTarget.leaseId) throw new Error("Plugin Runner credential lease unavailable");
|
||||
await registerPluginTaskCredentialHandoff(input.db, { companyId: input.execution.binding.companyId, leaseId: remoteTarget.leaseId },
|
||||
credentials.accountIdentity, input.managedAiCredentialWriteback);
|
||||
@@ -16251,7 +16350,8 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
releaseResumeVersion: description.releaseResumeVersion,
|
||||
authorityRunId: description.releaseResumeVersion === 1 ? input.execution.binding.runId : undefined,
|
||||
releasedSource: input.remoteWarmRecovery?.releasedSource,
|
||||
onResumed: async source => updateReleasedWarmRunner(input.execution, source, description.taskId, null),
|
||||
sessionHandoff: !!input.remoteWarmRecovery?.portableSourceTaskId,
|
||||
onResumed: async source => updateReleasedWarmRunner(input.execution, source, input.remoteWarmRecovery?.portableSourceTaskId ?? description.taskId, null),
|
||||
onRetired: async source => updateReleasedWarmRunner(input.execution, source, description.taskId, null),
|
||||
adoptIdentity: adoptionIdentity ?? (adopting && prior ? {
|
||||
runnerInstanceId: prior.runnerInstanceId, environmentLeaseId: prior.environmentLeaseId,
|
||||
|
||||
Reference in new issue
Block a user