Files
PaperClipAI/server/src/services/native-runtime/remote-warm-session-recovery.ts
T
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00

254 lines
17 KiB
TypeScript

import { and, eq } from "drizzle-orm";
import { agents, companies, environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import { parseNativeExecutionInput, type NativeExecutionInput, type PersistedNativeSession, type DurableRecoveryIdentity } from "../../vendor/paperclip-runner/index.js";
import { currentNativeControllerIdentity, evaluateNativeControllerTakeover, recordLegacyHostFallbackProof } from "./native-restart-recovery.js";
import { NativeControllerRecoveryWaitingError, readNativeControllerRecovery } from "./native-controller-recovery.js";
import { NativePredecessorRetirementUnprovenError } from "./predecessor-retirement.js";
import { readyPortableTaskSource } from "../plugin-task-ownership.js";
export type RemoteWarmSessionRecovery = {
identity: Record<string, string>;
checkpoint: PersistedNativeSession;
previousExecution: NativeExecutionInput;
reuseProcess: boolean;
releasedSource?: DurableRecoveryIdentity;
portableSourceTaskId?: string;
remote: { providerLeaseId: string; remoteCwd: string };
revalidate: (leaseOwner: string) => Promise<void>;
};
/** Protocol events that mean work started and may still be live or waiting. */
const POST_TERMINAL_LIVE_WORK = new Set(["turn.started", "turn.accepted", "run.attached", "tool.execution.started",
"semantic_tool.input", "runtime_request.created"]);
const record = (value: unknown): Record<string, unknown> =>
value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
function postTerminalEvents(control: Record<string, unknown>): unknown[] {
const events = Array.isArray(control.committedEvents) ? control.committedEvents : [];
const terminalIndex = events.findLastIndex(event => record(event).eventType === "run.terminal");
return terminalIndex < 0 ? [] : events.slice(terminalIndex + 1);
}
/** A session is settled when its commands finished and nothing is live or waiting.
* Events after the terminal describe the past, so refusing cannot undo them: they
* are reported instead. Every event must still authenticate as this session's. */
const RUNNER_LIFECYCLE_COMMAND_TYPES = new Set(["runner.suspend", "runner.drain", "runner.shutdown"]);
function outcomeUnknown(command: Record<string, unknown>): boolean {
return command.status === "indeterminate" ||
(command.status === "failed" && (record(command.result).code === "execution_indeterminate" ||
record(record(command.result).result).code === "execution_indeterminate"));
}
export function unsettledControllerCommands(commands: readonly Record<string, unknown>[]): Record<string, unknown>[] {
const lastCompletedTurnStart = Math.max(-1, ...commands
.filter(command => command.type === "turn.start" && command.status === "completed")
.map(command => Number(command.controllerSeq)));
return commands.filter(command => {
if (command.status === "completed") return false;
if (outcomeUnknown(command)) {
return !(command.type === "turn.start" && Number(command.controllerSeq) < lastCompletedTurnStart);
}
return !((command.status === "failed" || command.status === "rejected") &&
!RUNNER_LIFECYCLE_COMMAND_TYPES.has(String(command.type)));
});
}
export function isSettledRemoteWarmControlState(control: Record<string, unknown>, identity: Record<string, unknown>): boolean {
const commands = control.commands;
const events = control.committedEvents;
if (!Array.isArray(commands) || unsettledControllerCommands(commands.map(record)).length > 0 ||
!Array.isArray(events) || Object.entries(identity).some(([key, value]) => record(control.identity)[key] !== value)) return false;
const terminalIndex = events.findLastIndex(event => record(event).eventType === "run.terminal");
if (terminalIndex < 0) return false;
const matchesSession = (event: unknown) => {
const envelope = record(record(event).envelope);
return envelope.runId === identity.runId && envelope.normalizedSessionId === identity.normalizedSessionId;
};
if (!matchesSession(events[terminalIndex])) return false;
return events.slice(terminalIndex + 1).every(event => {
const envelope = record(record(event).envelope);
const notification = record(envelope.payload);
const payload = record(notification.payload);
const eventType = record(event).eventType;
// An active goal keeps a run going, and its next turn can start at any moment.
const live = (typeof eventType === "string" && POST_TERMINAL_LIVE_WORK.has(eventType)) ||
((eventType === "session.goal.snapshot" || eventType === "session.goal.updated") &&
(payload.workingNow !== false || record(payload.goal).status === "active"));
return !live && matchesSession(event) &&
Object.entries(identity).every(([key, value]) => envelope[key] === value) &&
(notification.schema === "paperclip.prp.event.v1" || notification.schema === "paperclip.prp.event.v2") &&
notification.eventType === eventType &&
notification.sourceInstanceId === identity.runnerInstanceId && notification.sourceKind === "runner" &&
notification.runId === identity.runId && notification.normalizedSessionId === identity.normalizedSessionId;
});
}
/** Event types recorded after the prior run's terminal, for the next run's log. */
export function summarizePostTerminalEvents(control: Record<string, unknown>): string | null {
const counts = new Map<string, number>();
for (const event of postTerminalEvents(control)) {
const type = String(record(event).eventType);
counts.set(type, (counts.get(type) ?? 0) + 1);
}
return counts.size === 0 ? null : [...counts].map(([type, count]) => `${type} x${count}`).join(", ");
}
/** A failed attachment suspends the Runner under the attaching run while the journal
* still names its predecessor. A same-run retry cannot prove either owner, so report
* the failure that caused this state rather than the resulting identity mismatch. */
export function describeOwnFailedAttachment(recoveryHistory: unknown): string {
const history = Array.isArray(recoveryHistory) ? recoveryHistory : [];
const cause = record(history.at(-1)).stderrTail;
return `runner_state_identity_mismatch: prior_attach_failed: ${typeof cause === "string" && cause ? cause : "unknown"}`;
}
const SETTLEMENT_DOC = "https://github.com/paperclipai/paperclip/blob/master/doc/execution-semantics.md#session-settlement-after-a-run";
/** Points whoever diagnoses a failed attachment at reviewed docs. The stored
* failure is already redacted and bounded; other failures produce no pointer. */
export function describeFailedAttachment(failureMessage: unknown): string | null {
if (typeof failureMessage !== "string" || !/\brun\.attach (failed|requires)\b/.test(failureMessage)) return null;
const encoded = /"message":("(?:[^"\\]|\\.)*")/.exec(failureMessage)?.[1];
let cause = failureMessage;
try { if (encoded) cause = JSON.parse(encoded) as string; } catch { /* Keep the whole failure. */ }
return `This run could not attach to its provider session: ${cause}. A later run may start a new provider thread. See ${SETTLEMENT_DOC}`;
}
type WarmRecoveryRefusal = "identity_incomplete" | "target_not_resumed_sandbox" | "owner_unsettled" | "execution_mismatch"
| "checkpoint_unsettled" | "journal_unsettled" | "controller_not_stopped" | "prior_lease_mismatch"
| "current_run_not_admitted" | "current_owner_changed" | "owner_changed" | "current_lease_changed";
export async function inspectRemoteWarmSessionRecovery(input: {
db: Db;
execution: NativeExecutionInput;
target: Extract<AdapterExecutionTarget, { kind: "remote" }>;
identity: Record<string, unknown>;
readControlState: () => Record<string, unknown>;
matchesExecution: (previous: NativeExecutionInput) => boolean;
loadCheckpoint: (previous: NativeExecutionInput) => PersistedNativeSession | null;
canReuseProcess: (previous: NativeExecutionInput) => boolean;
portableSourceTaskId?: string;
}): Promise<RemoteWarmSessionRecovery | null> {
const { binding } = input.execution;
const { target } = input;
let settledPredecessor: { runId: string; providerLeaseId: string } | null = null;
const deny = (reason: WarmRecoveryRefusal): never => {
const refusal = new Error(`runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; warm_recovery_refused=${reason}`);
if (settledPredecessor) {
throw new NativePredecessorRetirementUnprovenError(settledPredecessor.runId, settledPredecessor.providerLeaseId, refusal);
}
throw refusal;
};
const refuseCurrentRun = (reason: WarmRecoveryRefusal): never => {
throw new Error(`runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; warm_recovery_refused=${reason}`);
};
const identity: Record<string, string> = {};
for (const key of ["runId", "normalizedSessionId", "runnerInstanceId", "environmentLeaseId", "turnId", "itemId"]) {
const value = input.identity[key];
if (typeof value !== "string" || !value) return deny("identity_incomplete");
identity[key] = value;
}
// A Runner lives in the sandbox it started in. A created or replacement
// sandbox (for example after a restart or deploy replaced the controller)
// holds no Runner to adopt, so live adoption steps aside. Ordinary admission
// then retires the finished owner or keeps it held with a named reason.
if (!input.portableSourceTaskId && target.transport === "sandbox" && ["created", "replacement"].includes(target.sandboxLeaseAcquisition?.outcome ?? "")) {
return null;
}
if (target.transport !== "sandbox" || !target.leaseId || !target.environmentId ||
!target.providerKey || !target.sandboxLeaseAcquisition ||
(!input.portableSourceTaskId && target.sandboxLeaseAcquisition.outcome !== "resumed") ||
identity.runId === binding.runId) return deny("target_not_resumed_sandbox");
const providerLeaseId = target.sandboxLeaseAcquisition.providerLeaseId;
const sourceProviderLeaseId = input.portableSourceTaskId ?? providerLeaseId;
const [prior] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations })
.from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
.where(and(eq(heartbeatRuns.id, identity.runId), eq(heartbeatRuns.companyId, binding.companyId),
eq(heartbeatRuns.agentId, binding.agentId), eq(heartbeatRuns.nativeIssueId, binding.issueId)));
if (!prior || prior.run.status !== "succeeded" || !prior.owner.resultId || !prior.owner.controllerPid ||
!prior.owner.controllerProcessStartedAt || !prior.owner.controllerBootId) return deny("owner_unsettled");
const previous = parseNativeExecutionInput(prior.run.runnerProfileJson?.nativeExecutionInput);
if (previous.binding.runId !== identity.runId || !input.matchesExecution(previous)) return deny("execution_mismatch");
if (prior.owner.leaseOwner && prior.owner.leaseExpiresAt && prior.owner.leaseExpiresAt > new Date()) {
throw new NativeControllerRecoveryWaitingError(prior.owner.leaseExpiresAt);
}
if (prior.owner.phase !== "committed" || prior.owner.leaseOwner) return deny("owner_unsettled");
if (sourceProviderLeaseId === providerLeaseId) settledPredecessor = { runId: identity.runId, providerLeaseId };
const checkpoint = input.loadCheckpoint(previous);
if (!checkpoint || checkpoint.identity.runId !== identity.runId ||
checkpoint.identity.companyId !== binding.companyId || checkpoint.identity.issueId !== binding.issueId ||
checkpoint.identity.agentId !== binding.agentId || checkpoint.identity.sessionId !== identity.normalizedSessionId ||
checkpoint.driverKind !== previous.session.driverKind || !checkpoint.providerSessionId ||
checkpoint.activeTurnId !== null || checkpoint.pendingRuntimeRequests?.length !== 0) return deny("checkpoint_unsettled");
const control = input.readControlState();
if (!isSettledRemoteWarmControlState(control, identity)) return deny("journal_unsettled");
const assertPriorStopped = async () => {
const persistedRecovery = readNativeControllerRecovery(prior.run.runnerProfileJson?.nativeControllerRecovery,
prior.run.runnerProfileJson?.sessionCheckpoint);
// A PID in another webserver container cannot prove life or death. New
// durable controllers are fenced by the database lease and current turn.
const stopped = await evaluateNativeControllerTakeover({ owner: prior.owner, now: new Date(), requireStoppedController: true,
remoteLeaseFenced: persistedRecovery !== null });
if (!stopped.allowed) return deny("controller_not_stopped");
await recordLegacyHostFallbackProof(input.db, prior.run, stopped, "remote_warm_session_recovery");
};
await assertPriorStopped();
const priorLeases = await input.db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, binding.companyId), eq(environmentLeases.heartbeatRunId, identity.runId),
eq(environmentLeases.issueId, binding.issueId), eq(environmentLeases.environmentId, target.environmentId),
eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, sourceProviderLeaseId),
));
if (priorLeases.length !== 1 || priorLeases[0].metadata?.remoteCwd !== target.remoteCwd ||
!["released", "expired", "active", ...(input.portableSourceTaskId ? ["retained"] : [])].includes(priorLeases[0].status)) {
return deny("prior_lease_mismatch");
}
return {
identity, checkpoint, previousExecution: previous, reuseProcess: input.canReuseProcess(previous),
portableSourceTaskId: input.portableSourceTaskId,
remote: { providerLeaseId, remoteCwd: target.remoteCwd },
revalidate: async leaseOwner => {
await assertPriorStopped();
if (input.portableSourceTaskId &&
(await readyPortableTaskSource(input.db, binding.companyId, target.leaseId!, identity))?.taskId !== input.portableSourceTaskId) {
return deny("prior_lease_mismatch");
}
const controller = await currentNativeControllerIdentity();
const [current] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations, issue: issues, agent: agents, company: companies })
.from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
.innerJoin(issues, eq(issues.id, heartbeatRuns.nativeIssueId))
.innerJoin(agents, and(eq(agents.id, heartbeatRuns.agentId), eq(agents.companyId, heartbeatRuns.companyId)))
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
.where(and(eq(heartbeatRuns.id, binding.runId), eq(heartbeatRuns.companyId, binding.companyId),
eq(heartbeatRuns.agentId, binding.agentId), eq(heartbeatRuns.nativeIssueId, binding.issueId)));
const [lease] = await input.db.select().from(environmentLeases).where(and(
eq(environmentLeases.id, target.leaseId!), eq(environmentLeases.companyId, binding.companyId),
eq(environmentLeases.heartbeatRunId, binding.runId), eq(environmentLeases.issueId, binding.issueId),
));
const [priorOwner] = await input.db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.runId, identity.runId), eq(nativeRunFinalizations.companyId, binding.companyId),
eq(nativeRunFinalizations.issueId, binding.issueId),
));
if (!current || current.run.status !== "running" || current.run.resultJson?.nativeCancellation ||
current.company.status !== "active" || current.agent.lifecycleState !== "ready" ||
["paused", "terminated", "pending_approval"].includes(current.agent.status) ||
current.issue.executionRunId !== binding.runId || current.issue.assigneeAgentId !== binding.agentId ||
current.issue.status !== "in_progress") return refuseCurrentRun("current_run_not_admitted");
if (current.owner.leaseOwner !== leaseOwner || !current.owner.leaseExpiresAt ||
current.owner.leaseExpiresAt <= new Date() || current.owner.resultId ||
current.owner.controllerBootId !== controller.bootId || current.owner.controllerPid !== controller.pid) {
return refuseCurrentRun("current_owner_changed");
}
if (!priorOwner || priorOwner.leaseOwner || priorOwner.phase !== "committed" ||
priorOwner.resultId !== prior.owner.resultId || priorOwner.controllerBootId !== prior.owner.controllerBootId) {
return deny("owner_changed");
}
if (!lease || lease.status !== "active" ||
lease.environmentId !== target.environmentId || lease.provider !== target.providerKey ||
lease.providerLeaseId !== providerLeaseId || lease.metadata?.remoteCwd !== target.remoteCwd) return deny("current_lease_changed");
},
};
}