Files
PaperClipAI/server/src/services/native-runtime/native-runner-ownership.ts
T
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00

129 lines
7.0 KiB
TypeScript

import { createHash } from "node:crypto";
import { environmentLeases, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { and, desc, eq, inArray, isNull, or, sql } from "drizzle-orm";
import { remoteExecutionHasStopped } from "../remote-execution-termination.js";
import { readSystemRecovery } from "../system-recovery-state.js";
import { unlaunchedPortableTaskLease } from "../plugin-task-ownership.js";
type NativeRemoteExecutionScope = {
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "runnerProfileJson">;
issueId: string;
};
export async function readNativeRemoteExecutionOwnershipForRuns(db: Db, inputs: readonly NativeRemoteExecutionScope[]) {
const key = (companyId: string, issueId: string, runId: string) => `${companyId}:${issueId}:${runId}`;
const leases = inputs.length ? await db.select({ id: environmentLeases.id, companyId: environmentLeases.companyId,
issueId: environmentLeases.issueId, runId: environmentLeases.heartbeatRunId,
unlaunchedPortable: unlaunchedPortableTaskLease(),
metadata: sql<Record<string, unknown>>`jsonb_build_object('runnerTaskStartup', ${environmentLeases.metadata}->'runnerTaskStartup')`,
}).from(environmentLeases).where(and(
or(...inputs.map(input => and(eq(environmentLeases.companyId, input.run.companyId),
eq(environmentLeases.issueId, input.issueId), eq(environmentLeases.heartbeatRunId, input.run.id)))),
sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`,
)).orderBy(desc(environmentLeases.createdAt)) : [];
const byScope = new Map<string, typeof leases[number]>();
for (const lease of leases) {
const scope = key(lease.companyId, lease.issueId!, lease.runId!);
if (!byScope.has(scope)) byScope.set(scope, lease);
}
return inputs.map(input => {
const pluginTask = byScope.get(key(input.run.companyId, input.issueId, input.run.id));
return { pluginTask, remoteProcess: input.run.runnerProfileJson?.nativeWorkspaceSync != null
|| input.run.runnerProfileJson?.remoteWorkspace != null || !!pluginTask };
});
}
export async function readNativeRemoteExecutionOwnership(db: Db, input: NativeRemoteExecutionScope) {
return (await readNativeRemoteExecutionOwnershipForRuns(db, [input]))[0]!;
}
export const nativeManualAdmissionOwnerColumns = {
runId: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId,
issueId: nativeRunFinalizations.issueId, phase: nativeRunFinalizations.phase,
resultId: nativeRunFinalizations.resultId, failureCode: nativeRunFinalizations.failureCode,
failureDetail: sql<Record<string, unknown>>`jsonb_build_object('systemRecovery', ${nativeRunFinalizations.failureDetail}->'systemRecovery')`,
};
type ManualAdmissionOwner = Pick<typeof nativeRunFinalizations.$inferSelect,
"runId" | "companyId" | "issueId" | "phase" | "resultId" | "failureCode" | "failureDetail">;
export function nativeManualAdmissionHoldMatches(scope: { companyId: string; runId: string; issueId?: string },
owner: Partial<ManualAdmissionOwner> | null | undefined): boolean {
const policy = readSystemRecovery(owner?.failureDetail);
return Boolean(owner?.runId === scope.runId && owner.companyId === scope.companyId &&
(!scope.issueId || owner.issueId === scope.issueId) && owner.phase === "terminal_failure" && owner.resultId === null &&
["native_restart_recovery_blocked", "native_retry_cancelled"].includes(owner.failureCode ?? "") &&
policy?.state === "exhausted" && policy.sourceRunId === scope.runId);
}
export async function nativeManualAdmissionHoldsExecution(db: Db, scope: {
companyId: string; runId: string; issueId?: string;
}): Promise<boolean> {
const [hold] = await db.select(nativeManualAdmissionOwnerColumns).from(heartbeatRuns)
.innerJoin(nativeRunFinalizations, and(eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
eq(nativeRunFinalizations.runId, heartbeatRuns.id), eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId)))
.where(and(eq(heartbeatRuns.companyId, scope.companyId), eq(heartbeatRuns.id, scope.runId),
eq(heartbeatRuns.runtimeMode, "native"), scope.issueId ? eq(heartbeatRuns.nativeIssueId, scope.issueId) : undefined,
eq(nativeRunFinalizations.phase, "terminal_failure"), inArray(nativeRunFinalizations.failureCode, ["native_restart_recovery_blocked", "native_retry_cancelled"]),
isNull(nativeRunFinalizations.resultId),
)).limit(1);
return nativeManualAdmissionHoldMatches(scope, hold) && !await remoteExecutionHasStopped(db, scope.companyId, scope.runId);
}
export const NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE =
"native_execution_ownership_unverified";
export const NATIVE_ADOPTED_RUNNER_AUTHENTICATION_TIMEOUT =
"native_adopted_runner_authentication_timeout";
/** An unauthenticated retained process is not evidence that execution stopped. */
export class NativeRunnerOwnershipUnverifiedError extends Error {
constructor(
readonly reason:
| "adopted_runner_authentication_timeout"
| "native_chat_workspace_scope_mismatch" = "adopted_runner_authentication_timeout",
) {
super(NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE);
this.name = "NativeRunnerOwnershipUnverifiedError";
}
}
export function isNativeRunnerOwnershipHeld(run: {
runtimeMode?: string | null;
status?: string | null;
errorCode?: string | null;
nativePhase?: string | null;
}): boolean {
return (
run.runtimeMode === "native" &&
run.status === "running" &&
((run.errorCode === NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE &&
run.nativePhase === "terminal_failure") ||
run.errorCode === NATIVE_ADOPTED_RUNNER_AUTHENTICATION_TIMEOUT)
);
}
/** Null-safe database counterpart, evaluated atomically by claims and writes. */
export function nativeRunnerOwnershipNotHeldCondition() {
return sql`not coalesce(
${heartbeatRuns.runtimeMode} = 'native'
and ${heartbeatRuns.status} = 'running'
and (
(${heartbeatRuns.errorCode} = ${NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE}
and ${heartbeatRuns.nativePhase} = 'terminal_failure')
or ${heartbeatRuns.errorCode} = ${NATIVE_ADOPTED_RUNNER_AUTHENTICATION_TIMEOUT}
), false
)`;
}
/** Opaque binding for an operator's exact observed controller authority. Never
* expose the token itself. expiryEpoch must come from Postgres numeric epoch
* text, which is timezone independent and retains timestamp microseconds. */
export function nativeControllerAuthorityDigest(owner: Pick<typeof nativeRunFinalizations.$inferSelect,
"companyId" | "issueId" | "runId" | "leaseOwner" | "controllerGeneration" | "controllerBootId" |
"controllerHostHash" | "controllerPid" | "controllerProcessStartedAt">, expiryEpoch: string | null): string {
return createHash("sha256").update(JSON.stringify([
"paperclip.native-controller-authority.v1", owner.companyId, owner.issueId, owner.runId,
owner.leaseOwner, owner.controllerGeneration, owner.controllerBootId, owner.controllerHostHash,
owner.controllerPid, owner.controllerProcessStartedAt?.toISOString() ?? null, expiryEpoch,
])).digest("hex");
}