mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
129 lines
7.0 KiB
TypeScript
129 lines
7.0 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { environmentLeases, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
|
import { and, desc, eq, inArray, isNull, or, sql } from "drizzle-orm";
|
|
import { remoteExecutionHasStopped } from "../remote-execution-termination.js";
|
|
import { readSystemRecovery } from "../system-recovery-state.js";
|
|
import { unlaunchedPortableTaskLease } from "../plugin-task-ownership.js";
|
|
|
|
type NativeRemoteExecutionScope = {
|
|
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "runnerProfileJson">;
|
|
issueId: string;
|
|
};
|
|
|
|
export async function readNativeRemoteExecutionOwnershipForRuns(db: Db, inputs: readonly NativeRemoteExecutionScope[]) {
|
|
const key = (companyId: string, issueId: string, runId: string) => `${companyId}:${issueId}:${runId}`;
|
|
const leases = inputs.length ? await db.select({ id: environmentLeases.id, companyId: environmentLeases.companyId,
|
|
issueId: environmentLeases.issueId, runId: environmentLeases.heartbeatRunId,
|
|
unlaunchedPortable: unlaunchedPortableTaskLease(),
|
|
metadata: sql<Record<string, unknown>>`jsonb_build_object('runnerTaskStartup', ${environmentLeases.metadata}->'runnerTaskStartup')`,
|
|
}).from(environmentLeases).where(and(
|
|
or(...inputs.map(input => and(eq(environmentLeases.companyId, input.run.companyId),
|
|
eq(environmentLeases.issueId, input.issueId), eq(environmentLeases.heartbeatRunId, input.run.id)))),
|
|
sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`,
|
|
)).orderBy(desc(environmentLeases.createdAt)) : [];
|
|
const byScope = new Map<string, typeof leases[number]>();
|
|
for (const lease of leases) {
|
|
const scope = key(lease.companyId, lease.issueId!, lease.runId!);
|
|
if (!byScope.has(scope)) byScope.set(scope, lease);
|
|
}
|
|
return inputs.map(input => {
|
|
const pluginTask = byScope.get(key(input.run.companyId, input.issueId, input.run.id));
|
|
return { pluginTask, remoteProcess: input.run.runnerProfileJson?.nativeWorkspaceSync != null
|
|
|| input.run.runnerProfileJson?.remoteWorkspace != null || !!pluginTask };
|
|
});
|
|
}
|
|
|
|
export async function readNativeRemoteExecutionOwnership(db: Db, input: NativeRemoteExecutionScope) {
|
|
return (await readNativeRemoteExecutionOwnershipForRuns(db, [input]))[0]!;
|
|
}
|
|
|
|
export const nativeManualAdmissionOwnerColumns = {
|
|
runId: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId,
|
|
issueId: nativeRunFinalizations.issueId, phase: nativeRunFinalizations.phase,
|
|
resultId: nativeRunFinalizations.resultId, failureCode: nativeRunFinalizations.failureCode,
|
|
failureDetail: sql<Record<string, unknown>>`jsonb_build_object('systemRecovery', ${nativeRunFinalizations.failureDetail}->'systemRecovery')`,
|
|
};
|
|
type ManualAdmissionOwner = Pick<typeof nativeRunFinalizations.$inferSelect,
|
|
"runId" | "companyId" | "issueId" | "phase" | "resultId" | "failureCode" | "failureDetail">;
|
|
|
|
export function nativeManualAdmissionHoldMatches(scope: { companyId: string; runId: string; issueId?: string },
|
|
owner: Partial<ManualAdmissionOwner> | null | undefined): boolean {
|
|
const policy = readSystemRecovery(owner?.failureDetail);
|
|
return Boolean(owner?.runId === scope.runId && owner.companyId === scope.companyId &&
|
|
(!scope.issueId || owner.issueId === scope.issueId) && owner.phase === "terminal_failure" && owner.resultId === null &&
|
|
["native_restart_recovery_blocked", "native_retry_cancelled"].includes(owner.failureCode ?? "") &&
|
|
policy?.state === "exhausted" && policy.sourceRunId === scope.runId);
|
|
}
|
|
|
|
export async function nativeManualAdmissionHoldsExecution(db: Db, scope: {
|
|
companyId: string; runId: string; issueId?: string;
|
|
}): Promise<boolean> {
|
|
const [hold] = await db.select(nativeManualAdmissionOwnerColumns).from(heartbeatRuns)
|
|
.innerJoin(nativeRunFinalizations, and(eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
|
|
eq(nativeRunFinalizations.runId, heartbeatRuns.id), eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId)))
|
|
.where(and(eq(heartbeatRuns.companyId, scope.companyId), eq(heartbeatRuns.id, scope.runId),
|
|
eq(heartbeatRuns.runtimeMode, "native"), scope.issueId ? eq(heartbeatRuns.nativeIssueId, scope.issueId) : undefined,
|
|
eq(nativeRunFinalizations.phase, "terminal_failure"), inArray(nativeRunFinalizations.failureCode, ["native_restart_recovery_blocked", "native_retry_cancelled"]),
|
|
isNull(nativeRunFinalizations.resultId),
|
|
)).limit(1);
|
|
return nativeManualAdmissionHoldMatches(scope, hold) && !await remoteExecutionHasStopped(db, scope.companyId, scope.runId);
|
|
}
|
|
|
|
export const NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE =
|
|
"native_execution_ownership_unverified";
|
|
export const NATIVE_ADOPTED_RUNNER_AUTHENTICATION_TIMEOUT =
|
|
"native_adopted_runner_authentication_timeout";
|
|
|
|
/** An unauthenticated retained process is not evidence that execution stopped. */
|
|
export class NativeRunnerOwnershipUnverifiedError extends Error {
|
|
constructor(
|
|
readonly reason:
|
|
| "adopted_runner_authentication_timeout"
|
|
| "native_chat_workspace_scope_mismatch" = "adopted_runner_authentication_timeout",
|
|
) {
|
|
super(NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE);
|
|
this.name = "NativeRunnerOwnershipUnverifiedError";
|
|
}
|
|
}
|
|
|
|
export function isNativeRunnerOwnershipHeld(run: {
|
|
runtimeMode?: string | null;
|
|
status?: string | null;
|
|
errorCode?: string | null;
|
|
nativePhase?: string | null;
|
|
}): boolean {
|
|
return (
|
|
run.runtimeMode === "native" &&
|
|
run.status === "running" &&
|
|
((run.errorCode === NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE &&
|
|
run.nativePhase === "terminal_failure") ||
|
|
run.errorCode === NATIVE_ADOPTED_RUNNER_AUTHENTICATION_TIMEOUT)
|
|
);
|
|
}
|
|
|
|
/** Null-safe database counterpart, evaluated atomically by claims and writes. */
|
|
export function nativeRunnerOwnershipNotHeldCondition() {
|
|
return sql`not coalesce(
|
|
${heartbeatRuns.runtimeMode} = 'native'
|
|
and ${heartbeatRuns.status} = 'running'
|
|
and (
|
|
(${heartbeatRuns.errorCode} = ${NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE}
|
|
and ${heartbeatRuns.nativePhase} = 'terminal_failure')
|
|
or ${heartbeatRuns.errorCode} = ${NATIVE_ADOPTED_RUNNER_AUTHENTICATION_TIMEOUT}
|
|
), false
|
|
)`;
|
|
}
|
|
|
|
/** Opaque binding for an operator's exact observed controller authority. Never
|
|
* expose the token itself. expiryEpoch must come from Postgres numeric epoch
|
|
* text, which is timezone independent and retains timestamp microseconds. */
|
|
export function nativeControllerAuthorityDigest(owner: Pick<typeof nativeRunFinalizations.$inferSelect,
|
|
"companyId" | "issueId" | "runId" | "leaseOwner" | "controllerGeneration" | "controllerBootId" |
|
|
"controllerHostHash" | "controllerPid" | "controllerProcessStartedAt">, expiryEpoch: string | null): string {
|
|
return createHash("sha256").update(JSON.stringify([
|
|
"paperclip.native-controller-authority.v1", owner.companyId, owner.issueId, owner.runId,
|
|
owner.leaseOwner, owner.controllerGeneration, owner.controllerBootId, owner.controllerHostHash,
|
|
owner.controllerPid, owner.controllerProcessStartedAt?.toISOString() ?? null, expiryEpoch,
|
|
])).digest("hex");
|
|
}
|