mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
65 lines
4.9 KiB
TypeScript
65 lines
4.9 KiB
TypeScript
import { and, asc, eq, exists, inArray, or, sql } from "drizzle-orm";
|
|
import { environmentLeases, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
|
|
import { inspectDurablePrpCheckpointHeader, type DurablePrpCheckpoint } from "../../vendor/paperclip-runner/index.js";
|
|
import { nativeSha256 } from "./canonical.js";
|
|
|
|
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
const record = (value: unknown): Record<string, unknown> => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
|
|
|
/** Claims and recovered-source writers lock both coordinators in this order. */
|
|
export async function lockSourceRecoveryCoordinators(tx: Transaction, companyId: string, runIds: readonly string[]) {
|
|
return tx.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, companyId),
|
|
inArray(nativeRunFinalizations.runId, [...new Set(runIds)]))).orderBy(asc(nativeRunFinalizations.runId)).for("update");
|
|
}
|
|
|
|
export async function readNativeSourceConsumers(tx: Transaction, companyId: string, runId: string) {
|
|
const retired = exists(tx.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
|
|
eq(nativeRunnerCheckpoints.companyId, companyId), eq(nativeRunnerCheckpoints.runId, runId),
|
|
eq(nativeRunnerCheckpoints.part, "header"),
|
|
sql`${nativeRunnerCheckpoints.material}->>'schema' like 'paperclip.native-runner-checkpoint-retirement.%'`)));
|
|
return tx.select({ id: environmentLeases.id, metadata: environmentLeases.metadata, retired }).from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId),
|
|
or(sql`${environmentLeases.metadata}->'runnerTaskSourceConsumer' is not null`,
|
|
sql`${environmentLeases.metadata}->'portableTaskSuccessor' is not null`)));
|
|
}
|
|
|
|
/** Physical source consumption revokes the completed controller's late writer. */
|
|
export async function assertNativeSourceCheckpointConsumer(tx: Transaction, owner: {
|
|
companyId: string; runId: string; authorityRunId: string; bootId: string; generation: number;
|
|
}, checkpoint?: DurablePrpCheckpoint) {
|
|
const rows = await readNativeSourceConsumers(tx, owner.companyId, owner.runId);
|
|
if (rows.some(row => row.retired)) throw new Error("native_runner_checkpoint_retired");
|
|
if (!rows.length) return;
|
|
const deny = (): never => { throw new Error("native_runner_checkpoint_owner_lost"); };
|
|
if (owner.runId === owner.authorityRunId || rows.length !== 1 || !checkpoint) return deny();
|
|
const source = rows[0]!;
|
|
const header = checkpoint.records.get("header");
|
|
const identity = header ? inspectDurablePrpCheckpointHeader(header) : null;
|
|
const portable = record(source.metadata?.portableTaskSuccessor);
|
|
if (source.metadata?.portableTaskSuccessor !== undefined) {
|
|
if (typeof portable.targetLeaseId !== "string" || portable.sourceLeaseId !== source.id ||
|
|
!identity || nativeSha256(record(portable.operation).identity ?? null) !== nativeSha256(identity)) return deny();
|
|
const [target] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, owner.companyId),
|
|
eq(environmentLeases.id, portable.targetLeaseId)));
|
|
if (!target || target.heartbeatRunId !== owner.authorityRunId || target.status !== "active" || target.releasedAt ||
|
|
target.expiresAt && target.expiresAt.getTime() <= Date.now() ||
|
|
target.metadata?.portableTaskOwnershipReady !== true || target.metadata.portableTaskSuccessor !== undefined ||
|
|
nativeSha256(target.metadata.portableTaskPredecessor ?? null) !== nativeSha256(portable)) return deny();
|
|
return;
|
|
}
|
|
const consumer = record(source.metadata?.runnerTaskSourceConsumer);
|
|
if (typeof consumer.leaseId !== "string" || consumer.runId !== owner.authorityRunId) return deny();
|
|
const [lease] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, owner.companyId),
|
|
eq(environmentLeases.id, consumer.leaseId)));
|
|
const recovery = record(lease?.metadata?.runnerTaskSourceRecovery);
|
|
const expected = record(recovery.identity);
|
|
if (!lease || lease.heartbeatRunId !== owner.authorityRunId || lease.status !== "active" || lease.releasedAt ||
|
|
lease.expiresAt && lease.expiresAt.getTime() <= Date.now() ||
|
|
recovery.schema !== "paperclip.runner.source-recovery.v1" || recovery.sourceLeaseId !== source.id ||
|
|
recovery.consumerLeaseId !== lease.id || recovery.consumerRunId !== owner.authorityRunId ||
|
|
typeof recovery.accountIdentity !== "string" || recovery.accountIdentity !== consumer.accountIdentity ||
|
|
recovery.controllerBootId !== owner.bootId || recovery.controllerGeneration !== owner.generation ||
|
|
["runnerInstanceId", "environmentLeaseId", "runId", "normalizedSessionId", "turnId", "itemId"].some(key =>
|
|
typeof expected[key] !== "string" || expected[key] !== record(identity)[key] || expected[key] !== record(consumer.identity)[key])) return deny();
|
|
}
|