mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
120 lines
5.9 KiB
TypeScript
120 lines
5.9 KiB
TypeScript
import { and, eq, sql } from "drizzle-orm";
|
|
import { environmentLeases, type Db } from "@paperclipai/db";
|
|
import { z, type PluginEnvironmentTaskCredential } from "@paperclipai/plugin-sdk";
|
|
import { isDeepStrictEqual } from "node:util";
|
|
import { managedAiCredentialWritebackBindingSchema, syncManagedAiCredentials, type ManagedAiCredentialWritebackBinding } from "./managed-ai-credential-writeback.js";
|
|
|
|
const handoffSchema = z.object({
|
|
accountIdentity: z.string().min(1),
|
|
binding: managedAiCredentialWritebackBindingSchema,
|
|
state: z.enum(["pending", "complete"]),
|
|
outcome: z.enum(["rotated", "unchanged", "binding_changed", "not_dispatched", "no_credentials"]).optional(),
|
|
}).strict();
|
|
const key = "runnerTaskCredentialHandoff";
|
|
type Scope = { companyId: string; leaseId: string; expectedTaskId?: string };
|
|
|
|
export function canReadStoppedPluginTaskCredential(metadata: Record<string, unknown>, accountIdentity: string): boolean {
|
|
const handoff = handoffSchema.safeParse(metadata[key]);
|
|
return metadata.runnerTaskExecutionStopped === true && handoff.success &&
|
|
handoff.data.state === "pending" && handoff.data.accountIdentity === accountIdentity;
|
|
}
|
|
|
|
/** Save authority, never credential bytes or a controller-local temporary path. */
|
|
export async function registerPluginTaskCredentialHandoff(
|
|
db: Db,
|
|
scope: Scope,
|
|
accountIdentity: string,
|
|
binding: ManagedAiCredentialWritebackBinding,
|
|
) {
|
|
const handoff = handoffSchema.parse({ accountIdentity, binding, state: "pending" });
|
|
await db.transaction(async tx => {
|
|
const [lease] = await tx.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
|
|
)).for("update");
|
|
if (!lease || lease.status !== "active" || lease.metadata?.runnerTaskLease !== true ||
|
|
lease.metadata.runnerTaskRetiring === true) {
|
|
throw new Error("Plugin task credential lease unavailable");
|
|
}
|
|
const saved = lease.metadata[key];
|
|
if (saved) {
|
|
const prior = handoffSchema.parse(saved);
|
|
if (prior.accountIdentity !== accountIdentity || !isDeepStrictEqual(prior.binding, handoff.binding)) {
|
|
throw new Error("Plugin task credential authority changed");
|
|
}
|
|
return;
|
|
}
|
|
await tx.update(environmentLeases).set({ metadata: { ...lease.metadata, [key]: handoff } })
|
|
.where(eq(environmentLeases.id, lease.id));
|
|
});
|
|
}
|
|
|
|
/** Live PRP updates use durable authority, including while a warm task is idle. */
|
|
export async function syncPluginTaskCredentialHandoff(
|
|
db: Db,
|
|
scope: Scope,
|
|
fetchCredential: (accountIdentity: string) => Promise<PluginEnvironmentTaskCredential | null>,
|
|
) {
|
|
const [lease] = await db.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
|
|
scope.expectedTaskId === undefined ? undefined : eq(environmentLeases.providerLeaseId, scope.expectedTaskId),
|
|
));
|
|
if (!lease) throw new Error("Plugin task credential lease unavailable");
|
|
if (!lease.metadata?.[key]) return;
|
|
const handoff = handoffSchema.parse(lease.metadata[key]);
|
|
if (handoff.state === "complete") return;
|
|
const outcome = await writeBackCredentials(db, scope.companyId, handoff, fetchCredential);
|
|
if (outcome === "no_credentials") throw new Error("Plugin task credential snapshot unavailable");
|
|
return outcome;
|
|
}
|
|
|
|
async function writeBackCredentials(
|
|
db: Db,
|
|
companyId: string,
|
|
handoff: z.infer<typeof handoffSchema>,
|
|
fetchCredential: (accountIdentity: string) => Promise<PluginEnvironmentTaskCredential | null>,
|
|
) {
|
|
return syncManagedAiCredentials(db, companyId, handoff.binding, "codex", async () => {
|
|
const credential = await fetchCredential(handoff.accountIdentity);
|
|
if (credential === null) return null;
|
|
if (credential.type !== "codex-auth-json") throw new Error("Plugin task credential provider mismatch");
|
|
return credential.authJson;
|
|
});
|
|
}
|
|
|
|
/** Runs for live completion and the cleanup sweep, including after Core restarts. */
|
|
export async function settlePluginTaskCredentialHandoff(
|
|
db: Db,
|
|
scope: Scope,
|
|
metadata: Record<string, unknown>,
|
|
fetchCredential: (accountIdentity: string) => Promise<PluginEnvironmentTaskCredential | null>,
|
|
) {
|
|
if (!metadata[key]) return;
|
|
const handoff = handoffSchema.parse(metadata[key]);
|
|
if (handoff.state === "complete") return;
|
|
const dispatched = metadata.runnerTaskDispatched === true || !!metadata.runnerTaskIdentity;
|
|
if (dispatched && metadata.runnerTaskExecutionStopped !== true) {
|
|
throw new Error("Plugin task credential process termination is unconfirmed");
|
|
}
|
|
const outcome = dispatched
|
|
? await writeBackCredentials(db, scope.companyId, handoff, fetchCredential)
|
|
: "not_dispatched";
|
|
const settled = { ...handoff, state: "complete" as const, outcome };
|
|
const completed = await db.update(environmentLeases).set({
|
|
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ [key]: settled })}::jsonb`,
|
|
}).where(and(
|
|
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
|
|
scope.expectedTaskId === undefined ? undefined : eq(environmentLeases.providerLeaseId, scope.expectedTaskId),
|
|
sql`${environmentLeases.metadata}->${key} = ${JSON.stringify(handoff)}::jsonb`,
|
|
)).returning({ id: environmentLeases.id });
|
|
if (completed.length !== 1) {
|
|
const [current] = await db.select({ metadata: environmentLeases.metadata, taskId: environmentLeases.providerLeaseId }).from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
|
|
));
|
|
const saved = handoffSchema.safeParse(current?.metadata?.[key]);
|
|
if (scope.expectedTaskId !== undefined && current?.taskId !== scope.expectedTaskId ||
|
|
!saved.success || !isDeepStrictEqual({ ...saved.data, outcome: settled.outcome }, settled)) {
|
|
throw new Error("Plugin task credential handoff changed");
|
|
}
|
|
}
|
|
}
|