Files
PaperClipAI/server/src/services/plugin-task-credential-writeback.ts
T
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00

120 lines
5.9 KiB
TypeScript

import { and, eq, sql } from "drizzle-orm";
import { environmentLeases, type Db } from "@paperclipai/db";
import { z, type PluginEnvironmentTaskCredential } from "@paperclipai/plugin-sdk";
import { isDeepStrictEqual } from "node:util";
import { managedAiCredentialWritebackBindingSchema, syncManagedAiCredentials, type ManagedAiCredentialWritebackBinding } from "./managed-ai-credential-writeback.js";
const handoffSchema = z.object({
accountIdentity: z.string().min(1),
binding: managedAiCredentialWritebackBindingSchema,
state: z.enum(["pending", "complete"]),
outcome: z.enum(["rotated", "unchanged", "binding_changed", "not_dispatched", "no_credentials"]).optional(),
}).strict();
const key = "runnerTaskCredentialHandoff";
type Scope = { companyId: string; leaseId: string; expectedTaskId?: string };
export function canReadStoppedPluginTaskCredential(metadata: Record<string, unknown>, accountIdentity: string): boolean {
const handoff = handoffSchema.safeParse(metadata[key]);
return metadata.runnerTaskExecutionStopped === true && handoff.success &&
handoff.data.state === "pending" && handoff.data.accountIdentity === accountIdentity;
}
/** Save authority, never credential bytes or a controller-local temporary path. */
export async function registerPluginTaskCredentialHandoff(
db: Db,
scope: Scope,
accountIdentity: string,
binding: ManagedAiCredentialWritebackBinding,
) {
const handoff = handoffSchema.parse({ accountIdentity, binding, state: "pending" });
await db.transaction(async tx => {
const [lease] = await tx.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
)).for("update");
if (!lease || lease.status !== "active" || lease.metadata?.runnerTaskLease !== true ||
lease.metadata.runnerTaskRetiring === true) {
throw new Error("Plugin task credential lease unavailable");
}
const saved = lease.metadata[key];
if (saved) {
const prior = handoffSchema.parse(saved);
if (prior.accountIdentity !== accountIdentity || !isDeepStrictEqual(prior.binding, handoff.binding)) {
throw new Error("Plugin task credential authority changed");
}
return;
}
await tx.update(environmentLeases).set({ metadata: { ...lease.metadata, [key]: handoff } })
.where(eq(environmentLeases.id, lease.id));
});
}
/** Live PRP updates use durable authority, including while a warm task is idle. */
export async function syncPluginTaskCredentialHandoff(
db: Db,
scope: Scope,
fetchCredential: (accountIdentity: string) => Promise<PluginEnvironmentTaskCredential | null>,
) {
const [lease] = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
scope.expectedTaskId === undefined ? undefined : eq(environmentLeases.providerLeaseId, scope.expectedTaskId),
));
if (!lease) throw new Error("Plugin task credential lease unavailable");
if (!lease.metadata?.[key]) return;
const handoff = handoffSchema.parse(lease.metadata[key]);
if (handoff.state === "complete") return;
const outcome = await writeBackCredentials(db, scope.companyId, handoff, fetchCredential);
if (outcome === "no_credentials") throw new Error("Plugin task credential snapshot unavailable");
return outcome;
}
async function writeBackCredentials(
db: Db,
companyId: string,
handoff: z.infer<typeof handoffSchema>,
fetchCredential: (accountIdentity: string) => Promise<PluginEnvironmentTaskCredential | null>,
) {
return syncManagedAiCredentials(db, companyId, handoff.binding, "codex", async () => {
const credential = await fetchCredential(handoff.accountIdentity);
if (credential === null) return null;
if (credential.type !== "codex-auth-json") throw new Error("Plugin task credential provider mismatch");
return credential.authJson;
});
}
/** Runs for live completion and the cleanup sweep, including after Core restarts. */
export async function settlePluginTaskCredentialHandoff(
db: Db,
scope: Scope,
metadata: Record<string, unknown>,
fetchCredential: (accountIdentity: string) => Promise<PluginEnvironmentTaskCredential | null>,
) {
if (!metadata[key]) return;
const handoff = handoffSchema.parse(metadata[key]);
if (handoff.state === "complete") return;
const dispatched = metadata.runnerTaskDispatched === true || !!metadata.runnerTaskIdentity;
if (dispatched && metadata.runnerTaskExecutionStopped !== true) {
throw new Error("Plugin task credential process termination is unconfirmed");
}
const outcome = dispatched
? await writeBackCredentials(db, scope.companyId, handoff, fetchCredential)
: "not_dispatched";
const settled = { ...handoff, state: "complete" as const, outcome };
const completed = await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ [key]: settled })}::jsonb`,
}).where(and(
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
scope.expectedTaskId === undefined ? undefined : eq(environmentLeases.providerLeaseId, scope.expectedTaskId),
sql`${environmentLeases.metadata}->${key} = ${JSON.stringify(handoff)}::jsonb`,
)).returning({ id: environmentLeases.id });
if (completed.length !== 1) {
const [current] = await db.select({ metadata: environmentLeases.metadata, taskId: environmentLeases.providerLeaseId }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
));
const saved = handoffSchema.safeParse(current?.metadata?.[key]);
if (scope.expectedTaskId !== undefined && current?.taskId !== scope.expectedTaskId ||
!saved.success || !isDeepStrictEqual({ ...saved.data, outcome: settled.outcome }, settled)) {
throw new Error("Plugin task credential handoff changed");
}
}
}