mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 16:27:18 +02:00
## Thinking Path > - Paperclip coordinates AI agents and their runs. > - Environment plugins own remote capacity. > - A plugin needs a typed API to start and stop its installed Runner. > - Core must derive authority from the saved lease. > - A lost reply or controller restart must not lose cleanup authority. > - This PR adds the worker contract and the host task service. > - Native execution and Runner changes are separate PRs. ## Linked Issues or Issue Description **Subsystem affected** Plugin SDK and environment lease services. **Current behavior** Environment plugins have no typed Runner task hook. Core cannot dispatch these tasks or recover their credential handoff through a saved lease. **Proposed behavior** Add validated task operations, worker negotiation, company and project checks, retained-session authorization, and durable cleanup. Keep credentials out of lease metadata. **Additional context** This PR is the first part of a smaller series. Runner installed-home support is in #16977. Native execution and setup are in #16978. Both are stacked drafts. This PR does not enable native plugin execution, change default environments, add a route, or add a database migration. ## What Changed - Add `supportsTasks` and `onEnvironmentTask` with seven typed operations. - Validate worker requests and responses. Reject shell commands and file transfer operations. - Pin dispatch and cleanup to the saved company, lease, plugin, and driver. - Authorize projects and retained sessions. Validate the current run separately from the retained PRP identity. - Save dispatch before the provider call. Fence admission during cleanup. Require observed process termination. - Save account writeback authority on the lease. Recover Codex refreshes through the existing account and secret services. - Document the task API. Keep the Runner, heartbeat, setup, workspace, and provider changes in later PRs. ## Verification - Full `pnpm -r typecheck` and `pnpm build` passed on the separated API scope. - 201 focused tests passed: SDK worker dispatch, task validation, PostgreSQL task lineage, credential recovery, and existing environment cleanup. - The lineage test covers a third run, repeated same-run replacements, a stale submitting run, and exact termination receipts. - Initial commands hit sandbox socket/IPC restrictions. The permitted runs executed PostgreSQL and passed. A typecheck attempt lacked Cargo in PATH; the run with the installed toolchain passed. - The earlier combined implementation had a full local run with 22,286 passing tests, 120 skips, and eight failed assertions. Task fixture failures were fixed before this split. Remaining host Git and temporary-path assertions are recorded in the previous PR revision. This split does not claim a fresh full local suite pass. - The standard Greptile review scored this API head 5/5 with no unresolved threads. CI is green. The single Apex review also scored this head 5/5 with no actionable findings. - Fresh API CI lost a self-hosted runner during server shard 9. GitHub reported lost communication, with no failed test step. The affected jobs passed on retry without code changes. - CI on the prior combined head failed an unchanged workspace dev-server readiness case. No unrelated test fix is included here. Fresh CI is required for this smaller head. - This scope does not run a provider or a Cloud test. The user owns deployment and the intermediate Cloud test. ## Risks - The API is opt-in. Native plugin execution requires the later integrations. - Providers must make task submission idempotent and prove descendant termination. - A failed status read or credential export keeps the lease available for cleanup. - Codex refresh writeback requires the same active grant, secret reference, and session epoch. Revocation or account replacement discards the old refresh. ## Model Used OpenAI GPT-6 through Codex. Used source inspection, code execution, tests, and GitHub tools. The exact served model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run local checks and recorded failures and their results above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
144 lines
7.3 KiB
TypeScript
144 lines
7.3 KiB
TypeScript
import { aiConnectionMetadataSchema } from "@paperclipai/shared";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import { companySecrets, connectionGrants, environmentLeases, toolConnections, type Db } from "@paperclipai/db";
|
|
import { z, type PluginEnvironmentTaskCredential } from "@paperclipai/plugin-sdk";
|
|
import { mkdtemp, writeFile, rm, realpath } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { isDeepStrictEqual } from "node:util";
|
|
import { decideCodexAuthMerge, withAccountHomeSecretMutationLock } from "@paperclipai/adapter-codex-local/server";
|
|
import { WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
|
import { aiConnectionService } from "./ai-connections.js";
|
|
import { secretService } from "./secrets.js";
|
|
|
|
const bindingSchema = z.object({
|
|
grantId: z.string().uuid(),
|
|
secretId: z.string().uuid(),
|
|
sessionEpoch: z.number().int().nonnegative(),
|
|
versionSelector: z.union([z.literal("latest"), z.number().int().positive()]),
|
|
}).strict();
|
|
export type PluginTaskCredentialWritebackBinding = z.infer<typeof bindingSchema>;
|
|
const handoffSchema = z.object({
|
|
accountIdentity: z.string().min(1),
|
|
binding: bindingSchema,
|
|
state: z.enum(["pending", "complete"]),
|
|
}).strict();
|
|
const key = "runnerTaskCredentialHandoff";
|
|
type Scope = { companyId: string; leaseId: string };
|
|
|
|
/** Save authority, never credential bytes or a controller-local temporary path. */
|
|
export async function registerPluginTaskCredentialHandoff(
|
|
db: Db,
|
|
scope: Scope,
|
|
accountIdentity: string,
|
|
binding: PluginTaskCredentialWritebackBinding,
|
|
) {
|
|
const handoff = handoffSchema.parse({ accountIdentity, binding, state: "pending" });
|
|
await db.transaction(async tx => {
|
|
const [lease] = await tx.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
|
|
)).for("update");
|
|
if (!lease || lease.status !== "active" || lease.metadata?.runnerTaskLease !== true ||
|
|
lease.metadata.runnerTaskRetiring === true) {
|
|
throw new Error("Plugin task credential lease unavailable");
|
|
}
|
|
const saved = lease.metadata[key];
|
|
if (saved) {
|
|
const prior = handoffSchema.parse(saved);
|
|
if (prior.accountIdentity !== accountIdentity || !isDeepStrictEqual(prior.binding, handoff.binding)) {
|
|
throw new Error("Plugin task credential authority changed");
|
|
}
|
|
return;
|
|
}
|
|
await tx.update(environmentLeases).set({ metadata: { ...lease.metadata, [key]: handoff } })
|
|
.where(eq(environmentLeases.id, lease.id));
|
|
});
|
|
}
|
|
|
|
/** Runs for live completion and the cleanup sweep, including after Core restarts. */
|
|
export async function settlePluginTaskCredentialHandoff(
|
|
db: Db,
|
|
scope: Scope,
|
|
metadata: Record<string, unknown>,
|
|
fetchCredential: (accountIdentity: string) => Promise<PluginEnvironmentTaskCredential | null>,
|
|
) {
|
|
if (!metadata[key]) return;
|
|
const handoff = handoffSchema.parse(metadata[key]);
|
|
if (handoff.state === "complete") return;
|
|
if (metadata.runnerTaskDispatched === true && metadata.runnerTaskExecutionStopped !== true) {
|
|
throw new Error("Plugin task credential process termination is unconfirmed");
|
|
}
|
|
const binding = handoff.binding;
|
|
const home = await mkdtemp(join(await realpath(tmpdir()), "paperclip-task-credential-"));
|
|
try {
|
|
// Revocation or an explicit account replacement intentionally discards an old refresh.
|
|
const account = async (database: Db) => {
|
|
const [grant] = await database.select().from(connectionGrants).where(and(
|
|
eq(connectionGrants.companyId, scope.companyId), eq(connectionGrants.id, binding.grantId),
|
|
)).for("update");
|
|
if (!grant || grant.status !== "active") return null;
|
|
const ref = grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
|
|
if (!ref || ref.secretId !== binding.secretId ||
|
|
(ref.versionSelector ?? "latest") !== binding.versionSelector) return null;
|
|
const [secret] = await database.select().from(companySecrets).where(and(
|
|
eq(companySecrets.companyId, scope.companyId), eq(companySecrets.id, binding.secretId),
|
|
)).for("update");
|
|
if (!secret || secret.aiSessionEpoch !== binding.sessionEpoch) return null;
|
|
const [connection] = await database.select().from(toolConnections).where(and(
|
|
eq(toolConnections.companyId, scope.companyId), eq(toolConnections.id, grant.connectionId),
|
|
));
|
|
const ai = aiConnectionMetadataSchema.safeParse(connection?.config.ai);
|
|
if (!connection || !ai.success || ai.data.provider !== "openai" || ai.data.method !== "subscription") return null;
|
|
return { connection, grant };
|
|
};
|
|
const mutate = <T>(operation: (database: Db) => Promise<T>) => withAccountHomeSecretMutationLock(
|
|
undefined, scope.companyId, () => db.transaction(tx => operation(tx as unknown as Db)),
|
|
);
|
|
const current = metadata.runnerTaskDispatched === true && await mutate(account);
|
|
if (current) {
|
|
// Do not hold database or account locks while the provider exports its credential.
|
|
const credential = await fetchCredential(handoff.accountIdentity);
|
|
if (credential !== null) {
|
|
if (credential.type !== "codex-auth-json") throw new Error("Plugin task credential provider mismatch");
|
|
const incoming = join(home, "incoming.json"), destination = join(home, "current.json");
|
|
await writeFile(incoming, credential.authJson, { mode: 0o600 });
|
|
const writeBack = () => mutate(async database => {
|
|
const selected = await account(database);
|
|
if (!selected) return;
|
|
const value = await aiConnectionService(database).credential(selected);
|
|
await writeFile(destination, value, { mode: 0o600 });
|
|
if (await decideCodexAuthMerge(incoming, destination, { errorLabel: "AI account refresh" }) !== 10) return;
|
|
await secretService(database).rotate(binding.secretId, {
|
|
value: credential.authJson, preserveAiSessionEpoch: true,
|
|
}, { userId: selected.grant.subjectUserId });
|
|
});
|
|
for (let attempt = 0; ; attempt++) {
|
|
try {
|
|
await writeBack();
|
|
break;
|
|
} catch (error) {
|
|
if (attempt >= 2 || (error as NodeJS.ErrnoException)?.code !== WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE) throw error;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
const completed = await db.update(environmentLeases).set({
|
|
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ [key]: { ...handoff, state: "complete" } })}::jsonb`,
|
|
}).where(and(
|
|
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
|
|
sql`${environmentLeases.metadata}->${key} = ${JSON.stringify(handoff)}::jsonb`,
|
|
)).returning({ id: environmentLeases.id });
|
|
if (completed.length !== 1) {
|
|
const [current] = await db.select({ metadata: environmentLeases.metadata }).from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
|
|
));
|
|
if (!isDeepStrictEqual(handoffSchema.safeParse(current?.metadata?.[key]).data, { ...handoff, state: "complete" })) {
|
|
throw new Error("Plugin task credential handoff changed");
|
|
}
|
|
}
|
|
} finally {
|
|
// The provider retains the recoverable credential until completion succeeds.
|
|
await rm(home, { recursive: true, force: true });
|
|
}
|
|
}
|