Files
PaperClipAI/server/src/services/plugin-task-credential-writeback.ts
T
Nicky LeachandPaperclip 31bf6e4482 Pin plugin Runner cleanup to its physical task (#19300)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Plugin Runner owners manage a physical task through an environment
lease.
> - The existing replacement path can assign a new physical task to that
same lease.
> - A delayed old owner can then stop the replacement or mark it
stopped.
> - Cleanup must keep the original physical task identity across RPC and
database writes.
> - This pull request pins that identity and keeps cleanup available to
the new owner.

## Linked Issues or Issue Description

Refs #19134 and #19157. This fix is based on master and does not enable
repeated stopped-source recovery. A search found no duplicate fix.

**What happened?**

The maintained `createOwnedSandbox` replacement path changes
`providerLeaseId` on one lease. An old task owner uses that mutable
lease for a later Stop. A stopped-state response already in flight can
also update the new task's flags.

**Expected behavior**

The old owner must address its original task. Its late cleanup writes
must not alter the replacement. The replacement's own owner must still
stop and complete it.

**Steps to reproduce**

1. Create a plugin task owner for a stopped task.
2. Replace that task through the existing describe/startup path on the
same lease.
3. Resume an old checkpoint or a held old status response.
4. Observe a Stop for the replacement or an incorrect stopped flag
before this fix.

## What Changed

- Require each task owner to keep the physical ID returned by its
description. Reject a rebind to a different task.
- Carry the ID through the existing internal task context after startup
ends. Reject a changed physical task before RPC dispatch.
- Match the physical task in stopped-state, completion and
credential-handoff writes.
- Hold the exact lease row through the retained-session database update
and local checkpoint save.
- Cover the maintained replacement path with real PostgreSQL tests.
Preserve positive cleanup for the replacement owner. Document this
boundary.

## Verification

- Both replacement regressions failed on unmodified master `10a7d039c8`:
delayed old-owner Stop targeted the replacement, and a held old status
response marked it stopped.
- Six focused files passed 160 tests: task authority, task runtime, warm
tasks, execution target, credential writeback and task owner.
- Four selected executor factory cases passed. They cover task-only
providers and the physical task binding after the startup context ends.
- Runner TypeScript, plugin SDK and adapter-utils builds passed. Server
TypeScript check passed. Diff and added-line secret scans passed.
- All 15 setup-probe cases passed with the actual Runner and fake
provider binaries. The first local attempt lacked cargo on PATH. A
sandboxed native attempt timed out in seven process cases; the unchanged
run with loopback permission passed all 15. A separate initial DB
integration attempt was denied a loopback listener; its
permission-correct run passed.
- Review found three missed constructors in the existing SDK/Core
lifecycle fixture. They now use the actual describe result. All four
cases in that real PostgreSQL and worker RPC file passed.
- The first GitHub attempt failed in the unchanged GitHub recovery
pagination case in `chat-channels.integration.test.ts`: its zero-request
assertion observed a configuration GET. That exact case passed locally
without code or assertion changes. The one failed-job-only retry passed;
the initial failure log is retained.
- [GitHub workflow
42082587576](https://github.com/paperclipai/paperclip/actions/runs/42082587576)
passed on its second attempt. All CI checks completed, and Snyk passed.
Standard Greptile scored exact head `fb914d98b99e` 5/5 with no
unresolved threads. The branch is mergeable and ready for review.

## Risks

The task context change is internal. It adds no plugin wire field,
schema or Runner mode. A stale owner now gets an error and cannot clean
up a replacement. A new owner can still clean up that task. The retained
checkpoint write holds one existing lease row during a bounded local
file write.

Repeated stopped-source incarnation admission, runtime pins and
deployments are outside this change.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning. Used repository
tools, TypeScript and real PostgreSQL tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with version and capability
details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the bug template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change
- [x] I have run the focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no unresolved findings
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 12:14:09 -08:00

146 lines
7.6 KiB
TypeScript

import { aiConnectionMetadataSchema } from "@paperclipai/shared";
import { and, eq, sql } from "drizzle-orm";
import { companySecrets, connectionGrants, environmentLeases, toolConnections, type Db } from "@paperclipai/db";
import { z, type PluginEnvironmentTaskCredential } from "@paperclipai/plugin-sdk";
import { mkdtemp, writeFile, rm, realpath } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { isDeepStrictEqual } from "node:util";
import { decideCodexAuthMerge, withAccountHomeSecretMutationLock } from "@paperclipai/adapter-codex-local/server";
import { WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE } from "@paperclipai/adapter-utils/workspace-restore-merge";
import { aiConnectionService } from "./ai-connections.js";
import { secretService } from "./secrets.js";
const bindingSchema = z.object({
grantId: z.string().uuid(),
secretId: z.string().uuid(),
sessionEpoch: z.number().int().nonnegative(),
versionSelector: z.union([z.literal("latest"), z.number().int().positive()]),
}).strict();
export type PluginTaskCredentialWritebackBinding = z.infer<typeof bindingSchema>;
const handoffSchema = z.object({
accountIdentity: z.string().min(1),
binding: bindingSchema,
state: z.enum(["pending", "complete"]),
}).strict();
const key = "runnerTaskCredentialHandoff";
type Scope = { companyId: string; leaseId: string; expectedTaskId?: string };
/** Save authority, never credential bytes or a controller-local temporary path. */
export async function registerPluginTaskCredentialHandoff(
db: Db,
scope: Scope,
accountIdentity: string,
binding: PluginTaskCredentialWritebackBinding,
) {
const handoff = handoffSchema.parse({ accountIdentity, binding, state: "pending" });
await db.transaction(async tx => {
const [lease] = await tx.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
)).for("update");
if (!lease || lease.status !== "active" || lease.metadata?.runnerTaskLease !== true ||
lease.metadata.runnerTaskRetiring === true) {
throw new Error("Plugin task credential lease unavailable");
}
const saved = lease.metadata[key];
if (saved) {
const prior = handoffSchema.parse(saved);
if (prior.accountIdentity !== accountIdentity || !isDeepStrictEqual(prior.binding, handoff.binding)) {
throw new Error("Plugin task credential authority changed");
}
return;
}
await tx.update(environmentLeases).set({ metadata: { ...lease.metadata, [key]: handoff } })
.where(eq(environmentLeases.id, lease.id));
});
}
/** Runs for live completion and the cleanup sweep, including after Core restarts. */
export async function settlePluginTaskCredentialHandoff(
db: Db,
scope: Scope,
metadata: Record<string, unknown>,
fetchCredential: (accountIdentity: string) => Promise<PluginEnvironmentTaskCredential | null>,
) {
if (!metadata[key]) return;
const handoff = handoffSchema.parse(metadata[key]);
if (handoff.state === "complete") return;
if (metadata.runnerTaskDispatched === true && metadata.runnerTaskExecutionStopped !== true) {
throw new Error("Plugin task credential process termination is unconfirmed");
}
const binding = handoff.binding;
const home = await mkdtemp(join(await realpath(tmpdir()), "paperclip-task-credential-"));
try {
// Revocation or an explicit account replacement intentionally discards an old refresh.
const account = async (database: Db) => {
const [grant] = await database.select().from(connectionGrants).where(and(
eq(connectionGrants.companyId, scope.companyId), eq(connectionGrants.id, binding.grantId),
)).for("update");
if (!grant || grant.status !== "active") return null;
const ref = grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
if (!ref || ref.secretId !== binding.secretId ||
(ref.versionSelector ?? "latest") !== binding.versionSelector) return null;
const [secret] = await database.select().from(companySecrets).where(and(
eq(companySecrets.companyId, scope.companyId), eq(companySecrets.id, binding.secretId),
)).for("update");
if (!secret || secret.aiSessionEpoch !== binding.sessionEpoch) return null;
const [connection] = await database.select().from(toolConnections).where(and(
eq(toolConnections.companyId, scope.companyId), eq(toolConnections.id, grant.connectionId),
));
const ai = aiConnectionMetadataSchema.safeParse(connection?.config.ai);
if (!connection || !ai.success || ai.data.provider !== "openai" || ai.data.method !== "subscription") return null;
return { connection, grant };
};
const mutate = <T>(operation: (database: Db) => Promise<T>) => withAccountHomeSecretMutationLock(
undefined, scope.companyId, () => db.transaction(tx => operation(tx as unknown as Db)),
);
const current = metadata.runnerTaskDispatched === true && await mutate(account);
if (current) {
// Do not hold database or account locks while the provider exports its credential.
const credential = await fetchCredential(handoff.accountIdentity);
if (credential !== null) {
if (credential.type !== "codex-auth-json") throw new Error("Plugin task credential provider mismatch");
const incoming = join(home, "incoming.json"), destination = join(home, "current.json");
await writeFile(incoming, credential.authJson, { mode: 0o600 });
const writeBack = () => mutate(async database => {
const selected = await account(database);
if (!selected) return;
const value = await aiConnectionService(database).credential(selected);
await writeFile(destination, value, { mode: 0o600 });
if (await decideCodexAuthMerge(incoming, destination, { errorLabel: "AI account refresh" }) !== 10) return;
await secretService(database).rotate(binding.secretId, {
value: credential.authJson, preserveAiSessionEpoch: true,
}, { userId: selected.grant.subjectUserId });
});
for (let attempt = 0; ; attempt++) {
try {
await writeBack();
break;
} catch (error) {
if (attempt >= 2 || (error as NodeJS.ErrnoException)?.code !== WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE) throw error;
}
}
}
}
const completed = await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ [key]: { ...handoff, state: "complete" } })}::jsonb`,
}).where(and(
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
scope.expectedTaskId === undefined ? undefined : eq(environmentLeases.providerLeaseId, scope.expectedTaskId),
sql`${environmentLeases.metadata}->${key} = ${JSON.stringify(handoff)}::jsonb`,
)).returning({ id: environmentLeases.id });
if (completed.length !== 1) {
const [current] = await db.select({ metadata: environmentLeases.metadata, taskId: environmentLeases.providerLeaseId }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
));
if (scope.expectedTaskId !== undefined && current?.taskId !== scope.expectedTaskId ||
!isDeepStrictEqual(handoffSchema.safeParse(current?.metadata?.[key]).data, { ...handoff, state: "complete" })) {
throw new Error("Plugin task credential handoff changed");
}
}
} finally {
// The provider retains the recoverable credential until completion succeeds.
await rm(home, { recursive: true, force: true });
}
}