Add lease-scoped Runner task hooks to environment plugins (#15406)

## Thinking Path

> - Paperclip coordinates AI agents and their runs.
> - Environment plugins own remote capacity.
> - A plugin needs a typed API to start and stop its installed Runner.
> - Core must derive authority from the saved lease.
> - A lost reply or controller restart must not lose cleanup authority.
> - This PR adds the worker contract and the host task service.
> - Native execution and Runner changes are separate PRs.

## Linked Issues or Issue Description

**Subsystem affected**

Plugin SDK and environment lease services.

**Current behavior**

Environment plugins have no typed Runner task hook. Core cannot dispatch
these tasks or recover their credential handoff through a saved lease.

**Proposed behavior**

Add validated task operations, worker negotiation, company and project
checks, retained-session authorization, and durable cleanup. Keep
credentials out of lease metadata.

**Additional context**

This PR is the first part of a smaller series. Runner installed-home
support is in #16977. Native execution and setup are in #16978. Both are
stacked drafts. This PR does not enable native plugin execution, change
default environments, add a route, or add a database migration.

## What Changed

- Add `supportsTasks` and `onEnvironmentTask` with seven typed
operations.
- Validate worker requests and responses. Reject shell commands and file
transfer operations.
- Pin dispatch and cleanup to the saved company, lease, plugin, and
driver.
- Authorize projects and retained sessions. Validate the current run
separately from the retained PRP identity.
- Save dispatch before the provider call. Fence admission during
cleanup. Require observed process termination.
- Save account writeback authority on the lease. Recover Codex refreshes
through the existing account and secret services.
- Document the task API. Keep the Runner, heartbeat, setup, workspace,
and provider changes in later PRs.

## Verification

- Full `pnpm -r typecheck` and `pnpm build` passed on the separated API
scope.
- 201 focused tests passed: SDK worker dispatch, task validation,
PostgreSQL task lineage, credential recovery, and existing environment
cleanup.
- The lineage test covers a third run, repeated same-run replacements, a
stale submitting run, and exact termination receipts.
- Initial commands hit sandbox socket/IPC restrictions. The permitted
runs executed PostgreSQL and passed. A typecheck attempt lacked Cargo in
PATH; the run with the installed toolchain passed.
- The earlier combined implementation had a full local run with 22,286
passing tests, 120 skips, and eight failed assertions. Task fixture
failures were fixed before this split. Remaining host Git and
temporary-path assertions are recorded in the previous PR revision. This
split does not claim a fresh full local suite pass.
- The standard Greptile review scored this API head 5/5 with no
unresolved threads. CI is green. The single Apex review also scored this
head 5/5 with no actionable findings.
- Fresh API CI lost a self-hosted runner during server shard 9. GitHub
reported lost communication, with no failed test step. The affected jobs
passed on retry without code changes.
- CI on the prior combined head failed an unchanged workspace dev-server
readiness case. No unrelated test fix is included here. Fresh CI is
required for this smaller head.
- This scope does not run a provider or a Cloud test. The user owns
deployment and the intermediate Cloud test.

## Risks

- The API is opt-in. Native plugin execution requires the later
integrations.
- Providers must make task submission idempotent and prove descendant
termination.
- A failed status read or credential export keeps the lease available
for cleanup.
- Codex refresh writeback requires the same active grant, secret
reference, and session epoch. Revocation or account replacement discards
the old refresh.

## Model Used

OpenAI GPT-6 through Codex. Used source inspection, code execution,
tests, and GitHub tools. The exact served model ID and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run local checks and recorded failures and their results
above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed 2026-10-20 09:14:23 -07:00
1 parent 98d17da1e2
commit 33cc567f75
17 files changed
+1124 -15

No files matched your search

+7
View File
@@ -791,3 +791,10 @@ The host resets plugin state on account/company changes and keeps its built-in
menu when no unique contribution exists, discovery fails, the module is missing,
or rendering throws. The slot is a React-only contract; do not use a custom
element export. This replaces only the menu, not company policy or authorization.
## Runner task worker hooks
A plugin can declare `supportsTasks` on an environment driver and implement
`onEnvironmentTask`. See the [task API](PLUGIN_ENVIRONMENT_TASKS.md) for the
operations, validation, and lease rules. This hook does not add a shell or file
transfer API. Native agent execution is a separate integration.
+95
View File
@@ -0,0 +1,95 @@
# Environment task API
An environment driver can provide a Runner task through a worker hook.
The task API has no shell command or file transfer operation.
This API is a server and worker contract. It does not select an environment
for a native agent or connect a Runner to a run. Those integrations are separate.
## Declare the hook
Set `supportsTasks: true` on the environment driver in the plugin manifest.
Implement `onEnvironmentTask` with the SDK types:
```ts
import type {
PluginEnvironmentTaskParams,
PluginEnvironmentTaskResult,
} from "@paperclipai/plugin-sdk";
type TaskHook = (
params: PluginEnvironmentTaskParams,
) => Promise<PluginEnvironmentTaskResult>;
```
Core requires the `environment.drivers.register` capability, a ready plugin,
the manifest declaration, and the worker's `environmentTask` method.
The SDK validates each operation and its result before it returns the result.
See [the schemas](../../packages/plugins/sdk/src/environment-tasks.ts) for all fields.
## Operations
| Operation | Result | Required behavior |
| --- | --- | --- |
| `describe` | `description` | Return the installed Runner version and SHA-256 digest. |
| `submit` | `accepted` | Start the same opaque task at most once. Use the structured launch inputs. |
| `connection` | `connection` | Return an authenticated WSS endpoint and its generation. |
| `status` | `status` | Report the phase and observed process termination. |
| `credentials` | `credentials` | Return the selected account after process termination. |
| `stop` | `accepted` | Stop the task and its descendants. Fence a delayed submission. |
| `complete` | `accepted` | Release task resources after Core saves required results. |
Each result must contain the task ID from the request. Core rejects a result
with a different ID or result kind. A WSS URL must have no user information,
query, or fragment. Return authentication in the secret headers.
`submit` contains the PRP version range, Runner identity, bootstrap ticket,
authorized project IDs, and optional launch limits. It can also contain a saved
task ID for session reuse and a typed Codex or Claude credential. Core does not
supply an executable command or a guest path.
## Lease authority
Core calls `environmentRuntimeService.task` with a company ID, a saved lease ID,
and an operation. It derives the plugin and driver from the lease. A change to
the environment configuration cannot redirect cleanup to a new provider.
Before submission, Core checks the active lease, run, issue, and every project
in the company. It checks the submitting run ID separately from a retained PRP
identity. Session reuse requires the same agent, environment, workspace, and
provider. It also requires a retained source task and proof of termination.
An internal setup probe can use a private, time-limited lease without a run row.
Core checks the saved probe identity, rejects project mounts, and limits the
Runner lifetime to the remaining probe time.
## Failure and cleanup
A failed submission reply does not prove that the task failed to start.
Core saves the dispatch marker before it calls the worker. Retry against the
same task ID. Do not allocate another task to resolve an uncertain reply.
An accepted stop does not prove termination. Set `executionStopped: true`
only after the Runner and all descendants have stopped. A terminal phase alone
does not permit Core to release execution authority.
Core fences new submissions when cleanup starts. It stops the saved task,
checks termination, saves credential results, and completes the task before it
releases or destroys the environment lease. An uncertain result leaves a saved
lease for the cleanup sweep. Cleanup can use that lease after the environment
or run row is deleted.
## Credential writeback
Account selection and credential revision have separate identities. Keep the
selected account fixed for the task. A credential result of `null` means that
cancellation prevented task admission. It must not mean that an export failed.
Core saves the account grant, secret reference, and session epoch on the lease.
It saves no credential bytes or temporary path in lease metadata. After process
termination, it can recover a Codex refresh even if Core restarted. It uses the
existing account comparison and secret service. A revoked grant, replaced
account, or changed session epoch discards the old refresh.
Keep file synchronization and backup outside this API. The provider must retain
recoverable credential state until task completion succeeds.
@@ -1,3 +1,4 @@
import type { PluginEnvironmentTaskParams, PluginEnvironmentTaskResult } from "./environment-tasks.js";
import type { AgentLifecycleRequest, AgentLifecycleResult } from "@paperclipai/shared";
import type { AiConnectionRouterRequest, AiConnectionRouterResult } from "@paperclipai/shared";
/**
@@ -401,6 +402,9 @@ export interface PluginDefinition {
params: PluginEnvironmentValidateConfigParams,
): Promise<PluginEnvironmentValidationResult>;
/** Provide Runner execution: prepare resources, start Runner, connect via PRP, and manage task lifecycle. */
onEnvironmentTask?(params: PluginEnvironmentTaskParams): Promise<PluginEnvironmentTaskResult>;
/** Called to test reachability or readiness of a plugin-hosted environment. */
onEnvironmentProbe?(
params: PluginEnvironmentProbeParams,
@@ -0,0 +1,114 @@
import { z } from "zod";
import type { PluginEnvironmentDriverBaseParams, PluginEnvironmentLease } from "./protocol.js";
// Provider IDs are opaque; providers validate their own addressing constraints.
const providerId = z.string().min(1);
// PRP identity.schema.json stableId.
const runnerIdentity = z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._:-]{0,159}$/);
const identifier = z.string().regex(/^[a-zA-Z0-9][a-zA-Z0-9_-]{0,95}$/);
const secureUrl = z.string().url().refine(value => {
const url = new URL(value);
return url.protocol === "wss:" && !url.username && !url.password && !url.search && !url.hash;
}, "Expected a credential-free WSS URL");
const credentialValue = z.string().min(1).max(65_536).refine(value => !value.includes("\0"));
/** Only provider authentication is transferred. Paths and executable settings remain provider-owned. */
export const environmentTaskCredentialSchema = z.discriminatedUnion("type", [
z.object({ type: z.literal("codex-auth-json"), authJson: credentialValue }).strict(),
z.object({ type: z.literal("claude-env"), environment: z.object({
CLAUDE_CODE_OAUTH_TOKEN: credentialValue.optional(), ANTHROPIC_API_KEY: credentialValue.optional(),
}).strict().refine(env => Object.values(env).filter(Boolean).length === 1, "Select one Claude credential") }).strict(),
]);
export type PluginEnvironmentTaskCredential = z.infer<typeof environmentTaskCredentialSchema>;
/** One plugin-provided Runner execution attempt with a PRP connection. Secrets are transient RPC input. */
export const environmentTaskOperationSchema = z.discriminatedUnion("kind", [
z.object({ kind: z.literal("describe") }).strict(),
z.object({
kind: z.literal("submit"),
projectIds: z.array(z.string().uuid()).max(64).refine(ids => new Set(ids).size === ids.length, "Duplicate project IDs"),
runner: z.object({
/** Inclusive PRP version range supported by the submitting client. */
protocolMin: z.number().int().min(1).max(Number.MAX_SAFE_INTEGER),
protocolMax: z.number().int().min(1).max(Number.MAX_SAFE_INTEGER),
harness: identifier,
runnerId: runnerIdentity, leaseId: runnerIdentity, runId: runnerIdentity,
sessionId: runnerIdentity, turnId: runnerIdentity, itemId: runnerIdentity,
}).strict().refine(value => value.protocolMin <= value.protocolMax, "Invalid PRP version range"),
bootstrapTicket: z.string().min(1).max(65_536),
/** Opaque prior provider task; Core authorizes the saved session before supplying it. */
resumeFrom: z.string().min(1).optional(),
/** Require the externally installed account to match this Core-selected credential revision. */
credentialIdentity: z.string().min(1).max(1024).optional(),
/** Stable account selection; distinct from its current credential revision. */
accountIdentity: z.string().min(1).max(1024).optional(),
credential: environmentTaskCredentialSchema.optional(),
/** Core supplies these protected launch inputs for native agent runs. */
launch: z.object({
runnerVersion: z.string().min(1), runnerDigest: z.string().regex(/^sha256:[a-f0-9]{64}$/),
maxOutboxBytes: z.number().int().positive(), p0ReserveBytes: z.number().int().positive(),
maxRuntimeMs: z.number().int().nonnegative().optional(), maxLifetimeMs: z.number().int().nonnegative().optional(),
reconnectGraceMs: z.number().int().nonnegative().optional(),
lifecyclePolicy: z.object({ mode: z.literal("per_turn"), idleTimeoutMs: z.null() }).strict(),
}).strict().optional(),
}).strict(),
z.object({ kind: z.literal("credentials"), accountIdentity: z.string().min(1).max(1024) }).strict(),
z.object({ kind: z.literal("status") }).strict(),
z.object({ kind: z.literal("connection") }).strict(),
z.object({ kind: z.literal("complete") }).strict(),
z.object({ kind: z.literal("stop") }).strict(),
]);
export const environmentTaskResultSchema = z.discriminatedUnion("kind", [
z.object({ kind: z.literal("accepted"), taskId: providerId }).strict(),
z.object({ kind: z.literal("credentials"), taskId: providerId, /** Null proves that cancellation fenced admission before the task could start. It must not mean an unavailable export. */
credential: environmentTaskCredentialSchema.nullable() }).strict(),
z.object({ kind: z.literal("description"), taskId: providerId,
runner: z.object({ version: z.string().min(1), digest: z.string().regex(/^sha256:[a-f0-9]{64}$/) }).strict(),
}).strict(),
z.object({
kind: z.literal("status"), taskId: providerId,
phase: z.enum(["preparing", "running", "completed", "failed", "cancelled", "interrupted"]),
exitCode: z.number().int().optional(),
/** Provider observed the task and all descendants stopped; terminal phase alone is insufficient. */
executionStopped: z.boolean().optional(),
/** The provider retains this task's suspended session for a later launch. */
sessionStateAvailable: z.boolean().optional(),
}).strict(),
z.object({
kind: z.literal("connection"), taskId: providerId,
endpoint: z.object({
kind: z.literal("authenticated_websocket"), websocketUrl: secureUrl,
generation: providerId,
secretHeaders: z.array(z.object({
name: z.string().regex(/^[!#$%&'*+.^_`|~0-9A-Za-z-]+$/),
value: z.string().min(1).max(65_536).regex(/^[^\r\n]+$/),
}).strict()).max(16),
}).strict(),
}).strict(),
]);
export type PluginEnvironmentTaskOperation = z.infer<typeof environmentTaskOperationSchema>;
export type PluginEnvironmentTaskResult = z.infer<typeof environmentTaskResultSchema>;
export interface PluginEnvironmentTaskParams extends Omit<PluginEnvironmentDriverBaseParams, "environmentId"> {
/** Null during cleanup after the environment is deleted. */
environmentId: string | null;
lease: PluginEnvironmentLease;
/** Provider-issued task identifier persisted in the lease, stable across ambiguous submission retries. */
taskId: string;
/** Null during cleanup after the associated run is deleted. */
runId: string | null;
agentId: string | null;
/** Company-validated projects to prepare on submit; empty during cleanup. */
projectIds: string[];
operation: PluginEnvironmentTaskOperation;
}
/** An accepted stop is not proof of process termination. */
export function parseEnvironmentTaskResult(operation: PluginEnvironmentTaskOperation, taskId: string, value: unknown): PluginEnvironmentTaskResult {
const result = environmentTaskResultSchema.parse(value);
const expected = operation.kind === "describe" ? "description" : ["status", "connection", "credentials"].includes(operation.kind) ? operation.kind : "accepted";
if (result.taskId !== taskId || result.kind !== expected) throw new Error("Invalid plugin-provided Runner execution response");
return result;
}
+2
View File
@@ -463,4 +463,6 @@ export type { PluginEnvironmentSyncErrorDiagnostic } from "./environment-sync-er
export type { AiConnectionPool, AiConnectionPoolConfig, AiConnectionPoolMember, AiConnectionRouterRequest, AiConnectionRouterResult, AiConnectionRouterSelection } from "@paperclipai/shared";
export { environmentTaskOperationSchema, environmentTaskResultSchema, environmentTaskCredentialSchema, parseEnvironmentTaskResult } from "./environment-tasks.js";
export type { PluginEnvironmentTaskOperation, PluginEnvironmentTaskParams, PluginEnvironmentTaskResult, PluginEnvironmentTaskCredential } from "./environment-tasks.js";
export type { AgentLifecycleRequest, AgentLifecycleResult } from "@paperclipai/shared";
+3
View File
@@ -1,3 +1,4 @@
import type { PluginEnvironmentTaskParams, PluginEnvironmentTaskResult } from "./environment-tasks.js";
import type { PluginAgentRuntimeState } from "./types.js";
import type { AgentLifecycleRequest, AgentLifecycleResult } from "@paperclipai/shared";
import type { AiConnectionRouterRequest, AiConnectionRouterResult } from "@paperclipai/shared";
@@ -1393,6 +1394,7 @@ export interface HostToWorkerMethods {
params: PluginEnvironmentValidateConfigParams,
result: PluginEnvironmentValidationResult,
];
environmentTask: [params: PluginEnvironmentTaskParams, result: PluginEnvironmentTaskResult];
environmentProbe: [
params: PluginEnvironmentProbeParams,
result: PluginEnvironmentProbeResult,
@@ -1521,6 +1523,7 @@ export const HOST_TO_WORKER_OPTIONAL_METHODS: readonly HostToWorkerMethodName[]
"resolveExternalObject",
"refreshExternalObjects",
"environmentValidateConfig",
"environmentTask",
"environmentProbe",
"environmentAcquireLease",
"environmentResumeLease",
@@ -1,3 +1,4 @@
import { environmentTaskOperationSchema, parseEnvironmentTaskResult, type PluginEnvironmentTaskParams } from "./environment-tasks.js";
import type { AgentLifecycleRequest, AgentLifecycleResult } from "@paperclipai/shared";
import type { AiConnectionRouterRequest, AiConnectionRouterResult } from "@paperclipai/shared";
import { environmentCreationCleanupErrorData } from "./environment-creation-cleanup.js";
@@ -1682,6 +1683,13 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost
case "environmentValidateConfig":
return handleEnvironmentValidateConfig(params as PluginEnvironmentValidateConfigParams);
case "environmentTask": {
if (!plugin.definition.onEnvironmentTask) throw methodNotImplemented("environmentTask");
const input = params as PluginEnvironmentTaskParams;
const operation = environmentTaskOperationSchema.parse(input.operation);
return parseEnvironmentTaskResult(operation, input.taskId, await plugin.definition.onEnvironmentTask({ ...input, operation }));
}
case "environmentProbe":
return handleEnvironmentProbe(params as PluginEnvironmentProbeParams);
@@ -1801,6 +1809,7 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost
if (plugin.definition.onResolveExternalObject) supportedMethods.push("resolveExternalObject");
if (plugin.definition.onRefreshExternalObjects) supportedMethods.push("refreshExternalObjects");
if (plugin.definition.onEnvironmentValidateConfig) supportedMethods.push("environmentValidateConfig");
if (plugin.definition.onEnvironmentTask) supportedMethods.push("environmentTask");
if (plugin.definition.onEnvironmentProbe) supportedMethods.push("environmentProbe");
if (plugin.definition.onEnvironmentAcquireLease) supportedMethods.push("environmentAcquireLease");
if (plugin.definition.onEnvironmentResumeLease) supportedMethods.push("environmentResumeLease");
+2
View File
@@ -191,6 +191,8 @@ export interface SandboxProviderCapabilities {
}
export interface PluginEnvironmentDriverDeclaration {
/** Provides Runner execution, resource preparation, and PRP connections via environmentTask. */
supportsTasks?: boolean;
/** Stable driver key, unique within the plugin. Namespaced by plugin ID at runtime. */
driverKey: string;
/**
+1
View File
@@ -177,6 +177,7 @@ export type SandboxProviderCapabilitiesInput = z.infer<typeof sandboxProviderCap
// validation error, not a silently dropped field. A misspelled capability name
// (for example `supportsLoginPTY`) fails validation instead of dropping.
export const pluginEnvironmentDriverDeclarationSchema = z.object({
supportsTasks: z.boolean().optional(),
driverKey: z.string().min(1).regex(
/^[a-z0-9][a-z0-9._-]*$/,
"Environment driver key must start with a lowercase alphanumeric and contain only lowercase letters, digits, dots, hyphens, or underscores",
@@ -0,0 +1,108 @@
import { randomUUID } from "node:crypto";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { eq } from "drizzle-orm";
import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, plugins } from "@paperclipai/db";
import { resolvePluginTaskResume } from "../services/environment-task-runtime.js";
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
import { environmentRuntimeService } from "../services/environment-runtime.js";
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
const databaseSuite = support.supported ? describe : describe.skip;
databaseSuite("environment task lease authority", () => {
let db: ReturnType<typeof createDb>;
let stopDb: (() => Promise<void>) | undefined;
beforeAll(async () => {
const started = await startEmbeddedPostgresTestDatabase("environment-task-authority");
stopDb = started.stop; db = createDb(started.connectionString);
});
afterAll(async () => { await stopDb?.(); });
it("authorizes a new task lease only after scoped prior execution has stopped", async () => {
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID();
const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Lease transition", issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", role: "engineer", adapterType: "paperclip_runner" });
await db.insert(environments).values({ id: environmentId, companyId, name: `Fixture ${environmentId}`, driver: "plugin", config: {} });
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "fixture-continuation", version: "1.0.0", status: "ready", manifestJson: {
capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }],
} as never });
await db.insert(heartbeatRuns).values([
{ id: oldRunId, companyId, agentId, invocationSource: "manual", status: "succeeded" },
{ id: nextRunId, companyId, agentId, invocationSource: "manual", status: "running" },
]);
const pin = { driver: "plugin", pluginId, driverKey: "tasks", environmentId, runnerTaskLease: true };
const prior = { id: oldLeaseId, companyId, environmentId, heartbeatRunId: oldRunId, provider: "plugin:fixture:tasks", providerLeaseId: "old-task" };
await db.insert(environmentLeases).values([
{ ...prior, status: "released", leasePolicy: "ephemeral", cleanupStatus: "success", releasedAt: new Date(),
metadata: { ...pin, runnerTaskSessionRetained: true, runnerTaskIdentity: { runnerId: "runner", leaseId: oldLeaseId, runId: oldRunId, sessionId: "session" }, remoteExecutionTermination: remoteTerminationReceipt(prior, { providerLeaseId: "old-task", state: "stopped" }) } },
{ id: nextLeaseId, companyId, environmentId, heartbeatRunId: nextRunId, provider: prior.provider, providerLeaseId: "new-task",
status: "active", leasePolicy: "ephemeral", metadata: pin },
]);
const priorIdentity = { runnerInstanceId: "runner", normalizedSessionId: "session", environmentLeaseId: oldLeaseId, runId: oldRunId };
const authorize = () => resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity });
await expect(authorize()).resolves.toBe("old-task");
for (const field of ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "runId"]) {
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId,
priorIdentity: { ...priorIdentity, [field]: randomUUID() } })).rejects.toThrow("not authorized");
}
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, oldRunId));
await expect(authorize()).rejects.toThrow("not authorized");
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, oldRunId));
await expect(resolvePluginTaskResume(db, { companyId: randomUUID(), leaseId: nextLeaseId, priorIdentity })).rejects.toThrow("not authorized");
await db.update(environmentLeases).set({ metadata: { ...pin, pluginId: randomUUID() } }).where(eq(environmentLeases.id, nextLeaseId));
await expect(authorize()).rejects.toThrow("not authorized");
await db.update(environmentLeases).set({ metadata: pin }).where(eq(environmentLeases.id, nextLeaseId));
await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).rejects.toThrow("not authorized");
await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId));
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }),
call: async (_id: string, _method: string, params: any) => ({ kind: "accepted", taskId: params.taskId }),
} as unknown as PluginWorkerManager;
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
const runner = { runnerId: "runner", leaseId: oldLeaseId, sessionId: "session", runId: nextRunId,
turnId: "turn", itemId: "item", harness: "codex", protocolMin: 1, protocolMax: 3 };
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "submit", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
} })).resolves.toMatchObject({ kind: "accepted" });
const [second] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, nextLeaseId));
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),
metadata: { ...second.metadata, runnerTaskSessionRetained: true,
remoteExecutionTermination: remoteTerminationReceipt(second, { providerLeaseId: "new-task", state: "stopped" }) },
}).where(eq(environmentLeases.id, nextLeaseId));
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, nextRunId));
const thirdRunId = randomUUID(), thirdLeaseId = randomUUID();
await db.insert(heartbeatRuns).values({ id: thirdRunId, companyId, agentId, invocationSource: "manual", status: "running" });
await db.insert(environmentLeases).values({ id: thirdLeaseId, companyId, environmentId, heartbeatRunId: thirdRunId,
status: "active", provider: prior.provider, providerLeaseId: "third-task", metadata: pin });
await expect(resolvePluginTaskResume(db, { companyId, leaseId: thirdLeaseId,
priorIdentity: { ...priorIdentity, runId: nextRunId } })).resolves.toBe("new-task");
const submit = { kind: "submit", projectIds: [], bootstrapTicket: "fixture", runner: { ...runner, runId: thirdRunId }, resumeFrom: "new-task" } as const;
await expect(runtime.task({ companyId, leaseId: thirdLeaseId, operation: { ...submit, projectIds: [] } })).resolves.toMatchObject({ kind: "accepted" });
await expect(runtime.task({ companyId, leaseId: thirdLeaseId, operation: { ...submit, projectIds: [], runner } })).rejects.toThrow("identity mismatch");
// Same-run recovery must choose the latest task in the saved lineage, not an earlier physical lease.
const [third] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, thirdLeaseId));
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),
metadata: { ...third.metadata, runnerTaskSessionRetained: true,
remoteExecutionTermination: remoteTerminationReceipt(third, { providerLeaseId: "third-task", state: "stopped" }) },
}).where(eq(environmentLeases.id, thirdLeaseId));
const retryLeaseId = randomUUID();
await db.insert(environmentLeases).values({ id: retryLeaseId, companyId, environmentId, heartbeatRunId: thirdRunId,
status: "active", provider: prior.provider, providerLeaseId: "retry-task", metadata: pin });
await expect(runtime.task({ companyId, leaseId: retryLeaseId, operation: { ...submit, projectIds: [], resumeFrom: "third-task" } })).resolves.toMatchObject({ kind: "accepted" });
const [retry] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, retryLeaseId));
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),
metadata: { ...retry.metadata, runnerTaskSessionRetained: true,
remoteExecutionTermination: remoteTerminationReceipt(retry, { providerLeaseId: "retry-task", state: "stopped" }) },
}).where(eq(environmentLeases.id, retryLeaseId));
const finalLeaseId = randomUUID();
await db.insert(environmentLeases).values({ id: finalLeaseId, companyId, environmentId, heartbeatRunId: thirdRunId,
status: "active", provider: prior.provider, providerLeaseId: "final-task", metadata: pin });
await expect(resolvePluginTaskResume(db, { companyId, leaseId: finalLeaseId,
priorIdentity: { ...priorIdentity, runId: thirdRunId } })).resolves.toBe("retry-task");
});
});
@@ -0,0 +1,199 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { PgDialect } from "drizzle-orm/pg-core";
import type { Db } from "@paperclipai/db";
import { environmentTaskOperationSchema, parseEnvironmentTaskResult } from "@paperclipai/plugin-sdk";
import { executeEnvironmentTask } from "../services/environment-task-runtime.js";
const state = vi.hoisted(() => ({ plugin: {} as any }));
vi.mock("../services/plugin-registry.js", () => ({ pluginRegistryService: () => ({ getById: vi.fn(async () => state.plugin) }) }));
const leaseId = "10000000-0000-4000-8000-000000000001";
const projectId = "10000000-0000-4000-8000-000000000002";
const secondProjectId = "10000000-0000-4000-8000-000000000003";
const row = () => ({
lease: { id: leaseId, companyId: "company", environmentId: "environment", providerLeaseId: "attempt-1", heartbeatRunId: "run", issueId: "issue", status: "active", expiresAt: null as Date | null,
metadata: { driver: "plugin", pluginId: "original", driverKey: "tasks" } },
environment: { id: "environment", config: { pluginKey: "test.provider", driverKey: "tasks", driverConfig: {} } } as { id: string; config: Record<string, unknown> } | null,
run: { id: "run", agentId: "agent", status: "running" } as { id: string; agentId: string; status: string } | null,
});
function database(value: ReturnType<typeof row> | null = row(), projectRows = [{ id: projectId }]) {
const results = [value ? [value] : [], [{ id: "issue" }]];
const query: any = { from: () => query, leftJoin: () => query, where: vi.fn(() => query), limit: () => Promise.resolve(results.shift()), then: (resolve: (rows: typeof projectRows) => unknown) => Promise.resolve(projectRows).then(resolve) };
const update: any = { set: () => update, where: () => update,
returning: async () => [{ id: leaseId }], then: (resolve: (value: unknown) => unknown) => Promise.resolve([]).then(resolve) };
return { db: { select: () => query, update: () => update } as unknown as Db, query };
}
function worker(result: unknown = { kind: "accepted", taskId: "attempt-1" }) {
return { getWorker: vi.fn(() => ({ supportedMethods: ["environmentTask"] })), call: vi.fn(async () => result) };
}
const submit = { kind: "submit" as const, projectIds: [projectId], runner: { protocolMin: 1, protocolMax: 2, harness: "codex", runnerId: "runner", leaseId, runId: "run", sessionId: "session", turnId: "turn", itemId: "item" }, bootstrapTicket: "transient-test-ticket" };
beforeEach(() => {
state.plugin = { id: "original", pluginKey: "test.provider", status: "ready", manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } };
});
describe("plugin-provided Runner execution", () => {
it("accepts artifact identity and excludes filesystem transfer from the contract", () => {
const operation = environmentTaskOperationSchema.parse({ kind: "describe" });
const description = { kind: "description", taskId: "attempt-1", runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } };
expect(parseEnvironmentTaskResult(operation, "attempt-1", description)).toEqual(description);
for (const field of ["resources", "environment", "connection", "codexCommand"]) {
expect(() => parseEnvironmentTaskResult(operation, "attempt-1", { ...description, [field]: {} })).toThrow();
}
for (const kind of ["checkpoint", "read_file", "check_project_auth"]) {
expect(environmentTaskOperationSchema.safeParse({ kind }).success).toBe(false);
}
});
it("allows only the private, unexpired setup identity on an ad-hoc lease", async () => {
const value = row();
Object.assign(value.lease, { heartbeatRunId: null, issueId: null }); value.run = null;
Object.assign(value.lease.metadata, { runnerTaskProbe: { runId: "probe-run", sessionId: "probe-session", expiresAtMs: Date.now() + 60_000 } });
const operation = { ...submit, projectIds: [], runner: { ...submit.runner, runId: "probe-run", sessionId: "probe-session" },
launch: { runnerVersion: "fixture", runnerDigest: `sha256:${"a".repeat(64)}`,
maxOutboxBytes: 262144, p0ReserveBytes: 65536, maxRuntimeMs: 0, maxLifetimeMs: 0,
lifecyclePolicy: { mode: "per_turn" as const, idleTimeoutMs: null },
} };
const workers = worker();
await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation })).resolves.toMatchObject({ kind: "accepted" });
expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({ runId: "probe-run", projectIds: [] }), 15_000);
const lifetime = (workers.call.mock.calls[0] as unknown as [string, string, { operation: typeof operation }])[2].operation.launch.maxRuntimeMs;
expect(lifetime).toBeGreaterThan(0); expect(lifetime).toBeLessThanOrEqual(60_000);
expect((workers.call.mock.calls[0] as unknown as [string, string, { operation: typeof operation }])[2].operation.launch).not.toHaveProperty("maxLifetimeMs");
for (const invalid of [{ ...operation, projectIds: [projectId] }, { ...operation, runner: { ...operation.runner, sessionId: "other" } }, submit]) {
await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: invalid })).rejects.toThrow("identity mismatch");
}
Object.assign(value.lease.metadata, { runnerTaskProbe: { runId: "probe-run", sessionId: "probe-session", expiresAtMs: Date.now() - 1 } });
await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation })).rejects.toThrow("not active");
await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: { kind: "stop" } })).resolves.toMatchObject({ kind: "accepted" });
});
it("dispatches with host-derived scope and a persisted attempt identity", async () => {
const { db, query } = database(); const workers = worker();
await expect(executeEnvironmentTask(db, workers as never, { companyId: "company", leaseId, operation: submit })).resolves.toEqual({ kind: "accepted", taskId: "attempt-1" });
const scope = new PgDialect().sqlToQuery(query.where.mock.calls[0][0]);
expect(scope.sql).toContain('"environment_leases"."company_id" =');
expect(scope.sql).toContain('"environment_leases"."id" =');
expect(scope.params).toEqual([leaseId, "company"]);
expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({
taskId: "attempt-1", companyId: "company", agentId: "agent", projectIds: [projectId], runId: "run", operation: submit,
}), 15_000);
});
it("validates every requested project in the task company before dispatch", async () => {
const operation = { ...submit, projectIds: [projectId, secondProjectId] };
const { db, query } = database(row(), [{ id: projectId }, { id: secondProjectId }]);
const workers = worker();
await executeEnvironmentTask(db, workers as never, { companyId: "company", leaseId, operation });
expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({ projectIds: operation.projectIds }), 15_000);
const scope = new PgDialect().sqlToQuery(query.where.mock.calls[2][0]);
expect(scope.sql).toContain('"projects"."company_id" =');
expect(scope.params).toEqual(["company", projectId, secondProjectId]);
const rejected = worker();
await expect(executeEnvironmentTask(database(row(), [{ id: projectId }]).db, rejected as never, { companyId: "company", leaseId, operation })).rejects.toThrow("project unavailable");
expect(rejected.call).not.toHaveBeenCalled();
});
it("validates unique project IDs and allows tasks with no project mounts", async () => {
for (const projectIds of [[projectId, projectId], ["invalid"], Array(65).fill(projectId)]) {
expect(environmentTaskOperationSchema.safeParse({ ...submit, projectIds }).success).toBe(false);
}
const workers = worker();
await executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: { ...submit, projectIds: [] } });
expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({ projectIds: [] }), 15_000);
});
it("rejects an absent or cross-company lease before calling a worker", async () => {
const workers = worker();
await expect(executeEnvironmentTask(database(null).db, workers as never, { companyId: "other", leaseId, operation: submit })).rejects.toThrow("lease unavailable");
expect(workers.call).not.toHaveBeenCalled();
});
it("rejects mismatched Runner binding and inactive submission", async () => {
const workers = worker();
await expect(executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: { ...submit, runner: { ...submit.runner, runId: "other" } } })).rejects.toThrow("identity mismatch");
const value = row(); value.lease.status = "released";
await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: submit })).rejects.toThrow("not active");
expect(workers.call).not.toHaveBeenCalled();
});
it("uses the pinned provider for cleanup after environment edits", async () => {
const value = row(); value.lease.status = "released"; value.environment!.config.pluginKey = "replacement";
const workers = worker();
await executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: { kind: "stop" } });
expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({ config: {}, operation: { kind: "stop" } }), 15_000);
});
it("keeps cleanup available after environment, run, and issue deletion", async () => {
const value = row(); value.environment = null; value.run = null;
for (const kind of ["stop", "complete", "status"] as const) {
const workers = worker(kind === "status" ? { kind: "status", taskId: "attempt-1", phase: "cancelled" } : undefined);
await executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: { kind } });
expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({
config: {}, environmentId: null, runId: null, agentId: null, projectIds: [],
}), 15_000);
}
await expect(executeEnvironmentTask(database(value).db, worker() as never, { companyId: "company", leaseId, operation: submit })).rejects.toThrow("not active");
});
it("preserves opaque provider IDs in requests and receipts", async () => {
const value = row(); value.lease.providerLeaseId = `provider:attempt.${"x".repeat(120)}`;
const workers = worker({ kind: "accepted", taskId: value.lease.providerLeaseId });
await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: { kind: "stop" } })).resolves.toMatchObject({ taskId: value.lease.providerLeaseId });
});
it("returns a validated connection only while the lease and run are active", async () => {
const connection = { kind: "connection", taskId: "attempt-1", endpoint: {
kind: "authenticated_websocket", websocketUrl: "wss://runner.example.test/connect", generation: "attempt-1",
secretHeaders: [{ name: "Authorization", value: "Bearer fixture" }],
} };
const input = { companyId: "company", leaseId, operation: { kind: "connection" as const } };
await expect(executeEnvironmentTask(database().db, worker(connection) as never, input)).resolves.toEqual(connection);
for (const state of ["expired", "finished", "deleted"] as const) {
const value = row();
if (state === "expired") value.lease.expiresAt = new Date(0);
if (state === "finished") value.run!.status = "succeeded";
if (state === "deleted") value.environment = null;
const workers = worker(connection);
await expect(executeEnvironmentTask(database(value).db, workers as never, input)).rejects.toThrow("not active");
expect(workers.call).not.toHaveBeenCalled();
}
for (const endpoint of [
{ ...connection.endpoint, websocketUrl: "ws://runner.example.test/connect" },
{ ...connection.endpoint, websocketUrl: "wss://runner.example.test/connect?token=fixture" },
{ ...connection.endpoint, secretHeaders: [{ name: "Authorization", value: "fixture\r\nInjected: true" }] },
{ ...connection.endpoint, secretHeaders: [{ name: "Invalid Header", value: "fixture" }] },
]) {
await expect(executeEnvironmentTask(database().db, worker({ ...connection, endpoint }) as never, input)).rejects.toThrow("reconcile the same task");
}
});
it.each(["manifest", "worker"])("requires live %s support", async kind => {
const workers = worker();
if (kind === "manifest") state.plugin.manifestJson.environmentDrivers[0].supportsTasks = false;
else workers.getWorker.mockReturnValue({ supportedMethods: [] });
await expect(executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: submit })).rejects.toThrow("provider unavailable");
expect(workers.call).not.toHaveBeenCalled();
});
it("rejects wrong task receipts and sanitizes worker errors", async () => {
await expect(executeEnvironmentTask(database().db, worker({ kind: "accepted", taskId: "other" }) as never, { companyId: "company", leaseId, operation: submit })).rejects.toThrow("reconcile the same task");
const workers = worker(); workers.call.mockRejectedValue(new Error("private-credential"));
await expect(executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: submit })).rejects.toThrow(/^Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying$/);
});
it("accepts PRP identity characters and length limits", () => {
for (const field of ["runnerId", "leaseId", "runId", "sessionId", "turnId", "itemId"]) {
for (const value of ["session:part.1", "a".repeat(160)]) {
expect(environmentTaskOperationSchema.safeParse({ ...submit, runner: { ...submit.runner, [field]: value } }).success).toBe(true);
}
for (const value of ["a".repeat(161), "../escape", "a/b", ""]) {
expect(environmentTaskOperationSchema.safeParse({ ...submit, runner: { ...submit.runner, [field]: value } }).success).toBe(false);
}
}
});
it("validates the client's inclusive PRP version range", () => {
for (const range of [{ protocolMin: 1, protocolMax: 1 }, { protocolMin: 1, protocolMax: 2 }, { protocolMin: 3, protocolMax: 5 }]) {
expect(environmentTaskOperationSchema.safeParse({ ...submit, runner: { ...submit.runner, ...range } }).success).toBe(true);
}
for (const range of [
{ protocolMin: 0, protocolMax: 2 }, { protocolMin: 2, protocolMax: 1 },
{ protocolMin: 1.5, protocolMax: 2 }, { protocolMin: 1, protocolMax: 2.5 },
{ protocolMin: 1, protocolMax: Number.MAX_SAFE_INTEGER + 1 },
{ protocolMin: undefined, protocolMax: 2 }, { protocolMin: 1, protocolMax: undefined },
]) {
expect(environmentTaskOperationSchema.safeParse({ ...submit, runner: { ...submit.runner, ...range } }).success).toBe(false);
}
});
it("validates operation and result shape without making acceptance mean readiness", () => {
expect(environmentTaskOperationSchema.safeParse({ ...submit, surprise: true }).success).toBe(false);
expect(environmentTaskOperationSchema.safeParse({ ...submit, runner: { ...submit.runner, unknownField: true } }).success).toBe(false);
expect(() => parseEnvironmentTaskResult({ kind: "status" }, "attempt-1", { kind: "accepted", taskId: "attempt-1" })).toThrow();
expect(parseEnvironmentTaskResult(submit, "attempt-1", { kind: "accepted", taskId: "attempt-1" }).kind).toBe("accepted");
});
});
@@ -8,6 +8,7 @@ import {
parseMessage,
serializeMessage,
} from "../../../packages/plugins/sdk/src/protocol.js";
import type { PluginEnvironmentTaskParams } from "../../../packages/plugins/sdk/src/environment-tasks.js";
import { definePlugin } from "../../../packages/plugins/sdk/src/define-plugin.js";
import { startWorkerRpcHost } from "../../../packages/plugins/sdk/src/worker-rpc-host.js";
import { pluginManifestV1Schema, type PaperclipPluginManifestV1 } from "@paperclipai/shared";
@@ -94,6 +95,54 @@ describe("plugin environment driver seam", () => {
});
});
it("negotiates and validates plugin-provided Runner execution RPCs", async () => {
const calls: unknown[] = [];
const plugin = definePlugin({
async setup() {},
async onEnvironmentTask(input) {
calls.push(input);
return { kind: "accepted", taskId: input.taskId };
},
});
const stdin = new PassThrough();
const stdout = new PassThrough();
const host = startWorkerRpcHost({ plugin, stdin, stdout });
const responses: unknown[] = [];
stdout.on("data", chunk => {
for (const line of String(chunk).split("\n").filter(Boolean)) responses.push(parseMessage(line));
});
try {
const manifest = { ...baseManifest, environmentDrivers: [{ ...baseManifest.environmentDrivers![0], supportsTasks: true }] };
expect(pluginManifestV1Schema.parse(manifest).environmentDrivers![0].supportsTasks).toBe(true);
stdin.write(serializeMessage(createRequest("initialize", {
manifest, config: {}, instanceInfo: { instanceId: "instance-1", hostVersion: "1.0.0" }, apiVersion: 1,
}, 1)));
await waitForResponses(responses, 1);
const initialized = responses[0];
expect(isJsonRpcSuccessResponse(initialized)).toBe(true);
if (!isJsonRpcSuccessResponse(initialized)) return;
expect(initialized.result.supportedMethods).toContain("environmentTask");
const projectIds = ["10000000-0000-4000-8000-000000000001", "10000000-0000-4000-8000-000000000002"];
const params: PluginEnvironmentTaskParams = { driverKey: "fake-plugin", companyId: "company", environmentId: "environment", config: {},
taskId: "attempt", runId: "run", agentId: "agent", projectIds, lease: { providerLeaseId: "attempt" },
operation: { kind: "submit", projectIds, bootstrapTicket: "transient-test-ticket", runner: {
protocolMin: 1, protocolMax: 2, harness: "codex", runnerId: "runner", leaseId: "lease", runId: "run", sessionId: "session:part.1", turnId: "turn", itemId: "item",
} } };
stdin.write(serializeMessage(createRequest("environmentTask", params, 2)));
await waitForResponses(responses, 2);
expect(responses[1]).toMatchObject({ result: { kind: "accepted", taskId: "attempt" } });
expect(calls[0]).toMatchObject({ projectIds, operation: { kind: "submit", projectIds } });
stdin.write(serializeMessage(createRequest("environmentTask", { ...params, operation: { kind: "status" } }, 3)));
await waitForResponses(responses, 3);
expect(isJsonRpcErrorResponse(responses[2])).toBe(true); // Wrong response kind.
stdin.write(serializeMessage(createRequest("environmentTask", { ...params, operation: { kind: "shell", command: "echo test" } }, 4)));
await waitForResponses(responses, 4);
expect(isJsonRpcErrorResponse(responses[3])).toBe(true);
expect(calls).toHaveLength(2); // Invalid operation did not reach the plugin.
} finally { host.stop(); }
});
it("dispatches environment driver worker hooks and reports support", async () => {
const plugin = definePlugin({
async setup() {},
@@ -0,0 +1,117 @@
import { randomUUID } from "node:crypto";
import { mkdtemp, realpath, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { and, eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, companyMemberships, createDb, environmentLeases, environments, heartbeatRuns, plugins } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { aiConnectionService } from "../services/ai-connections.js";
import { prepareManagedAiRuntime } from "../services/ai-connection-runtime.js";
import { registerPluginTaskCredentialHandoff } from "../services/plugin-task-credential-writeback.js";
import { executeEnvironmentTask, settleDispatchedEnvironmentTask } from "../services/environment-task-runtime.js";
import { secretService } from "../services/secrets.js";
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
vi.mock("../services/subscription-refresh.js", () => ({ refreshSubscriptionConnection: async () => {} }));
const support = await getEmbeddedPostgresTestSupport();
const suite = support.supported ? describe : describe.skip;
const auth = (token: string, year = 2020) => JSON.stringify({ tokens: { account_id: "fixture-account", access_token: "fixture-access", refresh_token: token }, last_refresh: `${year}-01-01T00:00:00Z` });
suite("durable plugin task credential handoff", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>, db: ReturnType<typeof createDb>, home: string;
beforeAll(async () => {
home = await realpath(await mkdtemp(join(tmpdir(), "plugin-task-secret-")));
vi.stubEnv("PAPERCLIP_HOME", home); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "plugin-task-fixture");
database = await startEmbeddedPostgresTestDatabase("plugin-task-credential-"); db = createDb(database.connectionString);
}, 90_000);
afterAll(async () => { await database?.cleanup(); vi.unstubAllEnvs(); if (home) await rm(home, { recursive: true, force: true }); });
async function fixture(dispatched = true) {
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), pluginId = randomUUID(), environmentId = randomUUID(), leaseId = randomUUID();
const owner = `fixture-${randomUUID()}`;
await db.insert(companies).values({ id: companyId, name: "Task credential fixture", issuePrefix: `C${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" });
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const service = aiConnectionService(db);
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: "Fixture Codex", loginSessionId: "fixture", agentIds: [], allAgents: true }, auth("fixture-original"));
const [selection] = await service.quotaAccounts(companyId, owner);
const managed = await prepareManagedAiRuntime(db, { companyId, agentId, responsibleUserId: owner, adapterType: "paperclip_runner",
binding: { mode: "responsible_user", provider: "openai", method: "subscription" }, config: { provider: "codex" } });
expect(managed.credentialWriteback).toBeDefined();
await db.insert(environments).values({ id: environmentId, name: `Fixture ${environmentId}`, driver: "plugin", status: "active", config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } });
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "fixture-tasks", version: "1.0.0", status: "ready", manifestJson: {
capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }],
} as never });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "manual", status: "running" });
await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, heartbeatRunId: runId, provider: `plugin:${pluginId}:tasks`, providerLeaseId: "fixture-task", status: "active",
metadata: { driver: "plugin", pluginId, driverKey: "tasks", runnerTaskLease: true, runnerTaskDispatched: dispatched, runnerTaskExecutionStopped: false } });
await registerPluginTaskCredentialHandoff(db, { companyId, leaseId }, managed.sessionIdentity, managed.credentialWriteback!);
// Recovery must not depend on the original controller or its private home.
await rm(managed.home!, { recursive: true, force: true });
const calls: string[] = [];
let failExport = false;
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }),
call: async (_plugin: string, _method: string, params: any) => {
calls.push(params.operation.kind);
expect(params.companyId).toBe(companyId); expect(params.taskId).toBe("fixture-task");
if (params.operation.kind === "status") return { kind: "status", taskId: "fixture-task", phase: "cancelled", executionStopped: true };
if (params.operation.kind === "credentials") {
expect(params.operation.accountIdentity).toBe(managed.sessionIdentity);
if (failExport) throw new Error("fixture export unavailable");
return { kind: "credentials", taskId: "fixture-task", credential: { type: "codex-auth-json", authJson: auth("fixture-refreshed", 2021) } };
}
return { kind: "accepted", taskId: "fixture-task" };
} } as unknown as PluginWorkerManager;
const lease = () => db.select().from(environmentLeases).where(and(eq(environmentLeases.id, leaseId), eq(environmentLeases.companyId, companyId))).then(rows => rows[0]!);
return { companyId, agentId, runId, environmentId, leaseId, owner, selection: selection!, managed, workers, calls, lease,
failExport(value: boolean) { failExport = value; }, credential: () => service.credential(selection!),
settle: () => settleDispatchedEnvironmentTask(db, workers, { companyId, id: leaseId }) };
}
it("persists refreshed credentials before completing a stopped task, after controller files disappear", async () => {
const f = await fixture();
expect(await f.settle()).toBe(true);
expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
expect(f.calls.indexOf("credentials")).toBeLessThan(f.calls.indexOf("complete"));
const saved = await f.lease();
expect(saved.metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "complete" });
expect(JSON.stringify(saved.metadata)).not.toContain("fixture-refreshed");
expect(JSON.stringify(saved.metadata)).not.toContain(f.managed.home!);
});
it("keeps a failed transfer pending and recovers through a fresh cleanup after run and environment deletion", async () => {
const f = await fixture(); f.failExport(true);
await expect(f.settle()).rejects.toThrow("termination is unconfirmed");
expect(f.calls).not.toContain("complete");
expect((await f.lease()).metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "pending" });
expect(await f.credential()).toBe(auth("fixture-original"));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
await db.delete(environments).where(eq(environments.id, f.environmentId));
f.failExport(false);
expect(await f.settle()).toBe(true);
expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
expect(f.calls.at(-1)).toBe("complete");
});
it("does not overwrite an explicit account replacement", async () => {
const f = await fixture();
await secretService(db).rotate(f.managed.credentialWriteback!.secretId, { value: auth("fixture-reconnected", 2022) }, { userId: f.owner });
expect(await f.settle()).toBe(true);
expect(f.calls).not.toContain("credentials");
expect(await f.credential()).toBe(auth("fixture-reconnected", 2022));
});
it("allows concurrent cleanup to complete the same durable handoff", async () => {
const f = await fixture();
expect(await Promise.all([f.settle(), f.settle()])).toEqual([true, true]);
expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
expect((await f.lease()).metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "complete" });
});
it("retires an undispatched task without exporting credentials and blocks a late submit", async () => {
const f = await fixture(false);
expect(await f.settle()).toBe(false);
expect(f.calls).toEqual([]);
expect((await f.lease()).metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "complete" });
await expect(executeEnvironmentTask(db, f.workers, { companyId: f.companyId, leaseId: f.leaseId, operation: {
kind: "submit", projectIds: [], bootstrapTicket: "fixture-ticket", runner: { harness: "codex", protocolMin: 1, protocolMax: 3,
runnerId: "runner", leaseId: f.leaseId, runId: f.runId, sessionId: "session", turnId: "turn", itemId: "item" },
} })).rejects.toThrow("not active");
expect(f.calls).toEqual([]);
});
});
@@ -1,3 +1,4 @@
import type { PluginTaskCredentialWritebackBinding } from "./plugin-task-credential-writeback.js";
import { OPENROUTER_ATTRIBUTION_HEADERS, openRouterClaudeHeaders } from "@paperclipai/shared/openrouter-attribution";
import { createHash } from "node:crypto";
import { HttpError, unprocessable } from "../errors.js";
@@ -385,6 +386,10 @@ export async function prepareManagedAiRuntime(
const sessionIdentity = `${selection.grant.id}:${input.responsibleUserId ?? "shared"}:${noAuth ? "no-auth" : `${credentialRef!.secretId}:${freshness!.epoch}`}`;
return {
sessionIdentity,
...(subscriptionFile && input.binding.provider === "openai" && credentialRef && freshness ? {
credentialWriteback: { grantId: selection.grant.id, secretId: credentialRef.secretId, sessionEpoch: freshness.epoch,
versionSelector: credentialRef.versionSelector ?? "latest" } satisfies PluginTaskCredentialWritebackBinding,
} : {}),
config: {
...input.config,
...projected?.config,
+64 -15
View File
@@ -1,3 +1,5 @@
import { executeEnvironmentTask, pluginEnvironmentSupportsTasks, settleDispatchedEnvironmentTask } from "./environment-task-runtime.js";
import type { PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourceProtected } from "./legacy-shutdown-workspace-settlement.js";
import { nativeWorkspaceExportPendingForLease } from "./native-runtime/native-workspace-finalization-state.js";
import { beginIdleTrackedWork } from "./task-admission.js";
@@ -3594,16 +3596,16 @@ function createPluginEnvironmentDriver(
return { plugin, driver };
}
async function resolvePluginDriverForRelease(input: EnvironmentDriverReleaseInput) {
async function resolvePluginDriverForRelease(input: { environment: Environment | null; lease: EnvironmentDriverReleaseInput["lease"] }) {
const metadata = input.lease.metadata ?? {};
const metadataPluginId = readString(metadata.pluginId);
const metadataPluginKey = readString(metadata.pluginKey);
const metadataDriverKey = readString(metadata.driverKey);
const currentConfig = tryParseCurrentPluginConfig(input.environment);
const currentConfig = input.environment ? tryParseCurrentPluginConfig(input.environment) : null;
if (!metadataPluginId && !metadataPluginKey && !metadataDriverKey) {
if (!currentConfig) {
throw new Error(`Expected plugin environment config for driver "${input.environment.driver}".`);
throw new Error(`Expected pinned plugin environment config.`);
}
const { plugin } = await resolvePluginDriver(currentConfig);
return {
@@ -3698,6 +3700,8 @@ function createPluginEnvironmentDriver(
metadata: {
...(input.agentId ? { agentId: input.agentId } : {}),
providerMetadata: providerLease.metadata ?? {},
environmentId: input.environment.id,
...(input.adapterType === "paperclip_runner" && driver.supportsTasks === true ? { runnerTaskLease: true } : {}),
driver: input.environment.driver,
executionWorkspaceMode: input.executionWorkspaceMode,
pluginId: plugin.id,
@@ -3708,6 +3712,9 @@ function createPluginEnvironmentDriver(
},
async releaseRunLease(input) {
await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"pluginLeaseCleanupMethod":"release"}'::jsonb` }).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.lease.companyId)));
const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
const { plugin, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
await workerManager.call(plugin.id, "environmentReleaseLease", {
driverKey,
@@ -3718,17 +3725,18 @@ function createPluginEnvironmentDriver(
providerLeaseId: input.lease.providerLeaseId,
leaseMetadata: input.lease.metadata ?? undefined,
});
return await environmentsSvc.releaseLease(input.lease.id, input.status, { cleanupStatus: "success" });
return await environmentsSvc.releaseLease(input.lease.id, input.status, {
cleanupStatus: "success",
...(taskStopped ? { remoteExecutionTermination: remoteTerminationReceipt(input.lease,
{ providerLeaseId: input.lease.providerLeaseId, state: "stopped" }) } : {}),
});
},
async resumeRunLease(input) {
if (!input.lease.providerLeaseId) {
throw new Error(`Plugin environment lease "${input.lease.id}" does not have a provider lease id to resume.`);
}
const { pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease({
...input,
status: "released",
});
const { pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
return await resumePluginEnvironmentLease({
db,
workerManager,
@@ -3746,10 +3754,10 @@ function createPluginEnvironmentDriver(
},
async destroyRunLease(input) {
const { pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease({
...input,
status: "failed",
});
await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"pluginLeaseCleanupMethod":"destroy"}'::jsonb` }).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.lease.companyId)));
const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
const { pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
await destroyPluginEnvironmentLease({
db,
workerManager,
@@ -3769,14 +3777,37 @@ function createPluginEnvironmentDriver(
cleanupStatus: "success",
...(input.lease.status === "pending_cleanup" && typeof input.lease.metadata?.pendingCleanupAttemptId === "string"
? { expectedPendingCleanupAttemptId: input.lease.metadata.pendingCleanupAttemptId } : {}),
...(taskStopped ? { remoteExecutionTermination: remoteTerminationReceipt(input.lease,
{ providerLeaseId: input.lease.providerLeaseId, state: "stopped" }) } : {}),
});
},
async isPendingCleanupWorkerReady(input) {
try {
const { plugin } = await resolvePluginDriverForRelease(input);
const methods = workerManager.getWorker(plugin.id)?.supportedMethods ?? [];
return methods.includes(input.lease.metadata?.pluginLeaseCleanupMethod === "destroy" ? "environmentDestroyLease" : "environmentReleaseLease") &&
(input.lease.metadata?.runnerTaskDispatched !== true || methods.includes("environmentTask"));
} catch { return false; }
},
async retryPendingSandboxTeardown(input) {
const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
const { plugin, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
const environmentId = input.environment?.id ?? readString(input.lease.metadata?.environmentId);
if (!environmentId) throw new Error("Plugin lease cleanup has no saved environment identity");
const receipt = await workerManager.call(plugin.id, input.lease.metadata?.pluginLeaseCleanupMethod === "destroy" ? "environmentDestroyLease" : "environmentReleaseLease", {
driverKey, companyId: input.lease.companyId, environmentId, issueId: input.lease.issueId,
config: driverConfig, providerLeaseId: input.lease.providerLeaseId, leaseMetadata: input.lease.metadata ?? undefined,
});
return taskStopped ? { providerLeaseId: input.lease.providerLeaseId, state: "stopped" } : receipt;
},
async realizeWorkspace(input) {
const { plugin, pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease({
environment: input.environment,
lease: input.lease,
status: "released",
});
return await realizePluginEnvironmentWorkspace({
db,
@@ -3807,7 +3838,6 @@ function createPluginEnvironmentDriver(
const { plugin, pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease({
environment: input.environment,
lease: input.lease,
status: "released",
});
return await executePluginEnvironmentCommand({
db,
@@ -4051,6 +4081,17 @@ export function environmentRuntimeService(
return resolveSandboxDuplexBridgeInput(experimental);
},
async supportsTasks(input: { lease: EnvironmentLease }) {
return options.pluginWorkerManager
? pluginEnvironmentSupportsTasks(db, options.pluginWorkerManager, input.lease) : false;
},
/** Dispatch plugin-provided Runner execution, including resource preparation and task lifecycle. */
async task(input: { companyId: string; leaseId: string; operation: PluginEnvironmentTaskOperation }) {
if (!options.pluginWorkerManager) throw new Error("Plugin-provided Runner execution worker manager unavailable");
return executeEnvironmentTask(db, options.pluginWorkerManager, input);
},
async acquireRunLease(input: {
companyId: string;
environment: Environment;
@@ -4198,7 +4239,15 @@ export function environmentRuntimeService(
}
continue;
}
if (!environment) continue;
if (!environment) {
if (leaseSnapshot.metadata?.driver === "plugin" && options.pluginWorkerManager) {
const receipt = await requireDriverKey("plugin").retryPendingSandboxTeardown!({ environment: null, lease: leaseSnapshot });
await environmentsSvc.releaseLease(leaseSnapshot.id, status, { cleanupStatus: "success",
remoteExecutionTermination: remoteTerminationReceipt(leaseSnapshot, receipt),
});
}
continue;
}
if (
providerResourceDisposition === "keep_running" &&
leaseSnapshot.leasePolicy === "reuse_by_environment"
@@ -0,0 +1,202 @@
import { settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
import { and, eq, inArray, or, sql } from "drizzle-orm";
import { environmentLeases, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
import { environmentTaskOperationSchema, parseEnvironmentTaskResult, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
import { pluginRegistryService } from "./plugin-registry.js";
import { hasRemoteTerminationReceipt, remoteExecutionHasStopped } from "./remote-execution-termination.js";
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
export async function pluginEnvironmentSupportsTasks(db: Db, workers: PluginWorkerManager, lease: { companyId: string; metadata: Record<string, unknown> | null }) {
const metadata = lease.metadata ?? {};
if (metadata.driver !== "plugin" || typeof metadata.pluginId !== "string" || typeof metadata.driverKey !== "string") return false;
const plugin = await pluginRegistryService(db).getById(metadata.pluginId);
const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === metadata.driverKey);
const methods = workers.getWorker(metadata.pluginId)?.supportedMethods ?? [];
return plugin?.status === "ready" && driver?.supportsTasks === true &&
plugin.manifestJson.capabilities.includes("environment.drivers.register") &&
methods.includes("environmentTask");
}
/** Server-only dispatch for plugin-provided Runner execution.
* The caller supplies an authorized company and a persisted lease,
* never a plugin identity, resource endpoint, or agent/project authority.
* A timeout is ambiguous: retry the same operation against the same lease.
*/
export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManager, input: {
companyId: string;
leaseId: string;
operation: PluginEnvironmentTaskOperation;
}) {
const operation = environmentTaskOperationSchema.parse(input.operation);
const [row] = await db.select({ lease: environmentLeases, environment: environments, run: heartbeatRuns })
.from(environmentLeases)
.leftJoin(environments, eq(environments.id, environmentLeases.environmentId))
.leftJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, environmentLeases.companyId)))
.where(and(eq(environmentLeases.id, input.leaseId), eq(environmentLeases.companyId, input.companyId))).limit(1);
if (!row || !row.lease.providerLeaseId) throw new Error("Plugin-provided Runner execution lease unavailable");
const { lease, environment, run } = row;
const taskId = row.lease.providerLeaseId;
const metadata = lease.metadata ?? {};
const probe = metadata.runnerTaskProbe as { runId?: unknown; sessionId?: unknown; expiresAtMs?: unknown } | undefined;
const activeProbe = lease.heartbeatRunId === null && lease.issueId === null && probe
&& typeof probe.runId === "string" && typeof probe.sessionId === "string"
&& typeof probe.expiresAtMs === "number" && probe.expiresAtMs > Date.now();
if ((operation.kind === "describe" || operation.kind === "submit" || operation.kind === "connection") &&
(metadata.runnerTaskRetiring === true || !environment || (!activeProbe && run?.status !== "running") || lease.status !== "active" || (lease.expiresAt && lease.expiresAt.getTime() <= Date.now()))) throw new Error("Plugin-provided Runner execution lease is not active");
if (operation.kind === "submit" && (operation.runner.runId !== (activeProbe ? probe.runId : run?.id) || (!operation.resumeFrom && operation.runner.leaseId !== lease.id)
|| (activeProbe && (operation.runner.sessionId !== probe.sessionId || operation.projectIds.length > 0)))) {
throw new Error("Plugin-provided Runner execution identity mismatch");
}
if (operation.kind === "submit" && operation.resumeFrom) {
const sources = await db.select({ runId: environmentLeases.heartbeatRunId }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.provider, lease.provider!),
eq(environmentLeases.providerLeaseId, operation.resumeFrom),
)).limit(2);
const priorRunId = sources.length === 1 ? sources[0]!.runId : null;
if (activeProbe || !priorRunId || operation.resumeFrom !== await resolvePluginTaskResume(db, {
companyId: input.companyId, leaseId: lease.id,
priorIdentity: { runnerInstanceId: operation.runner.runnerId, environmentLeaseId: operation.runner.leaseId,
runId: priorRunId, normalizedSessionId: operation.runner.sessionId },
})) throw new Error("Plugin Runner resume identity mismatch");
}
if (metadata.driver !== "plugin" || typeof metadata.pluginId !== "string" || typeof metadata.driverKey !== "string") {
throw new Error("Plugin-provided Runner execution lease has no pinned plugin driver");
}
const plugin = await pluginRegistryService(db).getById(metadata.pluginId);
const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === metadata.driverKey);
if (!plugin || plugin.status !== "ready" || !driver?.supportsTasks ||
!plugin.manifestJson.capabilities.includes("environment.drivers.register") ||
!workers.getWorker(plugin.id)?.supportedMethods.includes("environmentTask")) {
throw new Error("Plugin-provided Runner execution provider unavailable");
}
const issue = operation.kind === "submit" && lease.issueId ? await db.select({ id: issues.id }).from(issues)
.where(and(eq(issues.id, lease.issueId), eq(issues.companyId, input.companyId))).limit(1).then(rows => rows[0]) : null;
if (operation.kind === "submit" && lease.issueId && !issue) throw new Error("Plugin-provided Runner execution issue unavailable");
const projectIds = operation.kind === "submit" ? operation.projectIds : [];
if (projectIds.length > 0) {
const authorizedProjects = await db.select({ id: projects.id }).from(projects)
.where(and(eq(projects.companyId, input.companyId), inArray(projects.id, projectIds)));
if (authorizedProjects.length !== projectIds.length) throw new Error("Plugin-provided Runner execution project unavailable");
}
// Provider identity comes from the lease. Editing the environment must never
// redirect status or cleanup to a replacement plugin.
const config = (environment?.config ?? {}) as Record<string, unknown>;
const driverConfig = config.pluginKey === plugin.pluginKey && config.driverKey === metadata.driverKey
? (config.driverConfig as Record<string, unknown> | undefined) ?? {} : {};
if (operation.kind === "submit" && activeProbe) {
const remainingMs = Number(probe.expiresAtMs) - Date.now();
if (remainingMs <= 0 || !operation.launch) throw new Error("Plugin-provided Runner probe lease is not active");
// Runner itself must stop if Core disappears during a setup probe.
const requestedLimits = [operation.launch.maxRuntimeMs, operation.launch.maxLifetimeMs]
.filter((value): value is number => typeof value === "number" && value > 0);
operation.launch.maxRuntimeMs = Math.min(remainingMs, ...requestedLimits);
delete operation.launch.maxLifetimeMs;
}
if (operation.kind === "submit") {
// No ticket, headers, environment values, or launch paths belong in metadata.
const marked = await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ runnerTaskLease: true, runnerTaskDispatched: true, runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false, runnerTaskIdentity: operation.runner, runnerTaskResumeFrom: operation.resumeFrom ?? null })}::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`)).returning({ id: environmentLeases.id });
if (marked.length !== 1) throw new Error("Plugin-provided Runner execution lease is not active");
}
try {
if (operation.kind === "complete") {
await settlePluginTaskCredentialHandoff(db, { companyId: input.companyId, leaseId: lease.id }, metadata, async accountIdentity => {
const snapshot = await executeEnvironmentTask(db, workers, { ...input, operation: { kind: "credentials", accountIdentity } });
if (snapshot.kind !== "credentials") throw new Error("Plugin Runner credential snapshot unavailable");
return snapshot.credential;
});
}
const result = await workers.call(plugin.id, "environmentTask", {
driverKey: metadata.driverKey, companyId: input.companyId, environmentId: environment?.id ?? null,
issueId: lease.issueId, config: driverConfig,
taskId, runId: run?.id ?? (activeProbe ? String(probe.runId) : null), agentId: run?.agentId ?? null, projectIds,
lease: { providerLeaseId: lease.providerLeaseId, metadata: lease.metadata ?? undefined }, operation,
}, 15_000);
const parsed = parseEnvironmentTaskResult(operation, taskId, result);
if (parsed.kind === "status" && parsed.executionStopped === true) {
await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ runnerTaskExecutionStopped: true, runnerTaskSessionRetained: parsed.sessionStateAvailable === true })}::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId)));
}
if (operation.kind === "complete") {
await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"runnerTaskDispatched":false}'::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
sql`${environmentLeases.metadata}->>'runnerTaskExecutionStopped' = 'true'`));
}
return parsed;
} catch {
// Worker errors and schema diagnostics can contain request credentials.
throw new Error("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying");
}
}
/** Failure leaves the dispatch marker and provider handle available to cleanup retries. */
export async function settleDispatchedEnvironmentTask(db: Db, workers: PluginWorkerManager, lease: { id: string; companyId: string }) {
const [current] = await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"runnerTaskRetiring":true}'::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId)))
.returning({ metadata: environmentLeases.metadata });
if (current?.metadata?.runnerTaskDispatched !== true) {
if (current?.metadata) await settlePluginTaskCredentialHandoff(db, { companyId: lease.companyId, leaseId: lease.id }, current.metadata,
async () => { throw new Error("Undispatched task must not request credentials"); });
return current?.metadata?.runnerTaskExecutionStopped === true;
}
const scope = { companyId: lease.companyId, leaseId: lease.id };
try {
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "stop" } });
const state = await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "status" } });
if (state.kind !== "status" || state.executionStopped !== true) throw new Error("Plugin Runner termination is unconfirmed");
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "complete" } });
return true;
} catch {
await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed",
failureReason: "plugin_runner_task_cleanup_unconfirmed", releasedAt: new Date(), updatedAt: new Date(),
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId)));
throw new Error("Plugin Runner termination is unconfirmed; the saved lease requires cleanup");
}
}
/** A saved provider task can resume only within the same authorized agent workspace. */
export async function resolvePluginTaskResume(db: Db, input: {
companyId: string; leaseId: string;
priorIdentity: { runnerInstanceId: string; environmentLeaseId: string; runId: string; normalizedSessionId: string };
}): Promise<string> {
const rows = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, input.companyId)))
.where(and(eq(environmentLeases.companyId, input.companyId),
or(eq(environmentLeases.id, input.leaseId), eq(heartbeatRuns.id, input.priorIdentity.runId))));
const next = rows.find(row => row.lease.id === input.leaseId);
// A retained PRP lease is logical; later provider tasks have different physical lease IDs.
const candidates = rows.filter(row => {
const identity = row.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
return row.lease.id !== input.leaseId && row.run.id === input.priorIdentity.runId && row.lease.metadata?.runnerTaskSessionRetained === true &&
identity?.runnerId === input.priorIdentity.runnerInstanceId && identity.leaseId === input.priorIdentity.environmentLeaseId &&
identity.sessionId === input.priorIdentity.normalizedSessionId && identity.runId === input.priorIdentity.runId &&
!rows.some(successor => successor.lease.id !== input.leaseId &&
successor.lease.metadata?.runnerTaskResumeFrom === row.lease.providerLeaseId);
});
const old = candidates.length === 1 ? candidates[0] : undefined;
const identity = old?.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
if (!old || !next || !old.lease.providerLeaseId || !identity ||
identity.runnerId !== input.priorIdentity.runnerInstanceId || identity.leaseId !== input.priorIdentity.environmentLeaseId ||
identity.sessionId !== input.priorIdentity.normalizedSessionId || identity.runId !== input.priorIdentity.runId ||
old.lease.metadata?.driver !== "plugin" || next.lease.metadata?.driver !== "plugin" ||
!next.lease.providerLeaseId || next.lease.providerLeaseId === old.lease.providerLeaseId ||
(old.run.id !== next.run.id && !["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes(old.run.status)) ||
old.run.agentId !== next.run.agentId || old.lease.issueId !== next.lease.issueId ||
old.lease.executionWorkspaceId !== next.lease.executionWorkspaceId ||
!old.lease.environmentId || old.lease.environmentId !== next.lease.environmentId ||
!next.lease.provider?.startsWith("plugin:") || old.lease.provider !== next.lease.provider ||
old.lease.metadata?.pluginId !== next.lease.metadata?.pluginId || old.lease.metadata?.driverKey !== next.lease.metadata?.driverKey ||
next.lease.status !== "active" || next.lease.releasedAt || next.run.status !== "running" ||
(next.lease.expiresAt && next.lease.expiresAt.getTime() <= Date.now()) ||
!(old.run.id === next.run.id
? rows.filter(row => row.lease.id !== next.lease.id).every(row => hasRemoteTerminationReceipt(row.lease))
: await remoteExecutionHasStopped(db, input.companyId, old.run.id))) {
throw new Error("Plugin Runner retained session is not authorized");
}
return old.lease.providerLeaseId;
}
@@ -0,0 +1,143 @@
import { aiConnectionMetadataSchema } from "@paperclipai/shared";
import { and, eq, sql } from "drizzle-orm";
import { companySecrets, connectionGrants, environmentLeases, toolConnections, type Db } from "@paperclipai/db";
import { z, type PluginEnvironmentTaskCredential } from "@paperclipai/plugin-sdk";
import { mkdtemp, writeFile, rm, realpath } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { isDeepStrictEqual } from "node:util";
import { decideCodexAuthMerge, withAccountHomeSecretMutationLock } from "@paperclipai/adapter-codex-local/server";
import { WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE } from "@paperclipai/adapter-utils/workspace-restore-merge";
import { aiConnectionService } from "./ai-connections.js";
import { secretService } from "./secrets.js";
const bindingSchema = z.object({
grantId: z.string().uuid(),
secretId: z.string().uuid(),
sessionEpoch: z.number().int().nonnegative(),
versionSelector: z.union([z.literal("latest"), z.number().int().positive()]),
}).strict();
export type PluginTaskCredentialWritebackBinding = z.infer<typeof bindingSchema>;
const handoffSchema = z.object({
accountIdentity: z.string().min(1),
binding: bindingSchema,
state: z.enum(["pending", "complete"]),
}).strict();
const key = "runnerTaskCredentialHandoff";
type Scope = { companyId: string; leaseId: string };
/** Save authority, never credential bytes or a controller-local temporary path. */
export async function registerPluginTaskCredentialHandoff(
db: Db,
scope: Scope,
accountIdentity: string,
binding: PluginTaskCredentialWritebackBinding,
) {
const handoff = handoffSchema.parse({ accountIdentity, binding, state: "pending" });
await db.transaction(async tx => {
const [lease] = await tx.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
)).for("update");
if (!lease || lease.status !== "active" || lease.metadata?.runnerTaskLease !== true ||
lease.metadata.runnerTaskRetiring === true) {
throw new Error("Plugin task credential lease unavailable");
}
const saved = lease.metadata[key];
if (saved) {
const prior = handoffSchema.parse(saved);
if (prior.accountIdentity !== accountIdentity || !isDeepStrictEqual(prior.binding, handoff.binding)) {
throw new Error("Plugin task credential authority changed");
}
return;
}
await tx.update(environmentLeases).set({ metadata: { ...lease.metadata, [key]: handoff } })
.where(eq(environmentLeases.id, lease.id));
});
}
/** Runs for live completion and the cleanup sweep, including after Core restarts. */
export async function settlePluginTaskCredentialHandoff(
db: Db,
scope: Scope,
metadata: Record<string, unknown>,
fetchCredential: (accountIdentity: string) => Promise<PluginEnvironmentTaskCredential | null>,
) {
if (!metadata[key]) return;
const handoff = handoffSchema.parse(metadata[key]);
if (handoff.state === "complete") return;
if (metadata.runnerTaskDispatched === true && metadata.runnerTaskExecutionStopped !== true) {
throw new Error("Plugin task credential process termination is unconfirmed");
}
const binding = handoff.binding;
const home = await mkdtemp(join(await realpath(tmpdir()), "paperclip-task-credential-"));
try {
// Revocation or an explicit account replacement intentionally discards an old refresh.
const account = async (database: Db) => {
const [grant] = await database.select().from(connectionGrants).where(and(
eq(connectionGrants.companyId, scope.companyId), eq(connectionGrants.id, binding.grantId),
)).for("update");
if (!grant || grant.status !== "active") return null;
const ref = grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
if (!ref || ref.secretId !== binding.secretId ||
(ref.versionSelector ?? "latest") !== binding.versionSelector) return null;
const [secret] = await database.select().from(companySecrets).where(and(
eq(companySecrets.companyId, scope.companyId), eq(companySecrets.id, binding.secretId),
)).for("update");
if (!secret || secret.aiSessionEpoch !== binding.sessionEpoch) return null;
const [connection] = await database.select().from(toolConnections).where(and(
eq(toolConnections.companyId, scope.companyId), eq(toolConnections.id, grant.connectionId),
));
const ai = aiConnectionMetadataSchema.safeParse(connection?.config.ai);
if (!connection || !ai.success || ai.data.provider !== "openai" || ai.data.method !== "subscription") return null;
return { connection, grant };
};
const mutate = <T>(operation: (database: Db) => Promise<T>) => withAccountHomeSecretMutationLock(
undefined, scope.companyId, () => db.transaction(tx => operation(tx as unknown as Db)),
);
const current = metadata.runnerTaskDispatched === true && await mutate(account);
if (current) {
// Do not hold database or account locks while the provider exports its credential.
const credential = await fetchCredential(handoff.accountIdentity);
if (credential !== null) {
if (credential.type !== "codex-auth-json") throw new Error("Plugin task credential provider mismatch");
const incoming = join(home, "incoming.json"), destination = join(home, "current.json");
await writeFile(incoming, credential.authJson, { mode: 0o600 });
const writeBack = () => mutate(async database => {
const selected = await account(database);
if (!selected) return;
const value = await aiConnectionService(database).credential(selected);
await writeFile(destination, value, { mode: 0o600 });
if (await decideCodexAuthMerge(incoming, destination, { errorLabel: "AI account refresh" }) !== 10) return;
await secretService(database).rotate(binding.secretId, {
value: credential.authJson, preserveAiSessionEpoch: true,
}, { userId: selected.grant.subjectUserId });
});
for (let attempt = 0; ; attempt++) {
try {
await writeBack();
break;
} catch (error) {
if (attempt >= 2 || (error as NodeJS.ErrnoException)?.code !== WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE) throw error;
}
}
}
}
const completed = await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ [key]: { ...handoff, state: "complete" } })}::jsonb`,
}).where(and(
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
sql`${environmentLeases.metadata}->${key} = ${JSON.stringify(handoff)}::jsonb`,
)).returning({ id: environmentLeases.id });
if (completed.length !== 1) {
const [current] = await db.select({ metadata: environmentLeases.metadata }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
));
if (!isDeepStrictEqual(handoffSchema.safeParse(current?.metadata?.[key]).data, { ...handoff, state: "complete" })) {
throw new Error("Plugin task credential handoff changed");
}
}
} finally {
// The provider retains the recoverable credential until completion succeeds.
await rm(home, { recursive: true, force: true });
}
}