mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment plugins can run a native Runner as a plugin task, and a warm session keeps that Runner between turns > - On each warm turn Core sends the plugin a `prepare` operation with the full launch requirements for the new run > - The requirements' `requirementsDigest` covers the run identity, and the run identity changes on every turn > - So a plugin cannot tell that only the run moved, and it must redo its whole preparation every warm turn > - This pull request gives the plugin two digests: one that stays the same while its preparation stays valid, and one that names the run > - The benefit is that a plugin can transfer only the run identity when nothing it reconciles changed ## Linked Issues or Issue Description **What existing behavior does this improve?** The `prepare` operation that Core sends to an environment plugin for a plugin Runner task, on cold starts and on warm attaches. **Subsystem affected** Native runtime, plugin Runner tasks, the Runner launch contract, and the plugin SDK environment task contract. **Current behavior** A warm attach sends the full requirements. The only digest available, `requirementsDigest`, includes `runId`, `turnId` and `itemId`, so it changes on every turn even when the launch is otherwise the same. **Proposed behavior** Core computes two digests for every `prepare`: - `preparationDigest`: the launch requirements an environment reconciles when it prepares a Runner: - the runner, lease and session identity; - provider, model and settings; - the prompt and instruction text; - the runtime context's instruction and skill bundles, working copy and connection instructions. It leaves out the `runId`, `turnId` and `itemId`, and the runtime context's MCP assignment and binding (with `aggregateDigest`, which hashes the assignment). MCP reaches the Runner through Core's tool authority on every run attachment, never through files the environment publishes. Core already continues a warm session across an MCP-only change (`nativeRuntimeContextsMatch` with tool refresh). The binding is also named per run (`native-mcp:<runId>`). So the digest stays the same across warm turns of one Runner, lease and session, including when the agent's MCP access changes. - `runDigest`: `preparationDigest` plus everything it leaves out: `runId`, `turnId`, `itemId`, and the MCP assignment and binding. It is unique to one run's launch. A plugin that advertises `launchDigestVersion: 1` in its task description gets both fields on `prepare`. **Reason and benefit** A plugin can compare `preparationDigest` with its saved preparation and transfer only the run identity when they match, instead of a full preparation. **Breaking changes** None. Plugins without `launchDigestVersion` get no new fields. ## What Changed - `packages/paperclip-runner/src/live/runner-provider-launch.ts`: add `runnerLaunchPreparationDigest` and `runnerLaunchRunDigest`. The preparation digest drops the MCP assignment and binding; the run digest takes the full requirements and includes them. Both use the same canonical JSON as `runnerLaunchRequirementsDigest`, with their own schema tags so the three digests never collide. Export them from the package and the server's vendored runner shim. - `packages/plugins/sdk/src/environment-tasks.ts`: add optional `preparationDigest` and `runDigest` to the `prepare` operation, and optional `launchDigestVersion: 1` to the task description. The capability gates both digests. - `server/src/services/native-runtime/plugin-runner-task.ts` and `native-session-executor.ts`: send both digests on `prepare` when the description advertises `launchDigestVersion: 1`. - `server/src/services/environment-task-runtime.ts`: recompute both digests from the requirements, and refuse a `prepare` whose digests are missing in part or do not match. - `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the SDK public source pin. - Tests: - the preparation digest is stable across runs and changes when the runner, lease, session, instructions, model, approval policy or settings change; - with a runtime context, an MCP-only change (the run's binding, no binding, a changed access set) keeps the preparation digest and changes the run digest; - a change to the instruction bundle, instruction entry, a skill bundle, the skill set, the working copy or the connection instructions changes the preparation digest; - the run digest changes per run; - Core sends the digests only when advertised; - the runtime refuses forged or partial digests before it calls the plugin. ## Verification - `npx vitest run src/live/runner-launch-preparation.test.ts` in `packages/paperclip-runner`: 26 passed. - `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts src/__tests__/environment-task-runtime.test.ts` in `server`: 160 passed. - `npx vitest run source-sdk-public-inputs.test.mjs source-sdk-evidence.test.ts source-sdk-public-type.test.ts` in `tests/runner-e2e`: 59 passed. - `pnpm typecheck` in `server` and in `packages/paperclip-runner`: clean. - Not run locally: the full suite. CI covers it. ## Risks Low risk. The digests are new optional fields behind a plugin capability version. Core computes them and checks them again before it calls the plugin, so a plugin can rely on them. They prove only that the launch requirements are unchanged. A plugin that skips work on a match must still check its own state, such as files it published into the sandbox. If the runtime context carries values that change per run, the preparation digest changes too, so a plugin does its full preparation. That is safe, but the plugin saves nothing. The MCP binding is the one such value Core sets per run today, and MCP is excluded entirely. A plugin that does publish MCP configuration into its sandbox must not rely on this digest for it. Values outside the requirements are not covered at all: the account identity and credentials, the GitHub broker, project IDs, and the plugin's own runtime revision. A plugin must keep checking those itself. A match lets a plugin skip publishing its launch configuration, not computing the descriptor: `prepare` must still return a launch whose `requirementsDigest` matches the current requirements, which carry the current MCP assignment and binding. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
655 lines
52 KiB
TypeScript
655 lines
52 KiB
TypeScript
import { pluginTaskResumeRefused, resolvePluginTaskResume } from "./plugin-task-resume.js";
|
|
export { resolvePluginTaskResume } from "./plugin-task-resume.js";
|
|
import { attachSandboxCreation, environmentService } from "./environments.js";
|
|
import { createOwnedSandbox, queueSandboxCreationCleanup, recordSandboxCreationResource } from "./sandbox-creation-operations.js";
|
|
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
|
import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js";
|
|
import { readWorkspaceProviderBinding } from "@paperclipai/shared";
|
|
import { canReadStoppedPluginTaskCredential, settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
|
|
import { and, eq, inArray, sql } from "drizzle-orm";
|
|
import { environmentLeases, executionWorkspaces, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
|
|
import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult,
|
|
environmentTaskErrorData, readEnvironmentTaskErrorDiagnostic, JsonRpcCallError, PLUGIN_RPC_ERROR_CODES,
|
|
environmentTaskErrorCodeData, PluginEnvironmentTaskError, readEnvironmentTaskErrorCode,
|
|
type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation, type PluginEnvironmentTaskResult } from "@paperclipai/plugin-sdk";
|
|
import { pluginRegistryService } from "./plugin-registry.js";
|
|
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
|
|
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
|
|
import { NativeExecutionContext } from "./native-runtime/native-execution-context.js";
|
|
import { assertPluginTaskSourceAttachment, assertPluginTaskSourceRecovery } from "./native-runtime/plugin-task-source-recovery.js";
|
|
import { pluginAgentRuntimeIdentity } from "./plugin-agent-runtime-state.js";
|
|
import { runnerLaunchPreparationDigest, runnerLaunchRunDigest } from "../vendor/paperclip-runner/index.js";
|
|
import { assertPluginTaskOwnershipDispatch, PluginTaskOwnershipPendingError, readyPortableTaskSource, reconcilePluginTaskAdmission } from "./plugin-task-ownership.js";
|
|
|
|
export type PluginTaskDescription = Extract<PluginEnvironmentTaskResult, { kind: "description" }>;
|
|
|
|
export function isSettledReadyRunnerState(state: Record<string, unknown>): boolean {
|
|
const commands = state.processedCommands;
|
|
return state.schema === "paperclip.runner.durable.state.v1" && state.lifecycle === "ready" &&
|
|
Array.isArray(state.outbox) && state.outbox.length === 0 && Number.isSafeInteger(state.nextSourceSeq) &&
|
|
Number(state.nextSourceSeq) > 0 && state.ackedSourceSeq === Number(state.nextSourceSeq) - 1 &&
|
|
state.pendingTerminalDelivery == null && state.pendingProviderCleanup == null && state.warmTransition == null &&
|
|
commands !== null && typeof commands === "object" && !Array.isArray(commands) &&
|
|
Object.values(commands).every(command => {
|
|
const status = command !== null && typeof command === "object" ? (command as { status?: unknown }).status : undefined;
|
|
return typeof status === "string" && status !== "" && status !== "pending" && status !== "indeterminate";
|
|
});
|
|
}
|
|
|
|
export async function pluginEnvironmentSupportsTasks(db: Db, workers: PluginWorkerManager, lease: { companyId: string; metadata: Record<string, unknown> | null }) {
|
|
const metadata = lease.metadata ?? {};
|
|
if (metadata.driver !== "plugin" || typeof metadata.pluginId !== "string" || typeof metadata.driverKey !== "string") return false;
|
|
const plugin = await pluginRegistryService(db).getById(metadata.pluginId);
|
|
const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === metadata.driverKey);
|
|
const methods = workers.getWorker(metadata.pluginId)?.supportedMethods ?? [];
|
|
return plugin?.status === "ready" && driver?.supportsTasks === true &&
|
|
plugin.manifestJson.capabilities.includes("environment.drivers.register") &&
|
|
methods.includes("environmentTask");
|
|
}
|
|
|
|
/** Server-only dispatch for plugin-provided Runner execution.
|
|
* The caller supplies an authorized company and a persisted lease,
|
|
* never a plugin identity, resource endpoint, or agent/project authority.
|
|
* A timeout is ambiguous: retry the same operation against the same lease.
|
|
*/
|
|
export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerManager, "getWorker" | "call">, input: {
|
|
companyId: string;
|
|
leaseId: string;
|
|
operation: PluginEnvironmentTaskOperation;
|
|
executionContext?: unknown;
|
|
ownershipSignal?: AbortSignal;
|
|
/** Physical task captured by an existing owner, independent of the mutable lease. */
|
|
expectedTaskId?: string;
|
|
}): Promise<PluginEnvironmentTaskResult> {
|
|
const operation = environmentTaskOperationSchema.parse(input.operation);
|
|
if (operation.kind === "prepare" && (operation.preparationDigest !== undefined || operation.runDigest !== undefined)) {
|
|
const preparationDigest = runnerLaunchPreparationDigest(operation.requirements as Parameters<typeof runnerLaunchPreparationDigest>[0]);
|
|
if (operation.preparationDigest !== preparationDigest || operation.runDigest !== runnerLaunchRunDigest(preparationDigest, operation.requirements as Parameters<typeof runnerLaunchRunDigest>[1])) {
|
|
throw new Error("Plugin Runner launch digest mismatch");
|
|
}
|
|
}
|
|
const [row] = await db.select({ lease: environmentLeases, environment: environments, run: heartbeatRuns })
|
|
.from(environmentLeases)
|
|
.leftJoin(environments, eq(environments.id, environmentLeases.environmentId))
|
|
.leftJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, environmentLeases.companyId)))
|
|
.where(and(eq(environmentLeases.id, input.leaseId), eq(environmentLeases.companyId, input.companyId))).limit(1);
|
|
if (!row || !row.lease.providerLeaseId) throw new Error("Plugin-provided Runner execution lease unavailable");
|
|
const { lease, environment, run } = row;
|
|
const context = input.executionContext;
|
|
if (context !== undefined) {
|
|
if (!(context instanceof NativeExecutionContext) || !run) {
|
|
throw new Error("native_execution_context_unavailable");
|
|
}
|
|
context.assertScope({ companyId: input.companyId, agentId: run.agentId, runId: run.id });
|
|
}
|
|
const taskId = row.lease.providerLeaseId;
|
|
if (input.expectedTaskId !== undefined && input.expectedTaskId !== taskId) throw new Error("Plugin Runner physical task changed");
|
|
const metadata = lease.metadata ?? {};
|
|
const ownership = operation.kind === "ensure_ownership";
|
|
if (operation.kind === "describe" && metadata.portableTaskOwnershipReady !== true && metadata.portableTaskSuccessor === undefined &&
|
|
(metadata.portableTaskPredecessor !== undefined || typeof metadata.portableTaskPreviousLeaseId === "string")) {
|
|
await reconcilePluginTaskAdmission(db, lease, request =>
|
|
executeEnvironmentTask(db, workers, { ...input, operation: request }), context?.signal ?? input.ownershipSignal);
|
|
return executeEnvironmentTask(db, workers, input);
|
|
}
|
|
const requiresOwnership = !["ensure_ownership", "status", "stop", "complete"].includes(operation.kind);
|
|
const stoppedCredentialHandoff = operation.kind === "credentials" && input.expectedTaskId === taskId &&
|
|
canReadStoppedPluginTaskCredential(metadata, operation.accountIdentity);
|
|
if (requiresOwnership && (metadata.portableTaskSuccessor !== undefined && !stoppedCredentialHandoff ||
|
|
(metadata.portableTaskPredecessor !== undefined || metadata.portableTaskPreviousLeaseId !== undefined) && metadata.portableTaskOwnershipReady !== true)) {
|
|
throw new Error("Plugin Runner session ownership unavailable");
|
|
}
|
|
if (ownership) {
|
|
if (!environment || lease.status !== "active" || lease.releasedAt || run?.status !== "running" ||
|
|
metadata.runnerTaskRetiring === true || lease.expiresAt && lease.expiresAt.getTime() <= Date.now()) {
|
|
throw new Error("Plugin-provided Runner execution lease is not active");
|
|
}
|
|
assertPluginTaskOwnershipDispatch(lease, operation);
|
|
}
|
|
// A different launch identity is authorized by the saved consumer claim, never by the mismatch alone.
|
|
const sourceRecovery = run && ((operation.kind === "prepare" && operation.requirements.identity.runId !== run.id && metadata.runnerTaskSourceRecovery) ||
|
|
(operation.kind === "submit" && operation.runner.runId !== run.id && metadata.runnerTaskSourceRecovery))
|
|
? await assertPluginTaskSourceRecovery(db, { companyId: input.companyId, leaseId: lease.id,
|
|
context: context instanceof NativeExecutionContext ? context : undefined, operation }) : undefined;
|
|
const probe = metadata.runnerTaskProbe as { runId?: unknown; sessionId?: unknown; expiresAtMs?: unknown } | undefined;
|
|
const activeProbe = lease.heartbeatRunId === null && lease.issueId === null && probe
|
|
&& typeof probe.runId === "string" && typeof probe.sessionId === "string"
|
|
&& typeof probe.expiresAtMs === "number" && probe.expiresAtMs > Date.now();
|
|
const checkpointIdentity = operation.kind === "read_state" || operation.kind === "read_file" ? operation.identity
|
|
: operation.kind === "archive_state" ? operation.priorIdentity : null;
|
|
const runner = checkpointIdentity ? { runnerId: checkpointIdentity.runnerInstanceId, leaseId: checkpointIdentity.environmentLeaseId,
|
|
runId: checkpointIdentity.runId, sessionId: checkpointIdentity.normalizedSessionId } : operation.kind === "submit" ? operation.runner : operation.kind === "prepare" ? {
|
|
runnerId: operation.requirements.identity.runnerInstanceId,
|
|
leaseId: operation.requirements.identity.environmentLeaseId,
|
|
runId: operation.requirements.identity.runId, sessionId: operation.requirements.identity.normalizedSessionId,
|
|
} : null;
|
|
const launches = operation.kind === "submit" || operation.kind === "prepare";
|
|
const scopesProjects = launches || operation.kind === "check_project_auth" || operation.kind === "read_file";
|
|
const retainedTask = operation.kind === "release" || operation.kind === "resume";
|
|
const resumeFrom = launches || checkpointIdentity ? (operation as { resumeFrom?: string }).resumeFrom : undefined;
|
|
const savedIdentity = metadata.runnerTaskIdentity as Record<string, unknown> | undefined;
|
|
const prior = operation.kind === "prepare" ? operation.priorIdentity : undefined;
|
|
const priorSaved = metadata.runnerTaskPriorIdentity as Record<string, unknown> | undefined;
|
|
const firstPortableResume = operation.kind === "resume" && !savedIdentity &&
|
|
metadata.portableTaskPredecessor !== undefined;
|
|
if (firstPortableResume && (run?.status !== "running" ||
|
|
!await readyPortableTaskSource(db, input.companyId, lease.id, operation.identity))) {
|
|
throw new Error("Plugin Runner session ownership unavailable");
|
|
}
|
|
const matches = (identity: typeof prior, saved: Record<string, unknown> | undefined) => !!identity && !!saved &&
|
|
identity.runnerInstanceId === saved.runnerId && identity.environmentLeaseId === saved.leaseId &&
|
|
identity.normalizedSessionId === saved.sessionId && identity.runId === saved.runId &&
|
|
identity.turnId === saved.turnId && identity.itemId === saved.itemId;
|
|
const portableCheckpoint = operation.kind === "read_state" && !resumeFrom && !savedIdentity &&
|
|
metadata.portableTaskPredecessor !== undefined
|
|
? (await readyPortableTaskSource(db, input.companyId, lease.id, operation.identity))?.taskId : undefined;
|
|
const ownCheckpoint = checkpointIdentity && !resumeFrom &&
|
|
(portableCheckpoint || matches(checkpointIdentity,savedIdentity) || operation.kind !== "read_file" && matches(checkpointIdentity,priorSaved));
|
|
if (checkpointIdentity && !resumeFrom && !ownCheckpoint) throw new Error("Plugin Runner state identity mismatch");
|
|
const retainedCheckpointRead = operation.kind === "read_state" && ownCheckpoint &&
|
|
lease.leasePolicy === "reuse_by_environment" && ["retained", "expired"].includes(lease.status) &&
|
|
run?.status === "succeeded" && metadata.runnerTaskDispatched === true &&
|
|
(metadata.runnerTaskLifecyclePolicy as {mode?:unknown}|undefined)?.mode === "warm";
|
|
const retainedConnection = operation.kind === "connection" && environment !== null &&
|
|
lease.leasePolicy === "reuse_by_environment" && lease.status === "retained" &&
|
|
run?.status === "succeeded" && savedIdentity?.runId === run.id &&
|
|
metadata.runnerTaskDispatched === true && metadata.runnerTaskExecutionStopped !== true &&
|
|
metadata.runnerTaskRetiring !== true && (!lease.expiresAt || lease.expiresAt.getTime() > Date.now()) &&
|
|
(metadata.runnerTaskLifecyclePolicy as { mode?: unknown } | undefined)?.mode === "warm";
|
|
if ((operation.kind === "read_file" || operation.kind === "describe" || operation.kind === "prepare" || operation.kind === "submit" || (operation.kind === "connection" && !retainedConnection) || (operation.kind === "read_state" && metadata.runnerTaskExecutionStopped !== true && !retainedCheckpointRead) || operation.kind === "archive_state" || operation.kind === "check_project_auth" || operation.kind === "copilot_metadata") &&
|
|
(metadata.runnerTaskRetiring === true || !environment || (!activeProbe && run?.status !== "running") || lease.status !== "active" || (lease.expiresAt && lease.expiresAt.getTime() <= Date.now()))) throw new Error("Plugin-provided Runner execution lease is not active");
|
|
if (operation.kind === "copilot_metadata" && !activeProbe) throw new Error("Copilot metadata requires a private setup lease");
|
|
const sourceReference = metadata.runnerTaskSourceRecovery as { identity?: { runId?: unknown } } | undefined;
|
|
const sourceAttachment = operation.kind === "prepare" && prior && sourceReference?.identity?.runId === prior.runId
|
|
? await assertPluginTaskSourceAttachment(db, { companyId: input.companyId, leaseId: lease.id,
|
|
context: context instanceof NativeExecutionContext ? context : undefined, operation }) : undefined;
|
|
const portableAttachment = prior && !sourceAttachment && metadata.runnerTaskReused !== true &&
|
|
metadata.portableTaskPredecessor !== undefined
|
|
? (await readyPortableTaskSource(db, input.companyId, lease.id, prior))?.taskId : undefined;
|
|
const warmAttachment = !!prior && (metadata.runnerTaskReused === true || !!sourceAttachment || !!portableAttachment) &&
|
|
(portableAttachment && !savedIdentity || matches(prior, savedIdentity) ||
|
|
matches(prior, priorSaved) && operation.kind === "prepare" && matches(operation.requirements.identity, savedIdentity)) &&
|
|
runner?.runnerId === prior.runnerInstanceId &&
|
|
runner?.leaseId === prior.environmentLeaseId && runner?.sessionId === prior.normalizedSessionId;
|
|
if (prior && !warmAttachment) throw new Error("Plugin warm task prior identity mismatch");
|
|
const portableAuthority = firstPortableResume || portableAttachment
|
|
? sql`${environmentLeases.metadata}->'portableTaskSuccessor' is null
|
|
and ${environmentLeases.metadata}->'portableTaskOwnershipReady' = 'true'::jsonb
|
|
and coalesce(${environmentLeases.metadata}->>'portableTaskCheckpointUnavailable', 'false') != 'true'
|
|
and ${environmentLeases.metadata}->'portableTaskPredecessor' = ${JSON.stringify(metadata.portableTaskPredecessor)}::jsonb`
|
|
: undefined;
|
|
if (runner && !sourceRecovery && ((!checkpointIdentity || !resumeFrom) && !(ownCheckpoint && operation.kind !== "read_file") && runner.runId !== (activeProbe ? probe.runId : run?.id) || (!resumeFrom && runner.leaseId !== lease.id && !ownCheckpoint && !warmAttachment)
|
|
|| (activeProbe && (runner.sessionId !== probe.sessionId || (launches && operation.projectIds.length > 0))))) {
|
|
throw new Error("Plugin-provided Runner execution identity mismatch");
|
|
}
|
|
let stoppedSourceCheckpoint = false;
|
|
if (runner && resumeFrom && !sourceRecovery) {
|
|
const inspectCheckpoint = operation.kind === "read_state";
|
|
const sources = await db.select({ id: environmentLeases.id, runId: environmentLeases.heartbeatRunId, metadata: environmentLeases.metadata }).from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.provider, lease.provider!),
|
|
eq(environmentLeases.providerLeaseId, resumeFrom),
|
|
inspectCheckpoint ? undefined : sql`${environmentLeases.metadata}->>'runnerTaskSessionRetained' = 'true'`,
|
|
checkpointIdentity ? eq(environmentLeases.heartbeatRunId, checkpointIdentity.runId) : undefined,
|
|
)).limit(2);
|
|
const priorRunId = sources.length === 1 ? sources[0]!.runId : null;
|
|
const sourceIdentity = sources.length === 1 ? sources[0]!.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined : undefined;
|
|
if (activeProbe || !priorRunId || (checkpointIdentity && (checkpointIdentity.runId !== priorRunId
|
|
|| checkpointIdentity.turnId !== sourceIdentity?.turnId || checkpointIdentity.itemId !== sourceIdentity?.itemId)) || resumeFrom !== await resolvePluginTaskResume(db, {
|
|
companyId: input.companyId, leaseId: lease.id,
|
|
priorIdentity: { runnerInstanceId: runner.runnerId, environmentLeaseId: runner.leaseId,
|
|
runId: priorRunId, normalizedSessionId: runner.sessionId },
|
|
inspectCheckpoint,
|
|
})) throw new Error("Plugin Runner resume identity mismatch");
|
|
// Older providers misclassified suspended checkpoints. A fresh provider
|
|
// attestation may repair availability, but never supplies stop/owner authority.
|
|
if (inspectCheckpoint && sources[0]!.metadata?.runnerTaskSessionRetained !== true) {
|
|
const status = await executeEnvironmentTask(db, workers, { companyId: input.companyId,
|
|
leaseId: sources[0]!.id, operation: { kind: "status" } });
|
|
if (status.kind !== "status" || status.executionStopped !== true || status.sessionStateAvailable !== true) {
|
|
throw pluginTaskResumeRefused("checkpoint_status_unconfirmed");
|
|
}
|
|
if (resumeFrom !== await resolvePluginTaskResume(db, { companyId: input.companyId, leaseId: lease.id,
|
|
priorIdentity: { runnerInstanceId: runner.runnerId, environmentLeaseId: runner.leaseId,
|
|
runId: priorRunId, normalizedSessionId: runner.sessionId } })) {
|
|
throw pluginTaskResumeRefused("resume_task_mismatch");
|
|
}
|
|
}
|
|
stoppedSourceCheckpoint = true;
|
|
}
|
|
if (metadata.driver !== "plugin" || typeof metadata.pluginId !== "string" || typeof metadata.driverKey !== "string") {
|
|
throw new Error("Plugin-provided Runner execution lease has no pinned plugin driver");
|
|
}
|
|
const plugin = await pluginRegistryService(db).getById(metadata.pluginId);
|
|
const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === metadata.driverKey);
|
|
if (!plugin || plugin.status !== "ready" || !driver?.supportsTasks ||
|
|
!plugin.manifestJson.capabilities.includes("environment.drivers.register")) {
|
|
throw new Error("Plugin-provided Runner execution provider unavailable");
|
|
}
|
|
const worker = workers.getWorker(plugin.id);
|
|
if (ownership && !worker && driver.sessionHandoffVersion === 1) throw new PluginTaskOwnershipPendingError();
|
|
if (!worker?.supportedMethods.includes("environmentTask")) {
|
|
throw new Error("Plugin-provided Runner execution provider unavailable");
|
|
}
|
|
if (operation.kind === "resume" && operation.handoffCredential && driver.sessionHandoffVersion !== 1) {
|
|
throw new Error("Plugin Runner credential handoff unavailable");
|
|
}
|
|
if (ownership && driver.sessionHandoffVersion !== 1) throw new Error("Plugin Runner ownership reconciliation unavailable");
|
|
const callWorker = async <M extends HostToWorkerMethodName>(method: M, params: HostToWorkerMethods[M][0], timeout?: number): Promise<HostToWorkerMethods[M][1]> => {
|
|
if (!context) {
|
|
const cleanup = metadata.runnerTaskStartupCleanup as { contextId?: unknown; requiresPreparationStop?: unknown } | undefined;
|
|
if (["stop", "status", "complete", "credentials"].includes(operation.kind) && typeof cleanup?.contextId === "string" && typeof cleanup.requiresPreparationStop === "boolean") {
|
|
return workers.call(plugin.id, method, params, timeout, undefined, undefined, cleanup.contextId);
|
|
}
|
|
return timeout === undefined ? workers.call(plugin.id, method, params) : workers.call(plugin.id, method, params, timeout);
|
|
}
|
|
const settled = context.trackWorker();
|
|
const result = await workers.call(plugin.id, method, params, Math.max(1, Math.min(timeout ?? 15_000, context.deadlineAtMs - Date.now())), undefined, settled, context.id);
|
|
context.assertOpen();
|
|
return result;
|
|
};
|
|
const issue = scopesProjects && lease.issueId ? await db.select({ id: issues.id }).from(issues)
|
|
.where(and(eq(issues.id, lease.issueId), eq(issues.companyId, input.companyId))).limit(1).then(rows => rows[0]) : null;
|
|
if (scopesProjects && lease.issueId && !issue) throw new Error("Plugin-provided Runner execution issue unavailable");
|
|
const workspaceBinding = readWorkspaceProviderBinding(lease.metadata);
|
|
if (metadata.workspaceProvider !== undefined && !workspaceBinding) throw new Error("Invalid managed workspace binding");
|
|
if (workspaceBinding) {
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.id, workspaceBinding.executionWorkspaceId), eq(executionWorkspaces.companyId, input.companyId)));
|
|
const saved = readWorkspaceProviderBinding(workspace?.metadata);
|
|
if (!workspace || workspace.id !== lease.executionWorkspaceId || workspace.providerType !== "plugin" || workspace.providerRef !== workspaceBinding.workspaceRef ||
|
|
(scopesProjects && !["active", "idle", "in_review"].includes(workspace.status)) ||
|
|
workspaceBinding.pluginId !== plugin.id || !saved || JSON.stringify(saved) !== JSON.stringify(workspaceBinding)) {
|
|
throw new Error("Managed workspace execution binding changed");
|
|
}
|
|
}
|
|
let freshWorkspace = metadata.freshWorkspace === undefined ? undefined : freshWorkspaceSchema.parse(metadata.freshWorkspace);
|
|
if (freshWorkspace && driver.freshWorkspaceVersion !== 1 && (launches || operation.kind === "describe" || operation.kind === "check_project_auth"))
|
|
throw new Error("Plugin-provided fresh workspace preparation unavailable");
|
|
if (freshWorkspace && operation.kind === "prepare") {
|
|
if (!lease.issueId) throw new Error("Fresh workspace task ownership is missing");
|
|
freshWorkspace = await stageProviderFreshWorkspace({ db, companyId: input.companyId,
|
|
issueId: lease.issueId, ...freshWorkspace, existing: freshWorkspace });
|
|
}
|
|
const savedProjectIds = metadata.runnerTaskProjectIds ?? [];
|
|
if (retainedTask && (!Array.isArray(savedProjectIds) || savedProjectIds.some(id => typeof id !== "string")))
|
|
throw new Error("Plugin Runner project scope changed");
|
|
const projectIds: string[] = scopesProjects ? operation.projectIds : retainedTask ? [...savedProjectIds as string[]] : [];
|
|
if (scopesProjects && Array.isArray(metadata.runnerTaskProjectIds) &&
|
|
JSON.stringify(projectIds) !== JSON.stringify(metadata.runnerTaskProjectIds)) throw new Error("Plugin Runner project scope changed");
|
|
if (activeProbe && projectIds.length > 0) throw new Error("Plugin Runner probe cannot use project mounts");
|
|
if (projectIds.length > 0) {
|
|
const authorizedProjects = await db.select({ id: projects.id }).from(projects)
|
|
.where(and(eq(projects.companyId, input.companyId), inArray(projects.id, projectIds)));
|
|
if (authorizedProjects.length !== projectIds.length) throw new Error("Plugin-provided Runner execution project unavailable");
|
|
}
|
|
// Provider identity comes from the lease. Editing the environment must never
|
|
// redirect status or cleanup to a replacement plugin.
|
|
const config = (environment?.config ?? {}) as Record<string, unknown>;
|
|
const driverConfig = config.pluginKey === plugin.pluginKey && config.driverKey === metadata.driverKey
|
|
? (config.driverConfig as Record<string, unknown> | undefined) ?? {} : {};
|
|
let requiredProjectAuthProof: Record<string, unknown> | undefined;
|
|
if (operation.kind === "read_file") {
|
|
const proof = metadata.runnerTaskProjectAuthValidation as { projectIds?: string[] } | undefined;
|
|
if (metadata.runnerTaskExecutionStopped === true || !Array.isArray(proof?.projectIds) ||
|
|
JSON.stringify([...proof.projectIds].sort()) !== JSON.stringify([...operation.projectIds].sort()))
|
|
throw new Error("Plugin Runner file scope unavailable");
|
|
}
|
|
if (operation.kind === "submit") {
|
|
const proof = metadata.runnerTaskProjectAuthValidation as { configurationId?: unknown; provider?: unknown; projectIds?: string[] } | undefined;
|
|
const provider = operation.runner.harness === "codex" ? "openai" : operation.runner.harness === "claude" ? "anthropic" : operation.runner.harness;
|
|
if (!provider || !operation.preparedLaunchId || proof?.configurationId !== operation.preparedLaunchId || proof.provider !== provider
|
|
|| (!Array.isArray(proof.projectIds) ? null : JSON.stringify([...proof.projectIds].sort())) !== JSON.stringify([...operation.projectIds].sort())
|
|
|| (operation.credential && (operation.credential.type === "codex-auth-json" ? "openai" : "anthropic") !== provider)) {
|
|
throw new Error("Plugin Runner managed project authentication validation unavailable");
|
|
}
|
|
requiredProjectAuthProof = proof as Record<string, unknown>;
|
|
}
|
|
if (operation.kind === "check_project_auth" || operation.kind === "prepare") {
|
|
await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskProjectAuthValidation'` })
|
|
.where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId)));
|
|
}
|
|
if (operation.kind === "submit" && activeProbe) {
|
|
const remainingMs = Number(probe.expiresAtMs) - Date.now();
|
|
if (remainingMs <= 0 || !operation.launch) throw new Error("Plugin-provided Runner probe lease is not active");
|
|
// Runner itself must stop if Core disappears during a setup probe.
|
|
const requestedLimits = [operation.launch.maxRuntimeMs, operation.launch.maxLifetimeMs]
|
|
.filter((value): value is number => typeof value === "number" && value > 0);
|
|
operation.launch.maxRuntimeMs = Math.min(remainingMs, ...requestedLimits);
|
|
delete operation.launch.maxLifetimeMs;
|
|
}
|
|
if (operation.kind === "submit" || operation.kind === "copilot_metadata" || firstPortableResume) {
|
|
// No ticket, headers, environment values, or launch paths belong in metadata.
|
|
// New execution invalidates this lease's earlier stop receipt with its stop flag.
|
|
const dispatched = {
|
|
runnerTaskLease: true, runnerTaskDispatched: true,
|
|
runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false,
|
|
...(operation.kind === "submit" ? {
|
|
runnerTaskIdentity: operation.runner,
|
|
runnerTaskResumeFrom: operation.resumeFrom ?? null,
|
|
runnerTaskAccountIdentity: operation.accountIdentity,
|
|
} : {}),
|
|
};
|
|
const marked = await db.update(environmentLeases).set({
|
|
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection' - 'remoteExecutionTermination') || ${JSON.stringify(dispatched)}::jsonb`,
|
|
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
|
|
requiredProjectAuthProof ? sql`${environmentLeases.metadata}->'runnerTaskProjectAuthValidation' = ${JSON.stringify(requiredProjectAuthProof)}::jsonb` : undefined,
|
|
firstPortableResume ? portableAuthority : undefined,
|
|
)).returning({ id: environmentLeases.id });
|
|
if (marked.length !== 1) throw new Error("Plugin-provided Runner execution lease is not active");
|
|
}
|
|
if (warmAttachment && operation.kind === "prepare") {
|
|
const i = operation.requirements.identity;
|
|
const startup = context?.pluginSupported ? { contextId: context.id, deadlineAtMs: context.deadlineAtMs, runId: i.runId } : undefined;
|
|
const marked = await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({
|
|
runnerTaskPriorIdentity: { runnerId: prior!.runnerInstanceId, leaseId: prior!.environmentLeaseId, runId: prior!.runId,
|
|
sessionId: prior!.normalizedSessionId, turnId: prior!.turnId, itemId: prior!.itemId },
|
|
runnerTaskIdentity: { runnerId: i.runnerInstanceId, leaseId: i.environmentLeaseId, runId: i.runId,
|
|
sessionId: i.normalizedSessionId, turnId: i.turnId, itemId: i.itemId },
|
|
...(startup ? { runnerTaskStartup: startup } : {}),
|
|
})}::jsonb` }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.status, "active"),
|
|
sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
|
|
portableAttachment ? portableAuthority : undefined,
|
|
savedIdentity ? sql`${environmentLeases.metadata}->'runnerTaskIdentity' = ${JSON.stringify(savedIdentity)}::jsonb`
|
|
: sql`${environmentLeases.metadata}->'runnerTaskIdentity' is null`)).returning({id:environmentLeases.id});
|
|
if (marked.length !== 1) throw new Error("Plugin warm task attachment lost authority");
|
|
}
|
|
async function recordTaskStop(result: Extract<ReturnType<typeof parseEnvironmentTaskResult>, {kind:"status" | "copilot_metadata"}>) {
|
|
await db.update(environmentLeases).set({
|
|
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ runnerTaskExecutionStopped: true,
|
|
runnerTaskSessionRetained: result.kind === "status" && result.sessionStateAvailable === true,
|
|
...(result.kind === "status" && result.preparationStopped === true ? { runnerTaskPreparationStopped: true } : {}) })}::jsonb`,
|
|
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.providerLeaseId, taskId)));
|
|
}
|
|
try {
|
|
const admissionRejection = metadata.runnerTaskAdmissionRejection as { taskId?: unknown } | undefined;
|
|
const stoppedWithoutSession = !!run && metadata.runnerTaskExecutionStopped === true &&
|
|
metadata.runnerTaskSessionRetained !== true && admissionRejection?.taskId !== taskId;
|
|
if (operation.kind === "describe" && (metadata.runnerTaskReused === true || stoppedWithoutSession)) {
|
|
const state = parseEnvironmentTaskResult({kind:"status"}, taskId, await callWorker("environmentTask", {
|
|
driverKey: metadata.driverKey, companyId: input.companyId, environmentId: environment?.id ?? null,
|
|
issueId: lease.issueId, config: driverConfig, taskId, runId: run?.id ?? null, agentId: run?.agentId ?? null,
|
|
projectIds, lease: {providerLeaseId: taskId, metadata}, operation:{kind:"status"},
|
|
}));
|
|
if (state.kind === "status" && state.executionStopped === true &&
|
|
(metadata.runnerTaskReused === true || state.sessionStateAvailable !== true)) {
|
|
await recordTaskStop(state);
|
|
await executeEnvironmentTask(db, workers, {...input, expectedTaskId:taskId, operation:{kind:"complete"}});
|
|
// Preserve the predecessor's stop proof for checkpoint authorization.
|
|
if (typeof metadata.runnerTaskPreviousLeaseId === "string") await db.update(environmentLeases).set({
|
|
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({runnerTaskExecutionStopped:true,
|
|
runnerTaskSessionRetained:state.sessionStateAvailable === true})}::jsonb`,
|
|
}).where(and(eq(environmentLeases.id, metadata.runnerTaskPreviousLeaseId), eq(environmentLeases.companyId, input.companyId)));
|
|
const replacementDriverKey = metadata.driverKey;
|
|
const creationWorker = context && driver.creationOperationVersion === 1
|
|
? workers.getWorker(plugin.id) : undefined;
|
|
const recoverableCreation = creationWorker?.supportedMethods.includes("environmentInspectLease") === true &&
|
|
metadata.providerMetadata && typeof metadata.providerMetadata === "object" && !Array.isArray(metadata.providerMetadata);
|
|
let abandoned = false;
|
|
const fresh = await createOwnedSandbox(db, {
|
|
companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId,
|
|
issueId: lease.issueId, executionWorkspaceId: lease.executionWorkspaceId, provider: lease.provider!,
|
|
pluginId: plugin.id, driverKey: metadata.driverKey, config: {}, requestKey: taskId,
|
|
metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: plugin.pluginKey, driverKey: metadata.driverKey,
|
|
runnerTaskLease: true, agentId: run!.agentId,
|
|
...(context ? { runnerTaskCreationContextId: context.id } : {}),
|
|
...(recoverableCreation ? { creationOperationVersion: 1, providerMetadata: metadata.providerMetadata } : {}) },
|
|
}, async identity => {
|
|
const params = { driverKey: replacementDriverKey, companyId: input.companyId, environmentId: environment!.id,
|
|
executionWorkspaceId: lease.executionWorkspaceId ?? undefined,
|
|
issueId: lease.issueId, config: driverConfig, runId: identity.runId, creationOperationId: identity.id,
|
|
agentId: run!.agentId, adapterType: "paperclip_runner",
|
|
...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? { requestedExpiresAt: metadata.runnerTaskRequestedExpiresAt } : {}) };
|
|
if (!context) return workers.call(plugin.id, "environmentAcquireLease", params, 30_000);
|
|
const settled = context.trackWorker();
|
|
let observedResource: Parameters<typeof queueSandboxCreationCleanup>[1] | undefined;
|
|
let dispatched = false;
|
|
try {
|
|
return await context.wait(() => {
|
|
dispatched = true;
|
|
const observe = async (result: HostToWorkerMethods["environmentAcquireLease"][1]) => {
|
|
if (!result.providerLeaseId || result.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
|
|
const resource = { id: identity.id, companyId: input.companyId, environmentId: environment!.id,
|
|
providerLeaseId: result.providerLeaseId, providerMetadata: result.metadata };
|
|
await recordSandboxCreationResource(db, resource);
|
|
observedResource = resource;
|
|
if (abandoned || !context.active || context.signal.aborted || Date.now() >= context.deadlineAtMs) {
|
|
await queueSandboxCreationCleanup(db, resource);
|
|
}
|
|
};
|
|
return recoverableCreation ? creationWorker!.call("environmentAcquireLease", params, 30_000, undefined, settled, context.id, observe)
|
|
: workers.call(plugin.id, "environmentAcquireLease", params, 30_000, undefined, settled, context.id, observe);
|
|
});
|
|
} catch (error) {
|
|
if (!dispatched) settled();
|
|
abandoned = true;
|
|
if (observedResource) await queueSandboxCreationCleanup(db, observedResource);
|
|
throw error;
|
|
}
|
|
});
|
|
if (!fresh.providerLeaseId || fresh.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
|
|
const {runnerTaskIdentity:_identity,runnerTaskPriorIdentity:_prior,runnerTaskCredentialHandoff:_handoff,
|
|
runnerTaskResumeFrom:_resume,runnerTaskPreviousLeaseId:_previous,remoteExecutionTermination:_stopped,...rest}=metadata;
|
|
const replacementMetadata={...rest,providerMetadata:fresh.metadata ?? {},runnerTaskReused:false,runnerTaskDispatched:false,
|
|
runnerTaskExecutionStopped:false,runnerTaskSessionRetained:false};
|
|
try {
|
|
context?.assertOpen();
|
|
if (!context) await recordSandboxCreationResource(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
|
|
environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId });
|
|
const expiresAt=fresh.expiresAt ? new Date(fresh.expiresAt) : null;
|
|
const deadline=typeof metadata.runnerTaskRequestedExpiresAt === "string" ? Date.parse(metadata.runnerTaskRequestedExpiresAt) : null;
|
|
if ((expiresAt && (!Number.isFinite(expiresAt.getTime()) || expiresAt.getTime() <= Date.now())) ||
|
|
(deadline !== null && (!Number.isFinite(deadline) || !expiresAt || expiresAt.getTime() > deadline)))
|
|
throw new Error("Plugin replacement task expiry is not bounded");
|
|
await attachSandboxCreation(db, String(fresh.metadata!.creationOperationId), { companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId }, async tx => {
|
|
context?.assertOpen();
|
|
const [updated] = await tx.update(environmentLeases).set({providerLeaseId:fresh.providerLeaseId,expiresAt,
|
|
metadata:{...replacementMetadata,creationOperationId:fresh.metadata!.creationOperationId},
|
|
}).where(and(eq(environmentLeases.id,lease.id),eq(environmentLeases.providerLeaseId,taskId),
|
|
eq(environmentLeases.status,"active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`)).returning({id:environmentLeases.id});
|
|
if (!updated) throw new Error("Plugin task replacement ownership changed");
|
|
const attached = (await environmentService(tx).getLeaseById(updated.id))!;
|
|
context?.assertOpen();
|
|
return attached;
|
|
});
|
|
} catch (error) {
|
|
if (context) await queueSandboxCreationCleanup(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
|
|
environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId, providerMetadata: fresh.metadata });
|
|
throw error;
|
|
}
|
|
return executeEnvironmentTask(db, workers, input);
|
|
}
|
|
}
|
|
if (operation.kind === "complete") {
|
|
await settlePluginTaskCredentialHandoff(db, { companyId: input.companyId, leaseId: lease.id, expectedTaskId: taskId }, metadata, async accountIdentity => {
|
|
const snapshot = await executeEnvironmentTask(db, workers, { ...input, expectedTaskId: taskId, operation: { kind: "credentials", accountIdentity } });
|
|
if (snapshot.kind !== "credentials") throw new Error("Plugin Runner credential snapshot unavailable");
|
|
return snapshot.credential;
|
|
});
|
|
}
|
|
const executionContext = context?.pluginSupported && ["describe", "prepare", "check_project_auth", "connection"].includes(operation.kind) ? await context.projection({ pluginId: plugin.id,
|
|
leaseId: lease.id, taskId, environmentId: environment!.id }, async () => {
|
|
try {
|
|
const [identity, configuration] = await Promise.all([
|
|
pluginAgentRuntimeIdentity(db, plugin.id, input.companyId, run!.agentId),
|
|
pluginRegistryService(db).getConfig(plugin.id, input.companyId),
|
|
]);
|
|
if (!identity) throw new Error("Missing transport binding");
|
|
return { instanceSecrets: identity.instanceSecrets, configuration: configuration?.configJson ?? {} };
|
|
} catch { throw new Error("Plugin task transport unavailable"); }
|
|
}) : undefined;
|
|
const params = {
|
|
driverKey: metadata.driverKey, companyId: input.companyId, environmentId: environment?.id ?? null,
|
|
issueId: lease.issueId, config: driverConfig,
|
|
taskId, runId: run?.id ?? (activeProbe ? String(probe.runId) : null), agentId: run?.agentId ?? null, projectIds,
|
|
lease: { providerLeaseId: lease.providerLeaseId, metadata: lease.metadata ?? undefined },
|
|
...(workspaceBinding ? { workspace: workspaceBinding } : {}),
|
|
...(freshWorkspace && (scopesProjects || retainedTask) ? { fresh: freshWorkspace } : {}), operation,
|
|
executionContextSupported: 1 as const,
|
|
...(executionContext ? { executionContext } : {}),
|
|
};
|
|
const timeout = operation.kind === "copilot_metadata" ? Math.max(1, Math.min(40_000, Number(probe?.expiresAtMs) - Date.now()))
|
|
: executionContext ? Math.max(1, context!.deadlineAtMs - Date.now())
|
|
: driver.sessionHandoffVersion === 1 && (ownership || operation.kind === "describe") ? 90_000 : 15_000;
|
|
const result = await callWorker("environmentTask", params, timeout).catch(error => {
|
|
// The plugin may have applied the saved claim before its reply was lost.
|
|
// Keep the same operation ID; the RPC response below still needs validation.
|
|
if (ownership && (!(error instanceof JsonRpcCallError) ||
|
|
error.code === PLUGIN_RPC_ERROR_CODES.TIMEOUT || error.code === PLUGIN_RPC_ERROR_CODES.WORKER_UNAVAILABLE)) throw new PluginTaskOwnershipPendingError();
|
|
throw error;
|
|
});
|
|
if (executionContext) context!.assertOpen();
|
|
const parsed = parseEnvironmentTaskResult(operation, taskId, result);
|
|
if (operation.kind === "submit" && parsed.kind === "admission_rejected") {
|
|
const recorded = await db.update(environmentLeases).set({
|
|
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskIdentity') || ${JSON.stringify({
|
|
runnerTaskAdmissionRejection: { taskId, reason: parsed.reason },
|
|
runnerTaskDispatched: false, runnerTaskExecutionStopped: true, runnerTaskSessionRetained: false,
|
|
})}::jsonb`,
|
|
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.providerLeaseId, taskId),
|
|
sql`${environmentLeases.metadata}->>'runnerTaskDispatched' = 'true'`,
|
|
sql`${environmentLeases.metadata}->'runnerTaskIdentity' = ${JSON.stringify(operation.runner)}::jsonb`,
|
|
)).returning({ id: environmentLeases.id });
|
|
if (recorded.length !== 1) throw new Error("Plugin Runner admission receipt is stale");
|
|
return parsed;
|
|
}
|
|
if (stoppedSourceCheckpoint && parsed.kind === "state" && isSettledReadyRunnerState(parsed.state)) {
|
|
parsed.state = { ...parsed.state, lifecycle: "suspended" };
|
|
}
|
|
if (sourceRecovery && (operation.kind === "prepare" || operation.kind === "submit")) {
|
|
await assertPluginTaskSourceRecovery(db, { companyId: input.companyId, leaseId: lease.id,
|
|
context: context instanceof NativeExecutionContext ? context : undefined, operation });
|
|
}
|
|
if (sourceAttachment && operation.kind === "prepare") {
|
|
await assertPluginTaskSourceAttachment(db, { companyId: input.companyId, leaseId: lease.id,
|
|
context: context instanceof NativeExecutionContext ? context : undefined, operation });
|
|
}
|
|
if (portableCheckpoint && operation.kind === "read_state" || portableAttachment && operation.kind === "prepare") {
|
|
const identity = operation.kind === "read_state" ? operation.identity : operation.priorIdentity;
|
|
const sourceTaskId = (await readyPortableTaskSource(db, input.companyId, lease.id, identity))?.taskId;
|
|
if (sourceTaskId !== (portableCheckpoint || portableAttachment)) throw new Error("Plugin Runner session ownership changed");
|
|
}
|
|
if (operation.kind === "read_file") {
|
|
const [current] = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases)
|
|
.leftJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, environmentLeases.companyId)))
|
|
.where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId))).limit(1);
|
|
if (!current || current.lease.status !== "active" || current.run?.status !== "running" ||
|
|
current.lease.metadata?.runnerTaskRetiring === true || current.lease.metadata?.runnerTaskExecutionStopped === true ||
|
|
current.lease.expiresAt && current.lease.expiresAt.getTime() <= Date.now()) throw new Error("Plugin Runner file lease expired during transfer");
|
|
}
|
|
if (freshWorkspace && parsed.kind === "description" && parsed.freshWorkspaceVersion !== 1)
|
|
throw new Error("Plugin-provided fresh workspace preparation unavailable");
|
|
if (operation.kind === "prepare" && parsed.kind === "prepared" && (parsed.projectAuthChecked || parsed.projectAuthCheckAtSubmit)) {
|
|
const harness = operation.requirements.provider === "acpx" ? operation.requirements.settings.acpxAgent : operation.requirements.provider;
|
|
if (!harness) throw new Error("Plugin Runner prepared provider unavailable");
|
|
const provider = harness === "codex" ? "openai" : harness === "claude" ? "anthropic" : harness;
|
|
const saved = await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({
|
|
runnerTaskProjectAuthValidation: { configurationId: parsed.launch.configurationId, provider, projectIds,
|
|
...(parsed.projectAuthChecked ? {} : { atSubmit: true }) },
|
|
})}::jsonb` }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`)).returning({ id: environmentLeases.id });
|
|
if (saved.length !== 1) throw new Error("Plugin Runner project validation lease expired");
|
|
}
|
|
if (operation.kind === "check_project_auth" && parsed.kind === "project_auth") {
|
|
if (parsed.configurationId !== operation.configurationId || parsed.provider !== operation.provider) throw new Error("Plugin Runner project validation identity mismatch");
|
|
if (parsed.allowed) {
|
|
const saved = await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({
|
|
runnerTaskProjectAuthValidation: { configurationId: parsed.configurationId, provider: parsed.provider, projectIds },
|
|
})}::jsonb` }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`)).returning({ id: environmentLeases.id });
|
|
if (saved.length !== 1) throw new Error("Plugin Runner project validation lease expired");
|
|
}
|
|
}
|
|
if ((parsed.kind === "status" || parsed.kind === "copilot_metadata") && parsed.executionStopped === true) {
|
|
await recordTaskStop(parsed);
|
|
}
|
|
if (operation.kind === "complete") {
|
|
if (freshWorkspace && metadata.runnerTaskExecutionStopped === true)
|
|
await releaseProviderRecoveryTransfers(input.companyId, freshWorkspace);
|
|
// The provider attested this task stopped: record the stop receipt now, not when
|
|
// the lease's resources are released. A reusable physical task can back several
|
|
// run leases, so confirm retirement for every owner.
|
|
if (metadata.runnerTaskExecutionStopped === true) {
|
|
const owners=lease.leasePolicy === "reuse_by_environment" ? await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId,input.companyId),
|
|
eq(environmentLeases.provider,lease.provider!),eq(environmentLeases.providerLeaseId,taskId))) : [lease];
|
|
for(const owner of owners) {
|
|
const receipt=remoteTerminationReceipt(owner,{providerLeaseId:taskId,state:"stopped"});
|
|
if(!receipt) continue;
|
|
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({
|
|
...(owner.id !== lease.id && owner.status === "retained" ? {status:"released" as const,releasedAt:new Date(),cleanupStatus:"success" as const}:{}),
|
|
metadata:sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({remoteExecutionTermination:receipt,runnerTaskExecutionStopped:true})}::jsonb`,
|
|
}).where(and(eq(environmentLeases.id,owner.id),eq(environmentLeases.providerLeaseId,taskId))), { companyId: owner.companyId, runId: owner.heartbeatRunId ?? undefined });
|
|
}
|
|
}
|
|
await db.update(environmentLeases).set({
|
|
metadata: sql`(case when ${environmentLeases.metadata}->>'runnerTaskPreparationStopped' = 'true'
|
|
then coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskStartup'
|
|
else coalesce(${environmentLeases.metadata}, '{}'::jsonb) end) || '{"runnerTaskDispatched":false}'::jsonb`,
|
|
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.providerLeaseId, taskId), sql`${environmentLeases.metadata}->>'runnerTaskExecutionStopped' = 'true'`));
|
|
}
|
|
return parsed;
|
|
} catch (error) {
|
|
if (error instanceof PluginTaskOwnershipPendingError) throw error;
|
|
const diagnostic = operation.kind === "submit" && operation.resumeFrom ? readEnvironmentTaskErrorDiagnostic(error) : null;
|
|
if (diagnostic) throw new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
|
|
message: "Plugin Runner source state unavailable",
|
|
data: environmentTaskErrorData({ status: diagnostic.httpStatus, code: diagnostic.errorCode, reason: diagnostic.reason }),
|
|
});
|
|
// Worker errors and schema diagnostics can contain request credentials.
|
|
const code = readEnvironmentTaskErrorCode(error);
|
|
throw Object.assign(new Error("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying"),
|
|
code ? { data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError(code)) } : {});
|
|
}
|
|
}
|
|
|
|
/** Failure leaves the dispatch marker and provider handle available to cleanup retries. */
|
|
export async function settleDispatchedEnvironmentTask(db: Db, workers: PluginWorkerManager, lease: { id: string; companyId: string },
|
|
options: { executionStopObserved?: boolean } = {}): Promise<false | "stopped" | "completed"> {
|
|
const [current] = await db.update(environmentLeases).set({
|
|
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"runnerTaskRetiring":true}'::jsonb`,
|
|
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId)))
|
|
.returning({ metadata: environmentLeases.metadata, heartbeatRunId: environmentLeases.heartbeatRunId, taskId: environmentLeases.providerLeaseId });
|
|
const startupCleanup = current?.metadata?.runnerTaskStartupCleanup as { requiresPreparationStop?: boolean } | undefined;
|
|
const startup = current?.metadata?.runnerTaskStartup;
|
|
if (startupCleanup && current?.metadata?.runnerTaskExecutionStopped === true && current.metadata.runnerTaskDispatched === false &&
|
|
!current.metadata.runnerTaskAdmissionRejection && !startup &&
|
|
(!startupCleanup.requiresPreparationStop || current.metadata.runnerTaskPreparationStopped === true)) return "completed";
|
|
if (current?.metadata?.runnerTaskDispatched !== true && !startup && !startupCleanup) {
|
|
if (current?.metadata) await settlePluginTaskCredentialHandoff(db, {
|
|
companyId: lease.companyId, leaseId: lease.id, expectedTaskId: current.taskId ?? undefined,
|
|
}, current.metadata, async accountIdentity => {
|
|
const snapshot = await executeEnvironmentTask(db, workers, { companyId: lease.companyId, leaseId: lease.id,
|
|
expectedTaskId: current.taskId ?? undefined, operation: { kind: "credentials", accountIdentity } });
|
|
if (snapshot.kind !== "credentials") throw new Error("Plugin Runner credential snapshot unavailable");
|
|
return snapshot.credential;
|
|
});
|
|
return current?.metadata?.runnerTaskExecutionStopped === true ? "stopped" : false;
|
|
}
|
|
const scope = { companyId: lease.companyId, leaseId: lease.id };
|
|
try {
|
|
const contextId = (startupCleanup as { contextId?: unknown } | undefined)?.contextId ?? (startup as { contextId?: unknown } | undefined)?.contextId;
|
|
const draining = typeof contextId === "string" ? workers.drainStartup?.(contextId) : undefined;
|
|
void draining?.catch(() => undefined);
|
|
const confirmed = (state?: Awaited<ReturnType<typeof executeEnvironmentTask>>) => state?.kind === "status" && state.executionStopped === true &&
|
|
(!(startup || startupCleanup?.requiresPreparationStop) || state.preparationStopped === true);
|
|
let state = options.executionStopObserved === true
|
|
? await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "status" } }) : undefined;
|
|
if (!confirmed(state)) {
|
|
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "stop" } });
|
|
state = await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "status" } });
|
|
}
|
|
if (!confirmed(state)) throw new Error("Plugin Runner termination is unconfirmed");
|
|
await draining;
|
|
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "complete" } });
|
|
return "completed";
|
|
} catch {
|
|
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed",
|
|
failureReason: "plugin_runner_task_cleanup_unconfirmed", releasedAt: new Date(), updatedAt: new Date(),
|
|
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId))), { companyId: lease.companyId, runId: current.heartbeatRunId ?? undefined });
|
|
throw new Error("Plugin Runner termination is unconfirmed; the saved lease requires cleanup");
|
|
}
|
|
}
|