Files
PaperClipAI/server/src/services/environment-task-runtime.ts
T
Nicky LeachandPaperclip 731b52453c fix(environments): verify workspaces on run leases and release settled tasks (#21289)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core owns run leases and delegates remote execution to environment
plugins.
> - A retry must verify its old checkout on the placement that will run
the task.
> - A completed task must still release its provider resources.
> - This change uses the existing lease for workspace verification and
always calls the release hook after settlement.
> - Saved settlement proof lets cleanup retry without repeating
execution work.

## Linked Issues or Issue Description

Refs #21237. That PR records stop proof before lease release. This PR
keeps provider cleanup after settlement and changes workspace
verification sequencing. Both touch task completion; preserve both
changes if the merge order requires conflict resolution.

**What happened?**

Workspace verification ran before a portable task had a lease, so a
provider could not verify the checkout on its selected placement. Core
also skipped the provider release hook after successful task completion.
This could leave resources allocated after the run ended.

**Expected behavior**

Verify the old checkout through the selected run lease. A confirmed
refusal can use the authorized fresh-workspace fallback. A transport
failure must remain retryable. Call the provider release hook after
settlement and retry failed cleanup without changing the run result.

**Steps to reproduce**

1. Retry a task whose prior checkout uses plugin-managed storage and
portable task placement.
2. Verify that the plugin receives the same task lease for verification
and preparation.
3. Complete a task, fail the provider release once, and reconstruct the
cleanup controller.
4. Confirm that cleanup calls only the release hook and retains the
completed run outcome.

## What Changed

- Acquire the logical run lease before provider workspace verification
and reuse it for preparation and execution.
- Preserve the existing session and workspace choices. Recompute
workspace-dependent inputs only after a definite refusal; acquire
credentials and managed runtime inputs once.
- Save the exact refusal and lease before cleanup. Use a
workspace-scoped creation key only for the resulting fresh attempt.
- Replay interrupted verification on the same unlaunched lease. Keep
transport errors retryable and reject replay after execution starts.
- Always call the provider release hook after task settlement. Pass
refreshed settlement proof and route failed releases through the
existing cleanup reaper.
- Add database regressions for call order, restart, retained writers,
refusal, transport errors, and settlement-only cleanup replay.

`plugin-workspaces.ts` grows from 997 to 1,001 lines. The four added
lines extend the existing workspace transaction and lease callbacks.
Keeping the checks there preserves the ownership assertion, provider
result validation, and binding update in one operation. Extracting only
the verification branch would split that transaction without removing a
separate responsibility.

## Verification

- Server TypeScript check passed: `tsc -p server/tsconfig.json
--noEmit`. The complete `server/scripts/typecheck-layers.mjs` check also
passed, including all test-tree layers.
- The first CI typecheck caught a readonly array in a test fixture that
the narrower source check did not include. Commit 8d9ec19e7a corrects
that fixture; all layered checks pass afterward.
- Added a public heartbeat refusal regression in 7c598a4da3. All ten
session-preparation tests pass; independent execution of the refusal
case also passes. It checks successful completion, the saved fresh
workspace, a new conversation, and one credential acquisition.
- Nine database/service suites passed 335 tests: plugin warm tasks,
plugin runner tasks, startup cleanup recovery, credential writeback,
plugin workspaces, session preparation, plugin environments, retry
workspace continuity, and heartbeat workspace preparation.
- After the final restart change, both targeted refusal/restart tests
passed. The primary agent independently reran those cases and the final
server typecheck. After integrating current master, the primary agent's
independent rerun passed all 237 tests across the warm-task,
plugin-workspace, session-preparation, and heartbeat-workspace suites.
- `pnpm verify` returned exit 2 and requested broad CI coverage because
the plugin execution contract documentation changed. No full local
Core-suite pass is claimed.
- Previous-revision CI was green on 8d9ec19e7a: 59 checks passed and 3
were skipped. The first run failed the unchanged legacy-continuation
browser assertion and credential-cleanup server shard. Each passed on
one targeted retry with no code changes. The credential-cleanup
integration suite also passed all 21 tests locally on Node 22. The
browser failure concerned an existing continuation metadata marker; the
server failure cascaded after a cleanup error left a test spy installed.
The successful retries do not establish the cause of the first failures.
- Local database fixtures use simulated provider workers. No live
provider or deployed environment was changed.

- Additional verification for 59c7a1dae8: the layered server typecheck
passed. Independent execution passed all 265 tests across durable task
retries, workspace preparation, and public heartbeat preparation. The
new shutdown regression preserves the fresh Retry choice, waits for
parent cleanup, and prevents the successor from inheriting the parent
lease reference. Same-run recovery retains its receipt. Required CI is
green on 59c7a1dae8: 58 checks passed and 3 were skipped. Standard
review scored 5/5 with no unresolved comments. The separate Superagent
Security Scan is neutral/inconclusive because its workflow failed; a
direct rerun request returned HTTP 404. That scan is not claimed as
passed.

## Risks

- Providers must use the supplied task lease for verification and return
a definite `binding_mismatch` only when the checkout is unconfirmed.
Transport errors retain the retry decision.
- A refused checkout waits for confirmed release before a fresh
allocation. Failed deletion remains owned by the existing cleanup
reaper.
- Task settlement and physical resource release stay separate. Retained
handoff leases do not enter this release path.
- No migration or SDK schema change is required.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) implemented the changes and tests.
The primary Codex agent independently reviewed the connected Core and
plugin paths and ran checks. A separate GPT-6.1 Sol agent reviewed
lifecycle and restart boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue using the bug template labels
- [x] I have not referenced internal or private project details
- [x] My branch name describes the change
- [x] I have added relevant tests and documentation
- [x] I have considered and documented risks
- [x] All required CI gates are green
- [x] Greptile is 5/5 with no unresolved comments
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 21:56:02 -08:00

647 lines
51 KiB
TypeScript

import { pluginTaskResumeRefused, resolvePluginTaskResume } from "./plugin-task-resume.js";
export { resolvePluginTaskResume } from "./plugin-task-resume.js";
import { attachSandboxCreation, environmentService } from "./environments.js";
import { createOwnedSandbox, queueSandboxCreationCleanup, recordSandboxCreationResource } from "./sandbox-creation-operations.js";
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js";
import { readWorkspaceProviderBinding } from "@paperclipai/shared";
import { canReadStoppedPluginTaskCredential, settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
import { and, eq, inArray, sql } from "drizzle-orm";
import { environmentLeases, executionWorkspaces, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult,
environmentTaskErrorData, readEnvironmentTaskErrorDiagnostic, JsonRpcCallError, PLUGIN_RPC_ERROR_CODES,
environmentTaskErrorCodeData, PluginEnvironmentTaskError, readEnvironmentTaskErrorCode,
type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation, type PluginEnvironmentTaskResult } from "@paperclipai/plugin-sdk";
import { pluginRegistryService } from "./plugin-registry.js";
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
import { NativeExecutionContext } from "./native-runtime/native-execution-context.js";
import { assertPluginTaskSourceAttachment, assertPluginTaskSourceRecovery } from "./native-runtime/plugin-task-source-recovery.js";
import { pluginAgentRuntimeIdentity } from "./plugin-agent-runtime-state.js";
import { assertPluginTaskOwnershipDispatch, PluginTaskOwnershipPendingError, readyPortableTaskSource, reconcilePluginTaskAdmission } from "./plugin-task-ownership.js";
export type PluginTaskDescription = Extract<PluginEnvironmentTaskResult, { kind: "description" }>;
export function isSettledReadyRunnerState(state: Record<string, unknown>): boolean {
const commands = state.processedCommands;
return state.schema === "paperclip.runner.durable.state.v1" && state.lifecycle === "ready" &&
Array.isArray(state.outbox) && state.outbox.length === 0 && Number.isSafeInteger(state.nextSourceSeq) &&
Number(state.nextSourceSeq) > 0 && state.ackedSourceSeq === Number(state.nextSourceSeq) - 1 &&
state.pendingTerminalDelivery == null && state.pendingProviderCleanup == null && state.warmTransition == null &&
commands !== null && typeof commands === "object" && !Array.isArray(commands) &&
Object.values(commands).every(command => {
const status = command !== null && typeof command === "object" ? (command as { status?: unknown }).status : undefined;
return typeof status === "string" && status !== "" && status !== "pending" && status !== "indeterminate";
});
}
export async function pluginEnvironmentSupportsTasks(db: Db, workers: PluginWorkerManager, lease: { companyId: string; metadata: Record<string, unknown> | null }) {
const metadata = lease.metadata ?? {};
if (metadata.driver !== "plugin" || typeof metadata.pluginId !== "string" || typeof metadata.driverKey !== "string") return false;
const plugin = await pluginRegistryService(db).getById(metadata.pluginId);
const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === metadata.driverKey);
const methods = workers.getWorker(metadata.pluginId)?.supportedMethods ?? [];
return plugin?.status === "ready" && driver?.supportsTasks === true &&
plugin.manifestJson.capabilities.includes("environment.drivers.register") &&
methods.includes("environmentTask");
}
/** Server-only dispatch for plugin-provided Runner execution.
* The caller supplies an authorized company and a persisted lease,
* never a plugin identity, resource endpoint, or agent/project authority.
* A timeout is ambiguous: retry the same operation against the same lease.
*/
export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerManager, "getWorker" | "call">, input: {
companyId: string;
leaseId: string;
operation: PluginEnvironmentTaskOperation;
executionContext?: unknown;
ownershipSignal?: AbortSignal;
/** Physical task captured by an existing owner, independent of the mutable lease. */
expectedTaskId?: string;
}): Promise<PluginEnvironmentTaskResult> {
const operation = environmentTaskOperationSchema.parse(input.operation);
const [row] = await db.select({ lease: environmentLeases, environment: environments, run: heartbeatRuns })
.from(environmentLeases)
.leftJoin(environments, eq(environments.id, environmentLeases.environmentId))
.leftJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, environmentLeases.companyId)))
.where(and(eq(environmentLeases.id, input.leaseId), eq(environmentLeases.companyId, input.companyId))).limit(1);
if (!row || !row.lease.providerLeaseId) throw new Error("Plugin-provided Runner execution lease unavailable");
const { lease, environment, run } = row;
const context = input.executionContext;
if (context !== undefined) {
if (!(context instanceof NativeExecutionContext) || !run) {
throw new Error("native_execution_context_unavailable");
}
context.assertScope({ companyId: input.companyId, agentId: run.agentId, runId: run.id });
}
const taskId = row.lease.providerLeaseId;
if (input.expectedTaskId !== undefined && input.expectedTaskId !== taskId) throw new Error("Plugin Runner physical task changed");
const metadata = lease.metadata ?? {};
const ownership = operation.kind === "ensure_ownership";
if (operation.kind === "describe" && metadata.portableTaskOwnershipReady !== true && metadata.portableTaskSuccessor === undefined &&
(metadata.portableTaskPredecessor !== undefined || typeof metadata.portableTaskPreviousLeaseId === "string")) {
await reconcilePluginTaskAdmission(db, lease, request =>
executeEnvironmentTask(db, workers, { ...input, operation: request }), context?.signal ?? input.ownershipSignal);
return executeEnvironmentTask(db, workers, input);
}
const requiresOwnership = !["ensure_ownership", "status", "stop", "complete"].includes(operation.kind);
const stoppedCredentialHandoff = operation.kind === "credentials" && input.expectedTaskId === taskId &&
canReadStoppedPluginTaskCredential(metadata, operation.accountIdentity);
if (requiresOwnership && (metadata.portableTaskSuccessor !== undefined && !stoppedCredentialHandoff ||
(metadata.portableTaskPredecessor !== undefined || metadata.portableTaskPreviousLeaseId !== undefined) && metadata.portableTaskOwnershipReady !== true)) {
throw new Error("Plugin Runner session ownership unavailable");
}
if (ownership) {
if (!environment || lease.status !== "active" || lease.releasedAt || run?.status !== "running" ||
metadata.runnerTaskRetiring === true || lease.expiresAt && lease.expiresAt.getTime() <= Date.now()) {
throw new Error("Plugin-provided Runner execution lease is not active");
}
assertPluginTaskOwnershipDispatch(lease, operation);
}
// A different launch identity is authorized by the saved consumer claim, never by the mismatch alone.
const sourceRecovery = run && ((operation.kind === "prepare" && operation.requirements.identity.runId !== run.id && metadata.runnerTaskSourceRecovery) ||
(operation.kind === "submit" && operation.runner.runId !== run.id && metadata.runnerTaskSourceRecovery))
? await assertPluginTaskSourceRecovery(db, { companyId: input.companyId, leaseId: lease.id,
context: context instanceof NativeExecutionContext ? context : undefined, operation }) : undefined;
const probe = metadata.runnerTaskProbe as { runId?: unknown; sessionId?: unknown; expiresAtMs?: unknown } | undefined;
const activeProbe = lease.heartbeatRunId === null && lease.issueId === null && probe
&& typeof probe.runId === "string" && typeof probe.sessionId === "string"
&& typeof probe.expiresAtMs === "number" && probe.expiresAtMs > Date.now();
const checkpointIdentity = operation.kind === "read_state" || operation.kind === "read_file" ? operation.identity
: operation.kind === "archive_state" ? operation.priorIdentity : null;
const runner = checkpointIdentity ? { runnerId: checkpointIdentity.runnerInstanceId, leaseId: checkpointIdentity.environmentLeaseId,
runId: checkpointIdentity.runId, sessionId: checkpointIdentity.normalizedSessionId } : operation.kind === "submit" ? operation.runner : operation.kind === "prepare" ? {
runnerId: operation.requirements.identity.runnerInstanceId,
leaseId: operation.requirements.identity.environmentLeaseId,
runId: operation.requirements.identity.runId, sessionId: operation.requirements.identity.normalizedSessionId,
} : null;
const launches = operation.kind === "submit" || operation.kind === "prepare";
const scopesProjects = launches || operation.kind === "check_project_auth" || operation.kind === "read_file";
const retainedTask = operation.kind === "release" || operation.kind === "resume";
const resumeFrom = launches || checkpointIdentity ? (operation as { resumeFrom?: string }).resumeFrom : undefined;
const savedIdentity = metadata.runnerTaskIdentity as Record<string, unknown> | undefined;
const prior = operation.kind === "prepare" ? operation.priorIdentity : undefined;
const priorSaved = metadata.runnerTaskPriorIdentity as Record<string, unknown> | undefined;
const firstPortableResume = operation.kind === "resume" && !savedIdentity &&
metadata.portableTaskPredecessor !== undefined;
if (firstPortableResume && (run?.status !== "running" ||
!await readyPortableTaskSource(db, input.companyId, lease.id, operation.identity))) {
throw new Error("Plugin Runner session ownership unavailable");
}
const matches = (identity: typeof prior, saved: Record<string, unknown> | undefined) => !!identity && !!saved &&
identity.runnerInstanceId === saved.runnerId && identity.environmentLeaseId === saved.leaseId &&
identity.normalizedSessionId === saved.sessionId && identity.runId === saved.runId &&
identity.turnId === saved.turnId && identity.itemId === saved.itemId;
const portableCheckpoint = operation.kind === "read_state" && !resumeFrom && !savedIdentity &&
metadata.portableTaskPredecessor !== undefined
? (await readyPortableTaskSource(db, input.companyId, lease.id, operation.identity))?.taskId : undefined;
const ownCheckpoint = checkpointIdentity && !resumeFrom &&
(portableCheckpoint || matches(checkpointIdentity,savedIdentity) || operation.kind !== "read_file" && matches(checkpointIdentity,priorSaved));
if (checkpointIdentity && !resumeFrom && !ownCheckpoint) throw new Error("Plugin Runner state identity mismatch");
const retainedCheckpointRead = operation.kind === "read_state" && ownCheckpoint &&
lease.leasePolicy === "reuse_by_environment" && ["retained", "expired"].includes(lease.status) &&
run?.status === "succeeded" && metadata.runnerTaskDispatched === true &&
(metadata.runnerTaskLifecyclePolicy as {mode?:unknown}|undefined)?.mode === "warm";
const retainedConnection = operation.kind === "connection" && environment !== null &&
lease.leasePolicy === "reuse_by_environment" && lease.status === "retained" &&
run?.status === "succeeded" && savedIdentity?.runId === run.id &&
metadata.runnerTaskDispatched === true && metadata.runnerTaskExecutionStopped !== true &&
metadata.runnerTaskRetiring !== true && (!lease.expiresAt || lease.expiresAt.getTime() > Date.now()) &&
(metadata.runnerTaskLifecyclePolicy as { mode?: unknown } | undefined)?.mode === "warm";
if ((operation.kind === "read_file" || operation.kind === "describe" || operation.kind === "prepare" || operation.kind === "submit" || (operation.kind === "connection" && !retainedConnection) || (operation.kind === "read_state" && metadata.runnerTaskExecutionStopped !== true && !retainedCheckpointRead) || operation.kind === "archive_state" || operation.kind === "check_project_auth" || operation.kind === "copilot_metadata") &&
(metadata.runnerTaskRetiring === true || !environment || (!activeProbe && run?.status !== "running") || lease.status !== "active" || (lease.expiresAt && lease.expiresAt.getTime() <= Date.now()))) throw new Error("Plugin-provided Runner execution lease is not active");
if (operation.kind === "copilot_metadata" && !activeProbe) throw new Error("Copilot metadata requires a private setup lease");
const sourceReference = metadata.runnerTaskSourceRecovery as { identity?: { runId?: unknown } } | undefined;
const sourceAttachment = operation.kind === "prepare" && prior && sourceReference?.identity?.runId === prior.runId
? await assertPluginTaskSourceAttachment(db, { companyId: input.companyId, leaseId: lease.id,
context: context instanceof NativeExecutionContext ? context : undefined, operation }) : undefined;
const portableAttachment = prior && !sourceAttachment && metadata.runnerTaskReused !== true &&
metadata.portableTaskPredecessor !== undefined
? (await readyPortableTaskSource(db, input.companyId, lease.id, prior))?.taskId : undefined;
const warmAttachment = !!prior && (metadata.runnerTaskReused === true || !!sourceAttachment || !!portableAttachment) &&
(portableAttachment && !savedIdentity || matches(prior, savedIdentity) ||
matches(prior, priorSaved) && operation.kind === "prepare" && matches(operation.requirements.identity, savedIdentity)) &&
runner?.runnerId === prior.runnerInstanceId &&
runner?.leaseId === prior.environmentLeaseId && runner?.sessionId === prior.normalizedSessionId;
if (prior && !warmAttachment) throw new Error("Plugin warm task prior identity mismatch");
const portableAuthority = firstPortableResume || portableAttachment
? sql`${environmentLeases.metadata}->'portableTaskSuccessor' is null
and ${environmentLeases.metadata}->'portableTaskOwnershipReady' = 'true'::jsonb
and coalesce(${environmentLeases.metadata}->>'portableTaskCheckpointUnavailable', 'false') != 'true'
and ${environmentLeases.metadata}->'portableTaskPredecessor' = ${JSON.stringify(metadata.portableTaskPredecessor)}::jsonb`
: undefined;
if (runner && !sourceRecovery && ((!checkpointIdentity || !resumeFrom) && !(ownCheckpoint && operation.kind !== "read_file") && runner.runId !== (activeProbe ? probe.runId : run?.id) || (!resumeFrom && runner.leaseId !== lease.id && !ownCheckpoint && !warmAttachment)
|| (activeProbe && (runner.sessionId !== probe.sessionId || (launches && operation.projectIds.length > 0))))) {
throw new Error("Plugin-provided Runner execution identity mismatch");
}
let stoppedSourceCheckpoint = false;
if (runner && resumeFrom && !sourceRecovery) {
const inspectCheckpoint = operation.kind === "read_state";
const sources = await db.select({ id: environmentLeases.id, runId: environmentLeases.heartbeatRunId, metadata: environmentLeases.metadata }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.provider, lease.provider!),
eq(environmentLeases.providerLeaseId, resumeFrom),
inspectCheckpoint ? undefined : sql`${environmentLeases.metadata}->>'runnerTaskSessionRetained' = 'true'`,
checkpointIdentity ? eq(environmentLeases.heartbeatRunId, checkpointIdentity.runId) : undefined,
)).limit(2);
const priorRunId = sources.length === 1 ? sources[0]!.runId : null;
const sourceIdentity = sources.length === 1 ? sources[0]!.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined : undefined;
if (activeProbe || !priorRunId || (checkpointIdentity && (checkpointIdentity.runId !== priorRunId
|| checkpointIdentity.turnId !== sourceIdentity?.turnId || checkpointIdentity.itemId !== sourceIdentity?.itemId)) || resumeFrom !== await resolvePluginTaskResume(db, {
companyId: input.companyId, leaseId: lease.id,
priorIdentity: { runnerInstanceId: runner.runnerId, environmentLeaseId: runner.leaseId,
runId: priorRunId, normalizedSessionId: runner.sessionId },
inspectCheckpoint,
})) throw new Error("Plugin Runner resume identity mismatch");
// Older providers misclassified suspended checkpoints. A fresh provider
// attestation may repair availability, but never supplies stop/owner authority.
if (inspectCheckpoint && sources[0]!.metadata?.runnerTaskSessionRetained !== true) {
const status = await executeEnvironmentTask(db, workers, { companyId: input.companyId,
leaseId: sources[0]!.id, operation: { kind: "status" } });
if (status.kind !== "status" || status.executionStopped !== true || status.sessionStateAvailable !== true) {
throw pluginTaskResumeRefused("checkpoint_status_unconfirmed");
}
if (resumeFrom !== await resolvePluginTaskResume(db, { companyId: input.companyId, leaseId: lease.id,
priorIdentity: { runnerInstanceId: runner.runnerId, environmentLeaseId: runner.leaseId,
runId: priorRunId, normalizedSessionId: runner.sessionId } })) {
throw pluginTaskResumeRefused("resume_task_mismatch");
}
}
stoppedSourceCheckpoint = true;
}
if (metadata.driver !== "plugin" || typeof metadata.pluginId !== "string" || typeof metadata.driverKey !== "string") {
throw new Error("Plugin-provided Runner execution lease has no pinned plugin driver");
}
const plugin = await pluginRegistryService(db).getById(metadata.pluginId);
const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === metadata.driverKey);
if (!plugin || plugin.status !== "ready" || !driver?.supportsTasks ||
!plugin.manifestJson.capabilities.includes("environment.drivers.register")) {
throw new Error("Plugin-provided Runner execution provider unavailable");
}
const worker = workers.getWorker(plugin.id);
if (ownership && !worker && driver.sessionHandoffVersion === 1) throw new PluginTaskOwnershipPendingError();
if (!worker?.supportedMethods.includes("environmentTask")) {
throw new Error("Plugin-provided Runner execution provider unavailable");
}
if (operation.kind === "resume" && operation.handoffCredential && driver.sessionHandoffVersion !== 1) {
throw new Error("Plugin Runner credential handoff unavailable");
}
if (ownership && driver.sessionHandoffVersion !== 1) throw new Error("Plugin Runner ownership reconciliation unavailable");
const callWorker = async <M extends HostToWorkerMethodName>(method: M, params: HostToWorkerMethods[M][0], timeout?: number): Promise<HostToWorkerMethods[M][1]> => {
if (!context) {
const cleanup = metadata.runnerTaskStartupCleanup as { contextId?: unknown; requiresPreparationStop?: unknown } | undefined;
if (["stop", "status", "complete", "credentials"].includes(operation.kind) && typeof cleanup?.contextId === "string" && typeof cleanup.requiresPreparationStop === "boolean") {
return workers.call(plugin.id, method, params, timeout, undefined, undefined, cleanup.contextId);
}
return timeout === undefined ? workers.call(plugin.id, method, params) : workers.call(plugin.id, method, params, timeout);
}
const settled = context.trackWorker();
const result = await workers.call(plugin.id, method, params, Math.max(1, Math.min(timeout ?? 15_000, context.deadlineAtMs - Date.now())), undefined, settled, context.id);
context.assertOpen();
return result;
};
const issue = scopesProjects && lease.issueId ? await db.select({ id: issues.id }).from(issues)
.where(and(eq(issues.id, lease.issueId), eq(issues.companyId, input.companyId))).limit(1).then(rows => rows[0]) : null;
if (scopesProjects && lease.issueId && !issue) throw new Error("Plugin-provided Runner execution issue unavailable");
const workspaceBinding = readWorkspaceProviderBinding(lease.metadata);
if (metadata.workspaceProvider !== undefined && !workspaceBinding) throw new Error("Invalid managed workspace binding");
if (workspaceBinding) {
const [workspace] = await db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.id, workspaceBinding.executionWorkspaceId), eq(executionWorkspaces.companyId, input.companyId)));
const saved = readWorkspaceProviderBinding(workspace?.metadata);
if (!workspace || workspace.id !== lease.executionWorkspaceId || workspace.providerType !== "plugin" || workspace.providerRef !== workspaceBinding.workspaceRef ||
(scopesProjects && !["active", "idle", "in_review"].includes(workspace.status)) ||
workspaceBinding.pluginId !== plugin.id || !saved || JSON.stringify(saved) !== JSON.stringify(workspaceBinding)) {
throw new Error("Managed workspace execution binding changed");
}
}
let freshWorkspace = metadata.freshWorkspace === undefined ? undefined : freshWorkspaceSchema.parse(metadata.freshWorkspace);
if (freshWorkspace && driver.freshWorkspaceVersion !== 1 && (launches || operation.kind === "describe" || operation.kind === "check_project_auth"))
throw new Error("Plugin-provided fresh workspace preparation unavailable");
if (freshWorkspace && operation.kind === "prepare") {
if (!lease.issueId) throw new Error("Fresh workspace task ownership is missing");
freshWorkspace = await stageProviderFreshWorkspace({ db, companyId: input.companyId,
issueId: lease.issueId, ...freshWorkspace, existing: freshWorkspace });
}
const savedProjectIds = metadata.runnerTaskProjectIds ?? [];
if (retainedTask && (!Array.isArray(savedProjectIds) || savedProjectIds.some(id => typeof id !== "string")))
throw new Error("Plugin Runner project scope changed");
const projectIds: string[] = scopesProjects ? operation.projectIds : retainedTask ? [...savedProjectIds as string[]] : [];
if (scopesProjects && Array.isArray(metadata.runnerTaskProjectIds) &&
JSON.stringify(projectIds) !== JSON.stringify(metadata.runnerTaskProjectIds)) throw new Error("Plugin Runner project scope changed");
if (activeProbe && projectIds.length > 0) throw new Error("Plugin Runner probe cannot use project mounts");
if (projectIds.length > 0) {
const authorizedProjects = await db.select({ id: projects.id }).from(projects)
.where(and(eq(projects.companyId, input.companyId), inArray(projects.id, projectIds)));
if (authorizedProjects.length !== projectIds.length) throw new Error("Plugin-provided Runner execution project unavailable");
}
// Provider identity comes from the lease. Editing the environment must never
// redirect status or cleanup to a replacement plugin.
const config = (environment?.config ?? {}) as Record<string, unknown>;
const driverConfig = config.pluginKey === plugin.pluginKey && config.driverKey === metadata.driverKey
? (config.driverConfig as Record<string, unknown> | undefined) ?? {} : {};
let requiredProjectAuthProof: Record<string, unknown> | undefined;
if (operation.kind === "read_file") {
const proof = metadata.runnerTaskProjectAuthValidation as { projectIds?: string[] } | undefined;
if (metadata.runnerTaskExecutionStopped === true || !Array.isArray(proof?.projectIds) ||
JSON.stringify([...proof.projectIds].sort()) !== JSON.stringify([...operation.projectIds].sort()))
throw new Error("Plugin Runner file scope unavailable");
}
if (operation.kind === "submit") {
const proof = metadata.runnerTaskProjectAuthValidation as { configurationId?: unknown; provider?: unknown; projectIds?: string[] } | undefined;
const provider = operation.runner.harness === "codex" ? "openai" : operation.runner.harness === "claude" ? "anthropic" : operation.runner.harness;
if (!provider || !operation.preparedLaunchId || proof?.configurationId !== operation.preparedLaunchId || proof.provider !== provider
|| (!Array.isArray(proof.projectIds) ? null : JSON.stringify([...proof.projectIds].sort())) !== JSON.stringify([...operation.projectIds].sort())
|| (operation.credential && (operation.credential.type === "codex-auth-json" ? "openai" : "anthropic") !== provider)) {
throw new Error("Plugin Runner managed project authentication validation unavailable");
}
requiredProjectAuthProof = proof as Record<string, unknown>;
}
if (operation.kind === "check_project_auth" || operation.kind === "prepare") {
await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskProjectAuthValidation'` })
.where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId)));
}
if (operation.kind === "submit" && activeProbe) {
const remainingMs = Number(probe.expiresAtMs) - Date.now();
if (remainingMs <= 0 || !operation.launch) throw new Error("Plugin-provided Runner probe lease is not active");
// Runner itself must stop if Core disappears during a setup probe.
const requestedLimits = [operation.launch.maxRuntimeMs, operation.launch.maxLifetimeMs]
.filter((value): value is number => typeof value === "number" && value > 0);
operation.launch.maxRuntimeMs = Math.min(remainingMs, ...requestedLimits);
delete operation.launch.maxLifetimeMs;
}
if (operation.kind === "submit" || operation.kind === "copilot_metadata" || firstPortableResume) {
// No ticket, headers, environment values, or launch paths belong in metadata.
// New execution invalidates this lease's earlier stop receipt with its stop flag.
const dispatched = {
runnerTaskLease: true, runnerTaskDispatched: true,
runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false,
...(operation.kind === "submit" ? {
runnerTaskIdentity: operation.runner,
runnerTaskResumeFrom: operation.resumeFrom ?? null,
runnerTaskAccountIdentity: operation.accountIdentity,
} : {}),
};
const marked = await db.update(environmentLeases).set({
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection' - 'remoteExecutionTermination') || ${JSON.stringify(dispatched)}::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
requiredProjectAuthProof ? sql`${environmentLeases.metadata}->'runnerTaskProjectAuthValidation' = ${JSON.stringify(requiredProjectAuthProof)}::jsonb` : undefined,
firstPortableResume ? portableAuthority : undefined,
)).returning({ id: environmentLeases.id });
if (marked.length !== 1) throw new Error("Plugin-provided Runner execution lease is not active");
}
if (warmAttachment && operation.kind === "prepare") {
const i = operation.requirements.identity;
const startup = context?.pluginSupported ? { contextId: context.id, deadlineAtMs: context.deadlineAtMs, runId: i.runId } : undefined;
const marked = await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({
runnerTaskPriorIdentity: { runnerId: prior!.runnerInstanceId, leaseId: prior!.environmentLeaseId, runId: prior!.runId,
sessionId: prior!.normalizedSessionId, turnId: prior!.turnId, itemId: prior!.itemId },
runnerTaskIdentity: { runnerId: i.runnerInstanceId, leaseId: i.environmentLeaseId, runId: i.runId,
sessionId: i.normalizedSessionId, turnId: i.turnId, itemId: i.itemId },
...(startup ? { runnerTaskStartup: startup } : {}),
})}::jsonb` }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.status, "active"),
sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
portableAttachment ? portableAuthority : undefined,
savedIdentity ? sql`${environmentLeases.metadata}->'runnerTaskIdentity' = ${JSON.stringify(savedIdentity)}::jsonb`
: sql`${environmentLeases.metadata}->'runnerTaskIdentity' is null`)).returning({id:environmentLeases.id});
if (marked.length !== 1) throw new Error("Plugin warm task attachment lost authority");
}
async function recordTaskStop(result: Extract<ReturnType<typeof parseEnvironmentTaskResult>, {kind:"status" | "copilot_metadata"}>) {
await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ runnerTaskExecutionStopped: true,
runnerTaskSessionRetained: result.kind === "status" && result.sessionStateAvailable === true,
...(result.kind === "status" && result.preparationStopped === true ? { runnerTaskPreparationStopped: true } : {}) })}::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.providerLeaseId, taskId)));
}
try {
const admissionRejection = metadata.runnerTaskAdmissionRejection as { taskId?: unknown } | undefined;
const stoppedWithoutSession = !!run && metadata.runnerTaskExecutionStopped === true &&
metadata.runnerTaskSessionRetained !== true && admissionRejection?.taskId !== taskId;
if (operation.kind === "describe" && (metadata.runnerTaskReused === true || stoppedWithoutSession)) {
const state = parseEnvironmentTaskResult({kind:"status"}, taskId, await callWorker("environmentTask", {
driverKey: metadata.driverKey, companyId: input.companyId, environmentId: environment?.id ?? null,
issueId: lease.issueId, config: driverConfig, taskId, runId: run?.id ?? null, agentId: run?.agentId ?? null,
projectIds, lease: {providerLeaseId: taskId, metadata}, operation:{kind:"status"},
}));
if (state.kind === "status" && state.executionStopped === true &&
(metadata.runnerTaskReused === true || state.sessionStateAvailable !== true)) {
await recordTaskStop(state);
await executeEnvironmentTask(db, workers, {...input, expectedTaskId:taskId, operation:{kind:"complete"}});
// Preserve the predecessor's stop proof for checkpoint authorization.
if (typeof metadata.runnerTaskPreviousLeaseId === "string") await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({runnerTaskExecutionStopped:true,
runnerTaskSessionRetained:state.sessionStateAvailable === true})}::jsonb`,
}).where(and(eq(environmentLeases.id, metadata.runnerTaskPreviousLeaseId), eq(environmentLeases.companyId, input.companyId)));
const replacementDriverKey = metadata.driverKey;
const creationWorker = context && driver.creationOperationVersion === 1
? workers.getWorker(plugin.id) : undefined;
const recoverableCreation = creationWorker?.supportedMethods.includes("environmentInspectLease") === true &&
metadata.providerMetadata && typeof metadata.providerMetadata === "object" && !Array.isArray(metadata.providerMetadata);
let abandoned = false;
const fresh = await createOwnedSandbox(db, {
companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId,
issueId: lease.issueId, executionWorkspaceId: lease.executionWorkspaceId, provider: lease.provider!,
pluginId: plugin.id, driverKey: metadata.driverKey, config: {}, requestKey: taskId,
metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: plugin.pluginKey, driverKey: metadata.driverKey,
runnerTaskLease: true, agentId: run!.agentId,
...(context ? { runnerTaskCreationContextId: context.id } : {}),
...(recoverableCreation ? { creationOperationVersion: 1, providerMetadata: metadata.providerMetadata } : {}) },
}, async identity => {
const params = { driverKey: replacementDriverKey, companyId: input.companyId, environmentId: environment!.id,
executionWorkspaceId: lease.executionWorkspaceId ?? undefined,
issueId: lease.issueId, config: driverConfig, runId: identity.runId, creationOperationId: identity.id,
agentId: run!.agentId, adapterType: "paperclip_runner",
...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? { requestedExpiresAt: metadata.runnerTaskRequestedExpiresAt } : {}) };
if (!context) return workers.call(plugin.id, "environmentAcquireLease", params, 30_000);
const settled = context.trackWorker();
let observedResource: Parameters<typeof queueSandboxCreationCleanup>[1] | undefined;
let dispatched = false;
try {
return await context.wait(() => {
dispatched = true;
const observe = async (result: HostToWorkerMethods["environmentAcquireLease"][1]) => {
if (!result.providerLeaseId || result.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
const resource = { id: identity.id, companyId: input.companyId, environmentId: environment!.id,
providerLeaseId: result.providerLeaseId, providerMetadata: result.metadata };
await recordSandboxCreationResource(db, resource);
observedResource = resource;
if (abandoned || !context.active || context.signal.aborted || Date.now() >= context.deadlineAtMs) {
await queueSandboxCreationCleanup(db, resource);
}
};
return recoverableCreation ? creationWorker!.call("environmentAcquireLease", params, 30_000, undefined, settled, context.id, observe)
: workers.call(plugin.id, "environmentAcquireLease", params, 30_000, undefined, settled, context.id, observe);
});
} catch (error) {
if (!dispatched) settled();
abandoned = true;
if (observedResource) await queueSandboxCreationCleanup(db, observedResource);
throw error;
}
});
if (!fresh.providerLeaseId || fresh.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
const {runnerTaskIdentity:_identity,runnerTaskPriorIdentity:_prior,runnerTaskCredentialHandoff:_handoff,
runnerTaskResumeFrom:_resume,runnerTaskPreviousLeaseId:_previous,remoteExecutionTermination:_stopped,...rest}=metadata;
const replacementMetadata={...rest,providerMetadata:fresh.metadata ?? {},runnerTaskReused:false,runnerTaskDispatched:false,
runnerTaskExecutionStopped:false,runnerTaskSessionRetained:false};
try {
context?.assertOpen();
if (!context) await recordSandboxCreationResource(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId });
const expiresAt=fresh.expiresAt ? new Date(fresh.expiresAt) : null;
const deadline=typeof metadata.runnerTaskRequestedExpiresAt === "string" ? Date.parse(metadata.runnerTaskRequestedExpiresAt) : null;
if ((expiresAt && (!Number.isFinite(expiresAt.getTime()) || expiresAt.getTime() <= Date.now())) ||
(deadline !== null && (!Number.isFinite(deadline) || !expiresAt || expiresAt.getTime() > deadline)))
throw new Error("Plugin replacement task expiry is not bounded");
await attachSandboxCreation(db, String(fresh.metadata!.creationOperationId), { companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId }, async tx => {
context?.assertOpen();
const [updated] = await tx.update(environmentLeases).set({providerLeaseId:fresh.providerLeaseId,expiresAt,
metadata:{...replacementMetadata,creationOperationId:fresh.metadata!.creationOperationId},
}).where(and(eq(environmentLeases.id,lease.id),eq(environmentLeases.providerLeaseId,taskId),
eq(environmentLeases.status,"active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`)).returning({id:environmentLeases.id});
if (!updated) throw new Error("Plugin task replacement ownership changed");
const attached = (await environmentService(tx).getLeaseById(updated.id))!;
context?.assertOpen();
return attached;
});
} catch (error) {
if (context) await queueSandboxCreationCleanup(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId, providerMetadata: fresh.metadata });
throw error;
}
return executeEnvironmentTask(db, workers, input);
}
}
if (operation.kind === "complete") {
await settlePluginTaskCredentialHandoff(db, { companyId: input.companyId, leaseId: lease.id, expectedTaskId: taskId }, metadata, async accountIdentity => {
const snapshot = await executeEnvironmentTask(db, workers, { ...input, expectedTaskId: taskId, operation: { kind: "credentials", accountIdentity } });
if (snapshot.kind !== "credentials") throw new Error("Plugin Runner credential snapshot unavailable");
return snapshot.credential;
});
}
const executionContext = context?.pluginSupported && ["describe", "prepare", "check_project_auth", "connection"].includes(operation.kind) ? await context.projection({ pluginId: plugin.id,
leaseId: lease.id, taskId, environmentId: environment!.id }, async () => {
try {
const [identity, configuration] = await Promise.all([
pluginAgentRuntimeIdentity(db, plugin.id, input.companyId, run!.agentId),
pluginRegistryService(db).getConfig(plugin.id, input.companyId),
]);
if (!identity) throw new Error("Missing transport binding");
return { instanceSecrets: identity.instanceSecrets, configuration: configuration?.configJson ?? {} };
} catch { throw new Error("Plugin task transport unavailable"); }
}) : undefined;
const params = {
driverKey: metadata.driverKey, companyId: input.companyId, environmentId: environment?.id ?? null,
issueId: lease.issueId, config: driverConfig,
taskId, runId: run?.id ?? (activeProbe ? String(probe.runId) : null), agentId: run?.agentId ?? null, projectIds,
lease: { providerLeaseId: lease.providerLeaseId, metadata: lease.metadata ?? undefined },
...(workspaceBinding ? { workspace: workspaceBinding } : {}),
...(freshWorkspace && (scopesProjects || retainedTask) ? { fresh: freshWorkspace } : {}), operation,
executionContextSupported: 1 as const,
...(executionContext ? { executionContext } : {}),
};
const timeout = operation.kind === "copilot_metadata" ? Math.max(1, Math.min(40_000, Number(probe?.expiresAtMs) - Date.now()))
: executionContext ? Math.max(1, context!.deadlineAtMs - Date.now())
: driver.sessionHandoffVersion === 1 && (ownership || operation.kind === "describe") ? 90_000 : 15_000;
const result = await callWorker("environmentTask", params, timeout).catch(error => {
// The plugin may have applied the saved claim before its reply was lost.
// Keep the same operation ID; the RPC response below still needs validation.
if (ownership && (!(error instanceof JsonRpcCallError) ||
error.code === PLUGIN_RPC_ERROR_CODES.TIMEOUT || error.code === PLUGIN_RPC_ERROR_CODES.WORKER_UNAVAILABLE)) throw new PluginTaskOwnershipPendingError();
throw error;
});
if (executionContext) context!.assertOpen();
const parsed = parseEnvironmentTaskResult(operation, taskId, result);
if (operation.kind === "submit" && parsed.kind === "admission_rejected") {
const recorded = await db.update(environmentLeases).set({
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskIdentity') || ${JSON.stringify({
runnerTaskAdmissionRejection: { taskId, reason: parsed.reason },
runnerTaskDispatched: false, runnerTaskExecutionStopped: true, runnerTaskSessionRetained: false,
})}::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.providerLeaseId, taskId),
sql`${environmentLeases.metadata}->>'runnerTaskDispatched' = 'true'`,
sql`${environmentLeases.metadata}->'runnerTaskIdentity' = ${JSON.stringify(operation.runner)}::jsonb`,
)).returning({ id: environmentLeases.id });
if (recorded.length !== 1) throw new Error("Plugin Runner admission receipt is stale");
return parsed;
}
if (stoppedSourceCheckpoint && parsed.kind === "state" && isSettledReadyRunnerState(parsed.state)) {
parsed.state = { ...parsed.state, lifecycle: "suspended" };
}
if (sourceRecovery && (operation.kind === "prepare" || operation.kind === "submit")) {
await assertPluginTaskSourceRecovery(db, { companyId: input.companyId, leaseId: lease.id,
context: context instanceof NativeExecutionContext ? context : undefined, operation });
}
if (sourceAttachment && operation.kind === "prepare") {
await assertPluginTaskSourceAttachment(db, { companyId: input.companyId, leaseId: lease.id,
context: context instanceof NativeExecutionContext ? context : undefined, operation });
}
if (portableCheckpoint && operation.kind === "read_state" || portableAttachment && operation.kind === "prepare") {
const identity = operation.kind === "read_state" ? operation.identity : operation.priorIdentity;
const sourceTaskId = (await readyPortableTaskSource(db, input.companyId, lease.id, identity))?.taskId;
if (sourceTaskId !== (portableCheckpoint || portableAttachment)) throw new Error("Plugin Runner session ownership changed");
}
if (operation.kind === "read_file") {
const [current] = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases)
.leftJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, environmentLeases.companyId)))
.where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId))).limit(1);
if (!current || current.lease.status !== "active" || current.run?.status !== "running" ||
current.lease.metadata?.runnerTaskRetiring === true || current.lease.metadata?.runnerTaskExecutionStopped === true ||
current.lease.expiresAt && current.lease.expiresAt.getTime() <= Date.now()) throw new Error("Plugin Runner file lease expired during transfer");
}
if (freshWorkspace && parsed.kind === "description" && parsed.freshWorkspaceVersion !== 1)
throw new Error("Plugin-provided fresh workspace preparation unavailable");
if (operation.kind === "prepare" && parsed.kind === "prepared" && parsed.projectAuthCheckAtSubmit) {
const harness = operation.requirements.provider === "acpx" ? operation.requirements.settings.acpxAgent : operation.requirements.provider;
if (!harness) throw new Error("Plugin Runner prepared provider unavailable");
const provider = harness === "codex" ? "openai" : harness === "claude" ? "anthropic" : harness;
const saved = await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({
runnerTaskProjectAuthValidation: { configurationId: parsed.launch.configurationId, provider, projectIds, atSubmit: true },
})}::jsonb` }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`)).returning({ id: environmentLeases.id });
if (saved.length !== 1) throw new Error("Plugin Runner project validation lease expired");
}
if (operation.kind === "check_project_auth" && parsed.kind === "project_auth") {
if (parsed.configurationId !== operation.configurationId || parsed.provider !== operation.provider) throw new Error("Plugin Runner project validation identity mismatch");
if (parsed.allowed) {
const saved = await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({
runnerTaskProjectAuthValidation: { configurationId: parsed.configurationId, provider: parsed.provider, projectIds },
})}::jsonb` }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`)).returning({ id: environmentLeases.id });
if (saved.length !== 1) throw new Error("Plugin Runner project validation lease expired");
}
}
if ((parsed.kind === "status" || parsed.kind === "copilot_metadata") && parsed.executionStopped === true) {
await recordTaskStop(parsed);
}
if (operation.kind === "complete") {
if (freshWorkspace && metadata.runnerTaskExecutionStopped === true)
await releaseProviderRecoveryTransfers(input.companyId, freshWorkspace);
// The provider attested this task stopped: record the stop receipt now, not when
// the lease's resources are released. A reusable physical task can back several
// run leases, so confirm retirement for every owner.
if (metadata.runnerTaskExecutionStopped === true) {
const owners=lease.leasePolicy === "reuse_by_environment" ? await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId,input.companyId),
eq(environmentLeases.provider,lease.provider!),eq(environmentLeases.providerLeaseId,taskId))) : [lease];
for(const owner of owners) {
const receipt=remoteTerminationReceipt(owner,{providerLeaseId:taskId,state:"stopped"});
if(!receipt) continue;
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({
...(owner.id !== lease.id && owner.status === "retained" ? {status:"released" as const,releasedAt:new Date(),cleanupStatus:"success" as const}:{}),
metadata:sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({remoteExecutionTermination:receipt,runnerTaskExecutionStopped:true})}::jsonb`,
}).where(and(eq(environmentLeases.id,owner.id),eq(environmentLeases.providerLeaseId,taskId))), { companyId: owner.companyId, runId: owner.heartbeatRunId ?? undefined });
}
}
await db.update(environmentLeases).set({
metadata: sql`(case when ${environmentLeases.metadata}->>'runnerTaskPreparationStopped' = 'true'
then coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskStartup'
else coalesce(${environmentLeases.metadata}, '{}'::jsonb) end) || '{"runnerTaskDispatched":false}'::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.providerLeaseId, taskId), sql`${environmentLeases.metadata}->>'runnerTaskExecutionStopped' = 'true'`));
}
return parsed;
} catch (error) {
if (error instanceof PluginTaskOwnershipPendingError) throw error;
const diagnostic = operation.kind === "submit" && operation.resumeFrom ? readEnvironmentTaskErrorDiagnostic(error) : null;
if (diagnostic) throw new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
message: "Plugin Runner source state unavailable",
data: environmentTaskErrorData({ status: diagnostic.httpStatus, code: diagnostic.errorCode, reason: diagnostic.reason }),
});
// Worker errors and schema diagnostics can contain request credentials.
const code = readEnvironmentTaskErrorCode(error);
throw Object.assign(new Error("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying"),
code ? { data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError(code)) } : {});
}
}
/** Failure leaves the dispatch marker and provider handle available to cleanup retries. */
export async function settleDispatchedEnvironmentTask(db: Db, workers: PluginWorkerManager, lease: { id: string; companyId: string },
options: { executionStopObserved?: boolean } = {}): Promise<false | "stopped" | "completed"> {
const [current] = await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"runnerTaskRetiring":true}'::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId)))
.returning({ metadata: environmentLeases.metadata, heartbeatRunId: environmentLeases.heartbeatRunId, taskId: environmentLeases.providerLeaseId });
const startupCleanup = current?.metadata?.runnerTaskStartupCleanup as { requiresPreparationStop?: boolean } | undefined;
const startup = current?.metadata?.runnerTaskStartup;
if (startupCleanup && current?.metadata?.runnerTaskExecutionStopped === true && current.metadata.runnerTaskDispatched === false &&
!current.metadata.runnerTaskAdmissionRejection && !startup &&
(!startupCleanup.requiresPreparationStop || current.metadata.runnerTaskPreparationStopped === true)) return "completed";
if (current?.metadata?.runnerTaskDispatched !== true && !startup && !startupCleanup) {
if (current?.metadata) await settlePluginTaskCredentialHandoff(db, {
companyId: lease.companyId, leaseId: lease.id, expectedTaskId: current.taskId ?? undefined,
}, current.metadata, async accountIdentity => {
const snapshot = await executeEnvironmentTask(db, workers, { companyId: lease.companyId, leaseId: lease.id,
expectedTaskId: current.taskId ?? undefined, operation: { kind: "credentials", accountIdentity } });
if (snapshot.kind !== "credentials") throw new Error("Plugin Runner credential snapshot unavailable");
return snapshot.credential;
});
return current?.metadata?.runnerTaskExecutionStopped === true ? "stopped" : false;
}
const scope = { companyId: lease.companyId, leaseId: lease.id };
try {
const contextId = (startupCleanup as { contextId?: unknown } | undefined)?.contextId ?? (startup as { contextId?: unknown } | undefined)?.contextId;
const draining = typeof contextId === "string" ? workers.drainStartup?.(contextId) : undefined;
void draining?.catch(() => undefined);
const confirmed = (state?: Awaited<ReturnType<typeof executeEnvironmentTask>>) => state?.kind === "status" && state.executionStopped === true &&
(!(startup || startupCleanup?.requiresPreparationStop) || state.preparationStopped === true);
let state = options.executionStopObserved === true
? await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "status" } }) : undefined;
if (!confirmed(state)) {
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "stop" } });
state = await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "status" } });
}
if (!confirmed(state)) throw new Error("Plugin Runner termination is unconfirmed");
await draining;
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "complete" } });
return "completed";
} catch {
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed",
failureReason: "plugin_runner_task_cleanup_unconfirmed", releasedAt: new Date(), updatedAt: new Date(),
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId))), { companyId: lease.companyId, runId: current.heartbeatRunId ?? undefined });
throw new Error("Plugin Runner termination is unconfirmed; the saved lease requires cleanup");
}
}