Files
PaperClipAI/server/src/services/environment-run-orchestrator.ts
T
Nicky LeachandPaperclip 731b52453c fix(environments): verify workspaces on run leases and release settled tasks (#21289)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core owns run leases and delegates remote execution to environment
plugins.
> - A retry must verify its old checkout on the placement that will run
the task.
> - A completed task must still release its provider resources.
> - This change uses the existing lease for workspace verification and
always calls the release hook after settlement.
> - Saved settlement proof lets cleanup retry without repeating
execution work.

## Linked Issues or Issue Description

Refs #21237. That PR records stop proof before lease release. This PR
keeps provider cleanup after settlement and changes workspace
verification sequencing. Both touch task completion; preserve both
changes if the merge order requires conflict resolution.

**What happened?**

Workspace verification ran before a portable task had a lease, so a
provider could not verify the checkout on its selected placement. Core
also skipped the provider release hook after successful task completion.
This could leave resources allocated after the run ended.

**Expected behavior**

Verify the old checkout through the selected run lease. A confirmed
refusal can use the authorized fresh-workspace fallback. A transport
failure must remain retryable. Call the provider release hook after
settlement and retry failed cleanup without changing the run result.

**Steps to reproduce**

1. Retry a task whose prior checkout uses plugin-managed storage and
portable task placement.
2. Verify that the plugin receives the same task lease for verification
and preparation.
3. Complete a task, fail the provider release once, and reconstruct the
cleanup controller.
4. Confirm that cleanup calls only the release hook and retains the
completed run outcome.

## What Changed

- Acquire the logical run lease before provider workspace verification
and reuse it for preparation and execution.
- Preserve the existing session and workspace choices. Recompute
workspace-dependent inputs only after a definite refusal; acquire
credentials and managed runtime inputs once.
- Save the exact refusal and lease before cleanup. Use a
workspace-scoped creation key only for the resulting fresh attempt.
- Replay interrupted verification on the same unlaunched lease. Keep
transport errors retryable and reject replay after execution starts.
- Always call the provider release hook after task settlement. Pass
refreshed settlement proof and route failed releases through the
existing cleanup reaper.
- Add database regressions for call order, restart, retained writers,
refusal, transport errors, and settlement-only cleanup replay.

`plugin-workspaces.ts` grows from 997 to 1,001 lines. The four added
lines extend the existing workspace transaction and lease callbacks.
Keeping the checks there preserves the ownership assertion, provider
result validation, and binding update in one operation. Extracting only
the verification branch would split that transaction without removing a
separate responsibility.

## Verification

- Server TypeScript check passed: `tsc -p server/tsconfig.json
--noEmit`. The complete `server/scripts/typecheck-layers.mjs` check also
passed, including all test-tree layers.
- The first CI typecheck caught a readonly array in a test fixture that
the narrower source check did not include. Commit 8d9ec19e7a corrects
that fixture; all layered checks pass afterward.
- Added a public heartbeat refusal regression in 7c598a4da3. All ten
session-preparation tests pass; independent execution of the refusal
case also passes. It checks successful completion, the saved fresh
workspace, a new conversation, and one credential acquisition.
- Nine database/service suites passed 335 tests: plugin warm tasks,
plugin runner tasks, startup cleanup recovery, credential writeback,
plugin workspaces, session preparation, plugin environments, retry
workspace continuity, and heartbeat workspace preparation.
- After the final restart change, both targeted refusal/restart tests
passed. The primary agent independently reran those cases and the final
server typecheck. After integrating current master, the primary agent's
independent rerun passed all 237 tests across the warm-task,
plugin-workspace, session-preparation, and heartbeat-workspace suites.
- `pnpm verify` returned exit 2 and requested broad CI coverage because
the plugin execution contract documentation changed. No full local
Core-suite pass is claimed.
- Previous-revision CI was green on 8d9ec19e7a: 59 checks passed and 3
were skipped. The first run failed the unchanged legacy-continuation
browser assertion and credential-cleanup server shard. Each passed on
one targeted retry with no code changes. The credential-cleanup
integration suite also passed all 21 tests locally on Node 22. The
browser failure concerned an existing continuation metadata marker; the
server failure cascaded after a cleanup error left a test spy installed.
The successful retries do not establish the cause of the first failures.
- Local database fixtures use simulated provider workers. No live
provider or deployed environment was changed.

- Additional verification for 59c7a1dae8: the layered server typecheck
passed. Independent execution passed all 265 tests across durable task
retries, workspace preparation, and public heartbeat preparation. The
new shutdown regression preserves the fresh Retry choice, waits for
parent cleanup, and prevents the successor from inheriting the parent
lease reference. Same-run recovery retains its receipt. Required CI is
green on 59c7a1dae8: 58 checks passed and 3 were skipped. Standard
review scored 5/5 with no unresolved comments. The separate Superagent
Security Scan is neutral/inconclusive because its workflow failed; a
direct rerun request returned HTTP 404. That scan is not claimed as
passed.

## Risks

- Providers must use the supplied task lease for verification and return
a definite `binding_mismatch` only when the checkout is unconfirmed.
Transport errors retain the retry decision.
- A refused checkout waits for confirmed release before a fresh
allocation. Failed deletion remains owned by the existing cleanup
reaper.
- Task settlement and physical resource release stay separate. Retained
handoff leases do not enter this release path.
- No migration or SDK schema change is required.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) implemented the changes and tests.
The primary Codex agent independently reviewed the connected Core and
plugin paths and ran checks. A separate GPT-6.1 Sol agent reviewed
lifecycle and restart boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue using the bug template labels
- [x] I have not referenced internal or private project details
- [x] My branch name describes the change
- [x] I have added relevant tests and documentation
- [x] I have considered and documented risks
- [x] All required CI gates are green
- [x] Greptile is 5/5 with no unresolved comments
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 21:56:02 -08:00

925 lines
41 KiB
TypeScript

import { RetainedSandboxAbsentError } from "./retained-sandbox-absent-error.js";
import { freshWorkspaceSchema } from "@paperclipai/plugin-sdk";
import { prepareRemoteWorkspace, readRemoteWorkspaceAuthority, remoteWorkspaceRepositoryPath } from "./remote-workspaces.js";
/**
* Centralized environment run orchestrator.
*
* Owns the full environment lifecycle for a heartbeat run:
* 1. Resolve selected environment
* 2. Validate environment is active and allowed
* 3. Acquire or resume lease
* 4. Realize workspace in the environment
* 5. Resolve execution target for the adapter
* 6. Release / retain / fail lease according to policy
* 7. Record activity and operator-visible status
*
* Heartbeat callers delegate to this service instead of inlining
* environment resolution, lease management, workspace realization,
* and transport logic.
*/
import { environmentLeases, type Db } from "@paperclipai/db";
import { and, eq } from "drizzle-orm";
import { remoteExecutionHasReleased } from "./remote-execution-termination.js";
import { remoteRecoveryLeaseMismatch, type RemoteRecoveryLeaseMismatchField } from "./native-runtime/remote-recovery-lease-mismatch.js";
import { EnvironmentProviderUnavailableError, readTransientLeaseAcquisitionFailure } from "./environment-provider-unavailable-error.js";
import { injectLeaseAcquisitionFault } from "./environment-lease-fault-injection.js";
import { findReusableSandboxResumeError } from "./reusable-sandbox-resume-error.js";
import { findPluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js";
import type {
Environment,
EnvironmentLease,
EnvironmentLeasePolicy,
EnvironmentLeaseStatus,
ExecutionWorkspace,
ExecutionWorkspaceConfig,
IssueExecutionWorkspaceSettings,
} from "@paperclipai/shared";
import {
resolveRunnerEnvironmentForRun,
resolveRunnerReconnectGraceMsForRun,
runnerLifecycleDefaultNeedsLeaseProbe,
resolveWarmRunnerIdleTimeoutMsForRun,
} from "./runner-environment-lifecycle.js";
import { assertEnvironmentCompanyBinding, environmentService, toEnvironmentLease } from "./environments.js";
import {
environmentRuntimeService,
buildEnvironmentLeaseContext,
type EnvironmentRuntimeLeaseRecord,
type EnvironmentRuntimeService,
type ProviderResourceDisposition,
} from "./environment-runtime.js";
import { ENVIRONMENT_DRIVER_TRAITS } from "./environment-driver-traits.js";
import type { PluginTaskDescription } from "./environment-task-runtime.js";
import { unlaunchedPortableTaskLease } from "./plugin-task-ownership.js";
import {
resolveEnvironmentExecutionTarget,
resolveEnvironmentExecutionTransport,
} from "./environment-execution-target.js";
import {
adapterExecutionTargetToRemoteSpec,
type AdapterExecutionTarget,
type AdapterRemoteExecutionSpec,
type AdapterWorkspaceRealization,
} from "@paperclipai/adapter-utils/execution-target";
import type { DuplexObservabilityRecorder } from "@paperclipai/adapter-utils/duplex-observability";
import { buildWorkspaceRealizationRequest } from "./workspace-realization.js";
import { executionWorkspaceService } from "./execution-workspaces.js";
import { logActivity } from "./activity-log.js";
import { logger } from "../middleware/logger.js";
import { parseObject } from "../adapters/utils.js";
import type { RealizedExecutionWorkspace } from "./workspace-runtime.js";
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
// ---------------------------------------------------------------------------
// Error types
// ---------------------------------------------------------------------------
export type EnvironmentErrorCode =
| "environment_not_found"
| "environment_inactive"
| "unsupported_environment"
| "unsupported_adapter_environment"
| "probe_failed"
| "lease_acquire_failed"
| "workspace_realization_failed"
| "transport_resolution_failed"
| "lease_release_failed"
| "lease_cleanup_failed";
export class EnvironmentRunError extends Error {
code: EnvironmentErrorCode;
environmentId?: string;
driver?: string;
provider?: string;
cause?: unknown;
constructor(
code: EnvironmentErrorCode,
message: string,
details?: {
environmentId?: string;
driver?: string;
provider?: string;
cause?: unknown;
},
) {
super(message);
this.name = "EnvironmentRunError";
this.code = code;
this.environmentId = details?.environmentId;
this.driver = details?.driver;
this.provider = details?.provider;
this.cause = details?.cause;
}
}
// ---------------------------------------------------------------------------
// Orchestration result types
// ---------------------------------------------------------------------------
export interface EnvironmentAcquisitionResult {
environment: Environment;
lease: EnvironmentLease;
leaseContext: ReturnType<typeof buildEnvironmentLeaseContext>;
executionTransport: Record<string, unknown> | null;
pluginTaskDescription?: PluginTaskDescription;
}
export interface EnvironmentRealizationResult {
lease: EnvironmentLease;
workspaceRealization: Record<string, unknown>;
executionTarget: AdapterExecutionTarget | null;
remoteExecution: AdapterRemoteExecutionSpec | null;
persistedExecutionWorkspace: ExecutionWorkspace | null;
}
export interface EnvironmentReleaseResult {
released: EnvironmentRuntimeLeaseRecord[];
errors: Array<{ leaseId: string; error: unknown }>;
}
function firstNonEmptyLine(text: string | null | undefined): string | null {
if (!text) return null;
for (const rawLine of text.split(/\r?\n/)) {
const line = rawLine.trim();
if (line) return line;
}
return null;
}
function formatProvisionFailureDetail(result: {
exitCode: number | null;
signal?: string | null;
timedOut: boolean;
stdout: string;
stderr: string;
}): string {
if (result.timedOut) {
return "provision command timed out";
}
const signal = typeof result.signal === "string" && result.signal.trim().length > 0
? ` (signal ${result.signal.trim()})`
: "";
const detail = firstNonEmptyLine(result.stderr) ?? firstNonEmptyLine(result.stdout);
const status = `exit code ${result.exitCode ?? "null"}${signal}`;
return detail ? `${status}: ${detail}` : status;
}
// ---------------------------------------------------------------------------
// Service factory
// ---------------------------------------------------------------------------
export function environmentRunOrchestrator(
db: Db,
options: {
pluginWorkerManager?: PluginWorkerManager;
environmentRuntime?: EnvironmentRuntimeService;
} = {},
) {
const environmentsSvc = environmentService(db);
const executionWorkspacesSvc = executionWorkspaceService(db);
const environmentRuntime = options.environmentRuntime ?? environmentRuntimeService(db, {
pluginWorkerManager: options.pluginWorkerManager,
});
/**
* Resolve the selected environment for a run. The caller passes the concrete
* selected environment id plus the built-in local fallback id used to lazily
* ensure the local environment row exists.
*/
async function resolveEnvironment(input: {
companyId: string;
selectedEnvironmentId: string;
localEnvironmentId: string;
}): Promise<Environment> {
const environmentId = input.selectedEnvironmentId || input.localEnvironmentId;
const environment =
environmentId === input.localEnvironmentId
? await environmentsSvc.ensureLocalEnvironment(input.companyId)
: await environmentsSvc.getById(environmentId);
if (!environment) {
throw new EnvironmentRunError("environment_not_found", `Environment "${environmentId}" not found.`, {
environmentId,
});
}
if (environment.status !== "active") {
throw new EnvironmentRunError("environment_inactive", `Environment "${environment.name}" is not active (status: ${environment.status}).`, {
environmentId: environment.id,
driver: environment.driver,
});
}
await assertEnvironmentCompanyBinding(db, environment.id, input.companyId);
return environment;
}
/**
* Acquire an environment lease for a heartbeat run.
* Wraps the runtime driver's acquire call with standardized error handling.
*/
async function acquireLease(input: {
companyId: string;
environment: Environment;
configuredEnvironment?: Environment;
issueId: string | null;
agentId: string;
heartbeatRunId: string;
persistedExecutionWorkspace: Pick<ExecutionWorkspace, "id" | "mode"> | null;
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
runnerTaskProjectIds?: readonly string[];
adapterType: string | null;
warmRunnerIdleTimeoutMs?: number;
runnerReconnectGraceMs?: number;
}): Promise<EnvironmentRuntimeLeaseRecord> {
try {
// Restart-chaos EP1 only; disarmed (always in production) this is one null check.
injectLeaseAcquisitionFault({ companyId: input.companyId, environmentId: input.environment.id,
heartbeatRunId: input.heartbeatRunId, issueId: input.issueId });
return await environmentRuntime.acquireRunLease({ ...input, assertCompanyBinding: true });
} catch (err) {
// Lease acquisition precedes any provider work, so a failure without a
// definite non-transient status (HTTP 5xx/429, errno, acquisition
// deadline, unfinished creation, database error) is retried by the setup
// path with its bounded bootstrap budget. Saved-sandbox resume and
// stopped-worker refusals keep their own, more specific classifications.
const transient = findReusableSandboxResumeError(err) || findPluginWorkerUnavailableError(err)
? null : readTransientLeaseAcquisitionFailure(err, { id: input.heartbeatRunId, companyId: input.companyId });
throw new EnvironmentRunError(
"lease_acquire_failed",
`Failed to acquire lease for environment "${input.environment.name}" (${input.environment.driver}): ${err instanceof Error ? err.message : String(err)}`,
{
environmentId: input.environment.id,
driver: input.environment.driver,
cause: transient
? new EnvironmentProviderUnavailableError({ environmentName: input.environment.name,
driver: input.environment.driver, evidence: transient, cause: err })
: err,
},
);
}
}
/**
* Resolve the execution transport for an adapter based on the acquired lease.
*/
async function resolveTransport(input: {
companyId: string;
adapterType: string;
environment: Environment;
leaseMetadata: Record<string, unknown> | null;
}): Promise<Record<string, unknown> | null> {
try {
return await resolveEnvironmentExecutionTransport({
db,
companyId: input.companyId,
adapterType: input.adapterType,
environment: input.environment,
leaseMetadata: input.leaseMetadata,
});
} catch (err) {
throw new EnvironmentRunError(
"transport_resolution_failed",
`Failed to resolve execution transport for "${input.environment.name}": ${err instanceof Error ? err.message : String(err)}`,
{
environmentId: input.environment.id,
driver: input.environment.driver,
cause: err,
},
);
}
}
/**
* Full acquisition flow: resolve environment, acquire lease, resolve transport.
* This is the primary entry point for heartbeat run setup.
*/
async function acquireForRun(input: {
companyId: string;
selectedEnvironmentId: string;
localEnvironmentId: string;
adapterType: string;
adapterConfig?: Record<string, unknown>;
admittedLifecycleMode?: "warm" | "per_turn";
managedAiCredentialIdentity?: string;
issueId: string | null;
heartbeatRunId: string;
agentId: string;
persistedExecutionWorkspace: Pick<ExecutionWorkspace, "id" | "mode"> | null;
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
runnerTaskProjectIds?: readonly string[];
runnerTaskResumeSessionId?: string | null;
/** Existing live runner recovery must use its original active lease, including ephemeral leases. */
reattachRemoteLease?: { kind?: "sandbox"; leaseId: string; providerLeaseId: string; remoteCwd: string }
| { kind: "plugin_task"; leaseId: string; preparation?: true };
/** A new controller must stop the previous plugin task before allocating a replacement. */
recoverPluginTask?: boolean;
replacementWorkspaceCreation?: string;
}): Promise<EnvironmentAcquisitionResult> {
// Step 1: Resolve environment
const selectedEnvironment = await resolveEnvironment({
companyId: input.companyId,
selectedEnvironmentId: input.selectedEnvironmentId,
localEnvironmentId: input.localEnvironmentId,
});
// The remote-sandbox warm default needs a provider that keeps reusable
// leases. Probe it only for a run the default can apply to; every other
// run resolves without it, exactly as before.
const reusableLeaseSupported = runnerLifecycleDefaultNeedsLeaseProbe(
selectedEnvironment, input.adapterType, input.adapterConfig, input.admittedLifecycleMode, input.managedAiCredentialIdentity,
) && await Promise.resolve(environmentRuntime.providerSupportsReusableLeases?.(selectedEnvironment)).then(
(supported) => supported === true, () => false);
const lifecycleDefaultContext = { reusableLeaseSupported, managedAiCredentialIdentity: input.managedAiCredentialIdentity };
const environment = resolveRunnerEnvironmentForRun(
selectedEnvironment, input.adapterType, input.adapterConfig, input.admittedLifecycleMode, input.managedAiCredentialIdentity,
lifecycleDefaultContext,
);
if (input.recoverPluginTask && environment.driver === "plugin" && input.adapterType === "paperclip_runner") {
const prior = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.heartbeatRunId, input.heartbeatRunId),
));
if (prior.some(lease => lease.metadata?.runnerTaskLease !== true || lease.environmentId !== environment.id ||
lease.issueId !== input.issueId || lease.executionWorkspaceId !== (input.persistedExecutionWorkspace?.id ?? null) ||
lease.metadata?.agentId !== input.agentId)) throw new Error("native_plugin_recovery_lease_mismatch");
const errors: unknown[] = [];
await environmentRuntime.releaseRunLeases(input.heartbeatRunId, "released", (_id, error) => { errors.push(error); });
// Same-run task resume requires every prior lease released with its receipt;
// a stop recorded before a held release is not enough here.
if (errors.length || (prior.length && !await remoteExecutionHasReleased(db, input.companyId, input.heartbeatRunId))) {
throw new Error("native_plugin_recovery_termination_unconfirmed", { cause: errors[0] });
}
}
// Step 2: Acquire lease
const warmRunnerIdleTimeoutMs = resolveWarmRunnerIdleTimeoutMsForRun(
selectedEnvironment, input.adapterType, input.adapterConfig, input.admittedLifecycleMode, lifecycleDefaultContext,
);
const runnerReconnectGraceMs = input.adapterType
? resolveRunnerReconnectGraceMsForRun(selectedEnvironment, input.adapterType)
: undefined;
const acquisitionInput = {
companyId: input.companyId,
environment,
configuredEnvironment: selectedEnvironment,
issueId: input.issueId,
agentId: input.agentId,
heartbeatRunId: input.heartbeatRunId,
persistedExecutionWorkspace: input.persistedExecutionWorkspace,
executionWorkspaceSettings: input.executionWorkspaceSettings,
runnerTaskProjectIds: input.runnerTaskProjectIds,
runnerTaskResumeSessionId: input.runnerTaskResumeSessionId,
replacementWorkspaceCreation: input.replacementWorkspaceCreation,
adapterType: input.adapterType ?? null,
warmRunnerIdleTimeoutMs,
runnerReconnectGraceMs,
};
let leaseRecord: EnvironmentRuntimeLeaseRecord;
const expected = input.reattachRemoteLease;
if (expected) {
// Reattachment keeps the original lease, including ephemeral leases.
// Never reacquire: acquisition can replace a non-reusable sandbox.
const replayPreparation = expected.kind === "plugin_task" && expected.preparation;
const lease = replayPreparation
? await db.select().from(environmentLeases).where(and(eq(environmentLeases.id, expected.leaseId),
eq(environmentLeases.companyId, input.companyId), unlaunchedPortableTaskLease()))
.then(([row]) => row ? toEnvironmentLease(row) : null)
: await environmentsSvc.getLeaseById(expected.leaseId);
const mismatch: RemoteRecoveryLeaseMismatchField | null = !lease ? "lease"
: lease.companyId !== input.companyId || lease.environmentId !== environment.id ||
lease.heartbeatRunId !== input.heartbeatRunId || lease.issueId !== input.issueId ||
lease.executionWorkspaceId !== (input.persistedExecutionWorkspace?.id ?? null) ||
lease.metadata?.agentId !== input.agentId ? "lease_scope"
: lease.status !== "active" || lease.releasedAt !== null || lease.cleanupStatus !== null ? "lease_status"
: lease.expiresAt !== null && new Date(lease.expiresAt).getTime() <= Date.now() ? "lease_expired"
: expected.kind === "plugin_task"
? environment.driver !== "plugin" ? "environment_driver"
: input.adapterType !== "paperclip_runner" ? "adapter_type"
: replayPreparation ? environment.config.pluginKey !== lease.metadata?.pluginKey ||
environment.config.driverKey !== lease.metadata?.driverKey ? "plugin_task" : null
: lease.metadata?.runnerTaskLease !== true || lease.metadata?.runnerTaskDispatched !== true
|| lease.metadata?.runnerTaskExecutionStopped === true ? "plugin_task" : null
: environment.driver !== "sandbox" ? "environment_driver"
: lease.provider !== environment.config.provider ? "provider"
: lease.providerLeaseId !== expected.providerLeaseId ? "provider_lease_id"
: lease.metadata?.remoteCwd !== expected.remoteCwd ? "remote_cwd" : null;
if (mismatch || !lease) {
throw remoteRecoveryLeaseMismatch("environment_acquire", mismatch ?? "lease");
}
if (expected.kind !== "plugin_task") {
// The provider may have archived this retained sandbox while its
// controller was offline. Ready the exact lease under its lifecycle
// budget before short preparation commands (such as project-auth
// checks) can run. This API cannot acquire a replacement.
const resumed = await environmentRuntime.resumeRunLease({
environment, lease, requestedCwd: expected.remoteCwd,
...(warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs } : {}),
...(runnerReconnectGraceMs !== undefined ? { runnerReconnectGraceMs } : {}),
}).catch((error: unknown) => {
// Confirmed provider absence is a named lease mismatch; other
// failures (for example a transient refusal) keep their own error.
if (error instanceof RetainedSandboxAbsentError) throw remoteRecoveryLeaseMismatch("environment_resume", "lease_expired");
throw error;
});
if (!resumed) throw remoteRecoveryLeaseMismatch("environment_resume", "lease");
if (resumed.providerLeaseId !== expected.providerLeaseId) {
throw remoteRecoveryLeaseMismatch("environment_resume", "provider_lease_id");
}
if (resumed.metadata?.expired === true) throw remoteRecoveryLeaseMismatch("environment_resume", "lease_expired");
// Resume confirms allocation readiness. Its optional metadata may
// describe the allocation root; keep the validated lease's workspace.
}
leaseRecord = { environment, lease, leaseContext: buildEnvironmentLeaseContext(input) };
} else {
leaseRecord = await acquireLease(acquisitionInput);
}
// Step 3: Log lease acquisition activity
await logActivity(db, {
companyId: input.companyId,
actorType: "agent",
actorId: input.agentId,
agentId: input.agentId,
runId: input.heartbeatRunId,
action: "environment.lease_acquired",
entityType: "environment_lease",
entityId: leaseRecord.lease.id,
issueId: input.issueId,
details: {
environmentId: environment.id,
driver: environment.driver,
leasePolicy: leaseRecord.lease.leasePolicy,
provider: leaseRecord.lease.provider,
executionWorkspaceId: leaseRecord.leaseContext.executionWorkspaceId,
issueId: input.issueId,
networkEgress: input.executionWorkspaceSettings?.networkEgress ?? null,
},
});
// Step 4: Resolve execution transport
const executionTransport = environment.driver === "plugin" && input.adapterType === "paperclip_runner" ? null : await resolveTransport({
companyId: input.companyId,
adapterType: input.adapterType,
environment,
leaseMetadata: leaseRecord.lease.metadata,
});
return {
environment,
lease: leaseRecord.lease,
leaseContext: leaseRecord.leaseContext,
executionTransport,
};
}
async function confirmWorkspaceTaskPlacement(acquisition: EnvironmentAcquisitionResult, input: {
companyId: string;
environmentId: string;
workspaceId: string;
signal?: AbortSignal;
}) {
const { environment, lease } = acquisition;
const driverKey = environment.config?.driverKey;
if (environment.id !== input.environmentId || typeof driverKey !== "string" || !lease.providerLeaseId) {
throw new Error("Workspace Launcher placement changed");
}
const description = await environmentRuntime.task({
companyId: input.companyId, leaseId: lease.id, operation: { kind: "describe" }, ownershipSignal: input.signal,
});
const current = await environmentsSvc.getLeaseById(lease.id);
if (description.kind !== "description" || !current || current.providerLeaseId !== description.taskId ||
current.status !== "active" || current.environmentId !== environment.id ||
current.executionWorkspaceId !== input.workspaceId) {
throw new Error("Workspace Launcher lease is no longer active");
}
return {
acquisition: { ...acquisition, lease: current, pluginTaskDescription: description },
taskLease: { driverKey, environmentId: environment.id, providerLeaseId: description.taskId },
};
}
/**
* Realize workspace in the environment and resolve the execution target.
*
* After lease acquisition, this method:
* 1. Builds a workspace realization request
* 2. Calls the environment runtime driver to realize the workspace
* 3. Persists realization metadata on the lease and execution workspace
* 4. Resolves the adapter execution target (local/ssh/sandbox)
*
* Returns the updated lease, realization metadata, and the execution
* target spec that the adapter needs to run.
*/
async function realizeForRun(input: {
environment: Environment;
lease: EnvironmentLease;
adapterType: string;
companyId: string;
issueId: string | null;
heartbeatRunId: string;
executionWorkspace: RealizedExecutionWorkspace;
effectiveExecutionWorkspaceMode: string | null;
persistedExecutionWorkspace: ExecutionWorkspace | null;
runnerTaskProjectIds?: readonly string[];
signal?: AbortSignal;
assertOwned?: () => Promise<void>;
/**
* The host duplex observability recorder for this run. The orchestrator threads
* it to `resolveEnvironmentExecutionTarget`, which stamps it on the sandbox
* target. Absent keeps the safe no-op default in the bridge.
*/
duplexObservabilityRecorder?: DuplexObservabilityRecorder | null;
}): Promise<EnvironmentRealizationResult> {
const {
environment,
adapterType,
companyId,
issueId,
heartbeatRunId,
executionWorkspace,
effectiveExecutionWorkspaceMode,
} = input;
let { lease, persistedExecutionWorkspace } = input;
const remoteAuthority = readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata);
if (remoteAuthority && persistedExecutionWorkspace) {
if (adapterType !== "paperclip_runner" || environment.driver !== "sandbox") {
throw new EnvironmentRunError("unsupported_adapter_environment", "Remote workspace storage requires a qualified native sandbox");
}
const target = await resolveEnvironmentExecutionTarget({
db, companyId, adapterType, environment, leaseId: lease.id, leaseMetadata: lease.metadata,
lease, environmentRuntime, duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
});
if (!target || target.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask) {
throw new EnvironmentRunError("transport_resolution_failed", "Remote workspace storage requires a sandbox file transport");
}
const prepared = await prepareRemoteWorkspace({
db, workspace: persistedExecutionWorkspace, runId: heartbeatRunId, lease, target,
source: remoteAuthority.source,
assertOwned: input.assertOwned,
provisionCommand: persistedExecutionWorkspace.config?.provisionCommand,
runtimeProvisionCommand: persistedExecutionWorkspace.config?.runtimeProvisionCommand,
});
lease = prepared.lease;
const workspaceRealization = {
version: 1, mode: "in_place", authoritativeRoot: prepared.remoteRoot,
pathAliases: [], outboundRestorePaths: [], provider: lease.provider,
environmentId: environment.id, leaseId: lease.id, providerLeaseId: lease.providerLeaseId,
additional: (remoteAuthority.source.repositories ?? []).filter(repository => repository.readOnly)
.map(repository => ({ path: remoteWorkspaceRepositoryPath(prepared.remoteRoot, repository),
projectId: repository.projectId ?? null, projectWorkspaceId: repository.projectWorkspaceId ?? null,
repoUrl: repository.repoUrl, repoRef: repository.ref })),
};
const updatedLease = await environmentsSvc.updateLeaseMetadata(lease.id, {
...lease.metadata, remoteCwd: prepared.remoteRoot, workspaceRealization,
});
if (!updatedLease) throw new Error("Remote workspace lease disappeared during preparation");
lease = updatedLease;
persistedExecutionWorkspace = await executionWorkspacesSvc.getStoredById(companyId, prepared.workspace.id);
if (!persistedExecutionWorkspace) throw new Error("Remote workspace disappeared during preparation");
input.executionWorkspace.cwd = prepared.remoteRoot;
const executionTarget: AdapterExecutionTarget = {
...target, remoteCwd: prepared.remoteRoot,
workspaceRealization: { mode: "in_place", authoritativeRoot: prepared.remoteRoot,
pathAliases: [], outboundRestorePaths: [] },
};
return { lease, workspaceRealization, executionTarget,
remoteExecution: adapterExecutionTargetToRemoteSpec(executionTarget), persistedExecutionWorkspace };
}
if (executionWorkspace.providerBinding) {
const binding = executionWorkspace.providerBinding;
if (environment.driver !== "plugin" || adapterType !== "paperclip_runner" || lease.metadata?.pluginId !== binding.pluginId || !await environmentRuntime.supportsTasks({ lease })) {
throw new EnvironmentRunError("workspace_realization_failed", "The execution plugin cannot access this managed workspace.", { environmentId: environment.id });
}
const updated = await environmentsSvc.updateLeaseMetadata(lease.id, { ...lease.metadata, workspaceProvider: binding });
if (!updated) throw new Error("Workspace execution lease disappeared");
lease = updated;
}
const freshWorkspace = freshWorkspaceSchema.safeParse(persistedExecutionWorkspace?.metadata?.workspaceProviderFresh ??
persistedExecutionWorkspace?.metadata?.runnerTaskFreshWorkspace);
if (environment.driver === "plugin" && freshWorkspace.success) {
const updated = await environmentsSvc.updateLeaseMetadata(lease.id, { ...lease.metadata, freshWorkspace: freshWorkspace.data });
if (!updated) throw new Error("Workspace execution lease disappeared");
lease = updated;
}
// Step 1: Build workspace realization request
const workspaceRealizationRequest = buildWorkspaceRealizationRequest({
adapterType,
companyId,
environmentId: environment.id,
executionWorkspaceId: persistedExecutionWorkspace?.id ?? null,
issueId,
heartbeatRunId,
requestedMode: persistedExecutionWorkspace?.mode ?? effectiveExecutionWorkspaceMode,
workspace: executionWorkspace,
workspaceConfig: persistedExecutionWorkspace?.config ?? null,
});
// Step 2: Realize workspace in the environment via the runtime driver
let workspaceRealization: Record<string, unknown> = {};
let realizedWorkspaceCwd: string | null = null;
const pluginTaskWorkspace = environment.driver === "plugin" && adapterType === "paperclip_runner";
if (pluginTaskWorkspace && !await environmentRuntime.supportsTasks({ lease })) {
throw new EnvironmentRunError("unsupported_adapter_environment", "Plugin environment does not provide native Runner task execution");
}
if (ENVIRONMENT_DRIVER_TRAITS[environment.driver].realizesWorkspace) {
try {
const remoteCwd =
typeof lease.metadata?.remoteCwd === "string" && lease.metadata.remoteCwd.trim().length > 0
? lease.metadata.remoteCwd
: undefined;
const workspaceRealizationResult = await environmentRuntime.realizeWorkspace({
environment,
lease,
workspace: {
localPath: executionWorkspace.cwd,
remotePath: remoteCwd,
mode: persistedExecutionWorkspace?.mode ?? effectiveExecutionWorkspaceMode ?? undefined,
metadata: {
workspaceRealizationRequest,
},
},
});
realizedWorkspaceCwd =
typeof workspaceRealizationResult.cwd === "string" && workspaceRealizationResult.cwd.trim().length > 0
? workspaceRealizationResult.cwd.trim()
: null;
workspaceRealization = parseObject(workspaceRealizationResult.metadata?.workspaceRealization);
} catch (err) {
throw new EnvironmentRunError(
"workspace_realization_failed",
`Failed to realize workspace for environment "${environment.name}" (${environment.driver}): ${err instanceof Error ? err.message : String(err)}`,
{
environmentId: environment.id,
driver: environment.driver,
cause: err,
},
);
}
}
const provisionCommand = workspaceRealizationRequest.runtimeOverlay.provisionCommand?.trim() ?? "";
const realizedCwd =
realizedWorkspaceCwd ??
(typeof lease.metadata?.remoteCwd === "string" && lease.metadata.remoteCwd.trim().length > 0
? lease.metadata.remoteCwd.trim()
: executionWorkspace.cwd);
// The host `provisionCommand` runs on the host worktree during the
// `workspace_provision` step, before the run reaches the environment.
// A `sandbox`-driver environment does not receive the repo tree here. The
// sandbox driver `realizeWorkspace` step only creates the remote folder.
// The adapter uploads the provisioned tree later, in its `stage.sync` step.
// So the host command must not run inside the still-empty sandbox; it fails
// there (exit 127). Skip the step for `sandbox`, and keep the existing skip
// for `local`. Keep the step for `ssh`, which runs the command on the
// remote host that shares the workspace path.
const driverSkipsHostProvision =
environment.driver === "local" || environment.driver === "sandbox" || pluginTaskWorkspace;
if (provisionCommand && environment.driver === "sandbox") {
logger.info(
{
environmentId: environment.id,
driver: environment.driver,
issueId,
heartbeatRunId,
},
"Skip host provisionCommand for sandbox-driver environment; the adapter stage.sync step delivers the provisioned tree",
);
}
if (provisionCommand && !executionWorkspace.providerBinding && !driverSkipsHostProvision) {
try {
const provisionResult = await environmentRuntime.execute({
environment,
lease,
command: "bash",
args: ["-lc", provisionCommand],
cwd: realizedCwd,
env: {
SHELL: "/bin/bash",
},
timeoutMs: 300_000,
// The provision command runs before the run opens its trace root, so it
// carries no run parent. A sandbox provider that opens a persistent
// session on the first command must not open the session here, or the
// session-setup span loses its parent and the span backend drops it.
// Bypass the session for this command; the session opens on the first
// in-run command instead, whose setup span parents to the run trace.
bypassSession: true,
});
if (provisionResult.exitCode !== 0 || provisionResult.timedOut) {
throw new Error(formatProvisionFailureDetail(provisionResult));
}
} catch (err) {
throw new EnvironmentRunError(
"workspace_realization_failed",
`Failed to provision workspace for environment "${environment.name}" (${environment.driver}): ${err instanceof Error ? err.message : String(err)}`,
{
environmentId: environment.id,
driver: environment.driver,
cause: err,
},
);
}
}
// Step 3: Persist realization metadata on lease and execution workspace
if (Object.keys(workspaceRealization).length > 0) {
const nextLeaseMetadata = {
...(lease.metadata ?? {}),
workspaceRealization,
};
const updatedLease = await environmentsSvc.updateLeaseMetadata(lease.id, nextLeaseMetadata);
if (updatedLease) {
lease = updatedLease;
}
if (persistedExecutionWorkspace) {
const updatedEw = await executionWorkspacesSvc.update(persistedExecutionWorkspace.id, {
metadata: {
...(persistedExecutionWorkspace.metadata ?? {}),
workspaceRealizationRequest,
workspaceRealization,
},
});
if (updatedEw) {
persistedExecutionWorkspace = updatedEw;
}
}
}
// Step 4: Resolve execution target for the adapter
let executionTarget: AdapterExecutionTarget | null;
try {
executionTarget = await resolveEnvironmentExecutionTarget({
db,
companyId,
adapterType,
environment,
leaseId: lease.id,
leaseMetadata: (lease.metadata as Record<string, unknown> | null) ?? null,
lease,
environmentRuntime,
signal: input.signal,
duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
});
// The native plugin task bridge owns guest paths. Never fall back to host execution.
if (executionWorkspace.providerBinding && (executionTarget?.kind !== "remote" ||
!("runnerTask" in executionTarget) || typeof executionTarget.runnerTask !== "function")) {
throw new Error("Managed workspace execution requires the native plugin task bridge");
}
const realizationMode = pluginTaskWorkspace || workspaceRealization.mode === "in_place" ? "in_place" : "copy";
const authoritativeRoot =
typeof workspaceRealization.authoritativeRoot === "string" && workspaceRealization.authoritativeRoot.trim().length > 0
? workspaceRealization.authoritativeRoot.trim()
: realizedCwd;
const workspaceTargetMetadata: AdapterWorkspaceRealization = {
mode: realizationMode,
authoritativeRoot,
pathAliases: Array.isArray(workspaceRealization.pathAliases)
? workspaceRealization.pathAliases.filter(
(entry): entry is { path: string; target: string } =>
typeof entry === "object" && entry !== null &&
typeof (entry as { path?: unknown }).path === "string" &&
typeof (entry as { target?: unknown }).target === "string",
)
: [],
outboundRestorePaths: Array.isArray(workspaceRealization.outboundRestorePaths)
? workspaceRealization.outboundRestorePaths.filter((entry): entry is string => typeof entry === "string")
: [],
};
if (executionTarget) {
executionTarget = {
...executionTarget,
...(executionTarget.kind === "remote" && executionTarget.transport === "sandbox" && executionTarget.runnerTask
? { runnerTaskProjectIds: [...new Set([executionWorkspace.projectId,
...input.runnerTaskProjectIds ?? [],
...(executionWorkspace.additionalWorkspaces ?? []).map(workspace => workspace.projectId)]
.filter((id): id is string => Boolean(id)))] } : {}),
...(executionTarget.kind === "remote" && realizationMode === "in_place"
? { remoteCwd: authoritativeRoot }
: {}),
workspaceRealization: workspaceTargetMetadata,
} as AdapterExecutionTarget;
}
} catch (err) {
throw new EnvironmentRunError(
"transport_resolution_failed",
`Failed to resolve execution target for "${environment.name}": ${err instanceof Error ? err.message : String(err)}`,
{
environmentId: environment.id,
driver: environment.driver,
cause: err,
},
);
}
return {
lease,
workspaceRealization,
executionTarget,
remoteExecution: adapterExecutionTargetToRemoteSpec(executionTarget),
persistedExecutionWorkspace,
};
}
/**
* Release all active leases for a heartbeat run.
* Tracks cleanup status per lease. Errors during individual lease release
* are captured but do not prevent other leases from being released.
* The original run failure (if any) is never hidden by cleanup errors.
*/
async function releaseForRun(input: {
heartbeatRunId: string;
companyId: string;
agentId: string;
status?: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed">;
failureReason?: string;
/** Explicit Stop during adapter startup; never used for ordinary cleanup. */
cancelActiveWork?: boolean;
respectCleanupDeadlines?: boolean;
/** Explicit paperclip_runner resource lifecycle. Omitted for legacy adapters. */
providerResourceDisposition?: ProviderResourceDisposition;
nativeLifecycleTelemetry?: {
provider: string;
harness: string;
lifecycleMode: "per_turn" | "warm";
sandboxResource:
| "keep_running"
| "stop_and_reuse"
| "destroy_after_turn";
};
}): Promise<EnvironmentReleaseResult> {
const status = input.status ?? "released";
const result: EnvironmentReleaseResult = { released: [], errors: [] };
let releasedLeases: EnvironmentRuntimeLeaseRecord[];
try {
const releaseArgs: Parameters<typeof environmentRuntime.releaseRunLeases> = [
input.heartbeatRunId,
status,
(leaseId, error) => result.errors.push({ leaseId, error }),
input.providerResourceDisposition,
];
if (input.cancelActiveWork) releaseArgs[4] = true;
if (input.respectCleanupDeadlines) releaseArgs[5] = { respectCleanupDeadlines: true };
releasedLeases = await environmentRuntime.releaseRunLeases(...releaseArgs);
} catch (err) {
result.errors.push({ leaseId: "*", error: err });
return result;
}
for (const released of releasedLeases) {
try {
await logActivity(db, {
companyId: input.companyId,
actorType: "agent",
actorId: input.agentId,
agentId: input.agentId,
runId: input.heartbeatRunId,
action: "environment.lease_released",
entityType: "environment_lease",
entityId: released.lease.id,
issueId: released.lease.issueId,
details: {
environmentId: released.lease.environmentId,
driver: released.environment.driver,
leasePolicy: released.lease.leasePolicy,
provider: released.lease.provider,
executionWorkspaceId: released.lease.executionWorkspaceId,
issueId: released.lease.issueId,
status: released.lease.status,
cleanupStatus: released.lease.cleanupStatus,
failureReason: input.failureReason ?? released.lease.failureReason,
providerResourceDisposition:
input.providerResourceDisposition ?? "legacy_default",
},
});
} catch {
// Activity logging failure should not block lease release
}
result.released.push(released);
}
return result;
}
return {
resolveEnvironment,
acquireLease,
resolveTransport,
acquireForRun,
confirmWorkspaceTaskPlacement,
realizeForRun,
releaseForRun,
// Expose the underlying runtime for cases that need direct driver access
runtime: environmentRuntime,
};
}
export type EnvironmentRunOrchestrator = ReturnType<typeof environmentRunOrchestrator>;