Files
PaperClipAI/server/src/services/environment-run-orchestrator.ts
T
DottaandPaperclip 08be8be414 fix(recovery): report exact retained sandbox resumes as resumed (#21259)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A controller restart must keep the same remote execution owner.
> - A retained sandbox lease stores the result of its first acquisition.
> - That result can say replacement after the same allocation resumes.
> - The Runner then rejects a valid resume as a changed sandbox.
> - This pull request reports the verified operation as resumed.
> - The saved acquisition history and identity checks stay intact.

## Linked Issues or Issue Description

Refs #18934. That change makes the retained sandbox ready before
preparation. This change corrects the acquisition receipt after that
resume.

**What happened?**
A remote native run resumed the exact retained sandbox after a
controller restart. The lease still carried a historical replacement
receipt. Runner verification rejected it with
`native_remote_recovery_lease_mismatch`, stage `runner_verification`,
field `replacement_sandbox`.

**What did you expect to happen?**
Report the verified exact-allocation operation as resumed. Preserve the
original saved history. Reject a changed, absent or expired allocation.

**Steps to reproduce**
Save an active lease with historical replacement metadata. Resume that
exact provider allocation through `acquireForRun`. Resolve its execution
target. Before the fix, the target still reports replacement and Runner
verification refuses it.

**Deployment mode**
Native Runner on a remote sandbox. The reported provider was Daytona.

## What Changed

- Return a separate operation receipt after the exact retained
allocation passes the existing resume checks.
- Pass the receipt through heartbeat preparation and apply it only to
the execution target.
- Keep the saved lease metadata and workspace unchanged.
- Test historical created, replacement and resumed receipts. Test
ordinary and remote-authority realization through the real execution
target resolver. Check every saved metadata write.
- Document the difference between saved acquisition history and the
current resume operation.

## Verification

- Head: `74d03e8ddadd1274891bea99a10455795c500557`.
- Red: three historical receipt cases fail without the resume receipt.
- Review regression red: realization writes resumed over the historical
replacement receipt. The saved metadata loses the previous allocation
and cause.
- Green: all 69 orchestrator tests pass. Ordinary and remote-authority
realization preserve history in every metadata write and return resumed
in the target.
- Green: 135 focused orchestrator and remote controller adoption cases
pass. Identity and changed-allocation refusals remain effective.
- Server typecheck passes, including its TypeScript build dependencies
and test layers.
- `git diff --check` passes.
- `pnpm verify` exits 2 because the execution-semantics document needs
broad CI. Required CI supplies that coverage.
- Linux x64, Node 25.9.0. Provider boundaries are simulated.
- The first post-merge attempt could not collect tests because the
compiled plugin SDK lacked the new taskLease schema. Shared and plugin
SDK builds passed, then the unchanged selection passed.
- Full preflight stopped because Cargo is absent. Dependency
installation completed. The later server TypeScript build and typecheck
passed.
- Live retained Runner reattachment is not verified. This workspace has
no provider credentials. No rollout or original-task recovery by this
patch is claimed.
- Required current-head CI and automated review remain pending.

## Risks

A wrong receipt could hide an actual replacement. The projection occurs
only after the existing company, task, run, agent, workspace, provider
allocation, expiry and directory checks pass. The provider must resume
the exact allocation. Runner process and identity verification still
runs. No replacement is acquired and no saved acquisition history is
changed. The heartbeat change only carries the typed receipt between the
owning orchestrator stages. Callers without a receipt keep their
existing behavior.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
execution and tool use. The exact backend snapshot and context window
are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:55:55 -06:00

938 lines
42 KiB
TypeScript

import { RetainedSandboxAbsentError } from "./retained-sandbox-absent-error.js";
import { freshWorkspaceSchema } from "@paperclipai/plugin-sdk";
import { prepareRemoteWorkspace, readRemoteWorkspaceAuthority, remoteWorkspaceRepositoryPath } from "./remote-workspaces.js";
/**
* Centralized environment run orchestrator.
*
* Owns the full environment lifecycle for a heartbeat run:
* 1. Resolve selected environment
* 2. Validate environment is active and allowed
* 3. Acquire or resume lease
* 4. Realize workspace in the environment
* 5. Resolve execution target for the adapter
* 6. Release / retain / fail lease according to policy
* 7. Record activity and operator-visible status
*
* Heartbeat callers delegate to this service instead of inlining
* environment resolution, lease management, workspace realization,
* and transport logic.
*/
import { environmentLeases, type Db } from "@paperclipai/db";
import { and, eq } from "drizzle-orm";
import { remoteExecutionHasReleased } from "./remote-execution-termination.js";
import { remoteRecoveryLeaseMismatch, type RemoteRecoveryLeaseMismatchField } from "./native-runtime/remote-recovery-lease-mismatch.js";
import { EnvironmentProviderUnavailableError, readTransientLeaseAcquisitionFailure } from "./environment-provider-unavailable-error.js";
import { injectLeaseAcquisitionFault } from "./environment-lease-fault-injection.js";
import { findReusableSandboxResumeError } from "./reusable-sandbox-resume-error.js";
import { findPluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js";
import type {
Environment,
EnvironmentLease,
EnvironmentLeasePolicy,
EnvironmentLeaseStatus,
ExecutionWorkspace,
ExecutionWorkspaceConfig,
IssueExecutionWorkspaceSettings,
} from "@paperclipai/shared";
import {
resolveRunnerEnvironmentForRun,
resolveRunnerReconnectGraceMsForRun,
runnerLifecycleDefaultNeedsLeaseProbe,
resolveWarmRunnerIdleTimeoutMsForRun,
} from "./runner-environment-lifecycle.js";
import { assertEnvironmentCompanyBinding, environmentService, toEnvironmentLease } from "./environments.js";
import {
environmentRuntimeService,
buildEnvironmentLeaseContext,
type EnvironmentRuntimeLeaseRecord,
type EnvironmentRuntimeService,
type ProviderResourceDisposition,
} from "./environment-runtime.js";
import { ENVIRONMENT_DRIVER_TRAITS } from "./environment-driver-traits.js";
import type { PluginTaskDescription } from "./environment-task-runtime.js";
import { unlaunchedPortableTaskLease } from "./plugin-task-ownership.js";
import {
resolveEnvironmentExecutionTarget,
resolveEnvironmentExecutionTransport,
} from "./environment-execution-target.js";
import {
adapterExecutionTargetToRemoteSpec,
type AdapterExecutionTarget,
type AdapterRemoteExecutionSpec,
type AdapterWorkspaceRealization,
type SandboxLeaseAcquisition,
} from "@paperclipai/adapter-utils/execution-target";
import type { DuplexObservabilityRecorder } from "@paperclipai/adapter-utils/duplex-observability";
import { buildWorkspaceRealizationRequest } from "./workspace-realization.js";
import { executionWorkspaceService } from "./execution-workspaces.js";
import { logActivity } from "./activity-log.js";
import { logger } from "../middleware/logger.js";
import { parseObject } from "../adapters/utils.js";
import type { RealizedExecutionWorkspace } from "./workspace-runtime.js";
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
// ---------------------------------------------------------------------------
// Error types
// ---------------------------------------------------------------------------
export type EnvironmentErrorCode =
| "environment_not_found"
| "environment_inactive"
| "unsupported_environment"
| "unsupported_adapter_environment"
| "probe_failed"
| "lease_acquire_failed"
| "workspace_realization_failed"
| "transport_resolution_failed"
| "lease_release_failed"
| "lease_cleanup_failed";
export class EnvironmentRunError extends Error {
code: EnvironmentErrorCode;
environmentId?: string;
driver?: string;
provider?: string;
cause?: unknown;
constructor(
code: EnvironmentErrorCode,
message: string,
details?: {
environmentId?: string;
driver?: string;
provider?: string;
cause?: unknown;
},
) {
super(message);
this.name = "EnvironmentRunError";
this.code = code;
this.environmentId = details?.environmentId;
this.driver = details?.driver;
this.provider = details?.provider;
this.cause = details?.cause;
}
}
// ---------------------------------------------------------------------------
// Orchestration result types
// ---------------------------------------------------------------------------
export interface EnvironmentAcquisitionResult {
environment: Environment;
lease: EnvironmentLease;
leaseContext: ReturnType<typeof buildEnvironmentLeaseContext>;
executionTransport: Record<string, unknown> | null;
pluginTaskDescription?: PluginTaskDescription;
sandboxLeaseAcquisition?: SandboxLeaseAcquisition;
}
export interface EnvironmentRealizationResult {
lease: EnvironmentLease;
workspaceRealization: Record<string, unknown>;
executionTarget: AdapterExecutionTarget | null;
remoteExecution: AdapterRemoteExecutionSpec | null;
persistedExecutionWorkspace: ExecutionWorkspace | null;
}
export interface EnvironmentReleaseResult {
released: EnvironmentRuntimeLeaseRecord[];
errors: Array<{ leaseId: string; error: unknown }>;
}
function firstNonEmptyLine(text: string | null | undefined): string | null {
if (!text) return null;
for (const rawLine of text.split(/\r?\n/)) {
const line = rawLine.trim();
if (line) return line;
}
return null;
}
function formatProvisionFailureDetail(result: {
exitCode: number | null;
signal?: string | null;
timedOut: boolean;
stdout: string;
stderr: string;
}): string {
if (result.timedOut) {
return "provision command timed out";
}
const signal = typeof result.signal === "string" && result.signal.trim().length > 0
? ` (signal ${result.signal.trim()})`
: "";
const detail = firstNonEmptyLine(result.stderr) ?? firstNonEmptyLine(result.stdout);
const status = `exit code ${result.exitCode ?? "null"}${signal}`;
return detail ? `${status}: ${detail}` : status;
}
// ---------------------------------------------------------------------------
// Service factory
// ---------------------------------------------------------------------------
export function environmentRunOrchestrator(
db: Db,
options: {
pluginWorkerManager?: PluginWorkerManager;
environmentRuntime?: EnvironmentRuntimeService;
} = {},
) {
const environmentsSvc = environmentService(db);
const executionWorkspacesSvc = executionWorkspaceService(db);
const environmentRuntime = options.environmentRuntime ?? environmentRuntimeService(db, {
pluginWorkerManager: options.pluginWorkerManager,
});
/**
* Resolve the selected environment for a run. The caller passes the concrete
* selected environment id plus the built-in local fallback id used to lazily
* ensure the local environment row exists.
*/
async function resolveEnvironment(input: {
companyId: string;
selectedEnvironmentId: string;
localEnvironmentId: string;
}): Promise<Environment> {
const environmentId = input.selectedEnvironmentId || input.localEnvironmentId;
const environment =
environmentId === input.localEnvironmentId
? await environmentsSvc.ensureLocalEnvironment(input.companyId)
: await environmentsSvc.getById(environmentId);
if (!environment) {
throw new EnvironmentRunError("environment_not_found", `Environment "${environmentId}" not found.`, {
environmentId,
});
}
if (environment.status !== "active") {
throw new EnvironmentRunError("environment_inactive", `Environment "${environment.name}" is not active (status: ${environment.status}).`, {
environmentId: environment.id,
driver: environment.driver,
});
}
await assertEnvironmentCompanyBinding(db, environment.id, input.companyId);
return environment;
}
/**
* Acquire an environment lease for a heartbeat run.
* Wraps the runtime driver's acquire call with standardized error handling.
*/
async function acquireLease(input: {
companyId: string;
environment: Environment;
configuredEnvironment?: Environment;
issueId: string | null;
agentId: string;
heartbeatRunId: string;
persistedExecutionWorkspace: Pick<ExecutionWorkspace, "id" | "mode"> | null;
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
runnerTaskProjectIds?: readonly string[];
adapterType: string | null;
warmRunnerIdleTimeoutMs?: number;
runnerReconnectGraceMs?: number;
}): Promise<EnvironmentRuntimeLeaseRecord> {
try {
// Restart-chaos EP1 only; disarmed (always in production) this is one null check.
injectLeaseAcquisitionFault({ companyId: input.companyId, environmentId: input.environment.id,
heartbeatRunId: input.heartbeatRunId, issueId: input.issueId });
return await environmentRuntime.acquireRunLease({ ...input, assertCompanyBinding: true });
} catch (err) {
// Lease acquisition precedes any provider work, so a failure without a
// definite non-transient status (HTTP 5xx/429, errno, acquisition
// deadline, unfinished creation, database error) is retried by the setup
// path with its bounded bootstrap budget. Saved-sandbox resume and
// stopped-worker refusals keep their own, more specific classifications.
const transient = findReusableSandboxResumeError(err) || findPluginWorkerUnavailableError(err)
? null : readTransientLeaseAcquisitionFailure(err, { id: input.heartbeatRunId, companyId: input.companyId });
throw new EnvironmentRunError(
"lease_acquire_failed",
`Failed to acquire lease for environment "${input.environment.name}" (${input.environment.driver}): ${err instanceof Error ? err.message : String(err)}`,
{
environmentId: input.environment.id,
driver: input.environment.driver,
cause: transient
? new EnvironmentProviderUnavailableError({ environmentName: input.environment.name,
driver: input.environment.driver, evidence: transient, cause: err })
: err,
},
);
}
}
/**
* Resolve the execution transport for an adapter based on the acquired lease.
*/
async function resolveTransport(input: {
companyId: string;
adapterType: string;
environment: Environment;
leaseMetadata: Record<string, unknown> | null;
}): Promise<Record<string, unknown> | null> {
try {
return await resolveEnvironmentExecutionTransport({
db,
companyId: input.companyId,
adapterType: input.adapterType,
environment: input.environment,
leaseMetadata: input.leaseMetadata,
});
} catch (err) {
throw new EnvironmentRunError(
"transport_resolution_failed",
`Failed to resolve execution transport for "${input.environment.name}": ${err instanceof Error ? err.message : String(err)}`,
{
environmentId: input.environment.id,
driver: input.environment.driver,
cause: err,
},
);
}
}
/**
* Full acquisition flow: resolve environment, acquire lease, resolve transport.
* This is the primary entry point for heartbeat run setup.
*/
async function acquireForRun(input: {
companyId: string;
selectedEnvironmentId: string;
localEnvironmentId: string;
adapterType: string;
adapterConfig?: Record<string, unknown>;
admittedLifecycleMode?: "warm" | "per_turn";
managedAiCredentialIdentity?: string;
issueId: string | null;
heartbeatRunId: string;
agentId: string;
persistedExecutionWorkspace: Pick<ExecutionWorkspace, "id" | "mode"> | null;
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
runnerTaskProjectIds?: readonly string[];
runnerTaskResumeSessionId?: string | null;
/** Existing live runner recovery must use its original active lease, including ephemeral leases. */
reattachRemoteLease?: { kind?: "sandbox"; leaseId: string; providerLeaseId: string; remoteCwd: string }
| { kind: "plugin_task"; leaseId: string; preparation?: true };
/** A new controller must stop the previous plugin task before allocating a replacement. */
recoverPluginTask?: boolean;
replacementWorkspaceCreation?: string;
}): Promise<EnvironmentAcquisitionResult> {
// Step 1: Resolve environment
const selectedEnvironment = await resolveEnvironment({
companyId: input.companyId,
selectedEnvironmentId: input.selectedEnvironmentId,
localEnvironmentId: input.localEnvironmentId,
});
// The remote-sandbox warm default needs a provider that keeps reusable
// leases. Probe it only for a run the default can apply to; every other
// run resolves without it, exactly as before.
const reusableLeaseSupported = runnerLifecycleDefaultNeedsLeaseProbe(
selectedEnvironment, input.adapterType, input.adapterConfig, input.admittedLifecycleMode, input.managedAiCredentialIdentity,
) && await Promise.resolve(environmentRuntime.providerSupportsReusableLeases?.(selectedEnvironment)).then(
(supported) => supported === true, () => false);
const lifecycleDefaultContext = { reusableLeaseSupported, managedAiCredentialIdentity: input.managedAiCredentialIdentity };
const environment = resolveRunnerEnvironmentForRun(
selectedEnvironment, input.adapterType, input.adapterConfig, input.admittedLifecycleMode, input.managedAiCredentialIdentity,
lifecycleDefaultContext,
);
if (input.recoverPluginTask && environment.driver === "plugin" && input.adapterType === "paperclip_runner") {
const prior = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.heartbeatRunId, input.heartbeatRunId),
));
if (prior.some(lease => lease.metadata?.runnerTaskLease !== true || lease.environmentId !== environment.id ||
lease.issueId !== input.issueId || lease.executionWorkspaceId !== (input.persistedExecutionWorkspace?.id ?? null) ||
lease.metadata?.agentId !== input.agentId)) throw new Error("native_plugin_recovery_lease_mismatch");
const errors: unknown[] = [];
await environmentRuntime.releaseRunLeases(input.heartbeatRunId, "released", (_id, error) => { errors.push(error); });
// Same-run task resume requires every prior lease released with its receipt;
// a stop recorded before a held release is not enough here.
if (errors.length || (prior.length && !await remoteExecutionHasReleased(db, input.companyId, input.heartbeatRunId))) {
throw new Error("native_plugin_recovery_termination_unconfirmed", { cause: errors[0] });
}
}
// Step 2: Acquire lease
const warmRunnerIdleTimeoutMs = resolveWarmRunnerIdleTimeoutMsForRun(
selectedEnvironment, input.adapterType, input.adapterConfig, input.admittedLifecycleMode, lifecycleDefaultContext,
);
const runnerReconnectGraceMs = input.adapterType
? resolveRunnerReconnectGraceMsForRun(selectedEnvironment, input.adapterType)
: undefined;
const acquisitionInput = {
companyId: input.companyId,
environment,
configuredEnvironment: selectedEnvironment,
issueId: input.issueId,
agentId: input.agentId,
heartbeatRunId: input.heartbeatRunId,
persistedExecutionWorkspace: input.persistedExecutionWorkspace,
executionWorkspaceSettings: input.executionWorkspaceSettings,
runnerTaskProjectIds: input.runnerTaskProjectIds,
runnerTaskResumeSessionId: input.runnerTaskResumeSessionId,
replacementWorkspaceCreation: input.replacementWorkspaceCreation,
adapterType: input.adapterType ?? null,
warmRunnerIdleTimeoutMs,
runnerReconnectGraceMs,
};
let leaseRecord: EnvironmentRuntimeLeaseRecord;
let sandboxLeaseAcquisition: SandboxLeaseAcquisition | undefined;
const expected = input.reattachRemoteLease;
if (expected) {
// Reattachment keeps the original lease, including ephemeral leases.
// Never reacquire: acquisition can replace a non-reusable sandbox.
const replayPreparation = expected.kind === "plugin_task" && expected.preparation;
const lease = replayPreparation
? await db.select().from(environmentLeases).where(and(eq(environmentLeases.id, expected.leaseId),
eq(environmentLeases.companyId, input.companyId), unlaunchedPortableTaskLease()))
.then(([row]) => row ? toEnvironmentLease(row) : null)
: await environmentsSvc.getLeaseById(expected.leaseId);
const mismatch: RemoteRecoveryLeaseMismatchField | null = !lease ? "lease"
: lease.companyId !== input.companyId || lease.environmentId !== environment.id ||
lease.heartbeatRunId !== input.heartbeatRunId || lease.issueId !== input.issueId ||
lease.executionWorkspaceId !== (input.persistedExecutionWorkspace?.id ?? null) ||
lease.metadata?.agentId !== input.agentId ? "lease_scope"
: lease.status !== "active" || lease.releasedAt !== null || lease.cleanupStatus !== null ? "lease_status"
: lease.expiresAt !== null && new Date(lease.expiresAt).getTime() <= Date.now() ? "lease_expired"
: expected.kind === "plugin_task"
? environment.driver !== "plugin" ? "environment_driver"
: input.adapterType !== "paperclip_runner" ? "adapter_type"
: replayPreparation ? environment.config.pluginKey !== lease.metadata?.pluginKey ||
environment.config.driverKey !== lease.metadata?.driverKey ? "plugin_task" : null
: lease.metadata?.runnerTaskLease !== true || lease.metadata?.runnerTaskDispatched !== true
|| lease.metadata?.runnerTaskExecutionStopped === true ? "plugin_task" : null
: environment.driver !== "sandbox" ? "environment_driver"
: lease.provider !== environment.config.provider ? "provider"
: lease.providerLeaseId !== expected.providerLeaseId ? "provider_lease_id"
: lease.metadata?.remoteCwd !== expected.remoteCwd ? "remote_cwd" : null;
if (mismatch || !lease) {
throw remoteRecoveryLeaseMismatch("environment_acquire", mismatch ?? "lease");
}
if (expected.kind !== "plugin_task") {
// The provider may have archived this retained sandbox while its
// controller was offline. Ready the exact lease under its lifecycle
// budget before short preparation commands (such as project-auth
// checks) can run. This API cannot acquire a replacement.
const resumed = await environmentRuntime.resumeRunLease({
environment, lease, requestedCwd: expected.remoteCwd,
...(warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs } : {}),
...(runnerReconnectGraceMs !== undefined ? { runnerReconnectGraceMs } : {}),
}).catch((error: unknown) => {
// Confirmed provider absence is a named lease mismatch; other
// failures (for example a transient refusal) keep their own error.
if (error instanceof RetainedSandboxAbsentError) throw remoteRecoveryLeaseMismatch("environment_resume", "lease_expired");
throw error;
});
if (!resumed) throw remoteRecoveryLeaseMismatch("environment_resume", "lease");
if (resumed.providerLeaseId !== expected.providerLeaseId) {
throw remoteRecoveryLeaseMismatch("environment_resume", "provider_lease_id");
}
if (resumed.metadata?.expired === true) throw remoteRecoveryLeaseMismatch("environment_resume", "lease_expired");
// Resume confirms allocation readiness. Its optional metadata may
// describe the allocation root; keep the validated lease's workspace.
sandboxLeaseAcquisition = { outcome: "resumed", providerLeaseId: expected.providerLeaseId };
}
leaseRecord = { environment, lease, leaseContext: buildEnvironmentLeaseContext(input) };
} else {
leaseRecord = await acquireLease(acquisitionInput);
}
// Step 3: Log lease acquisition activity
await logActivity(db, {
companyId: input.companyId,
actorType: "agent",
actorId: input.agentId,
agentId: input.agentId,
runId: input.heartbeatRunId,
action: "environment.lease_acquired",
entityType: "environment_lease",
entityId: leaseRecord.lease.id,
issueId: input.issueId,
details: {
environmentId: environment.id,
driver: environment.driver,
leasePolicy: leaseRecord.lease.leasePolicy,
provider: leaseRecord.lease.provider,
executionWorkspaceId: leaseRecord.leaseContext.executionWorkspaceId,
issueId: input.issueId,
networkEgress: input.executionWorkspaceSettings?.networkEgress ?? null,
},
});
// Step 4: Resolve execution transport
const executionTransport = environment.driver === "plugin" && input.adapterType === "paperclip_runner" ? null : await resolveTransport({
companyId: input.companyId,
adapterType: input.adapterType,
environment,
leaseMetadata: leaseRecord.lease.metadata,
});
return {
environment,
lease: leaseRecord.lease,
leaseContext: leaseRecord.leaseContext,
executionTransport,
...(sandboxLeaseAcquisition ? { sandboxLeaseAcquisition } : {}),
};
}
async function confirmWorkspaceTaskPlacement(acquisition: EnvironmentAcquisitionResult, input: {
companyId: string;
environmentId: string;
workspaceId: string;
signal?: AbortSignal;
}) {
const { environment, lease } = acquisition;
const driverKey = environment.config?.driverKey;
if (environment.id !== input.environmentId || typeof driverKey !== "string" || !lease.providerLeaseId) {
throw new Error("Workspace Launcher placement changed");
}
const description = await environmentRuntime.task({
companyId: input.companyId, leaseId: lease.id, operation: { kind: "describe" }, ownershipSignal: input.signal,
});
const current = await environmentsSvc.getLeaseById(lease.id);
if (description.kind !== "description" || !current || current.providerLeaseId !== description.taskId ||
current.status !== "active" || current.environmentId !== environment.id ||
current.executionWorkspaceId !== input.workspaceId) {
throw new Error("Workspace Launcher lease is no longer active");
}
return {
acquisition: { ...acquisition, lease: current, pluginTaskDescription: description },
taskLease: { driverKey, environmentId: environment.id, providerLeaseId: description.taskId },
};
}
/**
* Realize workspace in the environment and resolve the execution target.
*
* After lease acquisition, this method:
* 1. Builds a workspace realization request
* 2. Calls the environment runtime driver to realize the workspace
* 3. Persists realization metadata on the lease and execution workspace
* 4. Resolves the adapter execution target (local/ssh/sandbox)
*
* Returns the updated lease, realization metadata, and the execution
* target spec that the adapter needs to run.
*/
async function realizeForRun(input: {
environment: Environment;
lease: EnvironmentLease;
sandboxLeaseAcquisition?: SandboxLeaseAcquisition;
adapterType: string;
companyId: string;
issueId: string | null;
heartbeatRunId: string;
executionWorkspace: RealizedExecutionWorkspace;
effectiveExecutionWorkspaceMode: string | null;
persistedExecutionWorkspace: ExecutionWorkspace | null;
runnerTaskProjectIds?: readonly string[];
signal?: AbortSignal;
assertOwned?: () => Promise<void>;
/**
* The host duplex observability recorder for this run. The orchestrator threads
* it to `resolveEnvironmentExecutionTarget`, which stamps it on the sandbox
* target. Absent keeps the safe no-op default in the bridge.
*/
duplexObservabilityRecorder?: DuplexObservabilityRecorder | null;
}): Promise<EnvironmentRealizationResult> {
const {
environment,
adapterType,
companyId,
issueId,
heartbeatRunId,
executionWorkspace,
effectiveExecutionWorkspaceMode,
} = input;
let { lease, persistedExecutionWorkspace } = input;
const withAcquisitionReceipt = (target: AdapterExecutionTarget | null): AdapterExecutionTarget | null => {
if (!input.sandboxLeaseAcquisition || target?.kind !== "remote" || target.transport !== "sandbox") return target;
if (lease.providerLeaseId !== input.sandboxLeaseAcquisition.providerLeaseId) {
throw remoteRecoveryLeaseMismatch("environment_resume", "provider_lease_id");
}
return { ...target, sandboxLeaseAcquisition: input.sandboxLeaseAcquisition };
};
const remoteAuthority = readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata);
if (remoteAuthority && persistedExecutionWorkspace) {
if (adapterType !== "paperclip_runner" || environment.driver !== "sandbox") {
throw new EnvironmentRunError("unsupported_adapter_environment", "Remote workspace storage requires a qualified native sandbox");
}
const target = withAcquisitionReceipt(await resolveEnvironmentExecutionTarget({
db, companyId, adapterType, environment, leaseId: lease.id, leaseMetadata: lease.metadata,
lease, environmentRuntime, duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
}));
if (!target || target.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask) {
throw new EnvironmentRunError("transport_resolution_failed", "Remote workspace storage requires a sandbox file transport");
}
const prepared = await prepareRemoteWorkspace({
db, workspace: persistedExecutionWorkspace, runId: heartbeatRunId, lease, target,
source: remoteAuthority.source,
assertOwned: input.assertOwned,
provisionCommand: persistedExecutionWorkspace.config?.provisionCommand,
runtimeProvisionCommand: persistedExecutionWorkspace.config?.runtimeProvisionCommand,
});
lease = prepared.lease;
const workspaceRealization = {
version: 1, mode: "in_place", authoritativeRoot: prepared.remoteRoot,
pathAliases: [], outboundRestorePaths: [], provider: lease.provider,
environmentId: environment.id, leaseId: lease.id, providerLeaseId: lease.providerLeaseId,
additional: (remoteAuthority.source.repositories ?? []).filter(repository => repository.readOnly)
.map(repository => ({ path: remoteWorkspaceRepositoryPath(prepared.remoteRoot, repository),
projectId: repository.projectId ?? null, projectWorkspaceId: repository.projectWorkspaceId ?? null,
repoUrl: repository.repoUrl, repoRef: repository.ref })),
};
const updatedLease = await environmentsSvc.updateLeaseMetadata(lease.id, {
...lease.metadata, remoteCwd: prepared.remoteRoot, workspaceRealization,
});
if (!updatedLease) throw new Error("Remote workspace lease disappeared during preparation");
lease = updatedLease;
persistedExecutionWorkspace = await executionWorkspacesSvc.getStoredById(companyId, prepared.workspace.id);
if (!persistedExecutionWorkspace) throw new Error("Remote workspace disappeared during preparation");
input.executionWorkspace.cwd = prepared.remoteRoot;
const executionTarget: AdapterExecutionTarget = {
...target, remoteCwd: prepared.remoteRoot,
workspaceRealization: { mode: "in_place", authoritativeRoot: prepared.remoteRoot,
pathAliases: [], outboundRestorePaths: [] },
};
return { lease, workspaceRealization, executionTarget,
remoteExecution: adapterExecutionTargetToRemoteSpec(executionTarget), persistedExecutionWorkspace };
}
if (executionWorkspace.providerBinding) {
const binding = executionWorkspace.providerBinding;
if (environment.driver !== "plugin" || adapterType !== "paperclip_runner" || lease.metadata?.pluginId !== binding.pluginId || !await environmentRuntime.supportsTasks({ lease })) {
throw new EnvironmentRunError("workspace_realization_failed", "The execution plugin cannot access this managed workspace.", { environmentId: environment.id });
}
const updated = await environmentsSvc.updateLeaseMetadata(lease.id, { ...lease.metadata, workspaceProvider: binding });
if (!updated) throw new Error("Workspace execution lease disappeared");
lease = updated;
}
const freshWorkspace = freshWorkspaceSchema.safeParse(persistedExecutionWorkspace?.metadata?.workspaceProviderFresh ??
persistedExecutionWorkspace?.metadata?.runnerTaskFreshWorkspace);
if (environment.driver === "plugin" && freshWorkspace.success) {
const updated = await environmentsSvc.updateLeaseMetadata(lease.id, { ...lease.metadata, freshWorkspace: freshWorkspace.data });
if (!updated) throw new Error("Workspace execution lease disappeared");
lease = updated;
}
// Step 1: Build workspace realization request
const workspaceRealizationRequest = buildWorkspaceRealizationRequest({
adapterType,
companyId,
environmentId: environment.id,
executionWorkspaceId: persistedExecutionWorkspace?.id ?? null,
issueId,
heartbeatRunId,
requestedMode: persistedExecutionWorkspace?.mode ?? effectiveExecutionWorkspaceMode,
workspace: executionWorkspace,
workspaceConfig: persistedExecutionWorkspace?.config ?? null,
});
// Step 2: Realize workspace in the environment via the runtime driver
let workspaceRealization: Record<string, unknown> = {};
let realizedWorkspaceCwd: string | null = null;
const pluginTaskWorkspace = environment.driver === "plugin" && adapterType === "paperclip_runner";
if (pluginTaskWorkspace && !await environmentRuntime.supportsTasks({ lease })) {
throw new EnvironmentRunError("unsupported_adapter_environment", "Plugin environment does not provide native Runner task execution");
}
if (ENVIRONMENT_DRIVER_TRAITS[environment.driver].realizesWorkspace) {
try {
const remoteCwd =
typeof lease.metadata?.remoteCwd === "string" && lease.metadata.remoteCwd.trim().length > 0
? lease.metadata.remoteCwd
: undefined;
const workspaceRealizationResult = await environmentRuntime.realizeWorkspace({
environment,
lease,
workspace: {
localPath: executionWorkspace.cwd,
remotePath: remoteCwd,
mode: persistedExecutionWorkspace?.mode ?? effectiveExecutionWorkspaceMode ?? undefined,
metadata: {
workspaceRealizationRequest,
},
},
});
realizedWorkspaceCwd =
typeof workspaceRealizationResult.cwd === "string" && workspaceRealizationResult.cwd.trim().length > 0
? workspaceRealizationResult.cwd.trim()
: null;
workspaceRealization = parseObject(workspaceRealizationResult.metadata?.workspaceRealization);
} catch (err) {
throw new EnvironmentRunError(
"workspace_realization_failed",
`Failed to realize workspace for environment "${environment.name}" (${environment.driver}): ${err instanceof Error ? err.message : String(err)}`,
{
environmentId: environment.id,
driver: environment.driver,
cause: err,
},
);
}
}
const provisionCommand = workspaceRealizationRequest.runtimeOverlay.provisionCommand?.trim() ?? "";
const realizedCwd =
realizedWorkspaceCwd ??
(typeof lease.metadata?.remoteCwd === "string" && lease.metadata.remoteCwd.trim().length > 0
? lease.metadata.remoteCwd.trim()
: executionWorkspace.cwd);
// The host `provisionCommand` runs on the host worktree during the
// `workspace_provision` step, before the run reaches the environment.
// A `sandbox`-driver environment does not receive the repo tree here. The
// sandbox driver `realizeWorkspace` step only creates the remote folder.
// The adapter uploads the provisioned tree later, in its `stage.sync` step.
// So the host command must not run inside the still-empty sandbox; it fails
// there (exit 127). Skip the step for `sandbox`, and keep the existing skip
// for `local`. Keep the step for `ssh`, which runs the command on the
// remote host that shares the workspace path.
const driverSkipsHostProvision =
environment.driver === "local" || environment.driver === "sandbox" || pluginTaskWorkspace;
if (provisionCommand && environment.driver === "sandbox") {
logger.info(
{
environmentId: environment.id,
driver: environment.driver,
issueId,
heartbeatRunId,
},
"Skip host provisionCommand for sandbox-driver environment; the adapter stage.sync step delivers the provisioned tree",
);
}
if (provisionCommand && !executionWorkspace.providerBinding && !driverSkipsHostProvision) {
try {
const provisionResult = await environmentRuntime.execute({
environment,
lease,
command: "bash",
args: ["-lc", provisionCommand],
cwd: realizedCwd,
env: {
SHELL: "/bin/bash",
},
timeoutMs: 300_000,
// The provision command runs before the run opens its trace root, so it
// carries no run parent. A sandbox provider that opens a persistent
// session on the first command must not open the session here, or the
// session-setup span loses its parent and the span backend drops it.
// Bypass the session for this command; the session opens on the first
// in-run command instead, whose setup span parents to the run trace.
bypassSession: true,
});
if (provisionResult.exitCode !== 0 || provisionResult.timedOut) {
throw new Error(formatProvisionFailureDetail(provisionResult));
}
} catch (err) {
throw new EnvironmentRunError(
"workspace_realization_failed",
`Failed to provision workspace for environment "${environment.name}" (${environment.driver}): ${err instanceof Error ? err.message : String(err)}`,
{
environmentId: environment.id,
driver: environment.driver,
cause: err,
},
);
}
}
// Step 3: Persist realization metadata on lease and execution workspace
if (Object.keys(workspaceRealization).length > 0) {
const nextLeaseMetadata = {
...(lease.metadata ?? {}),
workspaceRealization,
};
const updatedLease = await environmentsSvc.updateLeaseMetadata(lease.id, nextLeaseMetadata);
if (updatedLease) {
lease = updatedLease;
}
if (persistedExecutionWorkspace) {
const updatedEw = await executionWorkspacesSvc.update(persistedExecutionWorkspace.id, {
metadata: {
...(persistedExecutionWorkspace.metadata ?? {}),
workspaceRealizationRequest,
workspaceRealization,
},
});
if (updatedEw) {
persistedExecutionWorkspace = updatedEw;
}
}
}
// Step 4: Resolve execution target for the adapter
let executionTarget: AdapterExecutionTarget | null;
try {
executionTarget = withAcquisitionReceipt(await resolveEnvironmentExecutionTarget({
db,
companyId,
adapterType,
environment,
leaseId: lease.id,
leaseMetadata: (lease.metadata as Record<string, unknown> | null) ?? null,
lease,
environmentRuntime,
signal: input.signal,
duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
}));
// The native plugin task bridge owns guest paths. Never fall back to host execution.
if (executionWorkspace.providerBinding && (executionTarget?.kind !== "remote" ||
!("runnerTask" in executionTarget) || typeof executionTarget.runnerTask !== "function")) {
throw new Error("Managed workspace execution requires the native plugin task bridge");
}
const realizationMode = pluginTaskWorkspace || workspaceRealization.mode === "in_place" ? "in_place" : "copy";
const authoritativeRoot =
typeof workspaceRealization.authoritativeRoot === "string" && workspaceRealization.authoritativeRoot.trim().length > 0
? workspaceRealization.authoritativeRoot.trim()
: realizedCwd;
const workspaceTargetMetadata: AdapterWorkspaceRealization = {
mode: realizationMode,
authoritativeRoot,
pathAliases: Array.isArray(workspaceRealization.pathAliases)
? workspaceRealization.pathAliases.filter(
(entry): entry is { path: string; target: string } =>
typeof entry === "object" && entry !== null &&
typeof (entry as { path?: unknown }).path === "string" &&
typeof (entry as { target?: unknown }).target === "string",
)
: [],
outboundRestorePaths: Array.isArray(workspaceRealization.outboundRestorePaths)
? workspaceRealization.outboundRestorePaths.filter((entry): entry is string => typeof entry === "string")
: [],
};
if (executionTarget) {
executionTarget = {
...executionTarget,
...(executionTarget.kind === "remote" && executionTarget.transport === "sandbox" && executionTarget.runnerTask
? { runnerTaskProjectIds: [...new Set([executionWorkspace.projectId,
...input.runnerTaskProjectIds ?? [],
...(executionWorkspace.additionalWorkspaces ?? []).map(workspace => workspace.projectId)]
.filter((id): id is string => Boolean(id)))] } : {}),
...(executionTarget.kind === "remote" && realizationMode === "in_place"
? { remoteCwd: authoritativeRoot }
: {}),
workspaceRealization: workspaceTargetMetadata,
} as AdapterExecutionTarget;
}
} catch (err) {
throw new EnvironmentRunError(
"transport_resolution_failed",
`Failed to resolve execution target for "${environment.name}": ${err instanceof Error ? err.message : String(err)}`,
{
environmentId: environment.id,
driver: environment.driver,
cause: err,
},
);
}
return {
lease,
workspaceRealization,
executionTarget,
remoteExecution: adapterExecutionTargetToRemoteSpec(executionTarget),
persistedExecutionWorkspace,
};
}
/**
* Release all active leases for a heartbeat run.
* Tracks cleanup status per lease. Errors during individual lease release
* are captured but do not prevent other leases from being released.
* The original run failure (if any) is never hidden by cleanup errors.
*/
async function releaseForRun(input: {
heartbeatRunId: string;
companyId: string;
agentId: string;
status?: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed">;
failureReason?: string;
/** Explicit Stop during adapter startup; never used for ordinary cleanup. */
cancelActiveWork?: boolean;
respectCleanupDeadlines?: boolean;
/** Explicit paperclip_runner resource lifecycle. Omitted for legacy adapters. */
providerResourceDisposition?: ProviderResourceDisposition;
nativeLifecycleTelemetry?: {
provider: string;
harness: string;
lifecycleMode: "per_turn" | "warm";
sandboxResource:
| "keep_running"
| "stop_and_reuse"
| "destroy_after_turn";
};
}): Promise<EnvironmentReleaseResult> {
const status = input.status ?? "released";
const result: EnvironmentReleaseResult = { released: [], errors: [] };
let releasedLeases: EnvironmentRuntimeLeaseRecord[];
try {
const releaseArgs: Parameters<typeof environmentRuntime.releaseRunLeases> = [
input.heartbeatRunId,
status,
(leaseId, error) => result.errors.push({ leaseId, error }),
input.providerResourceDisposition,
];
if (input.cancelActiveWork) releaseArgs[4] = true;
if (input.respectCleanupDeadlines) releaseArgs[5] = { respectCleanupDeadlines: true };
releasedLeases = await environmentRuntime.releaseRunLeases(...releaseArgs);
} catch (err) {
result.errors.push({ leaseId: "*", error: err });
return result;
}
for (const released of releasedLeases) {
try {
await logActivity(db, {
companyId: input.companyId,
actorType: "agent",
actorId: input.agentId,
agentId: input.agentId,
runId: input.heartbeatRunId,
action: "environment.lease_released",
entityType: "environment_lease",
entityId: released.lease.id,
issueId: released.lease.issueId,
details: {
environmentId: released.lease.environmentId,
driver: released.environment.driver,
leasePolicy: released.lease.leasePolicy,
provider: released.lease.provider,
executionWorkspaceId: released.lease.executionWorkspaceId,
issueId: released.lease.issueId,
status: released.lease.status,
cleanupStatus: released.lease.cleanupStatus,
failureReason: input.failureReason ?? released.lease.failureReason,
providerResourceDisposition:
input.providerResourceDisposition ?? "legacy_default",
},
});
} catch {
// Activity logging failure should not block lease release
}
result.released.push(released);
}
return result;
}
return {
resolveEnvironment,
acquireLease,
resolveTransport,
acquireForRun,
confirmWorkspaceTaskPlacement,
realizeForRun,
releaseForRun,
// Expose the underlying runtime for cases that need direct driver access
runtime: environmentRuntime,
};
}
export type EnvironmentRunOrchestrator = ReturnType<typeof environmentRunOrchestrator>;