Files
PaperClipAI/server/src/__tests__/environment-run-orchestrator.test.ts
T
DottaandClaude Opus 5.5 4afcbfc0c7 fix(native): retry transient provider refusals while reattaching a handed-off run (#22118)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A release swap hands live native runs from the outgoing server to
the incoming server. The incoming server reattaches each run to its
original sandbox.
> - Reattachment first resumes the exact sandbox lease through the
sandbox provider plugin.
> - In a release-swap wave, one reattach resume got an HTTP 502 from the
provider gateway. The host did not retry it. The run failed with the
generic `setup_failed` code after 13 minutes of agent work.
> - The task then stayed blocked with a Retry banner for about 10
minutes, until the fresh-continuation fallback reached its bound and
started a new run.
> - This pull request retries a transient provider refusal during
reattachment, with a bounded budget. It also reads the provider status
from the plugin RPC error envelope, so the host sees the 502.
> - The benefit is that a brief gateway failure during a swap is
invisible to the user. The same run continues and no Retry banner shows.

## Linked Issues or Issue Description

**What happened?**
During a graceful release swap, the incoming server reattached a live
native run to its retained sandbox. The `environmentResumeLease` plugin
call failed with an HTML "502 Bad Gateway" body 280 ms after the
reattach started. The other 18 handed-off runs in the same swap
reattached in about 8 seconds. The host treated the 502 as an unknown
error, so the run failed with `setup_failed`. The task stayed `blocked`
with `canRetry: true` until the fresh-continuation fallback created a
new run 10 minutes later.

**Expected behavior**
A transient provider refusal (HTTP 408/425/429/5xx, a connection errno,
or a plugin RPC deadline) during reattachment is retried within a
bounded budget. The same run continues hands-off. A definite refusal
(for example HTTP 409) or a confirmed missing sandbox still fails at
once with its own named reason.

**Steps to reproduce**
1. Start a remote native run on a reusable sandbox.
2. Send SIGTERM to the server so it hands the run to the next server.
3. Make the first `environmentResumeLease` call on the incoming server
fail with HTTP 502.
4. See that the run fails with `setup_failed` and the task shows Retry.

**Paperclip version or commit**
`master` at ca34aca777 (release-swap wave) and ea154ce55e (this branch
base).

Root causes:
- Why setup failed: `environmentRunOrchestrator.acquireForRun` called
`resumeRunLease` once for a reattachment. Acquisition already retries
transient resume refusals, but reattachment did not. Also,
`transientSandboxResumeFailure` read `status` and `statusCode` only from
the thrown error. A plugin refusal arrives as `JsonRpcCallError`, which
carries the provider status only in the validated
`paperclip/environment-resume-error/v1` data envelope. So a plugin 502
was not transient even on the acquisition path.
- Why recovery took 10 minutes: `setup_failed` on a native run with
provider work is not a restart-class code, so the same-run reattach lane
does not take it. Failed-workspace preservation recorded a restore hold.
The fresh-continuation fallback waits for that hold up to
`FRESH_CONTINUATION_PENDING_BOUND_MS` (10 minutes) when it sees no
progress. The continuation started at 23:23:40, which is 10 minutes and
2 seconds after the run finished at 23:13:38.
- Why a Retry banner showed: no fresh-continuation check row existed yet
and no system recovery was active, so the presentation fell through to
`failed` with `canRetry: true`. The banner did not stay after the task
was done. The issue rests at `done` with `canRetry: false`. The grader
reported 10482 s because it closed the banner at its final read hours
later. Its last sample that saw the banner was 272 s after the first.

## What Changed

- `environment-run-orchestrator.ts`: the exact-lease resume during
reattachment goes through `retryTransientResumeCall`. The first call
keeps the provider's configured start deadline. A first call that used
up that deadline (plugin RPC TIMEOUT) is not repeated. After a transient
refusal, a 90 s budget (`REUSABLE_SANDBOX_RESUME_BUDGET_MS`) bounds each
later wait and passes the remaining budget as `timeoutMs`. A confirmed
missing sandbox and a definite refusal are not retried.
- `packages/plugins/sdk/src/environment-resume-error.ts`: owns one retry
policy for provider failures: `isRetryableEnvironmentHttpStatus` (408,
425, 429, 500, 502-504 and gateway/edge 520-527, 529, 598, 599; never
501 or 505) and `isRetryableEnvironmentErrorCode(code, scope)`.
ABORT_ERR is never retried. ENOTFOUND is retried only in the
`known_host` scope (an exact reattach to a host this run already
reached, the DNS-outage case). Lease acquisition keeps its canonical
semantics in the `first_acquisition` scope.
- `environment-provider-unavailable-error.ts`: lease acquisition uses
the SDK policy instead of its own sets.
- `reusable-sandbox-resume-replacement.ts`: owns the resume retry helper
and `transientSandboxResumeFailure(error, scope)`, which reads only
structured evidence (status, envelope errno, plugin RPC deadline,
sandbox state transition) and a few transport phrases. The acquisition
resume loop uses the same classifier and backoff helpers, and
`resumeFailureTimedOut` (label or envelope ETIMEDOUT) for its idle-start
extension.
- `environment-runtime.ts`: `EnvironmentDriverResumeInput.timeoutMs`
caps the sandbox plugin driver's resume RPC. Other drivers keep their
own deadline.
- Cloudflare bridge client: a bridge request deadline throws
`CloudflareBridgeTimeoutError` with code `ETIMEDOUT`. Daytona:
`SandboxLivenessTimeoutError` carries code `ETIMEDOUT`. Both reach the
host as a structured errno in the resume envelope.
- `tests/restart-chaos/staging-mix.ts`: `bannerShownMs` counts a visible
Retry banner at most 2 minutes past the last sample that saw it. Before,
it counted the whole unsampled gap until the final read.

Previous-release story: this changes no persisted state, launch
configuration, or pack identity. A run saved by the previous release
reattaches the same way. Only the in-process retry around the resume
call is new.

## Verification

- `pnpm verify` (selected Vitest files), and focused runs of the four
changed server test files and the two restart-chaos test files.
- Red/green: `server/src/__tests__/environment-run-orchestrator.test.ts`
- "retries a provider gateway refusal during reattachment and keeps the
original allocation" fails on master (1 call, rejects with the 502) and
passes with the fix.
- "treats a provider HTTP %i refusal during reattachment as transient"
(500, 502, 520, 599).
- "gives the first reattach resume the provider deadline and bounds
every retry by the budget".
- "does not repeat a reattach resume that used up the provider's start
deadline", "does not retry a definite provider refusal during
reattachment", and "preserves a retained sandbox startup failure without
acquiring a replacement" (default budget, one call).
- `server/src/services/reusable-sandbox-resume-retry.test.ts`: a table
over every classifier branch, and the helper's deadline rules.
- `server/src/__tests__/environment-runtime.test.ts`: "caps the exact
resume call at the caller's remaining budget".
- `server/src/services/environment-provider-unavailable-error.test.ts`:
scope table and the extended status lists. Orchestrator reattach cases:
ENOTFOUND retried, ABORT_ERR and 501 not.
-
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`:
"applies the idle-start extension to a provider liveness deadline named
only by its errno" (red/green).
- Cloudflare `bridge-client.test.ts` and Daytona `plugin.test.ts`: the
deadline errors carry ETIMEDOUT in the resume envelope.
- Red/green integration:
`server/src/__tests__/heartbeat-native-remote-failure-retention.test.ts`
"keeps reattaching through a gateway refusal during a graceful hand-off
and continues the same run". The run's controller is lost, restart
recovery claims it to reattach, and the first resume fails with the 502
envelope. On master the run fails before the runner step (1 resume
call). With the fix the resume is retried, the same run reaches the
runner step at controller generation 2, no replacement allocation is
acquired, and no second run is created.
- `tests/restart-chaos/staging-mix.test.ts`: banner tests for a late
hide sample and for a banner still open at report time.

## Risks

- The acquisition resume loop no longer retries a refusal whose only
evidence is "timeout" or "timed out" text. Cloudflare and Daytona now
name their own deadlines ETIMEDOUT, and the plugin RPC deadline is
structured, so those stay retried. A provider that reports a timeout
only as text fails at once.
- Lease acquisition now also retries gateway statuses 520-527, 529, 598
and 599.
- A reattach resume can now take the provider deadline plus up to 90 s
when it gets transient refusals. Each retry is the same exact-lease
lifecycle resume. It cannot acquire or replace a sandbox, so a retry
cannot create a second executor.
- The aggregate recovery brake is unchanged. This change adds no new run
starts and no work before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:11:04 -06:00

1300 lines
61 KiB
TypeScript

import { RetainedSandboxAbsentError } from "../services/retained-sandbox-absent-error.js";
import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, environmentResumeErrorData } from "@paperclipai/plugin-sdk";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { randomUUID } from "node:crypto";
import * as remoteWorkspaces from "../services/remote-workspaces.js";
// ---------------------------------------------------------------------------
// Hoisted mocks — must be declared before any imports that reference them
// ---------------------------------------------------------------------------
const mockResolveEnvironmentExecutionTarget = vi.hoisted(() => vi.fn());
const mockAdapterExecutionTargetToRemoteSpec = vi.hoisted(() => vi.fn());
const mockBuildWorkspaceRealizationRequest = vi.hoisted(() => vi.fn());
const mockUpdateLeaseMetadata = vi.hoisted(() => vi.fn());
const mockUpdateExecutionWorkspace = vi.hoisted(() => vi.fn());
const mockGetStoredWorkspace = vi.hoisted(() => vi.fn());
const mockLogActivity = vi.hoisted(() => vi.fn());
const mockLoggerInfo = vi.hoisted(() => vi.fn());
const mockGetEnvironment = vi.hoisted(() => vi.fn());
const mockGetLease = vi.hoisted(() => vi.fn());
vi.mock("../services/environment-execution-target.js", () => ({
resolveEnvironmentExecutionTarget: mockResolveEnvironmentExecutionTarget,
resolveEnvironmentExecutionTransport: vi.fn().mockResolvedValue(null),
}));
vi.mock("@paperclipai/adapter-utils/execution-target", () => ({
adapterExecutionTargetToRemoteSpec: mockAdapterExecutionTargetToRemoteSpec,
}));
vi.mock("../services/workspace-realization.js", () => ({
buildWorkspaceRealizationRequest: mockBuildWorkspaceRealizationRequest,
}));
vi.mock("../services/environments.js", () => ({
assertEnvironmentCompanyBinding: vi.fn(async () => {}),
environmentService: vi.fn(() => ({
ensureLocalEnvironment: vi.fn(),
getById: mockGetEnvironment,
getLeaseById: mockGetLease,
acquireLease: vi.fn(),
releaseLease: vi.fn(),
updateLeaseMetadata: mockUpdateLeaseMetadata,
})),
}));
vi.mock("../services/execution-workspaces.js", () => ({
executionWorkspaceService: vi.fn(() => ({
update: mockUpdateExecutionWorkspace,
getStoredById: mockGetStoredWorkspace,
})),
}));
vi.mock("../services/activity-log.js", () => ({
logActivity: mockLogActivity,
}));
vi.mock("../middleware/logger.js", () => ({
logger: {
info: mockLoggerInfo,
warn: vi.fn(),
error: vi.fn(),
debug: vi.fn(),
},
}));
// ---------------------------------------------------------------------------
// Imports after mocks
// ---------------------------------------------------------------------------
import {
environmentRunOrchestrator,
EnvironmentRunError,
} from "../services/environment-run-orchestrator.ts";
import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "../services/native-runtime/native-runner-recovery-contract.js";
import type { Environment, EnvironmentLease, ExecutionWorkspace } from "@paperclipai/shared";
import type { RealizedExecutionWorkspace } from "../services/workspace-runtime.ts";
import type { EnvironmentRuntimeService } from "../services/environment-runtime.ts";
// ---------------------------------------------------------------------------
// Fixtures
// ---------------------------------------------------------------------------
function makeEnvironment(driver: string = "local"): Environment {
return {
id: "env-1",
name: "Test Environment",
description: null,
driver: driver as Environment["driver"],
status: "active",
config: {},
envVars: {},
metadata: null,
createdAt: new Date(),
updatedAt: new Date(),
};
}
function makeLease(overrides: Partial<EnvironmentLease> = {}): EnvironmentLease {
return {
id: "lease-1",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
status: "active",
leasePolicy: "ephemeral",
provider: "local",
providerLeaseId: null,
acquiredAt: new Date(),
lastUsedAt: new Date(),
expiresAt: null,
releasedAt: null,
failureReason: null,
cleanupStatus: null,
metadata: null,
createdAt: new Date(),
updatedAt: new Date(),
...overrides,
};
}
function makeExecutionWorkspace(cwd: string = "/workspace/project"): RealizedExecutionWorkspace {
return {
baseCwd: "/workspace",
source: "project_primary",
projectId: "project-1",
workspaceId: "ws-1",
repoUrl: null,
repoRef: null,
strategy: "project_primary",
cwd,
branchName: null,
worktreePath: null,
branchCreatedByRuntime: false,
warnings: [],
created: false,
};
}
function makePersistedExecutionWorkspace(
overrides: Partial<ExecutionWorkspace> = {},
): ExecutionWorkspace {
return {
id: "ew-1",
companyId: "company-1",
projectId: "project-1",
projectWorkspaceId: null,
sourceIssueId: null,
mode: "shared_workspace",
strategyType: "project_primary",
name: "workspace",
status: "active",
cwd: "/workspace/project",
repoUrl: null,
baseRef: null,
branchName: null,
providerType: "local_fs",
providerRef: null,
derivedFromExecutionWorkspaceId: null,
lastUsedAt: new Date(),
openedAt: new Date(),
closedAt: null,
cleanupEligibleAt: null,
cleanupReason: null,
config: null,
metadata: null,
deliveryState: "unknown",
createdAt: new Date(),
updatedAt: new Date(),
...overrides,
};
}
function makeRealizeInput(overrides: {
environment?: Environment;
lease?: EnvironmentLease;
persistedExecutionWorkspace?: ExecutionWorkspace | null;
} = {}): Parameters<ReturnType<typeof environmentRunOrchestrator>["realizeForRun"]>[0] {
return {
environment: overrides.environment ?? makeEnvironment("local"),
lease: overrides.lease ?? makeLease(),
adapterType: "claude_local",
companyId: "company-1",
issueId: null,
heartbeatRunId: "run-1",
executionWorkspace: makeExecutionWorkspace(),
effectiveExecutionWorkspaceMode: null,
persistedExecutionWorkspace: overrides.persistedExecutionWorkspace !== undefined
? overrides.persistedExecutionWorkspace
: null,
};
}
function makeMockRuntime(overrides: Partial<EnvironmentRuntimeService> = {}): EnvironmentRuntimeService {
return {
acquireRunLease: vi.fn(),
resumeRunLease: vi.fn(async ({ lease, requestedCwd }: Parameters<EnvironmentRuntimeService["resumeRunLease"]>[0]) => ({
providerLeaseId: lease.providerLeaseId,
metadata: { ...lease.metadata, remoteCwd: requestedCwd ?? lease.metadata?.remoteCwd },
})),
releaseRunLeases: vi.fn(),
execute: vi.fn().mockResolvedValue({
exitCode: 0,
signal: null,
timedOut: false,
stdout: "",
stderr: "",
}),
realizeWorkspace: vi.fn().mockResolvedValue({
cwd: "/workspace/project",
metadata: {
workspaceRealization: {
version: 1,
mode: "copy",
authoritativeRoot: "/workspace/project",
pathAliases: [],
outboundRestorePaths: [],
driver: "local",
cwd: "/workspace/project",
},
},
}),
...overrides,
} as unknown as EnvironmentRuntimeService;
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
describe("environmentRunOrchestrator — realizeForRun", () => {
const mockDb = {} as any;
beforeEach(() => {
vi.clearAllMocks();
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "claude_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: null,
},
});
mockAdapterExecutionTargetToRemoteSpec.mockReturnValue({
kind: "local",
environmentId: "env-1",
leaseId: "lease-1",
});
mockUpdateLeaseMetadata.mockResolvedValue(null);
mockUpdateExecutionWorkspace.mockResolvedValue(null);
mockLogActivity.mockResolvedValue(undefined);
});
it.each([false, true])("only requests active-work cancellation for explicit Stop: %s", async (cancelActiveWork) => {
const releaseRunLeases = vi.fn().mockResolvedValue([]);
const runtime = makeMockRuntime({ releaseRunLeases });
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await orchestrator.releaseForRun({
heartbeatRunId: "run-1", companyId: "company-1", agentId: "agent-1",
providerResourceDisposition: "stop_and_retain", cancelActiveWork,
});
expect(releaseRunLeases).toHaveBeenCalledWith(
"run-1", "released", expect.any(Function), "stop_and_retain",
...(cancelActiveWork ? [true] : []),
);
});
it("enforces company bindings when a caller acquires a lease directly", async () => {
const runtime = makeMockRuntime();
await environmentRunOrchestrator(mockDb, { environmentRuntime: runtime }).acquireLease({
companyId: "company-1", environment: makeEnvironment(), issueId: null, agentId: "agent-1", heartbeatRunId: "run-1",
persistedExecutionWorkspace: null, executionWorkspaceSettings: null, adapterType: null,
});
expect(runtime.acquireRunLease).toHaveBeenCalledWith(expect.objectContaining({ assertCompanyBinding: true }));
});
it.each(["sandbox", "ssh"] as const)("realizes an enrolled %s workspace in place and persists its authoritative remote root", async driver => {
const environment = { ...makeEnvironment(driver), id: randomUUID() };
const lease = makeLease({ id: randomUUID(), environmentId: environment.id, provider: "daytona", providerLeaseId: "sandbox-allocation" });
const workspace = makePersistedExecutionWorkspace({ id: randomUUID(), providerType: "cloud_sandbox", metadata: {
remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, source: { kind: "empty" },
writerRunId: randomUUID(), dirty: false, snapshotId: null, location: null },
} });
const remoteRoot = `/sandbox/.paperclip-workspaces/${workspace.id}`;
const savedLease = { ...lease, metadata: { remoteCwd: remoteRoot } };
const target = driver === "ssh"
? { kind: "remote", transport: "ssh", remoteCwd: "/sandbox", spec: { remoteCwd: "/sandbox" } }
: { kind: "remote", transport: "sandbox", remoteCwd: "/sandbox", runner: { execute: vi.fn() } };
mockResolveEnvironmentExecutionTarget.mockResolvedValue(target);
mockGetStoredWorkspace.mockResolvedValue({ ...workspace, cwd: remoteRoot });
mockUpdateLeaseMetadata.mockResolvedValue(savedLease);
const prepare = vi.spyOn(remoteWorkspaces, "prepareRemoteWorkspace").mockResolvedValue({
workspace, lease: savedLease, remoteRoot, reference: {} as never,
} as never);
try {
const runtime = makeMockRuntime();
const input = { ...makeRealizeInput({ environment, lease, persistedExecutionWorkspace: workspace }), adapterType: driver === "ssh" ? "claude_local" : "paperclip_runner" };
const result = await environmentRunOrchestrator(mockDb, { environmentRuntime: runtime }).realizeForRun(input);
expect(prepare).toHaveBeenCalledWith(expect.objectContaining({ workspace, target, source: { kind: "empty" } }));
expect(mockBuildWorkspaceRealizationRequest).not.toHaveBeenCalled();
expect(runtime.realizeWorkspace).not.toHaveBeenCalled();
expect(result.executionTarget).toMatchObject({ remoteCwd: remoteRoot,
workspaceRealization: { mode: "in_place", authoritativeRoot: remoteRoot, outboundRestorePaths: [] } });
expect(input.executionWorkspace.cwd).toBe(remoteRoot);
if (result.executionTarget?.kind === "remote" && result.executionTarget.transport === "ssh") {
expect(result.executionTarget.spec.remoteCwd).toBe(remoteRoot);
}
expect(mockUpdateLeaseMetadata).toHaveBeenCalledWith(lease.id, expect.objectContaining({ remoteCwd: remoteRoot }));
expect(result.lease.metadata?.remoteCwd).toBe(remoteRoot);
} finally { prepare.mockRestore(); }
});
it("happy path: returns lease, executionTarget, and remoteExecution on successful realization", async () => {
const executionTarget = { kind: "local", environmentId: "env-1", leaseId: "lease-1" };
const remoteExecution = { kind: "local", environmentId: "env-1", leaseId: "lease-1" };
mockResolveEnvironmentExecutionTarget.mockResolvedValue(executionTarget);
mockAdapterExecutionTargetToRemoteSpec.mockReturnValue(remoteExecution);
const runtime = makeMockRuntime();
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
const result = await orchestrator.realizeForRun(makeRealizeInput());
expect(result.lease).toBeDefined();
expect(result.executionTarget).toEqual({
...executionTarget,
workspaceRealization: {
mode: "copy",
authoritativeRoot: "/workspace/project",
pathAliases: [],
outboundRestorePaths: [],
},
});
expect(result.remoteExecution).toEqual(remoteExecution);
expect(result.workspaceRealization).toEqual(
expect.objectContaining({ version: 1, driver: "local" }),
);
expect(runtime.realizeWorkspace).toHaveBeenCalledOnce();
expect(mockResolveEnvironmentExecutionTarget).toHaveBeenCalledOnce();
});
it("uses an in-place authoritative root on the adapter execution target", async () => {
mockResolveEnvironmentExecutionTarget.mockResolvedValue({
kind: "remote",
transport: "sandbox",
remoteCwd: "/copied/workspace",
});
const runtime = makeMockRuntime({
realizeWorkspace: vi.fn().mockResolvedValue({
cwd: "/app",
metadata: {
workspaceRealization: {
version: 1,
mode: "in_place",
authoritativeRoot: "/app",
pathAliases: [],
outboundRestorePaths: [],
},
},
}),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
const result = await orchestrator.realizeForRun(
makeRealizeInput({ environment: makeEnvironment("sandbox") }),
);
expect(result.executionTarget).toEqual(expect.objectContaining({
kind: "remote",
transport: "sandbox",
remoteCwd: "/app",
workspaceRealization: {
mode: "in_place",
authoritativeRoot: "/app",
pathAliases: [],
outboundRestorePaths: [],
},
}));
});
it("realization failure: runtime.realizeWorkspace throws → EnvironmentRunError with code workspace_realization_failed", async () => {
const runtime = makeMockRuntime({
realizeWorkspace: vi.fn().mockRejectedValue(new Error("sandbox unreachable")),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await expect(orchestrator.realizeForRun(makeRealizeInput())).rejects.toSatisfy(
(err: unknown) =>
err instanceof EnvironmentRunError &&
err.code === "workspace_realization_failed" &&
err.environmentId === "env-1" &&
err.driver === "local",
);
expect(mockResolveEnvironmentExecutionTarget).not.toHaveBeenCalled();
});
it("target resolution failure: resolveEnvironmentExecutionTarget throws → EnvironmentRunError with code transport_resolution_failed", async () => {
mockResolveEnvironmentExecutionTarget.mockRejectedValue(new Error("network error"));
const runtime = makeMockRuntime();
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await expect(orchestrator.realizeForRun(makeRealizeInput())).rejects.toSatisfy(
(err: unknown) =>
err instanceof EnvironmentRunError &&
err.code === "transport_resolution_failed" &&
err.environmentId === "env-1",
);
});
it("non-sandbox driver skips workspace realization and goes straight to target resolution", async () => {
const environment = makeEnvironment("plugin" as Environment["driver"]);
const executionTarget = null;
mockResolveEnvironmentExecutionTarget.mockResolvedValue(executionTarget);
const runtime = makeMockRuntime();
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
const result = await orchestrator.realizeForRun(
makeRealizeInput({ environment }),
);
expect(runtime.realizeWorkspace).not.toHaveBeenCalled();
expect(result.workspaceRealization).toEqual({});
expect(result.executionTarget).toBeNull();
});
it("forwards authorized project IDs to task-only providers without staging directories", async () => {
const environment = makeEnvironment("plugin" as Environment["driver"]);
mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "remote", transport: "sandbox",
providerKey: "plugin", remoteCwd: "provider/project", runnerTask: vi.fn() });
const runtime = makeMockRuntime({ supportsTasks: vi.fn().mockResolvedValue(true) });
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
const input = { ...makeRealizeInput({ environment }), adapterType: "paperclip_runner",
runnerTaskProjectIds: ["anchor", "authorized-reference", "anchor"] };
input.executionWorkspace.projectId = "anchor";
const result = await orchestrator.realizeForRun(input);
expect(result.executionTarget).toMatchObject({ runnerTaskProjectIds: ["anchor", "authorized-reference"] });
expect(runtime.realizeWorkspace).not.toHaveBeenCalled();
expect(runtime.execute).not.toHaveBeenCalled();
});
it("persisted metadata is updated on lease and execution workspace after realization", async () => {
const persistedExecutionWorkspace = makePersistedExecutionWorkspace();
const updatedLease = makeLease({
metadata: { workspaceRealization: { version: 1, driver: "local", cwd: "/workspace/project" } },
});
const updatedEw = { ...persistedExecutionWorkspace, metadata: { workspaceRealizationRequest: {}, workspaceRealization: {} } };
mockUpdateLeaseMetadata.mockResolvedValue(updatedLease);
mockUpdateExecutionWorkspace.mockResolvedValue(updatedEw);
mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "local", environmentId: "env-1", leaseId: "lease-1" });
const runtime = makeMockRuntime();
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
const result = await orchestrator.realizeForRun(
makeRealizeInput({ persistedExecutionWorkspace }),
);
// Lease metadata should have been updated with workspaceRealization
expect(mockUpdateLeaseMetadata).toHaveBeenCalledOnce();
expect(mockUpdateLeaseMetadata).toHaveBeenCalledWith(
"lease-1",
expect.objectContaining({ workspaceRealization: expect.any(Object) }),
);
// Execution workspace metadata should have been updated
expect(mockUpdateExecutionWorkspace).toHaveBeenCalledOnce();
expect(mockUpdateExecutionWorkspace).toHaveBeenCalledWith(
"ew-1",
expect.objectContaining({
metadata: expect.objectContaining({
workspaceRealizationRequest: expect.any(Object),
workspaceRealization: expect.any(Object),
}),
}),
);
// The returned lease should reflect the updated value
expect(result.lease).toEqual(updatedLease);
expect(result.persistedExecutionWorkspace).toEqual(updatedEw);
});
it("runs a remote provision command after workspace realization when configured", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "claude_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: "npm install -g @anthropic-ai/claude-code",
},
});
mockResolveEnvironmentExecutionTarget.mockResolvedValue({
kind: "remote",
transport: "ssh",
remoteCwd: "/remote/workspace",
environmentId: "env-1",
leaseId: "lease-1",
spec: {
host: "ssh.example.test",
port: 22,
username: "ssh-user",
remoteCwd: "/remote/workspace",
remoteWorkspacePath: "/remote/workspace",
privateKey: null,
knownHosts: null,
strictHostKeyChecking: true,
},
});
const runtime = makeMockRuntime({
realizeWorkspace: vi.fn().mockResolvedValue({
cwd: "/remote/workspace",
metadata: {
workspaceRealization: {
version: 1,
transport: "ssh",
remote: { path: "/remote/workspace" },
},
},
}),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await orchestrator.realizeForRun(makeRealizeInput({
environment: makeEnvironment("ssh"),
}));
// The `ssh` driver runs the command on the remote host that shares the
// workspace path, so the configured provision command still runs there.
expect(runtime.execute).toHaveBeenCalledOnce();
expect(runtime.execute).toHaveBeenCalledWith(expect.objectContaining({
environment: expect.objectContaining({ driver: "ssh" }),
lease: expect.objectContaining({ id: "lease-1" }),
command: "bash",
args: ["-lc", "npm install -g @anthropic-ai/claude-code"],
cwd: "/remote/workspace",
env: {
SHELL: "/bin/bash",
},
}));
// The sandbox skip log is specific to the sandbox driver; ssh stays quiet.
expect(mockLoggerInfo).not.toHaveBeenCalled();
});
it("skips the host provision command for a sandbox environment and logs the skip", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "claude_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: "npm install -g @anthropic-ai/claude-code",
},
});
mockResolveEnvironmentExecutionTarget.mockResolvedValue({
kind: "remote",
transport: "sandbox",
providerKey: "e2b",
remoteCwd: "/remote/workspace",
environmentId: "env-1",
leaseId: "lease-1",
});
const runtime = makeMockRuntime({
realizeWorkspace: vi.fn().mockResolvedValue({
cwd: "/remote/workspace",
metadata: {
workspaceRealization: {
version: 1,
transport: "sandbox",
remote: { path: "/remote/workspace" },
},
},
}),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await orchestrator.realizeForRun(makeRealizeInput({
environment: makeEnvironment("sandbox"),
}));
// The sandbox receives the provisioned tree through the adapter stage.sync
// step, so the orchestrator must not run the host command in the sandbox.
expect(runtime.execute).not.toHaveBeenCalled();
// The skip is observable: exactly one log line records it with the driver.
expect(mockLoggerInfo).toHaveBeenCalledOnce();
expect(mockLoggerInfo).toHaveBeenCalledWith(
expect.objectContaining({ driver: "sandbox", environmentId: "env-1" }),
expect.stringContaining("Skip host provisionCommand"),
);
});
it("does not rerun the provision command during local environment realization", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "claude_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: "npm install -g @anthropic-ai/claude-code",
},
});
mockResolveEnvironmentExecutionTarget.mockResolvedValue({
kind: "local",
environmentId: "env-1",
leaseId: "lease-1",
});
const runtime = makeMockRuntime();
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await orchestrator.realizeForRun(makeRealizeInput({
environment: makeEnvironment("local"),
}));
// Local workspace provisioning already ran the command before realizeForRun.
expect(runtime.execute).not.toHaveBeenCalled();
// The sandbox skip log is specific to the sandbox driver; local stays quiet.
expect(mockLoggerInfo).not.toHaveBeenCalled();
});
it("runs project-level provision commands for ssh environments", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "gemini_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: "npm install -g @google/gemini-cli",
},
});
mockResolveEnvironmentExecutionTarget.mockResolvedValue({
kind: "remote",
transport: "ssh",
remoteCwd: "/remote/workspace",
environmentId: "env-1",
leaseId: "lease-1",
spec: {
host: "ssh.example.test",
port: 22,
username: "ssh-user",
remoteCwd: "/remote/workspace",
remoteWorkspacePath: "/remote/workspace",
privateKey: null,
knownHosts: null,
strictHostKeyChecking: true,
},
});
const runtime = makeMockRuntime({
realizeWorkspace: vi.fn().mockResolvedValue({
cwd: "/remote/workspace",
metadata: {
workspaceRealization: {
version: 1,
transport: "ssh",
remote: { path: "/remote/workspace" },
},
},
}),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await orchestrator.realizeForRun(makeRealizeInput({
environment: makeEnvironment("ssh"),
lease: makeLease({
provider: "ssh",
metadata: {
driver: "ssh",
remoteCwd: "/remote/workspace",
remoteWorkspacePath: "/remote/workspace",
host: "ssh.example.test",
port: 22,
username: "ssh-user",
},
}),
}));
expect(runtime.execute).toHaveBeenCalledWith(expect.objectContaining({
command: "bash",
args: ["-lc", "npm install -g @google/gemini-cli"],
}));
expect(mockResolveEnvironmentExecutionTarget).toHaveBeenCalledOnce();
});
it("surfaces remote provision command failures before resolving the adapter target", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "claude_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: "install-tool",
},
});
const runtime = makeMockRuntime({
execute: vi.fn().mockResolvedValue({
exitCode: 127,
signal: null,
timedOut: false,
stdout: "",
stderr: "/bin/sh: install-tool: not found\n",
}),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await expect(orchestrator.realizeForRun(makeRealizeInput({
environment: makeEnvironment("ssh"),
}))).rejects.toSatisfy(
(err: unknown) =>
err instanceof EnvironmentRunError &&
err.code === "workspace_realization_failed" &&
String(err.message).includes("install-tool: not found"),
);
expect(mockResolveEnvironmentExecutionTarget).not.toHaveBeenCalled();
});
});
describe("native runner lifecycle changes before lease acquisition", () => {
it.each([undefined, "account:generation"])("uses a warm plugin lease only with a managed credential identity: %s", async managedAiCredentialIdentity => {
const environment = { ...makeEnvironment("plugin"), config: { pluginKey: "fixture", driverKey: "tasks", driverConfig: {} } };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async input => ({ environment: input.environment,
lease: makeLease({ leasePolicy: input.environment.config.runnerLifecycleMode === "warm" ? "reuse_by_environment" : "ephemeral" }), leaseContext: { executionWorkspaceId: null, executionWorkspaceMode: null } }));
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) });
const result = await orchestrator.acquireForRun({ companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local",
adapterType: "paperclip_runner", adapterConfig: { lifecycleMode: "warm" }, managedAiCredentialIdentity,
issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1", persistedExecutionWorkspace: null, executionWorkspaceSettings: null });
expect(result.environment.config.runnerLifecycleMode).toBe(managedAiCredentialIdentity ? "warm" : "per_turn");
expect(result.lease.leasePolicy).toBe(managedAiCredentialIdentity ? "reuse_by_environment" : "ephemeral");
expect(environment.config).not.toHaveProperty("runnerLifecycleMode");
});
it("requests a reusable lease when an existing task switches from per-turn to warm", async () => {
const environment = { ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false } };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async (input) => ({
environment: input.environment,
lease: makeLease({ leasePolicy: input.environment.config.reuseLease ? "reuse_by_environment" : "ephemeral" }),
leaseContext: { executionWorkspaceId: "existing-workspace", executionWorkspaceMode: "shared_workspace" as const },
}));
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) });
const input = {
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local",
adapterType: "paperclip_runner", issueId: "existing-task", heartbeatRunId: "run-1", agentId: "agent-1",
runnerTaskProjectIds:["anchor","authorized-reference"],
persistedExecutionWorkspace: { id: "existing-workspace", mode: "shared_workspace" as const },
executionWorkspaceSettings: null,
};
const cold = await orchestrator.acquireForRun({ ...input, adapterConfig: { lifecycleMode: "per_turn" } });
expect(cold.lease.leasePolicy).toBe("ephemeral");
const warm = await orchestrator.acquireForRun({ ...input, heartbeatRunId: "run-2", adapterConfig: { lifecycleMode: "warm", idleTimeoutMs: 60_000 } });
expect(warm.lease.leasePolicy).toBe("reuse_by_environment");
expect(warm.environment.config).toEqual({ provider: "daytona", reuseLease: true });
expect(acquireRunLease.mock.calls[1][0].persistedExecutionWorkspace).toEqual(input.persistedExecutionWorkspace);
expect(acquireRunLease.mock.calls[1][0].configuredEnvironment).toBe(environment);
expect(acquireRunLease.mock.calls[1][0].runnerTaskProjectIds).toEqual(input.runnerTaskProjectIds);
// Run-scoped adaptation must not change the shared environment or other agents.
expect(environment.config).toEqual({ provider: "daytona", reuseLease: false });
const coldAgain = await orchestrator.acquireForRun({ ...input, heartbeatRunId: "run-3", adapterConfig: { lifecycleMode: "per_turn" } });
expect(coldAgain.lease.leasePolicy).toBe("ephemeral");
});
// The remote-sandbox warm default asks the provider before it requests a
// reusable lease, and asks only for a run whose lifecycle nobody chose.
it.each([
["a provider with reusable leases", {}, true, true, true],
["a provider without reusable leases (E2B, Modal, fake)", {}, false, true, false],
["an explicit per-turn agent", { lifecycleMode: "per_turn" }, true, false, false],
["a managed Claude account", { provider: "acpx", acpxAgent: "claude", model: "claude-sonnet-4-6", managed: true }, true, false, false],
] as const)("defaults an inheriting sandbox run to warm only on %s", async (_name, agent, supported, probed, warm) => {
const environment = { ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false } };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async (input) => ({ environment: input.environment, lease: makeLease(), leaseContext: { executionWorkspaceId: null, executionWorkspaceMode: null } }));
const providerSupportsReusableLeases = vi.fn(async () => supported);
const orchestrator = environmentRunOrchestrator({} as never, {
environmentRuntime: { ...makeMockRuntime({ acquireRunLease }), providerSupportsReusableLeases } as never,
});
const { managed, ...adapterConfig } = { managed: false, ...agent };
await orchestrator.acquireForRun({
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local", adapterType: "paperclip_runner",
adapterConfig, managedAiCredentialIdentity: managed ? "account:generation" : undefined,
issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
persistedExecutionWorkspace: null, executionWorkspaceSettings: null,
});
expect(providerSupportsReusableLeases).toHaveBeenCalledTimes(probed ? 1 : 0);
expect(acquireRunLease.mock.calls[0][0].environment.config.reuseLease).toBe(warm);
expect(acquireRunLease.mock.calls[0][0].warmRunnerIdleTimeoutMs).toBe(warm ? 300_000 : undefined);
expect(environment.config.reuseLease).toBe(false);
});
it.each([
["paperclip_runner", undefined, undefined, false],
["paperclip_runner", "inherit", "per_turn", false],
["paperclip_runner", "per_turn", "warm", false],
["paperclip_runner", "warm", "per_turn", true],
["paperclip_runner", "inherit", "warm", true],
["codex_local", "inherit", "warm", false],
])("respects environment precedence and adapter scope (%s, %s, %s)", async (adapterType, runnerLifecycleMode, lifecycleMode, expectedReuse) => {
const environment = { ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false, runnerLifecycleMode } };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async (input) => ({ environment: input.environment, lease: makeLease(), leaseContext: { executionWorkspaceId: null, executionWorkspaceMode: null } }));
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) });
await orchestrator.acquireForRun({
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local", adapterType,
adapterConfig: { lifecycleMode }, issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
persistedExecutionWorkspace: null, executionWorkspaceSettings: null,
});
expect(acquireRunLease.mock.calls[0][0].environment.config.reuseLease).toBe(expectedReuse);
});
// RC-54: a per-turn remote runner sandbox must outlive a controller outage
// the host still recovers, so acquisition forwards the reconnect grace.
it.each([
["paperclip_runner", "sandbox", REMOTE_RUNNER_RECONNECT_GRACE_MS],
["claude_local", "sandbox", undefined],
["paperclip_runner", "local", undefined],
] as const)("forwards the remote runner reconnect grace for %s on %s", async (adapterType, driver, expected) => {
const environment = { ...makeEnvironment(driver), config: driver === "sandbox"
? { provider: "daytona", reuseLease: false, runnerLifecycleMode: "per_turn" } : {} };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async (input) => ({ environment: input.environment, lease: makeLease(), leaseContext: { executionWorkspaceId: null, executionWorkspaceMode: null } }));
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) });
await orchestrator.acquireForRun({
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local", adapterType,
adapterConfig: {}, issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
persistedExecutionWorkspace: null, executionWorkspaceSettings: null,
});
expect(acquireRunLease.mock.calls[0][0].runnerReconnectGraceMs).toBe(expected);
});
});
describe("admitted native lifecycle recovery", () => {
it.each([
["warm", "per_turn", "per_turn", true],
["per_turn", "warm", "warm", false],
] as const)("preserves admitted %s over changed agent/environment settings", async (admittedLifecycleMode, lifecycleMode, runnerLifecycleMode, expectedReuse) => {
const environment = { ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false, runnerLifecycleMode } };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async (input) => ({ environment: input.environment, lease: makeLease(), leaseContext: { executionWorkspaceId: null, executionWorkspaceMode: null } }));
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) });
await orchestrator.acquireForRun({
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local",
adapterType: "paperclip_runner", adapterConfig: { lifecycleMode }, admittedLifecycleMode,
issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
persistedExecutionWorkspace: null, executionWorkspaceSettings: null,
});
expect(acquireRunLease.mock.calls[0][0].environment.config.reuseLease).toBe(expectedReuse);
expect(environment.config.reuseLease).toBe(false);
});
});
describe("live remote runner lease reattachment", () => {
const input = {
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local",
adapterType: "paperclip_runner", admittedLifecycleMode: "per_turn" as const,
issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
persistedExecutionWorkspace: { id: "ew-1", mode: "shared_workspace" as const },
executionWorkspaceSettings: null,
reattachRemoteLease: { leaseId: "lease-1", providerLeaseId: "original-sandbox", remoteCwd: "/remote/workspace" },
};
const originalLease = () => makeLease({
provider: "daytona", providerLeaseId: "original-sandbox", issueId: "task-1", executionWorkspaceId: "ew-1",
metadata: { agentId: "agent-1", remoteCwd: "/remote/workspace",
sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "original-sandbox" } },
});
beforeEach(() => {
vi.clearAllMocks();
mockGetEnvironment.mockResolvedValue({ ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false } });
mockGetLease.mockResolvedValue(originalLease());
});
it("readies the original active ephemeral lease without acquiring a replacement", async () => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime();
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
expect(mockGetLease).toHaveBeenCalledWith("lease-1");
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
// RC-54: the reattach resume carries the host reconnect grace, so the
// provider raises an older sandbox's shorter idle auto-stop.
expect(runtime.resumeRunLease).toHaveBeenCalledWith({
environment: result.environment, lease: savedLease, requestedCwd: "/remote/workspace",
runnerReconnectGraceMs: REMOTE_RUNNER_RECONNECT_GRACE_MS,
});
expect(result.lease).toEqual(savedLease);
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
expect(result.leaseContext.executionWorkspaceId).toBe("ew-1");
expect(result.environment.config.reuseLease).toBe(false);
});
it.each(["created", "replacement", "resumed"] as const)(
"reports exact-lease reattachment as resumed after historical %s acquisition", async outcome => {
const savedLease = originalLease();
savedLease.metadata!.sandboxLeaseAcquisition = {
outcome, providerLeaseId: "original-sandbox", previousProviderLeaseId: "older-sandbox", reason: "resume_failed",
};
const before = structuredClone(savedLease);
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime();
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
expect(result.lease.id).toBe(savedLease.id);
expect(result.lease.metadata?.remoteCwd).toBe("/remote/workspace");
expect(savedLease).toEqual(before);
expect(runtime.resumeRunLease).toHaveBeenCalledOnce();
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
},
);
it.each([false, true])("preserves acquisition history when realization persists workspace metadata (remote authority: %s)", async remoteAuthority => {
const savedLease = originalLease();
savedLease.metadata!.sandboxLeaseAcquisition = {
outcome: "replacement", providerLeaseId: "original-sandbox",
previousProviderLeaseId: "older-sandbox", reason: "resume_failed",
};
const history = structuredClone(savedLease.metadata!.sandboxLeaseAcquisition);
mockGetLease.mockResolvedValue(savedLease);
mockUpdateLeaseMetadata.mockImplementation(async (_id, metadata) => ({ ...savedLease, metadata }));
mockBuildWorkspaceRealizationRequest.mockReturnValue({ runtimeOverlay: {} });
const { resolveEnvironmentExecutionTarget } = await vi.importActual<typeof import("../services/environment-execution-target.js")>("../services/environment-execution-target.js");
mockResolveEnvironmentExecutionTarget.mockImplementation(resolveEnvironmentExecutionTarget);
const config = await import("../services/environment-config.js");
const resolveConfig = vi.spyOn(config, "resolveEnvironmentDriverConfigForRuntime").mockResolvedValue({
driver: "sandbox", config: { provider: "daytona", reuseLease: false },
} as never);
const workspace = makePersistedExecutionWorkspace({ metadata: remoteAuthority ? {
remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, source: { kind: "empty" },
writerRunId: randomUUID(), dirty: false, snapshotId: null, location: null },
} : null });
const prepare = vi.spyOn(remoteWorkspaces, "prepareRemoteWorkspace").mockImplementation(async args => ({
workspace, lease: args.lease, remoteRoot: "/remote/workspace", reference: {} as never,
} as never));
mockGetStoredWorkspace.mockResolvedValue(workspace);
try {
const runtime = makeMockRuntime({ supportsSync: () => false });
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: runtime });
const acquisition = await orchestrator.acquireForRun(input);
const result = await orchestrator.realizeForRun({
...makeRealizeInput(), ...acquisition, adapterType: input.adapterType, persistedExecutionWorkspace: workspace,
});
expect(mockUpdateLeaseMetadata).toHaveBeenCalled();
for (const [, metadata] of mockUpdateLeaseMetadata.mock.calls) {
expect(metadata.sandboxLeaseAcquisition).toEqual(history);
}
expect(result.lease.metadata!.sandboxLeaseAcquisition).toEqual(history);
expect(result.executionTarget).toMatchObject({ kind: "remote", transport: "sandbox",
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "original-sandbox" } });
} finally {
prepare.mockRestore();
resolveConfig.mockRestore();
mockResolveEnvironmentExecutionTarget.mockReset();
}
});
it("forwards the admitted warm idle window to the recovery resume", async () => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime();
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun({
...input, admittedLifecycleMode: "warm", adapterConfig: { lifecycleMode: "warm", idleTimeoutMs: 1_800_000 },
});
expect(runtime.resumeRunLease).toHaveBeenCalledWith({
environment: result.environment, lease: savedLease, requestedCwd: "/remote/workspace",
warmRunnerIdleTimeoutMs: 1_800_000, runnerReconnectGraceMs: REMOTE_RUNNER_RECONNECT_GRACE_MS,
});
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
});
it("does not return the retained sandbox to preparation while its lifecycle resume is pending", async () => {
let ready = false;
let release!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const savedLease = originalLease();
const before = structuredClone(savedLease);
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => {
await gate;
ready = true;
return { providerLeaseId: "original-sandbox", metadata: { remoteCwd: "/remote/workspace" } };
}) });
const pending = environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
try {
const state = await Promise.race([
pending.then(() => "returned"),
new Promise<string>(resolve => setImmediate(() => resolve("pending"))),
]);
expect(state).toBe("pending");
expect(ready).toBe(false);
} finally { release(); }
const result = await pending;
expect(result.lease).toEqual(before);
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
expect(savedLease).toEqual(before);
expect(ready).toBe(true);
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.execute).not.toHaveBeenCalled();
expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
});
it.each([
[null, "lease"],
[{ providerLeaseId: "replacement", metadata: { remoteCwd: "/remote/workspace" } }, "provider_lease_id"],
[{ providerLeaseId: "original-sandbox", metadata: { expired: true } }, "lease_expired"],
] as const)("holds recovery when the retained lease resume cannot confirm its original allocation (%j)", async (receipt, field) => {
const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => receipt) });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.rejects.toThrow(new Error(`native_remote_recovery_lease_mismatch: stage=environment_resume field=${field}`));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.execute).not.toHaveBeenCalled();
expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
});
it.each([undefined, { remoteCwd: "/allocation/root" }])(
"keeps the recorded workspace when resume returns allocation metadata (%j)", async metadata => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => ({
providerLeaseId: "original-sandbox", metadata,
})) });
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
expect(result.lease).toMatchObject({ id: savedLease.id, providerLeaseId: savedLease.providerLeaseId });
expect(result.lease.metadata?.remoteCwd).toBe("/remote/workspace");
expect(runtime.resumeRunLease).toHaveBeenCalledOnce();
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
},
);
it("names a provider-confirmed missing sandbox as a lease mismatch without acquiring a replacement", async () => {
const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => { throw new RetainedSandboxAbsentError(); }) });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.rejects.toThrow(new Error("native_remote_recovery_lease_mismatch: stage=environment_resume field=lease_expired"));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.execute).not.toHaveBeenCalled();
});
it("preserves a retained sandbox startup failure without acquiring a replacement", async () => {
const error = new Error("retained sandbox startup timed out");
const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => { throw error; }) });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.rejects.toBe(error);
expect(runtime.resumeRunLease).toHaveBeenCalledOnce();
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.execute).not.toHaveBeenCalled();
});
it("retries a provider gateway refusal during reattachment and keeps the original allocation", async () => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
const gateway = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
message: "<html><head><title>502 Bad Gateway</title></head></html>",
data: environmentResumeErrorData(Object.assign(new Error("gateway"), { statusCode: 502 })) });
const resumeRunLease = vi.fn()
.mockRejectedValueOnce(gateway)
.mockResolvedValueOnce({ providerLeaseId: "original-sandbox", metadata: { remoteCwd: "/remote/workspace" } });
const runtime = makeMockRuntime({ resumeRunLease });
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
expect(resumeRunLease).toHaveBeenCalledTimes(2);
expect(result.lease).toMatchObject({ id: savedLease.id, providerLeaseId: "original-sandbox" });
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
});
it("gives the first reattach resume the provider deadline and bounds every retry by the budget", async () => {
const gateway = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "web server returned an unknown error",
data: environmentResumeErrorData(Object.assign(new Error("gateway"), { statusCode: 520 })) });
const budgetMs = 1_500;
const timeouts: Array<number | undefined> = [];
const resumeRunLease = vi.fn(async ({ timeoutMs }: { timeoutMs?: number }) => {
timeouts.push(timeoutMs);
await new Promise(resolve => setTimeout(resolve, Math.min(timeoutMs ?? 300, 300)));
throw gateway;
});
const runtime = makeMockRuntime({ resumeRunLease });
const startedAt = Date.now();
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime, reattachResumeRetryBudgetMs: budgetMs })
.acquireForRun(input)).rejects.toBe(gateway);
expect(Date.now() - startedAt).toBeLessThan(300 + budgetMs + 250);
expect(timeouts[0]).toBeUndefined();
expect(timeouts.length).toBeGreaterThan(2);
const retries = timeouts.slice(1) as number[];
expect(retries.every(timeout => timeout <= budgetMs)).toBe(true);
expect(retries.every((timeout, index) => index === 0 || timeout < retries[index - 1]!)).toBe(true);
});
it("does not repeat a reattach resume that used up the provider's start deadline", async () => {
const deadline = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.TIMEOUT, message: "environmentResumeLease timed out" });
const resumeRunLease = vi.fn(async () => { throw deadline; });
const runtime = makeMockRuntime({ resumeRunLease });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input)).rejects.toBe(deadline);
expect(resumeRunLease).toHaveBeenCalledOnce();
});
it.each([500, 502, 520, 599])("treats a provider HTTP %i refusal during reattachment as transient", async statusCode => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
const refusal = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "gateway",
data: environmentResumeErrorData(Object.assign(new Error("gateway"), { statusCode })) });
const resumeRunLease = vi.fn()
.mockRejectedValueOnce(refusal)
.mockResolvedValueOnce({ providerLeaseId: "original-sandbox", metadata: { remoteCwd: "/remote/workspace" } });
const runtime = makeMockRuntime({ resumeRunLease });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.resolves.toMatchObject({ lease: { id: savedLease.id } });
expect(resumeRunLease).toHaveBeenCalledTimes(2);
});
it("retries a failed name lookup while reattaching to a host this run already reached", async () => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
const lookup = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "getaddrinfo ENOTFOUND app.daytona.io",
data: environmentResumeErrorData(Object.assign(new Error("dns"), { code: "ENOTFOUND" })) });
const resumeRunLease = vi.fn()
.mockRejectedValueOnce(lookup)
.mockResolvedValueOnce({ providerLeaseId: "original-sandbox", metadata: { remoteCwd: "/remote/workspace" } });
const runtime = makeMockRuntime({ resumeRunLease });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.resolves.toMatchObject({ lease: { id: savedLease.id } });
expect(resumeRunLease).toHaveBeenCalledTimes(2);
});
it.each([
["a caller's cancellation", { code: "ABORT_ERR" }],
["HTTP 501", { statusCode: 501 }],
])("does not retry %s during reattachment", async (_label, fields) => {
const refusal = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "refused",
data: environmentResumeErrorData(Object.assign(new Error("refused"), fields)) });
const resumeRunLease = vi.fn(async () => { throw refusal; });
const runtime = makeMockRuntime({ resumeRunLease });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input)).rejects.toBe(refusal);
expect(resumeRunLease).toHaveBeenCalledOnce();
});
it("does not retry a definite provider refusal during reattachment", async () => {
const refusal = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "conflict",
data: environmentResumeErrorData(Object.assign(new Error("conflict"), { statusCode: 409 })) });
const resumeRunLease = vi.fn(async () => { throw refusal; });
const runtime = makeMockRuntime({ resumeRunLease });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input)).rejects.toBe(refusal);
expect(resumeRunLease).toHaveBeenCalledOnce();
});
it("stops retrying a gateway refusal once the reattachment budget is spent", async () => {
const gateway = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "bad gateway",
data: environmentResumeErrorData(Object.assign(new Error("gateway"), { statusCode: 503 })) });
const resumeRunLease = vi.fn(async () => { throw gateway; });
const runtime = makeMockRuntime({ resumeRunLease });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime, reattachResumeRetryBudgetMs: 1_200 })
.acquireForRun(input)).rejects.toBe(gateway);
expect(resumeRunLease.mock.calls.length).toBeGreaterThan(1);
expect(resumeRunLease.mock.calls.length).toBeLessThan(6);
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
});
it.each(["active", "foreign-company", "revoked", "undispatched", "stopped"])("inspects the original plugin task lease (%s)", async state => {
mockGetEnvironment.mockResolvedValue(makeEnvironment("plugin"));
const lease = originalLease();
lease.metadata = { agentId: "agent-1", runnerTaskLease: true,
runnerTaskDispatched: state !== "undispatched", runnerTaskExecutionStopped: state === "stopped" };
if (state === "foreign-company") lease.companyId = "foreign";
if (state === "revoked") lease.status = "released";
mockGetLease.mockResolvedValue(lease);
const runtime = makeMockRuntime();
const acquired = environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun({
...input, reattachRemoteLease: { kind: "plugin_task", leaseId: lease.id },
});
if (state === "active") await expect(acquired).resolves.toMatchObject({ lease });
else await expect(acquired).rejects.toThrow(new Error("native_remote_recovery_lease_mismatch: stage=environment_acquire field="
+ (state === "foreign-company" ? "lease_scope" : state === "revoked" ? "lease_status" : "plugin_task")));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.resumeRunLease).not.toHaveBeenCalled();
});
it.each([
[{ companyId: "foreign" }, "lease_scope"], [{ environmentId: "foreign" }, "lease_scope"],
[{ heartbeatRunId: "another-run" }, "lease_scope"], [{ issueId: "another-task" }, "lease_scope"],
[{ executionWorkspaceId: "another-workspace" }, "lease_scope"],
[{ providerLeaseId: "replacement" }, "provider_lease_id"], [{ provider: "another-provider" }, "provider"],
[{ status: "released" }, "lease_status"], [{ expiresAt: new Date(0) }, "lease_expired"],
[{ releasedAt: new Date() }, "lease_status"], [{ cleanupStatus: "pending" }, "lease_status"],
[{ metadata: { agentId: "another-agent", remoteCwd: "/remote/workspace" } }, "lease_scope"],
[{ metadata: { agentId: "agent-1", remoteCwd: "/other/workspace" } }, "remote_cwd"],
] as [Partial<EnvironmentLease>, string][])("fails closed before provider acquisition for a mismatched lease %j", async (override, field) => {
mockGetLease.mockResolvedValue({ ...originalLease(), ...override });
const runtime = makeMockRuntime();
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.rejects.toThrow(new Error(`native_remote_recovery_lease_mismatch: stage=environment_acquire field=${field}`));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.resumeRunLease).not.toHaveBeenCalled();
});
it("never replaces a missing original lease", async () => {
mockGetLease.mockResolvedValue(null);
const runtime = makeMockRuntime();
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.rejects.toThrow(new Error("native_remote_recovery_lease_mismatch: stage=environment_acquire field=lease"));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
});
it("names a changed adapter type, not the environment driver, for a plugin task reattachment", async () => {
mockGetEnvironment.mockResolvedValue(makeEnvironment("plugin"));
const lease = originalLease();
lease.metadata = { agentId: "agent-1", runnerTaskLease: true, runnerTaskDispatched: true };
mockGetLease.mockResolvedValue(lease);
const runtime = makeMockRuntime();
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun({
...input, adapterType: "codex_local", reattachRemoteLease: { kind: "plugin_task", leaseId: lease.id },
})).rejects.toThrow(new Error("native_remote_recovery_lease_mismatch: stage=environment_acquire field=adapter_type"));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
});
});