Files
PaperClipAI/server/src/__tests__/environment-run-orchestrator.test.ts
T
DottaandPaperclip 08be8be414 fix(recovery): report exact retained sandbox resumes as resumed (#21259)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A controller restart must keep the same remote execution owner.
> - A retained sandbox lease stores the result of its first acquisition.
> - That result can say replacement after the same allocation resumes.
> - The Runner then rejects a valid resume as a changed sandbox.
> - This pull request reports the verified operation as resumed.
> - The saved acquisition history and identity checks stay intact.

## Linked Issues or Issue Description

Refs #18934. That change makes the retained sandbox ready before
preparation. This change corrects the acquisition receipt after that
resume.

**What happened?**
A remote native run resumed the exact retained sandbox after a
controller restart. The lease still carried a historical replacement
receipt. Runner verification rejected it with
`native_remote_recovery_lease_mismatch`, stage `runner_verification`,
field `replacement_sandbox`.

**What did you expect to happen?**
Report the verified exact-allocation operation as resumed. Preserve the
original saved history. Reject a changed, absent or expired allocation.

**Steps to reproduce**
Save an active lease with historical replacement metadata. Resume that
exact provider allocation through `acquireForRun`. Resolve its execution
target. Before the fix, the target still reports replacement and Runner
verification refuses it.

**Deployment mode**
Native Runner on a remote sandbox. The reported provider was Daytona.

## What Changed

- Return a separate operation receipt after the exact retained
allocation passes the existing resume checks.
- Pass the receipt through heartbeat preparation and apply it only to
the execution target.
- Keep the saved lease metadata and workspace unchanged.
- Test historical created, replacement and resumed receipts. Test
ordinary and remote-authority realization through the real execution
target resolver. Check every saved metadata write.
- Document the difference between saved acquisition history and the
current resume operation.

## Verification

- Head: `74d03e8ddadd1274891bea99a10455795c500557`.
- Red: three historical receipt cases fail without the resume receipt.
- Review regression red: realization writes resumed over the historical
replacement receipt. The saved metadata loses the previous allocation
and cause.
- Green: all 69 orchestrator tests pass. Ordinary and remote-authority
realization preserve history in every metadata write and return resumed
in the target.
- Green: 135 focused orchestrator and remote controller adoption cases
pass. Identity and changed-allocation refusals remain effective.
- Server typecheck passes, including its TypeScript build dependencies
and test layers.
- `git diff --check` passes.
- `pnpm verify` exits 2 because the execution-semantics document needs
broad CI. Required CI supplies that coverage.
- Linux x64, Node 25.9.0. Provider boundaries are simulated.
- The first post-merge attempt could not collect tests because the
compiled plugin SDK lacked the new taskLease schema. Shared and plugin
SDK builds passed, then the unchanged selection passed.
- Full preflight stopped because Cargo is absent. Dependency
installation completed. The later server TypeScript build and typecheck
passed.
- Live retained Runner reattachment is not verified. This workspace has
no provider credentials. No rollout or original-task recovery by this
patch is claimed.
- Required current-head CI and automated review remain pending.

## Risks

A wrong receipt could hide an actual replacement. The projection occurs
only after the existing company, task, run, agent, workspace, provider
allocation, expiry and directory checks pass. The provider must resume
the exact allocation. Runner process and identity verification still
runs. No replacement is acquired and no saved acquisition history is
changed. The heartbeat change only carries the typed receipt between the
owning orchestrator stages. Callers without a receipt keep their
existing behavior.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
execution and tool use. The exact backend snapshot and context window
are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:55:55 -06:00

1193 lines
54 KiB
TypeScript

import { RetainedSandboxAbsentError } from "../services/retained-sandbox-absent-error.js";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { randomUUID } from "node:crypto";
import * as remoteWorkspaces from "../services/remote-workspaces.js";
// ---------------------------------------------------------------------------
// Hoisted mocks — must be declared before any imports that reference them
// ---------------------------------------------------------------------------
const mockResolveEnvironmentExecutionTarget = vi.hoisted(() => vi.fn());
const mockAdapterExecutionTargetToRemoteSpec = vi.hoisted(() => vi.fn());
const mockBuildWorkspaceRealizationRequest = vi.hoisted(() => vi.fn());
const mockUpdateLeaseMetadata = vi.hoisted(() => vi.fn());
const mockUpdateExecutionWorkspace = vi.hoisted(() => vi.fn());
const mockGetStoredWorkspace = vi.hoisted(() => vi.fn());
const mockLogActivity = vi.hoisted(() => vi.fn());
const mockLoggerInfo = vi.hoisted(() => vi.fn());
const mockGetEnvironment = vi.hoisted(() => vi.fn());
const mockGetLease = vi.hoisted(() => vi.fn());
vi.mock("../services/environment-execution-target.js", () => ({
resolveEnvironmentExecutionTarget: mockResolveEnvironmentExecutionTarget,
resolveEnvironmentExecutionTransport: vi.fn().mockResolvedValue(null),
}));
vi.mock("@paperclipai/adapter-utils/execution-target", () => ({
adapterExecutionTargetToRemoteSpec: mockAdapterExecutionTargetToRemoteSpec,
}));
vi.mock("../services/workspace-realization.js", () => ({
buildWorkspaceRealizationRequest: mockBuildWorkspaceRealizationRequest,
}));
vi.mock("../services/environments.js", () => ({
assertEnvironmentCompanyBinding: vi.fn(async () => {}),
environmentService: vi.fn(() => ({
ensureLocalEnvironment: vi.fn(),
getById: mockGetEnvironment,
getLeaseById: mockGetLease,
acquireLease: vi.fn(),
releaseLease: vi.fn(),
updateLeaseMetadata: mockUpdateLeaseMetadata,
})),
}));
vi.mock("../services/execution-workspaces.js", () => ({
executionWorkspaceService: vi.fn(() => ({
update: mockUpdateExecutionWorkspace,
getStoredById: mockGetStoredWorkspace,
})),
}));
vi.mock("../services/activity-log.js", () => ({
logActivity: mockLogActivity,
}));
vi.mock("../middleware/logger.js", () => ({
logger: {
info: mockLoggerInfo,
warn: vi.fn(),
error: vi.fn(),
debug: vi.fn(),
},
}));
// ---------------------------------------------------------------------------
// Imports after mocks
// ---------------------------------------------------------------------------
import {
environmentRunOrchestrator,
EnvironmentRunError,
} from "../services/environment-run-orchestrator.ts";
import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "../services/native-runtime/native-runner-recovery-contract.js";
import type { Environment, EnvironmentLease, ExecutionWorkspace } from "@paperclipai/shared";
import type { RealizedExecutionWorkspace } from "../services/workspace-runtime.ts";
import type { EnvironmentRuntimeService } from "../services/environment-runtime.ts";
// ---------------------------------------------------------------------------
// Fixtures
// ---------------------------------------------------------------------------
function makeEnvironment(driver: string = "local"): Environment {
return {
id: "env-1",
name: "Test Environment",
description: null,
driver: driver as Environment["driver"],
status: "active",
config: {},
envVars: {},
metadata: null,
createdAt: new Date(),
updatedAt: new Date(),
};
}
function makeLease(overrides: Partial<EnvironmentLease> = {}): EnvironmentLease {
return {
id: "lease-1",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
status: "active",
leasePolicy: "ephemeral",
provider: "local",
providerLeaseId: null,
acquiredAt: new Date(),
lastUsedAt: new Date(),
expiresAt: null,
releasedAt: null,
failureReason: null,
cleanupStatus: null,
metadata: null,
createdAt: new Date(),
updatedAt: new Date(),
...overrides,
};
}
function makeExecutionWorkspace(cwd: string = "/workspace/project"): RealizedExecutionWorkspace {
return {
baseCwd: "/workspace",
source: "project_primary",
projectId: "project-1",
workspaceId: "ws-1",
repoUrl: null,
repoRef: null,
strategy: "project_primary",
cwd,
branchName: null,
worktreePath: null,
branchCreatedByRuntime: false,
warnings: [],
created: false,
};
}
function makePersistedExecutionWorkspace(
overrides: Partial<ExecutionWorkspace> = {},
): ExecutionWorkspace {
return {
id: "ew-1",
companyId: "company-1",
projectId: "project-1",
projectWorkspaceId: null,
sourceIssueId: null,
mode: "shared_workspace",
strategyType: "project_primary",
name: "workspace",
status: "active",
cwd: "/workspace/project",
repoUrl: null,
baseRef: null,
branchName: null,
providerType: "local_fs",
providerRef: null,
derivedFromExecutionWorkspaceId: null,
lastUsedAt: new Date(),
openedAt: new Date(),
closedAt: null,
cleanupEligibleAt: null,
cleanupReason: null,
config: null,
metadata: null,
deliveryState: "unknown",
createdAt: new Date(),
updatedAt: new Date(),
...overrides,
};
}
function makeRealizeInput(overrides: {
environment?: Environment;
lease?: EnvironmentLease;
persistedExecutionWorkspace?: ExecutionWorkspace | null;
} = {}): Parameters<ReturnType<typeof environmentRunOrchestrator>["realizeForRun"]>[0] {
return {
environment: overrides.environment ?? makeEnvironment("local"),
lease: overrides.lease ?? makeLease(),
adapterType: "claude_local",
companyId: "company-1",
issueId: null,
heartbeatRunId: "run-1",
executionWorkspace: makeExecutionWorkspace(),
effectiveExecutionWorkspaceMode: null,
persistedExecutionWorkspace: overrides.persistedExecutionWorkspace !== undefined
? overrides.persistedExecutionWorkspace
: null,
};
}
function makeMockRuntime(overrides: Partial<EnvironmentRuntimeService> = {}): EnvironmentRuntimeService {
return {
acquireRunLease: vi.fn(),
resumeRunLease: vi.fn(async ({ lease, requestedCwd }: Parameters<EnvironmentRuntimeService["resumeRunLease"]>[0]) => ({
providerLeaseId: lease.providerLeaseId,
metadata: { ...lease.metadata, remoteCwd: requestedCwd ?? lease.metadata?.remoteCwd },
})),
releaseRunLeases: vi.fn(),
execute: vi.fn().mockResolvedValue({
exitCode: 0,
signal: null,
timedOut: false,
stdout: "",
stderr: "",
}),
realizeWorkspace: vi.fn().mockResolvedValue({
cwd: "/workspace/project",
metadata: {
workspaceRealization: {
version: 1,
mode: "copy",
authoritativeRoot: "/workspace/project",
pathAliases: [],
outboundRestorePaths: [],
driver: "local",
cwd: "/workspace/project",
},
},
}),
...overrides,
} as unknown as EnvironmentRuntimeService;
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
describe("environmentRunOrchestrator — realizeForRun", () => {
const mockDb = {} as any;
beforeEach(() => {
vi.clearAllMocks();
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "claude_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: null,
},
});
mockAdapterExecutionTargetToRemoteSpec.mockReturnValue({
kind: "local",
environmentId: "env-1",
leaseId: "lease-1",
});
mockUpdateLeaseMetadata.mockResolvedValue(null);
mockUpdateExecutionWorkspace.mockResolvedValue(null);
mockLogActivity.mockResolvedValue(undefined);
});
it.each([false, true])("only requests active-work cancellation for explicit Stop: %s", async (cancelActiveWork) => {
const releaseRunLeases = vi.fn().mockResolvedValue([]);
const runtime = makeMockRuntime({ releaseRunLeases });
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await orchestrator.releaseForRun({
heartbeatRunId: "run-1", companyId: "company-1", agentId: "agent-1",
providerResourceDisposition: "stop_and_retain", cancelActiveWork,
});
expect(releaseRunLeases).toHaveBeenCalledWith(
"run-1", "released", expect.any(Function), "stop_and_retain",
...(cancelActiveWork ? [true] : []),
);
});
it("enforces company bindings when a caller acquires a lease directly", async () => {
const runtime = makeMockRuntime();
await environmentRunOrchestrator(mockDb, { environmentRuntime: runtime }).acquireLease({
companyId: "company-1", environment: makeEnvironment(), issueId: null, agentId: "agent-1", heartbeatRunId: "run-1",
persistedExecutionWorkspace: null, executionWorkspaceSettings: null, adapterType: null,
});
expect(runtime.acquireRunLease).toHaveBeenCalledWith(expect.objectContaining({ assertCompanyBinding: true }));
});
it("realizes an enrolled remote workspace in place and persists its authoritative remote root", async () => {
const environment = { ...makeEnvironment("sandbox"), id: randomUUID() };
const lease = makeLease({ id: randomUUID(), environmentId: environment.id, provider: "daytona", providerLeaseId: "sandbox-allocation" });
const workspace = makePersistedExecutionWorkspace({ id: randomUUID(), providerType: "cloud_sandbox", metadata: {
remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, source: { kind: "empty" },
writerRunId: randomUUID(), dirty: false, snapshotId: null, location: null },
} });
const remoteRoot = `/sandbox/.paperclip-workspaces/${workspace.id}`;
const savedLease = { ...lease, metadata: { remoteCwd: remoteRoot } };
const target = { kind: "remote", transport: "sandbox", remoteCwd: "/sandbox", runner: { execute: vi.fn() } };
mockResolveEnvironmentExecutionTarget.mockResolvedValue(target);
mockGetStoredWorkspace.mockResolvedValue({ ...workspace, cwd: remoteRoot });
mockUpdateLeaseMetadata.mockResolvedValue(savedLease);
const prepare = vi.spyOn(remoteWorkspaces, "prepareRemoteWorkspace").mockResolvedValue({
workspace, lease: savedLease, remoteRoot, reference: {} as never,
} as never);
try {
const runtime = makeMockRuntime();
const input = { ...makeRealizeInput({ environment, lease, persistedExecutionWorkspace: workspace }), adapterType: "paperclip_runner" };
const result = await environmentRunOrchestrator(mockDb, { environmentRuntime: runtime }).realizeForRun(input);
expect(prepare).toHaveBeenCalledWith(expect.objectContaining({ workspace, target, source: { kind: "empty" } }));
expect(mockBuildWorkspaceRealizationRequest).not.toHaveBeenCalled();
expect(runtime.realizeWorkspace).not.toHaveBeenCalled();
expect(result.executionTarget).toMatchObject({ remoteCwd: remoteRoot,
workspaceRealization: { mode: "in_place", authoritativeRoot: remoteRoot, outboundRestorePaths: [] } });
expect(input.executionWorkspace.cwd).toBe(remoteRoot);
expect(mockUpdateLeaseMetadata).toHaveBeenCalledWith(lease.id, expect.objectContaining({ remoteCwd: remoteRoot }));
expect(result.lease.metadata?.remoteCwd).toBe(remoteRoot);
} finally { prepare.mockRestore(); }
});
it("happy path: returns lease, executionTarget, and remoteExecution on successful realization", async () => {
const executionTarget = { kind: "local", environmentId: "env-1", leaseId: "lease-1" };
const remoteExecution = { kind: "local", environmentId: "env-1", leaseId: "lease-1" };
mockResolveEnvironmentExecutionTarget.mockResolvedValue(executionTarget);
mockAdapterExecutionTargetToRemoteSpec.mockReturnValue(remoteExecution);
const runtime = makeMockRuntime();
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
const result = await orchestrator.realizeForRun(makeRealizeInput());
expect(result.lease).toBeDefined();
expect(result.executionTarget).toEqual({
...executionTarget,
workspaceRealization: {
mode: "copy",
authoritativeRoot: "/workspace/project",
pathAliases: [],
outboundRestorePaths: [],
},
});
expect(result.remoteExecution).toEqual(remoteExecution);
expect(result.workspaceRealization).toEqual(
expect.objectContaining({ version: 1, driver: "local" }),
);
expect(runtime.realizeWorkspace).toHaveBeenCalledOnce();
expect(mockResolveEnvironmentExecutionTarget).toHaveBeenCalledOnce();
});
it("uses an in-place authoritative root on the adapter execution target", async () => {
mockResolveEnvironmentExecutionTarget.mockResolvedValue({
kind: "remote",
transport: "sandbox",
remoteCwd: "/copied/workspace",
});
const runtime = makeMockRuntime({
realizeWorkspace: vi.fn().mockResolvedValue({
cwd: "/app",
metadata: {
workspaceRealization: {
version: 1,
mode: "in_place",
authoritativeRoot: "/app",
pathAliases: [],
outboundRestorePaths: [],
},
},
}),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
const result = await orchestrator.realizeForRun(
makeRealizeInput({ environment: makeEnvironment("sandbox") }),
);
expect(result.executionTarget).toEqual(expect.objectContaining({
kind: "remote",
transport: "sandbox",
remoteCwd: "/app",
workspaceRealization: {
mode: "in_place",
authoritativeRoot: "/app",
pathAliases: [],
outboundRestorePaths: [],
},
}));
});
it("realization failure: runtime.realizeWorkspace throws → EnvironmentRunError with code workspace_realization_failed", async () => {
const runtime = makeMockRuntime({
realizeWorkspace: vi.fn().mockRejectedValue(new Error("sandbox unreachable")),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await expect(orchestrator.realizeForRun(makeRealizeInput())).rejects.toSatisfy(
(err: unknown) =>
err instanceof EnvironmentRunError &&
err.code === "workspace_realization_failed" &&
err.environmentId === "env-1" &&
err.driver === "local",
);
expect(mockResolveEnvironmentExecutionTarget).not.toHaveBeenCalled();
});
it("target resolution failure: resolveEnvironmentExecutionTarget throws → EnvironmentRunError with code transport_resolution_failed", async () => {
mockResolveEnvironmentExecutionTarget.mockRejectedValue(new Error("network error"));
const runtime = makeMockRuntime();
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await expect(orchestrator.realizeForRun(makeRealizeInput())).rejects.toSatisfy(
(err: unknown) =>
err instanceof EnvironmentRunError &&
err.code === "transport_resolution_failed" &&
err.environmentId === "env-1",
);
});
it("non-sandbox driver skips workspace realization and goes straight to target resolution", async () => {
const environment = makeEnvironment("plugin" as Environment["driver"]);
const executionTarget = null;
mockResolveEnvironmentExecutionTarget.mockResolvedValue(executionTarget);
const runtime = makeMockRuntime();
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
const result = await orchestrator.realizeForRun(
makeRealizeInput({ environment }),
);
expect(runtime.realizeWorkspace).not.toHaveBeenCalled();
expect(result.workspaceRealization).toEqual({});
expect(result.executionTarget).toBeNull();
});
it("forwards authorized project IDs to task-only providers without staging directories", async () => {
const environment = makeEnvironment("plugin" as Environment["driver"]);
mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "remote", transport: "sandbox",
providerKey: "plugin", remoteCwd: "provider/project", runnerTask: vi.fn() });
const runtime = makeMockRuntime({ supportsTasks: vi.fn().mockResolvedValue(true) });
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
const input = { ...makeRealizeInput({ environment }), adapterType: "paperclip_runner",
runnerTaskProjectIds: ["anchor", "authorized-reference", "anchor"] };
input.executionWorkspace.projectId = "anchor";
const result = await orchestrator.realizeForRun(input);
expect(result.executionTarget).toMatchObject({ runnerTaskProjectIds: ["anchor", "authorized-reference"] });
expect(runtime.realizeWorkspace).not.toHaveBeenCalled();
expect(runtime.execute).not.toHaveBeenCalled();
});
it("persisted metadata is updated on lease and execution workspace after realization", async () => {
const persistedExecutionWorkspace = makePersistedExecutionWorkspace();
const updatedLease = makeLease({
metadata: { workspaceRealization: { version: 1, driver: "local", cwd: "/workspace/project" } },
});
const updatedEw = { ...persistedExecutionWorkspace, metadata: { workspaceRealizationRequest: {}, workspaceRealization: {} } };
mockUpdateLeaseMetadata.mockResolvedValue(updatedLease);
mockUpdateExecutionWorkspace.mockResolvedValue(updatedEw);
mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "local", environmentId: "env-1", leaseId: "lease-1" });
const runtime = makeMockRuntime();
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
const result = await orchestrator.realizeForRun(
makeRealizeInput({ persistedExecutionWorkspace }),
);
// Lease metadata should have been updated with workspaceRealization
expect(mockUpdateLeaseMetadata).toHaveBeenCalledOnce();
expect(mockUpdateLeaseMetadata).toHaveBeenCalledWith(
"lease-1",
expect.objectContaining({ workspaceRealization: expect.any(Object) }),
);
// Execution workspace metadata should have been updated
expect(mockUpdateExecutionWorkspace).toHaveBeenCalledOnce();
expect(mockUpdateExecutionWorkspace).toHaveBeenCalledWith(
"ew-1",
expect.objectContaining({
metadata: expect.objectContaining({
workspaceRealizationRequest: expect.any(Object),
workspaceRealization: expect.any(Object),
}),
}),
);
// The returned lease should reflect the updated value
expect(result.lease).toEqual(updatedLease);
expect(result.persistedExecutionWorkspace).toEqual(updatedEw);
});
it("runs a remote provision command after workspace realization when configured", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "claude_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: "npm install -g @anthropic-ai/claude-code",
},
});
mockResolveEnvironmentExecutionTarget.mockResolvedValue({
kind: "remote",
transport: "ssh",
remoteCwd: "/remote/workspace",
environmentId: "env-1",
leaseId: "lease-1",
spec: {
host: "ssh.example.test",
port: 22,
username: "ssh-user",
remoteCwd: "/remote/workspace",
remoteWorkspacePath: "/remote/workspace",
privateKey: null,
knownHosts: null,
strictHostKeyChecking: true,
},
});
const runtime = makeMockRuntime({
realizeWorkspace: vi.fn().mockResolvedValue({
cwd: "/remote/workspace",
metadata: {
workspaceRealization: {
version: 1,
transport: "ssh",
remote: { path: "/remote/workspace" },
},
},
}),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await orchestrator.realizeForRun(makeRealizeInput({
environment: makeEnvironment("ssh"),
}));
// The `ssh` driver runs the command on the remote host that shares the
// workspace path, so the configured provision command still runs there.
expect(runtime.execute).toHaveBeenCalledOnce();
expect(runtime.execute).toHaveBeenCalledWith(expect.objectContaining({
environment: expect.objectContaining({ driver: "ssh" }),
lease: expect.objectContaining({ id: "lease-1" }),
command: "bash",
args: ["-lc", "npm install -g @anthropic-ai/claude-code"],
cwd: "/remote/workspace",
env: {
SHELL: "/bin/bash",
},
}));
// The sandbox skip log is specific to the sandbox driver; ssh stays quiet.
expect(mockLoggerInfo).not.toHaveBeenCalled();
});
it("skips the host provision command for a sandbox environment and logs the skip", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "claude_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: "npm install -g @anthropic-ai/claude-code",
},
});
mockResolveEnvironmentExecutionTarget.mockResolvedValue({
kind: "remote",
transport: "sandbox",
providerKey: "e2b",
remoteCwd: "/remote/workspace",
environmentId: "env-1",
leaseId: "lease-1",
});
const runtime = makeMockRuntime({
realizeWorkspace: vi.fn().mockResolvedValue({
cwd: "/remote/workspace",
metadata: {
workspaceRealization: {
version: 1,
transport: "sandbox",
remote: { path: "/remote/workspace" },
},
},
}),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await orchestrator.realizeForRun(makeRealizeInput({
environment: makeEnvironment("sandbox"),
}));
// The sandbox receives the provisioned tree through the adapter stage.sync
// step, so the orchestrator must not run the host command in the sandbox.
expect(runtime.execute).not.toHaveBeenCalled();
// The skip is observable: exactly one log line records it with the driver.
expect(mockLoggerInfo).toHaveBeenCalledOnce();
expect(mockLoggerInfo).toHaveBeenCalledWith(
expect.objectContaining({ driver: "sandbox", environmentId: "env-1" }),
expect.stringContaining("Skip host provisionCommand"),
);
});
it("does not rerun the provision command during local environment realization", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "claude_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: "npm install -g @anthropic-ai/claude-code",
},
});
mockResolveEnvironmentExecutionTarget.mockResolvedValue({
kind: "local",
environmentId: "env-1",
leaseId: "lease-1",
});
const runtime = makeMockRuntime();
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await orchestrator.realizeForRun(makeRealizeInput({
environment: makeEnvironment("local"),
}));
// Local workspace provisioning already ran the command before realizeForRun.
expect(runtime.execute).not.toHaveBeenCalled();
// The sandbox skip log is specific to the sandbox driver; local stays quiet.
expect(mockLoggerInfo).not.toHaveBeenCalled();
});
it("runs project-level provision commands for ssh environments", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "gemini_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: "npm install -g @google/gemini-cli",
},
});
mockResolveEnvironmentExecutionTarget.mockResolvedValue({
kind: "remote",
transport: "ssh",
remoteCwd: "/remote/workspace",
environmentId: "env-1",
leaseId: "lease-1",
spec: {
host: "ssh.example.test",
port: 22,
username: "ssh-user",
remoteCwd: "/remote/workspace",
remoteWorkspacePath: "/remote/workspace",
privateKey: null,
knownHosts: null,
strictHostKeyChecking: true,
},
});
const runtime = makeMockRuntime({
realizeWorkspace: vi.fn().mockResolvedValue({
cwd: "/remote/workspace",
metadata: {
workspaceRealization: {
version: 1,
transport: "ssh",
remote: { path: "/remote/workspace" },
},
},
}),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await orchestrator.realizeForRun(makeRealizeInput({
environment: makeEnvironment("ssh"),
lease: makeLease({
provider: "ssh",
metadata: {
driver: "ssh",
remoteCwd: "/remote/workspace",
remoteWorkspacePath: "/remote/workspace",
host: "ssh.example.test",
port: 22,
username: "ssh-user",
},
}),
}));
expect(runtime.execute).toHaveBeenCalledWith(expect.objectContaining({
command: "bash",
args: ["-lc", "npm install -g @google/gemini-cli"],
}));
expect(mockResolveEnvironmentExecutionTarget).toHaveBeenCalledOnce();
});
it("surfaces remote provision command failures before resolving the adapter target", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
adapterType: "claude_local",
companyId: "company-1",
environmentId: "env-1",
executionWorkspaceId: null,
issueId: null,
heartbeatRunId: "run-1",
requestedMode: null,
source: {
kind: "project_primary",
localPath: "/workspace/project",
projectId: null,
projectWorkspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
branchName: null,
worktreePath: null,
},
runtimeOverlay: {
provisionCommand: "install-tool",
},
});
const runtime = makeMockRuntime({
execute: vi.fn().mockResolvedValue({
exitCode: 127,
signal: null,
timedOut: false,
stdout: "",
stderr: "/bin/sh: install-tool: not found\n",
}),
});
const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
await expect(orchestrator.realizeForRun(makeRealizeInput({
environment: makeEnvironment("ssh"),
}))).rejects.toSatisfy(
(err: unknown) =>
err instanceof EnvironmentRunError &&
err.code === "workspace_realization_failed" &&
String(err.message).includes("install-tool: not found"),
);
expect(mockResolveEnvironmentExecutionTarget).not.toHaveBeenCalled();
});
});
describe("native runner lifecycle changes before lease acquisition", () => {
it.each([undefined, "account:generation"])("uses a warm plugin lease only with a managed credential identity: %s", async managedAiCredentialIdentity => {
const environment = { ...makeEnvironment("plugin"), config: { pluginKey: "fixture", driverKey: "tasks", driverConfig: {} } };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async input => ({ environment: input.environment,
lease: makeLease({ leasePolicy: input.environment.config.runnerLifecycleMode === "warm" ? "reuse_by_environment" : "ephemeral" }), leaseContext: { executionWorkspaceId: null, executionWorkspaceMode: null } }));
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) });
const result = await orchestrator.acquireForRun({ companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local",
adapterType: "paperclip_runner", adapterConfig: { lifecycleMode: "warm" }, managedAiCredentialIdentity,
issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1", persistedExecutionWorkspace: null, executionWorkspaceSettings: null });
expect(result.environment.config.runnerLifecycleMode).toBe(managedAiCredentialIdentity ? "warm" : "per_turn");
expect(result.lease.leasePolicy).toBe(managedAiCredentialIdentity ? "reuse_by_environment" : "ephemeral");
expect(environment.config).not.toHaveProperty("runnerLifecycleMode");
});
it("requests a reusable lease when an existing task switches from per-turn to warm", async () => {
const environment = { ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false } };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async (input) => ({
environment: input.environment,
lease: makeLease({ leasePolicy: input.environment.config.reuseLease ? "reuse_by_environment" : "ephemeral" }),
leaseContext: { executionWorkspaceId: "existing-workspace", executionWorkspaceMode: "shared_workspace" as const },
}));
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) });
const input = {
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local",
adapterType: "paperclip_runner", issueId: "existing-task", heartbeatRunId: "run-1", agentId: "agent-1",
runnerTaskProjectIds:["anchor","authorized-reference"],
persistedExecutionWorkspace: { id: "existing-workspace", mode: "shared_workspace" as const },
executionWorkspaceSettings: null,
};
const cold = await orchestrator.acquireForRun({ ...input, adapterConfig: { lifecycleMode: "per_turn" } });
expect(cold.lease.leasePolicy).toBe("ephemeral");
const warm = await orchestrator.acquireForRun({ ...input, heartbeatRunId: "run-2", adapterConfig: { lifecycleMode: "warm", idleTimeoutMs: 60_000 } });
expect(warm.lease.leasePolicy).toBe("reuse_by_environment");
expect(warm.environment.config).toEqual({ provider: "daytona", reuseLease: true });
expect(acquireRunLease.mock.calls[1][0].persistedExecutionWorkspace).toEqual(input.persistedExecutionWorkspace);
expect(acquireRunLease.mock.calls[1][0].configuredEnvironment).toBe(environment);
expect(acquireRunLease.mock.calls[1][0].runnerTaskProjectIds).toEqual(input.runnerTaskProjectIds);
// Run-scoped adaptation must not change the shared environment or other agents.
expect(environment.config).toEqual({ provider: "daytona", reuseLease: false });
const coldAgain = await orchestrator.acquireForRun({ ...input, heartbeatRunId: "run-3", adapterConfig: { lifecycleMode: "per_turn" } });
expect(coldAgain.lease.leasePolicy).toBe("ephemeral");
});
// The remote-sandbox warm default asks the provider before it requests a
// reusable lease, and asks only for a run whose lifecycle nobody chose.
it.each([
["a provider with reusable leases", {}, true, true, true],
["a provider without reusable leases (E2B, Modal, fake)", {}, false, true, false],
["an explicit per-turn agent", { lifecycleMode: "per_turn" }, true, false, false],
["a managed Claude account", { provider: "acpx", acpxAgent: "claude", model: "claude-sonnet-4-6", managed: true }, true, false, false],
] as const)("defaults an inheriting sandbox run to warm only on %s", async (_name, agent, supported, probed, warm) => {
const environment = { ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false } };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async (input) => ({ environment: input.environment, lease: makeLease(), leaseContext: { executionWorkspaceId: null, executionWorkspaceMode: null } }));
const providerSupportsReusableLeases = vi.fn(async () => supported);
const orchestrator = environmentRunOrchestrator({} as never, {
environmentRuntime: { ...makeMockRuntime({ acquireRunLease }), providerSupportsReusableLeases } as never,
});
const { managed, ...adapterConfig } = { managed: false, ...agent };
await orchestrator.acquireForRun({
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local", adapterType: "paperclip_runner",
adapterConfig, managedAiCredentialIdentity: managed ? "account:generation" : undefined,
issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
persistedExecutionWorkspace: null, executionWorkspaceSettings: null,
});
expect(providerSupportsReusableLeases).toHaveBeenCalledTimes(probed ? 1 : 0);
expect(acquireRunLease.mock.calls[0][0].environment.config.reuseLease).toBe(warm);
expect(acquireRunLease.mock.calls[0][0].warmRunnerIdleTimeoutMs).toBe(warm ? 300_000 : undefined);
expect(environment.config.reuseLease).toBe(false);
});
it.each([
["paperclip_runner", undefined, undefined, false],
["paperclip_runner", "inherit", "per_turn", false],
["paperclip_runner", "per_turn", "warm", false],
["paperclip_runner", "warm", "per_turn", true],
["paperclip_runner", "inherit", "warm", true],
["codex_local", "inherit", "warm", false],
])("respects environment precedence and adapter scope (%s, %s, %s)", async (adapterType, runnerLifecycleMode, lifecycleMode, expectedReuse) => {
const environment = { ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false, runnerLifecycleMode } };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async (input) => ({ environment: input.environment, lease: makeLease(), leaseContext: { executionWorkspaceId: null, executionWorkspaceMode: null } }));
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) });
await orchestrator.acquireForRun({
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local", adapterType,
adapterConfig: { lifecycleMode }, issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
persistedExecutionWorkspace: null, executionWorkspaceSettings: null,
});
expect(acquireRunLease.mock.calls[0][0].environment.config.reuseLease).toBe(expectedReuse);
});
// RC-54: a per-turn remote runner sandbox must outlive a controller outage
// the host still recovers, so acquisition forwards the reconnect grace.
it.each([
["paperclip_runner", "sandbox", REMOTE_RUNNER_RECONNECT_GRACE_MS],
["claude_local", "sandbox", undefined],
["paperclip_runner", "local", undefined],
] as const)("forwards the remote runner reconnect grace for %s on %s", async (adapterType, driver, expected) => {
const environment = { ...makeEnvironment(driver), config: driver === "sandbox"
? { provider: "daytona", reuseLease: false, runnerLifecycleMode: "per_turn" } : {} };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async (input) => ({ environment: input.environment, lease: makeLease(), leaseContext: { executionWorkspaceId: null, executionWorkspaceMode: null } }));
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) });
await orchestrator.acquireForRun({
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local", adapterType,
adapterConfig: {}, issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
persistedExecutionWorkspace: null, executionWorkspaceSettings: null,
});
expect(acquireRunLease.mock.calls[0][0].runnerReconnectGraceMs).toBe(expected);
});
});
describe("admitted native lifecycle recovery", () => {
it.each([
["warm", "per_turn", "per_turn", true],
["per_turn", "warm", "warm", false],
] as const)("preserves admitted %s over changed agent/environment settings", async (admittedLifecycleMode, lifecycleMode, runnerLifecycleMode, expectedReuse) => {
const environment = { ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false, runnerLifecycleMode } };
mockGetEnvironment.mockResolvedValue(environment);
const acquireRunLease = vi.fn(async (input) => ({ environment: input.environment, lease: makeLease(), leaseContext: { executionWorkspaceId: null, executionWorkspaceMode: null } }));
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) });
await orchestrator.acquireForRun({
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local",
adapterType: "paperclip_runner", adapterConfig: { lifecycleMode }, admittedLifecycleMode,
issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
persistedExecutionWorkspace: null, executionWorkspaceSettings: null,
});
expect(acquireRunLease.mock.calls[0][0].environment.config.reuseLease).toBe(expectedReuse);
expect(environment.config.reuseLease).toBe(false);
});
});
describe("live remote runner lease reattachment", () => {
const input = {
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local",
adapterType: "paperclip_runner", admittedLifecycleMode: "per_turn" as const,
issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
persistedExecutionWorkspace: { id: "ew-1", mode: "shared_workspace" as const },
executionWorkspaceSettings: null,
reattachRemoteLease: { leaseId: "lease-1", providerLeaseId: "original-sandbox", remoteCwd: "/remote/workspace" },
};
const originalLease = () => makeLease({
provider: "daytona", providerLeaseId: "original-sandbox", issueId: "task-1", executionWorkspaceId: "ew-1",
metadata: { agentId: "agent-1", remoteCwd: "/remote/workspace",
sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "original-sandbox" } },
});
beforeEach(() => {
vi.clearAllMocks();
mockGetEnvironment.mockResolvedValue({ ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false } });
mockGetLease.mockResolvedValue(originalLease());
});
it("readies the original active ephemeral lease without acquiring a replacement", async () => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime();
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
expect(mockGetLease).toHaveBeenCalledWith("lease-1");
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
// RC-54: the reattach resume carries the host reconnect grace, so the
// provider raises an older sandbox's shorter idle auto-stop.
expect(runtime.resumeRunLease).toHaveBeenCalledWith({
environment: result.environment, lease: savedLease, requestedCwd: "/remote/workspace",
runnerReconnectGraceMs: REMOTE_RUNNER_RECONNECT_GRACE_MS,
});
expect(result.lease).toEqual(savedLease);
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
expect(result.leaseContext.executionWorkspaceId).toBe("ew-1");
expect(result.environment.config.reuseLease).toBe(false);
});
it.each(["created", "replacement", "resumed"] as const)(
"reports exact-lease reattachment as resumed after historical %s acquisition", async outcome => {
const savedLease = originalLease();
savedLease.metadata!.sandboxLeaseAcquisition = {
outcome, providerLeaseId: "original-sandbox", previousProviderLeaseId: "older-sandbox", reason: "resume_failed",
};
const before = structuredClone(savedLease);
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime();
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
expect(result.lease.id).toBe(savedLease.id);
expect(result.lease.metadata?.remoteCwd).toBe("/remote/workspace");
expect(savedLease).toEqual(before);
expect(runtime.resumeRunLease).toHaveBeenCalledOnce();
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
},
);
it.each([false, true])("preserves acquisition history when realization persists workspace metadata (remote authority: %s)", async remoteAuthority => {
const savedLease = originalLease();
savedLease.metadata!.sandboxLeaseAcquisition = {
outcome: "replacement", providerLeaseId: "original-sandbox",
previousProviderLeaseId: "older-sandbox", reason: "resume_failed",
};
const history = structuredClone(savedLease.metadata!.sandboxLeaseAcquisition);
mockGetLease.mockResolvedValue(savedLease);
mockUpdateLeaseMetadata.mockImplementation(async (_id, metadata) => ({ ...savedLease, metadata }));
mockBuildWorkspaceRealizationRequest.mockReturnValue({ runtimeOverlay: {} });
const { resolveEnvironmentExecutionTarget } = await vi.importActual<typeof import("../services/environment-execution-target.js")>("../services/environment-execution-target.js");
mockResolveEnvironmentExecutionTarget.mockImplementation(resolveEnvironmentExecutionTarget);
const config = await import("../services/environment-config.js");
const resolveConfig = vi.spyOn(config, "resolveEnvironmentDriverConfigForRuntime").mockResolvedValue({
driver: "sandbox", config: { provider: "daytona", reuseLease: false },
} as never);
const workspace = makePersistedExecutionWorkspace({ metadata: remoteAuthority ? {
remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, source: { kind: "empty" },
writerRunId: randomUUID(), dirty: false, snapshotId: null, location: null },
} : null });
const prepare = vi.spyOn(remoteWorkspaces, "prepareRemoteWorkspace").mockImplementation(async args => ({
workspace, lease: args.lease, remoteRoot: "/remote/workspace", reference: {} as never,
} as never));
mockGetStoredWorkspace.mockResolvedValue(workspace);
try {
const runtime = makeMockRuntime({ supportsSync: () => false });
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: runtime });
const acquisition = await orchestrator.acquireForRun(input);
const result = await orchestrator.realizeForRun({
...makeRealizeInput(), ...acquisition, adapterType: input.adapterType, persistedExecutionWorkspace: workspace,
});
expect(mockUpdateLeaseMetadata).toHaveBeenCalled();
for (const [, metadata] of mockUpdateLeaseMetadata.mock.calls) {
expect(metadata.sandboxLeaseAcquisition).toEqual(history);
}
expect(result.lease.metadata!.sandboxLeaseAcquisition).toEqual(history);
expect(result.executionTarget).toMatchObject({ kind: "remote", transport: "sandbox",
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "original-sandbox" } });
} finally {
prepare.mockRestore();
resolveConfig.mockRestore();
mockResolveEnvironmentExecutionTarget.mockReset();
}
});
it("forwards the admitted warm idle window to the recovery resume", async () => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime();
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun({
...input, admittedLifecycleMode: "warm", adapterConfig: { lifecycleMode: "warm", idleTimeoutMs: 1_800_000 },
});
expect(runtime.resumeRunLease).toHaveBeenCalledWith({
environment: result.environment, lease: savedLease, requestedCwd: "/remote/workspace",
warmRunnerIdleTimeoutMs: 1_800_000, runnerReconnectGraceMs: REMOTE_RUNNER_RECONNECT_GRACE_MS,
});
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
});
it("does not return the retained sandbox to preparation while its lifecycle resume is pending", async () => {
let ready = false;
let release!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const savedLease = originalLease();
const before = structuredClone(savedLease);
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => {
await gate;
ready = true;
return { providerLeaseId: "original-sandbox", metadata: { remoteCwd: "/remote/workspace" } };
}) });
const pending = environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
try {
const state = await Promise.race([
pending.then(() => "returned"),
new Promise<string>(resolve => setImmediate(() => resolve("pending"))),
]);
expect(state).toBe("pending");
expect(ready).toBe(false);
} finally { release(); }
const result = await pending;
expect(result.lease).toEqual(before);
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
expect(savedLease).toEqual(before);
expect(ready).toBe(true);
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.execute).not.toHaveBeenCalled();
expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
});
it.each([
[null, "lease"],
[{ providerLeaseId: "replacement", metadata: { remoteCwd: "/remote/workspace" } }, "provider_lease_id"],
[{ providerLeaseId: "original-sandbox", metadata: { expired: true } }, "lease_expired"],
] as const)("holds recovery when the retained lease resume cannot confirm its original allocation (%j)", async (receipt, field) => {
const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => receipt) });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.rejects.toThrow(new Error(`native_remote_recovery_lease_mismatch: stage=environment_resume field=${field}`));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.execute).not.toHaveBeenCalled();
expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
});
it.each([undefined, { remoteCwd: "/allocation/root" }])(
"keeps the recorded workspace when resume returns allocation metadata (%j)", async metadata => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => ({
providerLeaseId: "original-sandbox", metadata,
})) });
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
expect(result.lease).toMatchObject({ id: savedLease.id, providerLeaseId: savedLease.providerLeaseId });
expect(result.lease.metadata?.remoteCwd).toBe("/remote/workspace");
expect(runtime.resumeRunLease).toHaveBeenCalledOnce();
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
},
);
it("names a provider-confirmed missing sandbox as a lease mismatch without acquiring a replacement", async () => {
const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => { throw new RetainedSandboxAbsentError(); }) });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.rejects.toThrow(new Error("native_remote_recovery_lease_mismatch: stage=environment_resume field=lease_expired"));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.execute).not.toHaveBeenCalled();
});
it("preserves a retained sandbox startup failure without acquiring a replacement", async () => {
const error = new Error("retained sandbox startup timed out");
const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => { throw error; }) });
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.rejects.toBe(error);
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.execute).not.toHaveBeenCalled();
});
it.each(["active", "foreign-company", "revoked", "undispatched", "stopped"])("inspects the original plugin task lease (%s)", async state => {
mockGetEnvironment.mockResolvedValue(makeEnvironment("plugin"));
const lease = originalLease();
lease.metadata = { agentId: "agent-1", runnerTaskLease: true,
runnerTaskDispatched: state !== "undispatched", runnerTaskExecutionStopped: state === "stopped" };
if (state === "foreign-company") lease.companyId = "foreign";
if (state === "revoked") lease.status = "released";
mockGetLease.mockResolvedValue(lease);
const runtime = makeMockRuntime();
const acquired = environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun({
...input, reattachRemoteLease: { kind: "plugin_task", leaseId: lease.id },
});
if (state === "active") await expect(acquired).resolves.toMatchObject({ lease });
else await expect(acquired).rejects.toThrow(new Error("native_remote_recovery_lease_mismatch: stage=environment_acquire field="
+ (state === "foreign-company" ? "lease_scope" : state === "revoked" ? "lease_status" : "plugin_task")));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.resumeRunLease).not.toHaveBeenCalled();
});
it.each([
[{ companyId: "foreign" }, "lease_scope"], [{ environmentId: "foreign" }, "lease_scope"],
[{ heartbeatRunId: "another-run" }, "lease_scope"], [{ issueId: "another-task" }, "lease_scope"],
[{ executionWorkspaceId: "another-workspace" }, "lease_scope"],
[{ providerLeaseId: "replacement" }, "provider_lease_id"], [{ provider: "another-provider" }, "provider"],
[{ status: "released" }, "lease_status"], [{ expiresAt: new Date(0) }, "lease_expired"],
[{ releasedAt: new Date() }, "lease_status"], [{ cleanupStatus: "pending" }, "lease_status"],
[{ metadata: { agentId: "another-agent", remoteCwd: "/remote/workspace" } }, "lease_scope"],
[{ metadata: { agentId: "agent-1", remoteCwd: "/other/workspace" } }, "remote_cwd"],
] as [Partial<EnvironmentLease>, string][])("fails closed before provider acquisition for a mismatched lease %j", async (override, field) => {
mockGetLease.mockResolvedValue({ ...originalLease(), ...override });
const runtime = makeMockRuntime();
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.rejects.toThrow(new Error(`native_remote_recovery_lease_mismatch: stage=environment_acquire field=${field}`));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.resumeRunLease).not.toHaveBeenCalled();
});
it("never replaces a missing original lease", async () => {
mockGetLease.mockResolvedValue(null);
const runtime = makeMockRuntime();
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
.rejects.toThrow(new Error("native_remote_recovery_lease_mismatch: stage=environment_acquire field=lease"));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
});
it("names a changed adapter type, not the environment driver, for a plugin task reattachment", async () => {
mockGetEnvironment.mockResolvedValue(makeEnvironment("plugin"));
const lease = originalLease();
lease.metadata = { agentId: "agent-1", runnerTaskLease: true, runnerTaskDispatched: true };
mockGetLease.mockResolvedValue(lease);
const runtime = makeMockRuntime();
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun({
...input, adapterType: "codex_local", reattachRemoteLease: { kind: "plugin_task", leaseId: lease.id },
})).rejects.toThrow(new Error("native_remote_recovery_lease_mismatch: stage=environment_acquire field=adapter_type"));
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
});
});