Files
PaperClipAI/server/src/__tests__/plugin-workspaces.integration.test.ts
T
Nicky LeachandClaude Opus 5.5 1140984b82 feat(plugins): tell an opted-in workspace provider the run's environment (#21512)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin can be a project workspace provider and also an environment
driver for the agent runs that use those workspaces
> - Core asks the workspace provider to prepare the project workspace
before the run acquires its environment lease
> - The provider knows the run, agent, project, and execution workspace,
but not the environment that the run selected
> - A provider whose compute exists per run (for example, a sandbox that
it creates from the environment's driver config) cannot start or find
the run's compute for the workspace work
> - This pull request lets a provider opt in to receive the run's
selected environment, when that environment is one of the provider's own
drivers
> - The benefit is that such a provider can prepare the workspace on the
run's own compute, with no change to the order of operations in Core

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Plugin workspace preparation for runs whose environment is a driver of
the same plugin.

**Subsystem affected**
Plugin SDK workspace provider params, the plugin manifest workspace
provider declaration, and `callWorkspaceProvider` /
`preparePluginWorkspace` in the server.

**Current behavior**
`prepare` params carry `runId`, `agentId`, `projectId`, and
`executionWorkspaceId`. They carry `taskLease` only when Core acquired
the lease first. They never name the run's environment.

**Proposed behavior**
A provider that declares `workspaceProvider.workspaceEnvironmentVersion:
1` receives `environment: { id, driverKey, config }` when the run's
selected environment is a plugin environment of that same provider.
`config` is the environment's `driverConfig`, the value that the
driver's acquire receives.

## What Changed

- SDK: optional strict `environment` field on
`pluginWorkspaceParamsSchema`.
- Manifest: optional `workspaceProvider.workspaceEnvironmentVersion: 1`,
next to the sibling version fields. Core sends the new field only to
providers that declare it, because providers parse the params with the
strict schema.
- Server: `callWorkspaceProvider` adds `environment` only when the
provider opted in, the environment driver is `plugin`, its `pluginKey`
is this provider, and it names a `driverKey`. The field is not part of
the operation ID.
- Server: `preparePluginWorkspace` passes the run's environment to its
prepare call. This covers normal, verify-existing, fresh, and
replacement prepares.
- Calls that do not have the run's environment do not send it: finalize,
inspect, remove, reservation reconciliation, and the retry-candidate
verification in `fresh-workspace-continuation.ts`.
- Lease acquisition, the `sessionHandoffVersion` gate, and the order of
operations do not change.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
capability and its exceptions.

## Verification

- `packages/plugins/sdk/tests/workspace-provider.test.ts`: 11 passed.
The schema accepts the field and rejects malformed values.
- `packages/shared/src/validators/plugin.test.ts`: 65 passed. The
manifest accepts only version 1.
- `server`: `plugin-workspaces.integration`,
`plugin-fresh-workspaces.integration`, `project-environments`, and
`heartbeat/workspace-preparation`: 226 passed, against embedded
Postgres. The new integration test checks the exact shape, an unchanged
operation ID, verification prepares, and omission for an undeclared
provider, another plugin's environment, non-plugin drivers, a missing
`driverKey`, and no environment. It fails when the server stops passing
the environment.
- `pnpm typecheck` passes in `packages/shared`, `packages/plugins/sdk`,
and `server`.
- The full suite was not run locally. CI runs it.

## Risks

- Low risk. Providers that do not declare the capability get the same
params as before.
- `config` is the stored `driverConfig`, in the same form that the
driver's acquire gets. Secret references in it reach the provider in
that same form.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:43:10 -08:00

1493 lines
71 KiB
TypeScript

import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, PluginWorkspaceOperationError, workspaceOperationErrorData } from "@paperclipai/plugin-sdk";
import { isPluginWorkspaceReplay } from "../services/plugin-workspace-replay-context.js";
import { createPluginWorkspaceFixture } from "./helpers/plugin-workspaces.js";
import { randomUUID } from "node:crypto";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { and, eq, sql } from "drizzle-orm";
import {
createDb,
subscribeDatabaseWork,
environmentLeases,
environments,
executionWorkspaces,
heartbeatRuns,
issues,
issueWorkProducts,
plugins,
pluginCompanySettings,
} from "@paperclipai/db";
import {
callWorkspaceProvider,
preparePluginWorkspace,
inspectPluginWorkspace,
finalizePluginWorkspace,
removePluginWorkspace,
type WorkspaceWorkers,
} from "../services/plugin-workspaces.js";
import { verifyRetryWorkspace } from "../services/fresh-workspace-continuation.js";
import { executionWorkspaceService } from "../services/execution-workspaces.js";
import { WorkspaceProviderFailureError } from "../services/workspace-provider-failure.js";
import { findPluginWorkerUnavailableError, PluginWorkerUnavailableError } from "../services/plugin-worker-unavailable-error.js";
import { pluginRegistryService } from "../services/plugin-registry.js";
import { projectService } from "../services/projects.js";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./helpers/embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)(
"plugin workspace ownership",
() => {
let db: ReturnType<typeof createDb>,
stopDb: (() => Promise<void>) | undefined;
beforeAll(async () => {
const started =
await startEmbeddedPostgresTestDatabase("plugin-workspaces");
db = createDb(started.connectionString, { maxConnections: 1 });
stopDb = started.cleanup;
});
afterAll(async () => {
await db?.$client.end();
await stopDb?.();
});
const fixture = (companyId?: string) =>
createPluginWorkspaceFixture(db, companyId);
it.each(["ready", "lost_reply", "unconfirmed", "transport", "worker_lookup", "live_writer", "older_retained", "unsupported"] as const)(
"verifies the same checkout before choosing a fresh retry (%s)", async (state) => {
const f = await fixture();
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
if (state !== "unsupported") await db.update(plugins).set({ manifestJson: {
...plugin.manifestJson, workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, workspaceVerificationVersion: 1 },
} }).where(eq(plugins.id, f.pluginId));
if (state === "lost_reply") {
f.lose();
await expect(preparePluginWorkspace(db, f.workers, f.input)).rejects.toThrow();
} else {
await preparePluginWorkspace(db, f.workers, f.input);
}
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
const workspaceId = task.executionWorkspaceId!;
await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
if (state !== "lost_reply") await db.insert(environmentLeases).values({
companyId: f.companyId, issueId: f.issueId, heartbeatRunId: f.runId,
provider: "plugin:test:tasks", executionWorkspaceId: workspaceId, leasePolicy: "ephemeral",
status: state === "live_writer" ? "active" : "released",
releasedAt: state === "live_writer" ? null : new Date(),
});
if (state === "older_retained") {
const [older] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.input.agentId,
issueId: f.issueId, status: "succeeded", finishedAt: new Date() }).returning();
await db.insert(environmentLeases).values({
companyId: f.companyId, issueId: f.issueId, heartbeatRunId: older.id,
provider: "plugin:test:tasks", executionWorkspaceId: workspaceId, leasePolicy: "reuse_by_environment",
status: "retained", releasedAt: null,
});
}
const retryIntentId = randomUUID();
const [retry] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.input.agentId,
issueId: f.issueId, status: "running", contextSnapshot: { taskRetryRequestId: retryIntentId },
}).returning();
const fresh = { mode: "fresh_isolated_workspace" as const, retryIntentId, sourceRunId: f.runId, retainedSourceIds: [] };
const call = vi.fn<WorkspaceWorkers["call"]>(async (...args) => {
const result = await f.workers.call(...args) as { kind: string };
if (state === "transport") throw new Error("connection closed");
if (state === "unconfirmed") throw new JsonRpcCallError({
code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
message: "checkout incomplete",
data: workspaceOperationErrorData(new PluginWorkspaceOperationError("binding_mismatch")),
});
return { ...result, created: false };
});
const workers: WorkspaceWorkers = {
getWorker: (pluginId) => {
if (state === "worker_lookup") throw new Error("worker lookup timed out");
const worker = f.workers.getWorker(pluginId);
return worker && { ...worker, supportedMethods: ["workspaceOperation", "workspacePrepare"] };
},
call,
};
const verificationStarted = Date.now();
const result = verifyRetryWorkspace(db, workers, retry, f.issueId, null, fresh);
if (state === "transport" || state === "worker_lookup") {
if (state === "transport") await expect(result).rejects.toMatchObject({ kind: "transport", retryable: true });
else await expect(result).rejects.toThrow("worker lookup timed out");
return;
}
const verified = state === "ready" || state === "lost_reply";
expect(await result).toEqual(verified
? { mode: "reuse_issue_workspace", retryIntentId, sourceRunId: f.runId, workspaceId, cwd: null }
: fresh);
expect(f.calls.slice(1)).toHaveLength(["live_writer", "older_retained", "unsupported"].includes(state) ? 0 : 1);
if (f.calls.length > 1) {
expect(call).toHaveBeenCalledExactlyOnceWith(f.pluginId, "workspaceOperation", expect.objectContaining({
executionWorkspaceId: workspaceId, runId: retry.id, operation: expect.objectContaining({ verifyExisting: true }),
}), 12_000);
const deadline = Date.parse(f.calls[1].deadline);
expect(deadline).toBeGreaterThanOrEqual(verificationStarted + 10_000);
expect(deadline).toBeLessThanOrEqual(Date.now() + 10_000);
}
const [retained] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
expect(retained.providerRef).toBe(`checkout:${workspaceId}`);
});
it("sends the run's own-driver environment only to a provider that declares it", async () => {
const f = await fixture();
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
const declare = (workspaceEnvironmentVersion?: 1) => db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
capabilities: [...plugin.manifestJson.capabilities, "environment.drivers.register"],
environmentDrivers: [{ driverKey: "tasks", displayName: "Tasks", configSchema: { type: "object" }, supportsTasks: true }],
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, workspaceVerificationVersion: 1,
...(workspaceEnvironmentVersion ? { workspaceEnvironmentVersion } : {}) },
} }).where(eq(plugins.id, f.pluginId));
await declare(1);
const [row] = await db.insert(environments).values({ name: `Snapshot tasks ${f.pluginId}`, driver: "plugin", status: "active",
config: { pluginKey: plugin.pluginKey, driverKey: "tasks", driverConfig: { snapshot: "base", cpu: 2 } } }).returning();
const environment = { ...row, driver: "plugin" as const };
const expected = { id: row.id, driverKey: "tasks", config: { snapshot: "base", cpu: 2 } };
const prepared = await preparePluginWorkspace(db, f.workers, { ...f.input, environment });
expect(f.calls).toHaveLength(1);
expect(f.calls[0].environment).toEqual(expected);
// Retry verification of the retained checkout carries the same environment.
await preparePluginWorkspace(db, f.workers, { ...f.input, environment, existingWorkspaceId: prepared!.id, verifyExisting: true });
expect(f.calls[1]).toMatchObject({ environment: expected, operation: { verifyExisting: true } });
const operation = f.calls[0].operation;
const send = async (selected: Parameters<typeof callWorkspaceProvider>[2]["environment"]) => {
await callWorkspaceProvider(db, f.workers, { workspace: prepared!, runId: f.runId, agentId: f.input.agentId,
environment: selected, operation });
return f.calls.at(-1)!;
};
const withEnvironment = await send(environment);
expect(withEnvironment.environment).toEqual(expected);
const without = await send(undefined);
expect(without).not.toHaveProperty("environment");
// The environment is context only; the operation identity does not change.
expect(withEnvironment.operationId).toBe(without.operationId);
expect(await send({ ...environment, config: { pluginKey: plugin.pluginKey, driverKey: "tasks" } }))
.toMatchObject({ environment: { id: row.id, driverKey: "tasks", config: {} } });
for (const omitted of [
{ ...environment, config: { ...environment.config, pluginKey: "other.plugin" } },
{ ...environment, config: { pluginKey: plugin.pluginKey, driverConfig: { snapshot: "base" } } },
{ id: row.id, driver: "local" as const, config: {} },
{ id: row.id, driver: "sandbox" as const, config: { provider: "e2b", pluginKey: plugin.pluginKey, driverKey: "tasks" } },
null,
]) {
const request = await send(omitted);
expect(request).not.toHaveProperty("environment");
expect(request.operationId).toBe(without.operationId);
}
await declare();
const undeclared = await send(environment);
expect(undeclared).not.toHaveProperty("environment");
expect(undeclared.operationId).toBe(without.operationId);
});
it("polls bounded requests using the provider's advertised preparation deadline", async () => {
const f = await fixture();
const deadline = new Date(Date.now() + 30_000).toISOString();
const phases: string[] = [];
const workers = {
getWorker: () => ({ supportedMethods: ["workspaceOperation", "workspacePrepare"] }),
call: async (pluginId: string, method: string, params: any, timeout: number) => {
expect(method).toBe("workspacePrepare");
expect(timeout).toBeLessThanOrEqual(15_000);
phases.push(params.phase);
if (params.phase === "submit") return { kind: "pending", deadline };
expect(params.deadline).toBe(deadline);
if (phases.length === 2) throw new Error("lost status reply");
const { phase: _phase, ...original } = params;
return f.workers.call(pluginId, "workspaceOperation", original);
},
} as unknown as WorkspaceWorkers;
expect(await preparePluginWorkspace(db, workers, f.input)).toBeTruthy();
expect(phases).toEqual(["submit", "poll", "poll"]);
expect(f.calls).toHaveLength(1);
});
it("stops preparation at the provider deadline before issuing a poll", async () => {
const f = await fixture();
const workers = {
getWorker: () => ({ supportedMethods: ["workspaceOperation", "workspacePrepare"] }),
call: async (_pluginId: string, method: string, params: { phase: string }) => {
expect(method).toBe("workspacePrepare");
expect(params.phase).toBe("submit");
return { kind: "pending", deadline: new Date(Date.now() - 1).toISOString() };
},
} as unknown as WorkspaceWorkers;
await expect(preparePluginWorkspace(db, workers, f.input)).rejects.toMatchObject({
kind: "provider", reason: "deadline_exceeded", retryable: true,
});
});
it("rejects an oversized provider deadline before it can retain the workspace lifecycle lock", async () => {
const f = await fixture();
const workers = {
getWorker: () => ({ supportedMethods: ["workspaceOperation", "workspacePrepare"] }),
call: async (_pluginId: string, method: string, params: { phase: string }) => {
expect(method).toBe("workspacePrepare");
expect(params.phase).toBe("submit");
return { kind: "pending", deadline: "2099-01-01T00:00:00Z" };
},
} as unknown as WorkspaceWorkers;
await expect(preparePluginWorkspace(db, workers, f.input)).rejects.toMatchObject({
kind: "provider", reason: "unsupported_operation", retryable: false,
});
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
expect(task?.executionWorkspaceId).toBeTruthy();
expect(await preparePluginWorkspace(db, f.workers, { ...f.input, existingWorkspaceId: task!.executionWorkspaceId })).toBeTruthy();
});
it.each([
["missing", undefined],
["stopped", { status: "stopped", supportedMethods: [] }],
["starting", { status: "starting", supportedMethods: [] }],
] as const)("retries preparation when the ready provider's worker is %s, before sending a request", async (_state, worker) => {
const f = await fixture();
const workers = { getWorker: () => worker, call: f.workers.call.bind(f.workers) } as unknown as WorkspaceWorkers;
const failure = await preparePluginWorkspace(db, workers, f.input).then(() => null, (error: unknown) => error);
expect(failure).toBeInstanceOf(WorkspaceProviderFailureError);
expect((failure as WorkspaceProviderFailureError).summary()).toEqual({ kind: "worker_unavailable", reason: null, retryable: true });
expect(findPluginWorkerUnavailableError(failure)).toBeInstanceOf(PluginWorkerUnavailableError);
expect(f.calls).toHaveLength(0);
});
it("keeps a running worker without workspace operations an unavailable provider", async () => {
const f = await fixture();
const workers = { getWorker: () => ({ status: "running", supportedMethods: [] }), call: f.workers.call.bind(f.workers) } as unknown as WorkspaceWorkers;
const failure = await preparePluginWorkspace(db, workers, f.input).then(() => null, (error: unknown) => error);
expect(failure).not.toBeInstanceOf(WorkspaceProviderFailureError);
expect((failure as Error).message).toBe("The workspace storage plugin is unavailable");
expect(f.calls).toHaveLength(0);
});
it.each([1, undefined])("gates recovery hints by provider capability (%s)", async (version) => {
const f = await fixture();
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, recoveryPullRequestVersion: version as 1 | undefined } } }).where(eq(plugins.id, f.pluginId));
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
const recoveryPullRequest = { number: 42, headSha: "a".repeat(40) };
await inspectPluginWorkspace(db, f.workers, workspace!, recoveryPullRequest);
expect(f.calls.at(-1).operation.recoveryPullRequest).toEqual(version === 1 ? recoveryPullRequest : undefined);
let freshInspectionCompleted = false;
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), recoveryPullRequest, async () => {
freshInspectionCompleted = f.calls.at(-1).operation.kind === "inspect";
throw new Error("delivery changed");
})).rejects.toThrow("delivery changed");
expect(freshInspectionCompleted).toBe(true);
const [unchanged] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
expect(unchanged.metadata?.workspaceProviderRemoval).toBeUndefined();
f.loseRemoval();
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), recoveryPullRequest)).rejects.toThrow();
const [pending] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
expect((pending.metadata?.workspaceProviderRemoval as { operation: { recoveryPullRequest?: unknown } }).operation.recoveryPullRequest)
.toEqual(version === 1 ? recoveryPullRequest : undefined);
await removePluginWorkspace(db, f.workers, pending, "a".repeat(40), recoveryPullRequest, async () => { throw new Error("must not reauthorize pending removal"); });
expect(f.calls.at(-1).operation.recoveryPullRequest).toEqual(version === 1 ? recoveryPullRequest : undefined);
});
it.each([1, undefined])("replays a lost removal acknowledgement unchanged after capability flip (%s)", async (version) => {
const f = await fixture();
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
const setCapability = (recoveryPullRequestVersion: 1 | undefined) => db.update(plugins).set({ manifestJson: {
...plugin.manifestJson, workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, recoveryPullRequestVersion },
} }).where(eq(plugins.id, f.pluginId));
await setCapability(version as 1 | undefined);
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
f.loseRemoval();
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), { number: 42, headSha: "a".repeat(40) })).rejects.toThrow();
const first = f.calls.at(-1);
await db.update(plugins).set({ status: "disabled" }).where(eq(plugins.id, f.pluginId));
const count = f.calls.length;
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40))).rejects.toThrow("unavailable");
expect(f.calls).toHaveLength(count);
await db.update(plugins).set({ status: "ready" }).where(eq(plugins.id, f.pluginId));
await setCapability(version === 1 ? undefined : 1);
const [pending] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
await removePluginWorkspace(db, f.workers, pending, "a".repeat(40));
expect(f.calls.at(-1).operation).toEqual(first.operation);
expect(f.calls.at(-1).operationId).toBe(first.operationId);
expect(pending.metadata?.workspaceProviderRemoval).toMatchObject({ operation: first.operation, operationId: first.operationId });
const [removed] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
expect(removed.metadata?.workspaceProviderRemoved).toBe(true);
});
it.each(["legacy", "ambiguous", "mismatched"] as const)("fails closed for uncertain removal identities (%s)", async (identity) => {
const hint = identity !== "legacy";
const f = await fixture();
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, recoveryPullRequestVersion: 1 } } }).where(eq(plugins.id, f.pluginId));
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
f.loseRemoval();
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), hint ? { number: 42, headSha: "a".repeat(40) } : undefined)).rejects.toThrow();
const first = f.calls.at(-1);
const [pending] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
const intent = pending.metadata!.workspaceProviderRemoval as Record<string, unknown>;
if (identity === "mismatched") intent.operationId = "0".repeat(64);
else delete intent.operationId;
await db.update(executionWorkspaces).set({ metadata: { ...pending.metadata, workspaceProviderRemoval: intent } }).where(eq(executionWorkspaces.id, pending.id));
if (hint) {
const count = f.calls.length;
await expect(removePluginWorkspace(db, f.workers, pending, "a".repeat(40))).rejects.toThrow(identity === "ambiguous" ? "identity is ambiguous" : "identity does not match");
expect(f.calls).toHaveLength(count);
} else {
await removePluginWorkspace(db, f.workers, pending, "a".repeat(40));
expect(f.calls.at(-1).operationId).toBe(first.operationId);
expect(f.calls.at(-1).operation).toEqual(first.operation);
}
});
it.each(["refs/heads/main", null, undefined])("persists the actual prepared destination (%s)", async (destinationRef) => {
const f = await fixture();
const workers: WorkspaceWorkers = { ...f.workers, call: async (...args: Parameters<typeof f.workers.call>) => {
const result = await f.workers.call(...args) as { kind?: string };
return result.kind === "prepared" ? { ...result, ...(destinationRef !== undefined ? { destinationRef } : {}) } : result;
} };
const workspace = await preparePluginWorkspace(db, workers, f.input);
expect(workspace!.metadata?.workspaceProviderDestinationRef).toBe(destinationRef);
});
it.each([true, undefined])("wakes parked cleanup when a provider upgrade enables ancestry (%s)", async (supportsMergeAncestry) => {
const f = await fixture();
await preparePluginWorkspace(db, f.workers, f.input);
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId));
const svc = executionWorkspaceService(db, { pluginWorkerManager: f.workers, workspaceReaperCooldownDays: 0 });
await svc.sweepTerminalWorkspaces(100);
expect(await svc.nextTerminalWorkspaceSweepAt()).toBeNull();
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(false);
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.terminalWorkspaces, (event, hints) => {
events.push(event);
if (event === "settled") svc.terminalWorkspaceWorkSettled(hints);
});
try {
await pluginRegistryService(db).update(f.pluginId, { manifest: { ...plugin.manifestJson,
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry } } });
expect(events).toEqual(["intent", "settled"]);
} finally { unsubscribe(); }
await svc.sweepTerminalWorkspaces(100);
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(true);
expect(f.calls.some(call => call.operation.kind === "remove")).toBe(true);
});
it("wakes cleanup only for provider contract changes and activation", async () => {
const f = await fixture();
const registry = pluginRegistryService(db);
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.terminalWorkspaces, event => events.push(event));
try {
await registry.update(f.pluginId, { manifest: { ...plugin.manifestJson, description: "Updated description" } });
await registry.updateStatus(f.pluginId, { status: "disabled" });
expect(events).toEqual([]);
await registry.updateStatus(f.pluginId, { status: "ready" });
expect(events.splice(0)).toEqual(["intent", "settled"]);
await registry.updateStatus(f.pluginId, { status: "ready" });
await registry.upsertCompanySettings(f.pluginId, f.companyId, { enabled: false, settingsJson: {} });
expect(events).toEqual([]);
await registry.upsertCompanySettings(f.pluginId, f.companyId, { enabled: true, settingsJson: {} });
expect(events.splice(0)).toEqual(["intent", "settled"]);
await registry.upsertCompanySettings(f.pluginId, f.companyId, { enabled: true, settingsJson: {} });
expect(events).toEqual([]);
await db.transaction(async tx => {
await pluginRegistryService(tx as unknown as typeof db).update(f.pluginId, {
manifest: { ...plugin.manifestJson, workspaceProvider: undefined },
});
expect(events).toEqual(["intent"]);
});
expect(events.splice(0)).toEqual(["intent", "settled"]);
await registry.updateStatus(f.pluginId, { status: "disabled" });
await registry.updateStatus(f.pluginId, { status: "ready" });
await registry.update(f.pluginId, { manifest: { ...plugin.manifestJson, workspaceProvider: undefined, description: "Other plugin" } });
expect(events).toEqual([]);
} finally { unsubscribe(); }
});
it("keeps storage idle without a settled completion HEAD, even with a recorded PR", async () => {
const f = await fixture();
await preparePluginWorkspace(db, f.workers, f.input);
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId));
await db.insert(issueWorkProducts).values({ companyId: f.companyId, issueId: f.issueId,
type: "pull_request", provider: "github", title: "Work", status: "active",
url: "https://github.com/example/repo/pull/1" });
const options = { pluginWorkerManager: f.workers, workspaceReaperCooldownDays: 0,
resolveWorkspaceDelivery: async () => { throw new Error("No delivery lookup without a completion HEAD"); } };
const svc = executionWorkspaceService(db, options);
await svc.sweepTerminalWorkspaces(100);
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(false);
expect(await svc.nextTerminalWorkspaceSweepAt()).toBeNull();
svc.terminalWorkspaceWorkSettled(new Set([`issue:${randomUUID()}`]));
await svc.sweepTerminalWorkspaces(100);
expect(await svc.nextTerminalWorkspaceSweepAt()).toBeNull();
await executionWorkspaceService(db, options).sweepTerminalWorkspaces(100);
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(false);
});
it("uses bounded finalization only for opted-in providers, preserving maintenance inspection", async () => {
const f = await fixture();
const workspace = (await preparePluginWorkspace(db, f.workers, f.input))!;
const context = { runId: f.runId, agentId: f.input.agentId };
expect((await finalizePluginWorkspace(db, f.workers, workspace, context)).kind).toBe("inspection");
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, finalizationVersion: 1 } } }).where(eq(plugins.id, f.pluginId));
const completionIds: string[] = [];
const call = f.workers.call.bind(f.workers);
f.workers.call = async (...args) => {
const p = args[2] as any;
if (p.operation.kind !== "finalize") return call(...args);
expect(p.agentId).toBe(context.agentId);
completionIds.push(p.operationId);
expect(Date.parse(p.deadline) - Date.now()).toBeLessThanOrEqual(30_000);
expect(args[3]).toBe(40_000);
return { kind: "finalized", branchName: workspace.branchName, headSha: "a".repeat(40) };
};
expect((await finalizePluginWorkspace(db, f.workers, workspace, context)).kind).toBe("finalized");
await finalizePluginWorkspace(db, f.workers, workspace, context);
await finalizePluginWorkspace(db, f.workers, workspace, { ...context, runId: randomUUID() });
expect(completionIds[1]).toBe(completionIds[0]);
expect(completionIds[2]).not.toBe(completionIds[0]);
expect((await inspectPluginWorkspace(db, f.workers, workspace)).kind).toBe("inspection");
f.workers.call = async () => { throw new Error("sync failed"); };
await expect(finalizePluginWorkspace(db, f.workers, workspace, context)).rejects.toThrow("incomplete");
});
it("keeps the single-connection pool responsive while provider preparation holds its lifecycle lock", async () => {
const f = await fixture();
let entered!: () => void, release!: () => void;
const started = new Promise<void>((resolve) => {
entered = resolve;
});
const resume = new Promise<void>((resolve) => {
release = resolve;
});
const call = f.workers.call.bind(f.workers);
f.workers.call = async (...args) => {
if ((args[2] as any).operation.kind === "prepare") {
entered();
await resume;
}
return call(...args);
};
const preparing = preparePluginWorkspace(db, f.workers, f.input);
await started;
try {
await expect(
projectService(db).updateWorkspace(
f.input.projectId,
f.input.projectWorkspaceId,
{ repoRef: "next" },
),
).rejects.toThrow("Workspace lifecycle is busy");
expect(await db.execute(sql`select 1 as usable`)).toMatchObject([
{ usable: 1 },
]);
} finally {
release();
}
const workspace = await preparing;
await expect(
inspectPluginWorkspace(db, f.workers, workspace!),
).resolves.toMatchObject({ kind: "inspection" });
});
it("preserves a scheduled selection through the first provider binding and next admission", async () => {
const f = await fixture();
const svc = executionWorkspaceService(db, {
pluginWorkerManager: f.workers,
});
const actor = {
type: "board" as const,
source: "local_implicit" as const,
isInstanceAdmin: true,
};
const request = {
companyId: f.companyId,
issueId: f.issueId,
actor,
selection: { kind: "task_directory" as const },
expectedBindingRevision: 0,
requestKey: "during-prepare",
};
await svc.selectTaskWorkspace(request);
await preparePluginWorkspace(db, f.workers, f.input);
expect(
await svc.inspectTaskWorkspace(f.companyId, f.issueId, actor),
).toMatchObject({
bindingRevision: 2,
pendingSelection: {
expectedBindingRevision: 2,
intent: { request: { expectedBindingRevision: 0 } },
},
});
await db
.update(heartbeatRuns)
.set({ status: "succeeded" })
.where(eq(heartbeatRuns.id, f.runId));
expect(
await svc.applyPendingTaskWorkspaceSelection({
companyId: f.companyId,
issueId: f.issueId,
runId: randomUUID(),
actor,
}),
).toBe(true);
expect(await svc.selectTaskWorkspace(request)).toMatchObject({
kind: "applied",
});
});
it("does not retire reusable storage for rejected or normalized unchanged source edits", async () => {
const f = await fixture(),
workspace = await preparePluginWorkspace(db, f.workers, f.input);
await db
.update(executionWorkspaces)
.set({ status: "archived", closedAt: new Date() })
.where(eq(executionWorkspaces.id, workspace!.id));
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
expect(
await projectService(db).updateWorkspace(
f.input.projectId,
f.input.projectWorkspaceId,
{ repoRef: " " },
),
).toMatchObject({ repoRef: null });
expect(
await projectService(db).updateWorkspace(
f.input.projectId,
f.input.projectWorkspaceId,
{ repoUrl: null },
),
).toBeNull();
const [row] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, workspace!.id));
expect(row.metadata?.workspaceProviderRetired).not.toBe(true);
expect(
await executionWorkspaceService(db, {
pluginWorkerManager: f.workers,
}).reopenClosedIsolatedExecutionWorkspaceForIssue({
workspaceId: workspace!.id,
issue: {
id: f.issueId,
companyId: f.companyId,
projectId: f.input.projectId,
},
actor: { actorType: "user", agentId: null },
authorizationActor: {
type: "board",
source: "local_implicit",
isInstanceAdmin: true,
},
}),
).toMatchObject({ ok: true, reopened: true });
});
it("uses new inspection identities even when the provider caches operation results", async () => {
const f = await fixture(),
workspace = await preparePluginWorkspace(db, f.workers, f.input);
const call = f.workers.call.bind(f.workers),
cache = new Map<string, unknown>();
f.workers.call = async (...args) => {
const params = args[2] as any;
if (cache.has(params.operationId))
return cache.get(params.operationId) as any;
const result = await call(...args);
cache.set(params.operationId, result);
return result;
};
expect(
await inspectPluginWorkspace(db, f.workers, workspace!),
).toMatchObject({ hasDirtyTrackedFiles: false });
f.dirty();
expect(
await inspectPluginWorkspace(db, f.workers, workspace!),
).toMatchObject({ hasDirtyTrackedFiles: true });
expect(cache.size).toBe(2);
});
it("retries closed storage cleanup when the pre-delete inspection failed before recording removal", async () => {
const f = await fixture(),
workspace = await preparePluginWorkspace(db, f.workers, f.input);
await db
.update(issues)
.set({ status: "done", completedAt: new Date() })
.where(eq(issues.id, f.issueId));
await db
.update(heartbeatRuns)
.set({ status: "succeeded" })
.where(eq(heartbeatRuns.id, f.runId));
const call = f.workers.call.bind(f.workers);
let inspections = 0;
const workers = {
getWorker: (id: string) =>
id === f.pluginId
? { supportedMethods: ["workspaceOperation"] }
: undefined,
call: async (...args: Parameters<WorkspaceWorkers["call"]>) => {
if (
(args[2] as any).operation.kind === "inspect" &&
++inspections === 2
)
throw new Error("temporary inspection outage");
return call(...args);
},
} as unknown as WorkspaceWorkers;
let clock = Date.now();
const svc = executionWorkspaceService(db, {
pluginWorkerManager: workers,
workspaceReaperCooldownDays: 0,
now: () => new Date(clock),
});
const sweep = () => svc.sweepTerminalWorkspaces(100);
expect(await sweep()).toMatchObject({ cleanupFailed: 1 });
const [failed] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, workspace!.id));
expect(failed.status).toBe("cleanup_failed");
expect(failed.metadata?.workspaceProviderRemoval).toBeUndefined();
expect(failed.metadata?.terminalCleanupRetry).toMatchObject({ failures: 1, failure: { kind: "transport" },
retryAt: new Date(clock + 300_000).toISOString() });
expect(await sweep()).toMatchObject({ skippedBackoff: 1 });
clock += 300_000;
await sweep();
const [removed] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, workspace!.id));
expect(removed.metadata?.workspaceProviderRemoved).toBe(true);
expect(removed.status).toBe("archived");
expect(removed.metadata).not.toHaveProperty("terminalCleanupRetry");
});
it("prepares a durable reference without a local checkout or agent allocation", async () => {
const f = await fixture();
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
expect(workspace).toMatchObject({
providerType: "plugin",
cwd: null,
status: "active",
});
expect(f.calls[0].operation.source).not.toHaveProperty("cwd");
const [task] = await db
.select()
.from(issues)
.where(eq(issues.id, f.issueId));
expect(task.executionWorkspaceId).toBe(workspace!.id);
expect(
await inspectPluginWorkspace(db, f.workers, workspace!),
).toMatchObject({ remotelyRecoverable: true });
});
it("retains the reservation and retries the same operation after a lost response", async () => {
const f = await fixture();
f.lose();
await expect(
preparePluginWorkspace(db, f.workers, f.input),
).rejects.toThrow("incomplete");
const [task] = await db
.select()
.from(issues)
.where(eq(issues.id, f.issueId));
const result = await preparePluginWorkspace(db, f.workers, {
...f.input,
existingWorkspaceId: task.executionWorkspaceId,
});
expect(result!.id).toBe(task.executionWorkspaceId);
expect(f.calls[1].operationId).toBe(f.calls[0].operationId);
});
it("reconciles a lost preparation result after the task run stops", async () => {
const f = await fixture();
f.lose();
await expect(
preparePluginWorkspace(db, f.workers, f.input),
).rejects.toThrow("incomplete");
await db
.update(heartbeatRuns)
.set({ status: "cancelled" })
.where(eq(heartbeatRuns.id, f.runId));
const [task] = await db
.select()
.from(issues)
.where(eq(issues.id, f.issueId));
const [reservation] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
expect(reservation.providerRef).toBeNull();
await expect(
inspectPluginWorkspace(db, f.workers, reservation),
).resolves.toMatchObject({ remotelyRecoverable: true });
expect(f.calls[1]).toMatchObject({
runId: f.runId,
agentId: f.input.agentId,
operation: { kind: "prepare" },
});
expect(f.calls[1].operationId).toBe(f.calls[0].operationId);
const [recorded] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, reservation.id));
expect(recorded.providerRef).not.toBeNull();
});
it.each(["inspect", "retry", "replacement"] as const)("recovers a lost Launcher prepare through %s", async (recovery) => {
const f = await fixture();
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
capabilities: [...plugin.manifestJson.capabilities, "environment.drivers.register"],
environmentDrivers: [{ driverKey: "tasks", displayName: "Tasks", configSchema: { type: "object" },
supportsTasks: true, sessionHandoffVersion: 1 }],
} }).where(eq(plugins.id, f.pluginId));
const [environment] = await db.insert(environments).values({ name: `Workspace Launcher ${f.pluginId}`, driver: "plugin",
status: "active", config: { pluginKey: plugin.pluginKey, driverKey: "tasks" } }).returning();
const taskLease = { environmentId: environment.id, driverKey: "tasks", providerLeaseId: randomUUID() };
const leaseId = randomUUID();
const input = { ...f.input, environment: { ...environment, driver: "plugin" as const },
acquireTaskLease: async () => ({ leaseId, taskLease }) };
let deleted = false;
const replacementTaskLease = { ...taskLease, providerLeaseId: randomUUID() };
const replacementWorkers = { ...f.workers, call: vi.fn<WorkspaceWorkers["call"]>(async (...args) => {
const request = args[2] as (typeof f.calls)[number];
if (deleted && request.taskLease?.providerLeaseId === taskLease.providerLeaseId)
throw new Error("Previous Launcher allocation is gone");
const result = await f.workers.call(...args) as { kind: string };
if (!deleted) throw new Error("Lost committed preparation result");
return { ...result, created: false } as never;
}) };
if (recovery !== "replacement") f.lose();
await expect(preparePluginWorkspace(db, recovery === "replacement" ? replacementWorkers : f.workers, input)).rejects.toThrow("incomplete");
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
const [reservation] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
expect(reservation.providerRef).toBeNull();
expect(reservation.metadata?.workspaceProviderPreparationPlacement).toMatchObject({
generation: 0, runId: f.runId, agentId: f.input.agentId, taskLease,
});
if (recovery === "inspect") {
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.runId));
const workers = { ...f.workers, call: vi.fn<WorkspaceWorkers["call"]>(async (...args) => {
const request = args[2] as (typeof f.calls)[number];
expect(isPluginWorkspaceReplay({
pluginId: f.pluginId, companyId: f.companyId, projectId: f.input.projectId,
projectWorkspaceId: f.input.projectWorkspaceId, executionWorkspaceId: reservation.id,
})).toBe(request.operation.kind === "prepare");
return f.workers.call(...args);
}) };
await expect(inspectPluginWorkspace(db, workers, reservation)).resolves.toMatchObject({ remotelyRecoverable: true });
} else if (recovery === "retry") {
await expect(preparePluginWorkspace(db, f.workers, { ...input, existingWorkspaceId: reservation.id })).resolves.toMatchObject({ id: reservation.id });
} else {
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, f.runId));
const [nextRun] = await db.insert(heartbeatRuns).values({
companyId: f.companyId, issueId: f.issueId, agentId: f.input.agentId, status: "running",
}).returning();
deleted = true;
await expect(preparePluginWorkspace(db, replacementWorkers, {
...input, runId: nextRun.id, existingWorkspaceId: reservation.id,
acquireTaskLease: async () => ({ leaseId: randomUUID(), taskLease: replacementTaskLease }),
})).resolves.toMatchObject({ id: reservation.id });
expect(replacementWorkers.call).toHaveBeenCalledTimes(2);
expect(f.calls[1]).toMatchObject({ runId: nextRun.id, agentId: f.input.agentId, taskLease: replacementTaskLease });
expect(f.calls[1].operationId).not.toBe(f.calls[0].operationId);
}
if (recovery !== "replacement") expect(f.calls[1]).toMatchObject({
runId: f.runId, agentId: f.input.agentId, taskLease, operationId: f.calls[0].operationId,
});
expect(f.calls[1].operation).toEqual(f.calls[0].operation);
const [prepared] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, reservation.id));
expect(prepared.providerRef).not.toBeNull();
expect(prepared.metadata?.workspaceProviderPreparationPlacement).toBeUndefined();
});
it("reconciles an unfinished reservation before applying PR-only cleanup eligibility", async () => {
const f = await fixture();
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
f.lose();
await expect(preparePluginWorkspace(db, f.workers, f.input)).rejects.toThrow("incomplete");
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.runId));
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
f.dirty();
await executionWorkspaceService(db, { pluginWorkerManager: f.workers, workspaceReaperCooldownDays: 0 }).sweepTerminalWorkspaces(100);
expect(f.calls.filter(call => call.operation.kind === "prepare")).toHaveLength(2);
expect(f.calls.filter(call => call.operation.kind === "inspect")).toHaveLength(1);
expect(f.calls.some(call => call.operation.kind === "remove")).toBe(false);
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
expect(workspace.providerRef).not.toBeNull();
});
it("fails closed for a disabled plugin, stopped run, and cross-company source", async () => {
const f = await fixture();
await db.insert(pluginCompanySettings).values({
pluginId: f.pluginId,
companyId: f.companyId,
enabled: false,
});
await expect(
preparePluginWorkspace(db, f.workers, f.input),
).rejects.toThrow("unavailable");
await db
.update(pluginCompanySettings)
.set({ enabled: true })
.where(eq(pluginCompanySettings.pluginId, f.pluginId));
await db
.update(heartbeatRuns)
.set({ status: "cancelled" })
.where(eq(heartbeatRuns.id, f.runId));
await expect(
preparePluginWorkspace(db, f.workers, f.input),
).rejects.toThrow();
expect(f.calls).toHaveLength(0);
await expect(
preparePluginWorkspace(db, f.workers, {
...f.input,
companyId: randomUUID(),
}),
).resolves.toBeNull();
});
it("keeps source ownership immutable and preserves bindings through metadata edits", async () => {
const f = await fixture();
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
const svc = projectService(db);
await expect(
svc.update(f.input.projectId, {
executionWorkspacePolicy: { storageProviderPluginId: null },
}),
).rejects.toMatchObject({ status: 409, details: { code: "project_storage_locked" } });
await expect(
svc.updateWorkspace(f.input.projectId, f.input.projectWorkspaceId, {
repoUrl: "https://github.com/example/other.git",
}),
).rejects.toThrow("repository source");
await expect(
svc.removeWorkspace(f.input.projectId, f.input.projectWorkspaceId),
).rejects.toThrow("repository source");
const placement = { generation: 0, runId: f.runId, agentId: f.input.agentId,
taskLease: { environmentId: randomUUID(), driverKey: "tasks", providerLeaseId: randomUUID() } };
await db.update(executionWorkspaces).set({
metadata: { ...workspace!.metadata, workspaceProviderPreparationPlacement: placement },
}).where(eq(executionWorkspaces.id, workspace!.id));
const execution = executionWorkspaceService(db);
const updated = await execution.update(workspace!.id, {
metadata: {
note: "caller edit",
workspaceProvider: { workspaceRef: "foreign" },
workspaceProviderPreparationPlacement: { ...placement, runId: randomUUID() },
},
});
expect(updated!.metadata?.workspaceProviderPreparationPlacement).toBeUndefined();
const [saved] = await db.select({ metadata: executionWorkspaces.metadata })
.from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
expect(saved.metadata?.workspaceProviderPreparationPlacement).toEqual(placement);
expect(updated!.metadata?.workspaceProvider).toEqual(
workspace!.metadata?.workspaceProvider,
);
await expect(
execution.update(workspace!.id, { providerType: "local_fs" }),
).rejects.toThrow("storage plugin");
});
it("describes provider removal without promising host cleanup commands", async () => {
const f = await fixture();
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
await db
.update(issues)
.set({ status: "done" })
.where(eq(issues.id, f.issueId));
const readiness = await executionWorkspaceService(db, {
pluginWorkerManager: f.workers,
}).getCloseReadiness(workspace!.id);
expect(readiness!.plannedActions).toContainEqual(
expect.objectContaining({
kind: "remove_provider_workspace",
command: null,
}),
);
expect(
readiness!.plannedActions.some(
(action) => action.kind === "git_branch_delete",
),
).toBe(false);
});
it("reopens through the same provider without requiring a live run or a host checkout", async () => {
const f = await fixture();
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
await db
.update(heartbeatRuns)
.set({ status: "succeeded" })
.where(eq(heartbeatRuns.id, f.runId));
await db
.update(executionWorkspaces)
.set({ status: "archived", closedAt: new Date() })
.where(eq(executionWorkspaces.id, workspace!.id));
const result = await executionWorkspaceService(db, {
pluginWorkerManager: f.workers,
}).reopenClosedIsolatedExecutionWorkspaceForIssue({
workspaceId: workspace!.id,
issue: {
id: f.issueId,
companyId: f.companyId,
projectId: f.input.projectId,
},
actor: { actorType: "user", agentId: null },
authorizationActor: {
type: "board",
source: "local_implicit",
isInstanceAdmin: true,
},
});
expect(result).toMatchObject({
ok: true,
reopened: true,
generation: 1,
workspace: { cwd: null, providerType: "plugin" },
});
expect(f.calls.at(-1)).toMatchObject({
runId: null,
agentId: null,
operation: { kind: "prepare", branchMode: "existing" },
});
expect(f.calls.at(-1).operation.existing.workspaceRef).toBe(
workspace!.providerRef,
);
});
it("keeps a relocated restoration reference consistent through subsequent removal", async () => {
const f = await fixture();
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
await db
.update(heartbeatRuns)
.set({ status: "succeeded" })
.where(eq(heartbeatRuns.id, f.runId));
await db
.update(executionWorkspaces)
.set({ status: "archived", closedAt: new Date() })
.where(eq(executionWorkspaces.id, workspace!.id));
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
f.relocate();
const svc = executionWorkspaceService(db, {
pluginWorkerManager: f.workers,
});
const result = await svc.reopenClosedIsolatedExecutionWorkspaceForIssue({
workspaceId: workspace!.id,
issue: {
id: f.issueId,
companyId: f.companyId,
projectId: f.input.projectId,
},
actor: { actorType: "user", agentId: null },
authorizationActor: {
type: "board",
source: "local_implicit",
isInstanceAdmin: true,
},
});
expect(result).toMatchObject({ ok: true, reopened: true });
const [restored] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, workspace!.id));
expect(restored.providerRef).toBe(`${workspace!.providerRef}:restored`);
expect(restored.metadata?.workspaceProvider).toMatchObject({
workspaceRef: restored.providerRef,
});
await removePluginWorkspace(db, f.workers, restored, "a".repeat(40));
expect(f.calls.at(-1).operation).toMatchObject({
kind: "remove",
binding: { workspaceRef: restored.providerRef },
});
});
it("reads workspace metadata without provider inspection", async () => {
const f = await fixture();
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
await db
.update(issues)
.set({ status: "done" })
.where(eq(issues.id, f.issueId));
f.workers.call = async () => { throw new Error("Storage unavailable"); };
const svc = executionWorkspaceService(db, {
pluginWorkerManager: f.workers,
inspectGitCloseReadiness: async () => {
throw new Error("Host inspection must not run for provider storage");
},
});
expect(await svc.getById(workspace!.id)).toMatchObject({
deliveryState: "unknown",
});
expect(await svc.getById(workspace!.id)).toMatchObject({
deliveryState: "unknown",
});
expect(
f.calls.filter((call) => call.operation.kind === "inspect"),
).toHaveLength(0);
});
it("replays a durable removal after a lost result without inspecting the missing checkout", async () => {
const f = await fixture();
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
f.loseRemoval();
await expect(
removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40)),
).rejects.toThrow("incomplete");
const inspections = f.calls.filter(
(call) => call.operation.kind === "inspect",
).length;
await removePluginWorkspace(db, f.workers, workspace!, null);
expect(
f.calls.filter((call) => call.operation.kind === "inspect"),
).toHaveLength(inspections);
const removals = f.calls.filter(
(call) => call.operation.kind === "remove",
);
expect(removals).toHaveLength(2);
expect(removals[0].operationId).toBe(removals[1].operationId);
const [row] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, workspace!.id));
expect(row.metadata).toMatchObject({
workspaceProviderRemoved: true,
workspaceProviderRemoval: null,
});
});
it("retires cleaned historical bindings when the storage source changes and refuses later reopen", async () => {
const f = await fixture();
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
await db
.update(executionWorkspaces)
.set({ status: "archived", closedAt: new Date() })
.where(eq(executionWorkspaces.id, workspace!.id));
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
await expect(
projectService(db).updateWorkspace(
f.input.projectId,
f.input.projectWorkspaceId,
{ repoUrl: "https://github.com/example/new.git" },
),
).resolves.not.toBeNull();
const before = f.calls.length;
const result = await executionWorkspaceService(db, {
pluginWorkerManager: f.workers,
}).reopenClosedIsolatedExecutionWorkspaceForIssue({
workspaceId: workspace!.id,
issue: {
id: f.issueId,
companyId: f.companyId,
projectId: f.input.projectId,
},
actor: { actorType: "user", agentId: null },
authorizationActor: {
type: "board",
source: "local_implicit",
isInstanceAdmin: true,
},
});
expect(result).toMatchObject({ ok: false, code: "not_reopenable" });
expect(f.calls).toHaveLength(before);
await expect(
projectService(db).update(f.input.projectId, {
executionWorkspacePolicy: {},
}),
).resolves.not.toBeNull();
await expect(
projectService(db).removeWorkspace(
f.input.projectId,
f.input.projectWorkspaceId,
),
).resolves.not.toBeNull();
});
it("keeps sweeping later workspaces when one storage plugin is unavailable", async () => {
const first = await fixture(),
second = await fixture();
await preparePluginWorkspace(db, first.workers, first.input);
await preparePluginWorkspace(db, second.workers, second.input);
await db.insert(pluginCompanySettings).values({
pluginId: first.pluginId,
companyId: first.companyId,
enabled: false,
});
for (const f of [first, second])
await db
.update(issues)
.set({ status: "done", completedAt: new Date() })
.where(eq(issues.id, f.issueId));
const workers = {
getWorker: (id: string) =>
([first.pluginId, second.pluginId] as string[]).includes(id)
? { supportedMethods: ["workspaceOperation"] }
: undefined,
call: (id: string, ...args: unknown[]) =>
(id === first.pluginId ? first.workers : second.workers).call(
id,
...(args as [any, any, any]),
),
} as unknown as WorkspaceWorkers;
await expect(
executionWorkspaceService(db, {
pluginWorkerManager: workers,
workspaceReaperCooldownDays: 0,
}).sweepTerminalWorkspaces(100),
).resolves.toBeDefined();
expect(
second.calls.some((call) => call.operation.kind === "inspect"),
).toBe(true);
});
it("retries an ambiguous deletion through the existing terminal sweep", async () => {
const f = await fixture();
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
f.loseRemoval();
await db
.update(executionWorkspaces)
.set({ status: "cleanup_failed", closedAt: new Date() })
.where(eq(executionWorkspaces.id, workspace!.id));
await db
.update(issues)
.set({ status: "done", completedAt: new Date() })
.where(eq(issues.id, f.issueId));
await db
.update(heartbeatRuns)
.set({ status: "cancelled" })
.where(eq(heartbeatRuns.id, f.runId));
await expect(
removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40)),
).rejects.toThrow("incomplete");
const workers = {
getWorker: (id: string) =>
id === f.pluginId
? { supportedMethods: ["workspaceOperation"] }
: undefined,
call: f.workers.call.bind(f.workers),
} as unknown as WorkspaceWorkers;
await executionWorkspaceService(db, {
pluginWorkerManager: workers,
workspaceReaperCooldownDays: 0,
}).sweepTerminalWorkspaces(100);
const [row] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, workspace!.id));
expect(row).toMatchObject({
status: "archived",
metadata: {
workspaceProviderRemoved: true,
workspaceProviderRemoval: null,
},
});
expect(
f.calls.filter((call) => call.operation.kind === "remove"),
).toHaveLength(2);
});
it("keeps restoration intent durable when reopening loses the provider response", async () => {
const f = await fixture();
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
await db
.update(executionWorkspaces)
.set({ status: "archived", closedAt: new Date() })
.where(eq(executionWorkspaces.id, workspace!.id));
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
f.lose();
const request = {
workspaceId: workspace!.id,
issue: {
id: f.issueId,
companyId: f.companyId,
projectId: f.input.projectId,
},
actor: { actorType: "user", agentId: null },
authorizationActor: {
type: "board" as const,
source: "local_implicit" as const,
isInstanceAdmin: true,
},
};
const svc = executionWorkspaceService(db, {
pluginWorkerManager: f.workers,
});
expect(
await svc.reopenClosedIsolatedExecutionWorkspaceForIssue(request),
).toMatchObject({ ok: false, code: "rebuild_failed" });
const [pending] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, workspace!.id));
expect(pending.metadata).toMatchObject({
workspaceProviderRemoved: false,
workspaceProviderRestoration: { generation: 1 },
});
await expect(
projectService(db).updateWorkspace(
f.input.projectId,
f.input.projectWorkspaceId,
{ repoRef: "new" },
),
).rejects.toThrow("storage owner");
expect(
await svc.reopenClosedIsolatedExecutionWorkspaceForIssue(request),
).toMatchObject({ ok: true, generation: 1 });
});
it("reconciles a remotely completed restoration before reaping and never reuses an older removal receipt", async () => {
const f = await fixture();
const call = f.workers.call.bind(f.workers),
cache = new Map<string, unknown>();
let loseRestoreReply = false;
f.workers.call = async (...args) => {
const params = args[2] as any;
if (params.operation.kind === "inspect") return call(...args);
if (cache.has(params.operationId))
return cache.get(params.operationId) as any;
const result = await call(...args);
cache.set(params.operationId, result);
if (
loseRestoreReply &&
params.operation.kind === "prepare" &&
params.operation.existing
) {
loseRestoreReply = false;
throw new Error("lost completed restoration reply");
}
return result;
};
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
await db
.update(heartbeatRuns)
.set({ status: "succeeded" })
.where(eq(heartbeatRuns.id, f.runId));
await db
.update(issues)
.set({ status: "done", completedAt: new Date() })
.where(eq(issues.id, f.issueId));
const svc = executionWorkspaceService(db, {
pluginWorkerManager: f.workers,
workspaceReaperCooldownDays: 0,
});
await svc.archiveWorkspaceUnderLifecycleLock({
id: workspace!.id,
patch: {},
closedAt: new Date(),
});
const read = () =>
db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, workspace!.id))
.then((rows) => rows[0]!);
await removePluginWorkspace(db, f.workers, await read(), "a".repeat(40));
const firstRemoval = f.calls.find(
(call) => call.operation.kind === "remove",
);
loseRestoreReply = true;
const restoreEvents: string[] = [];
const stopRestoreEvents = subscribeDatabaseWork(
db,
DELIVERY_QUEUES.terminalWorkspaces,
(event, hints) => {
if (hints?.has(`workspace:${workspace!.id}`))
restoreEvents.push(event);
},
);
try {
expect(
await svc.reopenClosedIsolatedExecutionWorkspaceForIssue({
workspaceId: workspace!.id,
issue: {
id: f.issueId,
companyId: f.companyId,
projectId: f.input.projectId,
},
actor: { actorType: "user", agentId: null },
authorizationActor: {
type: "board",
source: "local_implicit",
isInstanceAdmin: true,
},
}),
).toMatchObject({ ok: false, code: "rebuild_failed" });
} finally {
stopRestoreEvents();
}
expect(restoreEvents).toEqual(["intent", "settled"]);
const pending = await read();
expect(pending.metadata?.workspaceProviderRestoration).toMatchObject({
generation: firstRemoval.generation + 1,
});
await expect(
removePluginWorkspace(db, f.workers, pending, "a".repeat(40)),
).rejects.toThrow("restoration");
const settled: Array<ReturnType<typeof read>> = [];
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(
db,
DELIVERY_QUEUES.terminalWorkspaces,
(event, hints) => {
if (!hints?.has(`workspace:${workspace!.id}`)) return;
events.push(event);
if (event === "settled") settled.push(read());
},
);
try {
await svc.sweepTerminalWorkspaces(100);
} finally {
unsubscribe();
}
expect(events).toEqual(["intent", "settled"]);
expect(await settled[0]).toMatchObject({
metadata: {
workspaceProviderRestoration: null,
lifecycleGeneration: firstRemoval.generation + 1,
workspaceProviderRemoved: false,
},
});
const recovered = await read();
expect(recovered.metadata).toMatchObject({
workspaceProviderRemoved: false,
workspaceProviderRestoration: null,
lifecycleGeneration: firstRemoval.generation + 1,
});
await expect(
projectService(db).updateWorkspace(
f.input.projectId,
f.input.projectWorkspaceId,
{ repoRef: "after-recovery" },
),
).rejects.toThrow("storage owner");
await svc.sweepTerminalWorkspaces(100);
expect((await read()).metadata?.workspaceProviderRemoved).toBe(true);
const removals = f.calls.filter(
(call) => call.operation.kind === "remove",
);
expect(removals).toHaveLength(2);
expect(removals[1].generation).toBe(firstRemoval.generation + 1);
expect(removals[1].operationId).not.toBe(firstRemoval.operationId);
});
it("rejects unsupported provider modes before making a reservation", async () => {
const f = await fixture();
await expect(
preparePluginWorkspace(db, f.workers, {
...f.input,
mode: "shared_workspace",
strategy: "project_primary",
}),
).rejects.toThrow("does not support");
const [task] = await db
.select()
.from(issues)
.where(eq(issues.id, f.issueId));
expect(task.executionWorkspaceId).toBeNull();
expect(f.calls).toHaveLength(0);
});
it("checks the saved mode when project defaults change on a reused workspace", async () => {
const f = await fixture();
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
const reused = await preparePluginWorkspace(db, f.workers, {
...f.input,
existingWorkspaceId: workspace!.id,
mode: "shared_workspace",
strategy: "project_primary",
});
expect(reused!.id).toBe(workspace!.id);
expect(f.calls.at(-1).operation).toMatchObject({
mode: "isolated_workspace",
strategy: "git_worktree",
});
});
it("preserves a saved null branch when reuse defaults change", async () => {
const f = await fixture();
const [plugin] = await db
.select()
.from(plugins)
.where(eq(plugins.id, f.pluginId));
await db
.update(plugins)
.set({
manifestJson: {
...plugin.manifestJson,
workspaceProvider: {
modes: ["shared_workspace", "isolated_workspace"],
strategies: ["project_primary", "git_worktree"],
},
},
})
.where(eq(plugins.id, f.pluginId));
const workspace = await preparePluginWorkspace(db, f.workers, {
...f.input,
mode: "shared_workspace",
strategy: "project_primary",
branchName: null,
});
expect(workspace!.branchName).toBeNull();
await preparePluginWorkspace(db, f.workers, {
...f.input,
existingWorkspaceId: workspace!.id,
branchName: "new-default",
});
expect(f.calls.at(-1).operation).toMatchObject({
mode: "shared_workspace",
strategy: "project_primary",
branchName: null,
});
});
it("does not share inspection results across opaque references owned by different providers", async () => {
const first = await fixture(),
second = await fixture(first.companyId);
for (const f of [first, second]) {
const workers = {
getWorker: f.workers.getWorker.bind(f.workers),
call: async (...args: any[]) => {
const result = await (f.workers.call as any)(...args);
return result.kind === "prepared"
? {
...result,
binding: { ...result.binding, workspaceRef: "opaque" },
}
: result;
},
} as unknown as WorkspaceWorkers;
await preparePluginWorkspace(db, workers, f.input);
await db
.update(issues)
.set({ status: "done", completedAt: new Date() })
.where(eq(issues.id, f.issueId));
await db
.update(heartbeatRuns)
.set({ status: "cancelled" })
.where(eq(heartbeatRuns.id, f.runId));
}
second.dirty();
const workers = {
getWorker: (id: string) =>
([first.pluginId, second.pluginId] as string[]).includes(id)
? { supportedMethods: ["workspaceOperation"] }
: undefined,
call: (id: string, ...args: any[]) =>
(
(id === first.pluginId ? first.workers : second.workers).call as any
)(id, ...args),
} as unknown as WorkspaceWorkers;
await executionWorkspaceService(db, {
pluginWorkerManager: workers,
workspaceReaperCooldownDays: 0,
}).sweepTerminalWorkspaces(100);
const [task] = await db
.select()
.from(issues)
.where(eq(issues.id, second.issueId));
const [retained] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
expect(retained.status).toBe("active");
expect(
second.calls.some((call) => call.operation.kind === "inspect"),
).toBe(true);
});
it("never removes a checkout with local-only changes", async () => {
const f = await fixture();
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
f.dirty();
await expect(
removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40)),
).rejects.toThrow("local changes");
expect(f.calls.some((call) => call.operation.kind === "remove")).toBe(
false,
);
});
},
);