mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin can be a project workspace provider and also an environment
driver for the agent runs that use those workspaces
> - Core asks the workspace provider to prepare the project workspace
before the run acquires its environment lease
> - The provider knows the run, agent, project, and execution workspace,
but not the environment that the run selected
> - A provider whose compute exists per run (for example, a sandbox that
it creates from the environment's driver config) cannot start or find
the run's compute for the workspace work
> - This pull request lets a provider opt in to receive the run's
selected environment, when that environment is one of the provider's own
drivers
> - The benefit is that such a provider can prepare the workspace on the
run's own compute, with no change to the order of operations in Core
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Plugin workspace preparation for runs whose environment is a driver of
the same plugin.
**Subsystem affected**
Plugin SDK workspace provider params, the plugin manifest workspace
provider declaration, and `callWorkspaceProvider` /
`preparePluginWorkspace` in the server.
**Current behavior**
`prepare` params carry `runId`, `agentId`, `projectId`, and
`executionWorkspaceId`. They carry `taskLease` only when Core acquired
the lease first. They never name the run's environment.
**Proposed behavior**
A provider that declares `workspaceProvider.workspaceEnvironmentVersion:
1` receives `environment: { id, driverKey, config }` when the run's
selected environment is a plugin environment of that same provider.
`config` is the environment's `driverConfig`, the value that the
driver's acquire receives.
## What Changed
- SDK: optional strict `environment` field on
`pluginWorkspaceParamsSchema`.
- Manifest: optional `workspaceProvider.workspaceEnvironmentVersion: 1`,
next to the sibling version fields. Core sends the new field only to
providers that declare it, because providers parse the params with the
strict schema.
- Server: `callWorkspaceProvider` adds `environment` only when the
provider opted in, the environment driver is `plugin`, its `pluginKey`
is this provider, and it names a `driverKey`. The field is not part of
the operation ID.
- Server: `preparePluginWorkspace` passes the run's environment to its
prepare call. This covers normal, verify-existing, fresh, and
replacement prepares.
- Calls that do not have the run's environment do not send it: finalize,
inspect, remove, reservation reconciliation, and the retry-candidate
verification in `fresh-workspace-continuation.ts`.
- Lease acquisition, the `sessionHandoffVersion` gate, and the order of
operations do not change.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
capability and its exceptions.
## Verification
- `packages/plugins/sdk/tests/workspace-provider.test.ts`: 11 passed.
The schema accepts the field and rejects malformed values.
- `packages/shared/src/validators/plugin.test.ts`: 65 passed. The
manifest accepts only version 1.
- `server`: `plugin-workspaces.integration`,
`plugin-fresh-workspaces.integration`, `project-environments`, and
`heartbeat/workspace-preparation`: 226 passed, against embedded
Postgres. The new integration test checks the exact shape, an unchanged
operation ID, verification prepares, and omission for an undeclared
provider, another plugin's environment, non-plugin drivers, a missing
`driverKey`, and no environment. It fails when the server stops passing
the environment.
- `pnpm typecheck` passes in `packages/shared`, `packages/plugins/sdk`,
and `server`.
- The full suite was not run locally. CI runs it.
## Risks
- Low risk. Providers that do not declare the capability get the same
params as before.
- `config` is the stored `driverConfig`, in the same form that the
driver's acquire gets. Secret references in it reach the provider in
that same form.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1493 lines
71 KiB
TypeScript
1493 lines
71 KiB
TypeScript
import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, PluginWorkspaceOperationError, workspaceOperationErrorData } from "@paperclipai/plugin-sdk";
|
|
import { isPluginWorkspaceReplay } from "../services/plugin-workspace-replay-context.js";
|
|
import { createPluginWorkspaceFixture } from "./helpers/plugin-workspaces.js";
|
|
import { randomUUID } from "node:crypto";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import {
|
|
createDb,
|
|
subscribeDatabaseWork,
|
|
environmentLeases,
|
|
environments,
|
|
executionWorkspaces,
|
|
heartbeatRuns,
|
|
issues,
|
|
issueWorkProducts,
|
|
plugins,
|
|
pluginCompanySettings,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
callWorkspaceProvider,
|
|
preparePluginWorkspace,
|
|
inspectPluginWorkspace,
|
|
finalizePluginWorkspace,
|
|
removePluginWorkspace,
|
|
type WorkspaceWorkers,
|
|
} from "../services/plugin-workspaces.js";
|
|
import { verifyRetryWorkspace } from "../services/fresh-workspace-continuation.js";
|
|
import { executionWorkspaceService } from "../services/execution-workspaces.js";
|
|
import { WorkspaceProviderFailureError } from "../services/workspace-provider-failure.js";
|
|
import { findPluginWorkerUnavailableError, PluginWorkerUnavailableError } from "../services/plugin-worker-unavailable-error.js";
|
|
import { pluginRegistryService } from "../services/plugin-registry.js";
|
|
import { projectService } from "../services/projects.js";
|
|
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
|
import {
|
|
getEmbeddedPostgresTestSupport,
|
|
startEmbeddedPostgresTestDatabase,
|
|
} from "./helpers/embedded-postgres.js";
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
(support.supported ? describe : describe.skip)(
|
|
"plugin workspace ownership",
|
|
() => {
|
|
let db: ReturnType<typeof createDb>,
|
|
stopDb: (() => Promise<void>) | undefined;
|
|
beforeAll(async () => {
|
|
const started =
|
|
await startEmbeddedPostgresTestDatabase("plugin-workspaces");
|
|
db = createDb(started.connectionString, { maxConnections: 1 });
|
|
stopDb = started.cleanup;
|
|
});
|
|
afterAll(async () => {
|
|
await db?.$client.end();
|
|
await stopDb?.();
|
|
});
|
|
const fixture = (companyId?: string) =>
|
|
createPluginWorkspaceFixture(db, companyId);
|
|
it.each(["ready", "lost_reply", "unconfirmed", "transport", "worker_lookup", "live_writer", "older_retained", "unsupported"] as const)(
|
|
"verifies the same checkout before choosing a fresh retry (%s)", async (state) => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
if (state !== "unsupported") await db.update(plugins).set({ manifestJson: {
|
|
...plugin.manifestJson, workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, workspaceVerificationVersion: 1 },
|
|
} }).where(eq(plugins.id, f.pluginId));
|
|
if (state === "lost_reply") {
|
|
f.lose();
|
|
await expect(preparePluginWorkspace(db, f.workers, f.input)).rejects.toThrow();
|
|
} else {
|
|
await preparePluginWorkspace(db, f.workers, f.input);
|
|
}
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
const workspaceId = task.executionWorkspaceId!;
|
|
await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
|
|
if (state !== "lost_reply") await db.insert(environmentLeases).values({
|
|
companyId: f.companyId, issueId: f.issueId, heartbeatRunId: f.runId,
|
|
provider: "plugin:test:tasks", executionWorkspaceId: workspaceId, leasePolicy: "ephemeral",
|
|
status: state === "live_writer" ? "active" : "released",
|
|
releasedAt: state === "live_writer" ? null : new Date(),
|
|
});
|
|
if (state === "older_retained") {
|
|
const [older] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.input.agentId,
|
|
issueId: f.issueId, status: "succeeded", finishedAt: new Date() }).returning();
|
|
await db.insert(environmentLeases).values({
|
|
companyId: f.companyId, issueId: f.issueId, heartbeatRunId: older.id,
|
|
provider: "plugin:test:tasks", executionWorkspaceId: workspaceId, leasePolicy: "reuse_by_environment",
|
|
status: "retained", releasedAt: null,
|
|
});
|
|
}
|
|
const retryIntentId = randomUUID();
|
|
const [retry] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.input.agentId,
|
|
issueId: f.issueId, status: "running", contextSnapshot: { taskRetryRequestId: retryIntentId },
|
|
}).returning();
|
|
const fresh = { mode: "fresh_isolated_workspace" as const, retryIntentId, sourceRunId: f.runId, retainedSourceIds: [] };
|
|
const call = vi.fn<WorkspaceWorkers["call"]>(async (...args) => {
|
|
const result = await f.workers.call(...args) as { kind: string };
|
|
if (state === "transport") throw new Error("connection closed");
|
|
if (state === "unconfirmed") throw new JsonRpcCallError({
|
|
code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
|
|
message: "checkout incomplete",
|
|
data: workspaceOperationErrorData(new PluginWorkspaceOperationError("binding_mismatch")),
|
|
});
|
|
return { ...result, created: false };
|
|
});
|
|
const workers: WorkspaceWorkers = {
|
|
getWorker: (pluginId) => {
|
|
if (state === "worker_lookup") throw new Error("worker lookup timed out");
|
|
const worker = f.workers.getWorker(pluginId);
|
|
return worker && { ...worker, supportedMethods: ["workspaceOperation", "workspacePrepare"] };
|
|
},
|
|
call,
|
|
};
|
|
const verificationStarted = Date.now();
|
|
const result = verifyRetryWorkspace(db, workers, retry, f.issueId, null, fresh);
|
|
if (state === "transport" || state === "worker_lookup") {
|
|
if (state === "transport") await expect(result).rejects.toMatchObject({ kind: "transport", retryable: true });
|
|
else await expect(result).rejects.toThrow("worker lookup timed out");
|
|
return;
|
|
}
|
|
const verified = state === "ready" || state === "lost_reply";
|
|
expect(await result).toEqual(verified
|
|
? { mode: "reuse_issue_workspace", retryIntentId, sourceRunId: f.runId, workspaceId, cwd: null }
|
|
: fresh);
|
|
expect(f.calls.slice(1)).toHaveLength(["live_writer", "older_retained", "unsupported"].includes(state) ? 0 : 1);
|
|
if (f.calls.length > 1) {
|
|
expect(call).toHaveBeenCalledExactlyOnceWith(f.pluginId, "workspaceOperation", expect.objectContaining({
|
|
executionWorkspaceId: workspaceId, runId: retry.id, operation: expect.objectContaining({ verifyExisting: true }),
|
|
}), 12_000);
|
|
const deadline = Date.parse(f.calls[1].deadline);
|
|
expect(deadline).toBeGreaterThanOrEqual(verificationStarted + 10_000);
|
|
expect(deadline).toBeLessThanOrEqual(Date.now() + 10_000);
|
|
}
|
|
const [retained] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
|
|
expect(retained.providerRef).toBe(`checkout:${workspaceId}`);
|
|
});
|
|
|
|
it("sends the run's own-driver environment only to a provider that declares it", async () => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
const declare = (workspaceEnvironmentVersion?: 1) => db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
capabilities: [...plugin.manifestJson.capabilities, "environment.drivers.register"],
|
|
environmentDrivers: [{ driverKey: "tasks", displayName: "Tasks", configSchema: { type: "object" }, supportsTasks: true }],
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, workspaceVerificationVersion: 1,
|
|
...(workspaceEnvironmentVersion ? { workspaceEnvironmentVersion } : {}) },
|
|
} }).where(eq(plugins.id, f.pluginId));
|
|
await declare(1);
|
|
const [row] = await db.insert(environments).values({ name: `Snapshot tasks ${f.pluginId}`, driver: "plugin", status: "active",
|
|
config: { pluginKey: plugin.pluginKey, driverKey: "tasks", driverConfig: { snapshot: "base", cpu: 2 } } }).returning();
|
|
const environment = { ...row, driver: "plugin" as const };
|
|
const expected = { id: row.id, driverKey: "tasks", config: { snapshot: "base", cpu: 2 } };
|
|
|
|
const prepared = await preparePluginWorkspace(db, f.workers, { ...f.input, environment });
|
|
expect(f.calls).toHaveLength(1);
|
|
expect(f.calls[0].environment).toEqual(expected);
|
|
// Retry verification of the retained checkout carries the same environment.
|
|
await preparePluginWorkspace(db, f.workers, { ...f.input, environment, existingWorkspaceId: prepared!.id, verifyExisting: true });
|
|
expect(f.calls[1]).toMatchObject({ environment: expected, operation: { verifyExisting: true } });
|
|
|
|
const operation = f.calls[0].operation;
|
|
const send = async (selected: Parameters<typeof callWorkspaceProvider>[2]["environment"]) => {
|
|
await callWorkspaceProvider(db, f.workers, { workspace: prepared!, runId: f.runId, agentId: f.input.agentId,
|
|
environment: selected, operation });
|
|
return f.calls.at(-1)!;
|
|
};
|
|
const withEnvironment = await send(environment);
|
|
expect(withEnvironment.environment).toEqual(expected);
|
|
const without = await send(undefined);
|
|
expect(without).not.toHaveProperty("environment");
|
|
// The environment is context only; the operation identity does not change.
|
|
expect(withEnvironment.operationId).toBe(without.operationId);
|
|
expect(await send({ ...environment, config: { pluginKey: plugin.pluginKey, driverKey: "tasks" } }))
|
|
.toMatchObject({ environment: { id: row.id, driverKey: "tasks", config: {} } });
|
|
for (const omitted of [
|
|
{ ...environment, config: { ...environment.config, pluginKey: "other.plugin" } },
|
|
{ ...environment, config: { pluginKey: plugin.pluginKey, driverConfig: { snapshot: "base" } } },
|
|
{ id: row.id, driver: "local" as const, config: {} },
|
|
{ id: row.id, driver: "sandbox" as const, config: { provider: "e2b", pluginKey: plugin.pluginKey, driverKey: "tasks" } },
|
|
null,
|
|
]) {
|
|
const request = await send(omitted);
|
|
expect(request).not.toHaveProperty("environment");
|
|
expect(request.operationId).toBe(without.operationId);
|
|
}
|
|
|
|
await declare();
|
|
const undeclared = await send(environment);
|
|
expect(undeclared).not.toHaveProperty("environment");
|
|
expect(undeclared.operationId).toBe(without.operationId);
|
|
});
|
|
|
|
it("polls bounded requests using the provider's advertised preparation deadline", async () => {
|
|
const f = await fixture();
|
|
const deadline = new Date(Date.now() + 30_000).toISOString();
|
|
const phases: string[] = [];
|
|
const workers = {
|
|
getWorker: () => ({ supportedMethods: ["workspaceOperation", "workspacePrepare"] }),
|
|
call: async (pluginId: string, method: string, params: any, timeout: number) => {
|
|
expect(method).toBe("workspacePrepare");
|
|
expect(timeout).toBeLessThanOrEqual(15_000);
|
|
phases.push(params.phase);
|
|
if (params.phase === "submit") return { kind: "pending", deadline };
|
|
expect(params.deadline).toBe(deadline);
|
|
if (phases.length === 2) throw new Error("lost status reply");
|
|
const { phase: _phase, ...original } = params;
|
|
return f.workers.call(pluginId, "workspaceOperation", original);
|
|
},
|
|
} as unknown as WorkspaceWorkers;
|
|
expect(await preparePluginWorkspace(db, workers, f.input)).toBeTruthy();
|
|
expect(phases).toEqual(["submit", "poll", "poll"]);
|
|
expect(f.calls).toHaveLength(1);
|
|
});
|
|
|
|
it("stops preparation at the provider deadline before issuing a poll", async () => {
|
|
const f = await fixture();
|
|
const workers = {
|
|
getWorker: () => ({ supportedMethods: ["workspaceOperation", "workspacePrepare"] }),
|
|
call: async (_pluginId: string, method: string, params: { phase: string }) => {
|
|
expect(method).toBe("workspacePrepare");
|
|
expect(params.phase).toBe("submit");
|
|
return { kind: "pending", deadline: new Date(Date.now() - 1).toISOString() };
|
|
},
|
|
} as unknown as WorkspaceWorkers;
|
|
await expect(preparePluginWorkspace(db, workers, f.input)).rejects.toMatchObject({
|
|
kind: "provider", reason: "deadline_exceeded", retryable: true,
|
|
});
|
|
});
|
|
|
|
it("rejects an oversized provider deadline before it can retain the workspace lifecycle lock", async () => {
|
|
const f = await fixture();
|
|
const workers = {
|
|
getWorker: () => ({ supportedMethods: ["workspaceOperation", "workspacePrepare"] }),
|
|
call: async (_pluginId: string, method: string, params: { phase: string }) => {
|
|
expect(method).toBe("workspacePrepare");
|
|
expect(params.phase).toBe("submit");
|
|
return { kind: "pending", deadline: "2099-01-01T00:00:00Z" };
|
|
},
|
|
} as unknown as WorkspaceWorkers;
|
|
await expect(preparePluginWorkspace(db, workers, f.input)).rejects.toMatchObject({
|
|
kind: "provider", reason: "unsupported_operation", retryable: false,
|
|
});
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
expect(task?.executionWorkspaceId).toBeTruthy();
|
|
expect(await preparePluginWorkspace(db, f.workers, { ...f.input, existingWorkspaceId: task!.executionWorkspaceId })).toBeTruthy();
|
|
});
|
|
|
|
it.each([
|
|
["missing", undefined],
|
|
["stopped", { status: "stopped", supportedMethods: [] }],
|
|
["starting", { status: "starting", supportedMethods: [] }],
|
|
] as const)("retries preparation when the ready provider's worker is %s, before sending a request", async (_state, worker) => {
|
|
const f = await fixture();
|
|
const workers = { getWorker: () => worker, call: f.workers.call.bind(f.workers) } as unknown as WorkspaceWorkers;
|
|
const failure = await preparePluginWorkspace(db, workers, f.input).then(() => null, (error: unknown) => error);
|
|
expect(failure).toBeInstanceOf(WorkspaceProviderFailureError);
|
|
expect((failure as WorkspaceProviderFailureError).summary()).toEqual({ kind: "worker_unavailable", reason: null, retryable: true });
|
|
expect(findPluginWorkerUnavailableError(failure)).toBeInstanceOf(PluginWorkerUnavailableError);
|
|
expect(f.calls).toHaveLength(0);
|
|
});
|
|
it("keeps a running worker without workspace operations an unavailable provider", async () => {
|
|
const f = await fixture();
|
|
const workers = { getWorker: () => ({ status: "running", supportedMethods: [] }), call: f.workers.call.bind(f.workers) } as unknown as WorkspaceWorkers;
|
|
const failure = await preparePluginWorkspace(db, workers, f.input).then(() => null, (error: unknown) => error);
|
|
expect(failure).not.toBeInstanceOf(WorkspaceProviderFailureError);
|
|
expect((failure as Error).message).toBe("The workspace storage plugin is unavailable");
|
|
expect(f.calls).toHaveLength(0);
|
|
});
|
|
it.each([1, undefined])("gates recovery hints by provider capability (%s)", async (version) => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, recoveryPullRequestVersion: version as 1 | undefined } } }).where(eq(plugins.id, f.pluginId));
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
const recoveryPullRequest = { number: 42, headSha: "a".repeat(40) };
|
|
await inspectPluginWorkspace(db, f.workers, workspace!, recoveryPullRequest);
|
|
expect(f.calls.at(-1).operation.recoveryPullRequest).toEqual(version === 1 ? recoveryPullRequest : undefined);
|
|
let freshInspectionCompleted = false;
|
|
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), recoveryPullRequest, async () => {
|
|
freshInspectionCompleted = f.calls.at(-1).operation.kind === "inspect";
|
|
throw new Error("delivery changed");
|
|
})).rejects.toThrow("delivery changed");
|
|
expect(freshInspectionCompleted).toBe(true);
|
|
const [unchanged] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(unchanged.metadata?.workspaceProviderRemoval).toBeUndefined();
|
|
f.loseRemoval();
|
|
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), recoveryPullRequest)).rejects.toThrow();
|
|
const [pending] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect((pending.metadata?.workspaceProviderRemoval as { operation: { recoveryPullRequest?: unknown } }).operation.recoveryPullRequest)
|
|
.toEqual(version === 1 ? recoveryPullRequest : undefined);
|
|
await removePluginWorkspace(db, f.workers, pending, "a".repeat(40), recoveryPullRequest, async () => { throw new Error("must not reauthorize pending removal"); });
|
|
expect(f.calls.at(-1).operation.recoveryPullRequest).toEqual(version === 1 ? recoveryPullRequest : undefined);
|
|
});
|
|
it.each([1, undefined])("replays a lost removal acknowledgement unchanged after capability flip (%s)", async (version) => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
const setCapability = (recoveryPullRequestVersion: 1 | undefined) => db.update(plugins).set({ manifestJson: {
|
|
...plugin.manifestJson, workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, recoveryPullRequestVersion },
|
|
} }).where(eq(plugins.id, f.pluginId));
|
|
await setCapability(version as 1 | undefined);
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
f.loseRemoval();
|
|
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), { number: 42, headSha: "a".repeat(40) })).rejects.toThrow();
|
|
const first = f.calls.at(-1);
|
|
await db.update(plugins).set({ status: "disabled" }).where(eq(plugins.id, f.pluginId));
|
|
const count = f.calls.length;
|
|
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40))).rejects.toThrow("unavailable");
|
|
expect(f.calls).toHaveLength(count);
|
|
await db.update(plugins).set({ status: "ready" }).where(eq(plugins.id, f.pluginId));
|
|
await setCapability(version === 1 ? undefined : 1);
|
|
const [pending] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
await removePluginWorkspace(db, f.workers, pending, "a".repeat(40));
|
|
expect(f.calls.at(-1).operation).toEqual(first.operation);
|
|
expect(f.calls.at(-1).operationId).toBe(first.operationId);
|
|
expect(pending.metadata?.workspaceProviderRemoval).toMatchObject({ operation: first.operation, operationId: first.operationId });
|
|
const [removed] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(removed.metadata?.workspaceProviderRemoved).toBe(true);
|
|
});
|
|
it.each(["legacy", "ambiguous", "mismatched"] as const)("fails closed for uncertain removal identities (%s)", async (identity) => {
|
|
const hint = identity !== "legacy";
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, recoveryPullRequestVersion: 1 } } }).where(eq(plugins.id, f.pluginId));
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
f.loseRemoval();
|
|
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), hint ? { number: 42, headSha: "a".repeat(40) } : undefined)).rejects.toThrow();
|
|
const first = f.calls.at(-1);
|
|
const [pending] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
const intent = pending.metadata!.workspaceProviderRemoval as Record<string, unknown>;
|
|
if (identity === "mismatched") intent.operationId = "0".repeat(64);
|
|
else delete intent.operationId;
|
|
await db.update(executionWorkspaces).set({ metadata: { ...pending.metadata, workspaceProviderRemoval: intent } }).where(eq(executionWorkspaces.id, pending.id));
|
|
if (hint) {
|
|
const count = f.calls.length;
|
|
await expect(removePluginWorkspace(db, f.workers, pending, "a".repeat(40))).rejects.toThrow(identity === "ambiguous" ? "identity is ambiguous" : "identity does not match");
|
|
expect(f.calls).toHaveLength(count);
|
|
} else {
|
|
await removePluginWorkspace(db, f.workers, pending, "a".repeat(40));
|
|
expect(f.calls.at(-1).operationId).toBe(first.operationId);
|
|
expect(f.calls.at(-1).operation).toEqual(first.operation);
|
|
}
|
|
});
|
|
it.each(["refs/heads/main", null, undefined])("persists the actual prepared destination (%s)", async (destinationRef) => {
|
|
const f = await fixture();
|
|
const workers: WorkspaceWorkers = { ...f.workers, call: async (...args: Parameters<typeof f.workers.call>) => {
|
|
const result = await f.workers.call(...args) as { kind?: string };
|
|
return result.kind === "prepared" ? { ...result, ...(destinationRef !== undefined ? { destinationRef } : {}) } : result;
|
|
} };
|
|
const workspace = await preparePluginWorkspace(db, workers, f.input);
|
|
expect(workspace!.metadata?.workspaceProviderDestinationRef).toBe(destinationRef);
|
|
});
|
|
it.each([true, undefined])("wakes parked cleanup when a provider upgrade enables ancestry (%s)", async (supportsMergeAncestry) => {
|
|
const f = await fixture();
|
|
await preparePluginWorkspace(db, f.workers, f.input);
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId));
|
|
const svc = executionWorkspaceService(db, { pluginWorkerManager: f.workers, workspaceReaperCooldownDays: 0 });
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
expect(await svc.nextTerminalWorkspaceSweepAt()).toBeNull();
|
|
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(false);
|
|
const events: string[] = [];
|
|
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.terminalWorkspaces, (event, hints) => {
|
|
events.push(event);
|
|
if (event === "settled") svc.terminalWorkspaceWorkSettled(hints);
|
|
});
|
|
try {
|
|
await pluginRegistryService(db).update(f.pluginId, { manifest: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry } } });
|
|
expect(events).toEqual(["intent", "settled"]);
|
|
} finally { unsubscribe(); }
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(true);
|
|
expect(f.calls.some(call => call.operation.kind === "remove")).toBe(true);
|
|
});
|
|
it("wakes cleanup only for provider contract changes and activation", async () => {
|
|
const f = await fixture();
|
|
const registry = pluginRegistryService(db);
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
const events: string[] = [];
|
|
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.terminalWorkspaces, event => events.push(event));
|
|
try {
|
|
await registry.update(f.pluginId, { manifest: { ...plugin.manifestJson, description: "Updated description" } });
|
|
await registry.updateStatus(f.pluginId, { status: "disabled" });
|
|
expect(events).toEqual([]);
|
|
await registry.updateStatus(f.pluginId, { status: "ready" });
|
|
expect(events.splice(0)).toEqual(["intent", "settled"]);
|
|
await registry.updateStatus(f.pluginId, { status: "ready" });
|
|
await registry.upsertCompanySettings(f.pluginId, f.companyId, { enabled: false, settingsJson: {} });
|
|
expect(events).toEqual([]);
|
|
await registry.upsertCompanySettings(f.pluginId, f.companyId, { enabled: true, settingsJson: {} });
|
|
expect(events.splice(0)).toEqual(["intent", "settled"]);
|
|
await registry.upsertCompanySettings(f.pluginId, f.companyId, { enabled: true, settingsJson: {} });
|
|
expect(events).toEqual([]);
|
|
await db.transaction(async tx => {
|
|
await pluginRegistryService(tx as unknown as typeof db).update(f.pluginId, {
|
|
manifest: { ...plugin.manifestJson, workspaceProvider: undefined },
|
|
});
|
|
expect(events).toEqual(["intent"]);
|
|
});
|
|
expect(events.splice(0)).toEqual(["intent", "settled"]);
|
|
await registry.updateStatus(f.pluginId, { status: "disabled" });
|
|
await registry.updateStatus(f.pluginId, { status: "ready" });
|
|
await registry.update(f.pluginId, { manifest: { ...plugin.manifestJson, workspaceProvider: undefined, description: "Other plugin" } });
|
|
expect(events).toEqual([]);
|
|
} finally { unsubscribe(); }
|
|
});
|
|
it("keeps storage idle without a settled completion HEAD, even with a recorded PR", async () => {
|
|
const f = await fixture();
|
|
await preparePluginWorkspace(db, f.workers, f.input);
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId));
|
|
await db.insert(issueWorkProducts).values({ companyId: f.companyId, issueId: f.issueId,
|
|
type: "pull_request", provider: "github", title: "Work", status: "active",
|
|
url: "https://github.com/example/repo/pull/1" });
|
|
const options = { pluginWorkerManager: f.workers, workspaceReaperCooldownDays: 0,
|
|
resolveWorkspaceDelivery: async () => { throw new Error("No delivery lookup without a completion HEAD"); } };
|
|
const svc = executionWorkspaceService(db, options);
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(false);
|
|
expect(await svc.nextTerminalWorkspaceSweepAt()).toBeNull();
|
|
svc.terminalWorkspaceWorkSettled(new Set([`issue:${randomUUID()}`]));
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
expect(await svc.nextTerminalWorkspaceSweepAt()).toBeNull();
|
|
await executionWorkspaceService(db, options).sweepTerminalWorkspaces(100);
|
|
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(false);
|
|
});
|
|
it("uses bounded finalization only for opted-in providers, preserving maintenance inspection", async () => {
|
|
const f = await fixture();
|
|
const workspace = (await preparePluginWorkspace(db, f.workers, f.input))!;
|
|
const context = { runId: f.runId, agentId: f.input.agentId };
|
|
expect((await finalizePluginWorkspace(db, f.workers, workspace, context)).kind).toBe("inspection");
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, finalizationVersion: 1 } } }).where(eq(plugins.id, f.pluginId));
|
|
const completionIds: string[] = [];
|
|
const call = f.workers.call.bind(f.workers);
|
|
f.workers.call = async (...args) => {
|
|
const p = args[2] as any;
|
|
if (p.operation.kind !== "finalize") return call(...args);
|
|
expect(p.agentId).toBe(context.agentId);
|
|
completionIds.push(p.operationId);
|
|
expect(Date.parse(p.deadline) - Date.now()).toBeLessThanOrEqual(30_000);
|
|
expect(args[3]).toBe(40_000);
|
|
return { kind: "finalized", branchName: workspace.branchName, headSha: "a".repeat(40) };
|
|
};
|
|
expect((await finalizePluginWorkspace(db, f.workers, workspace, context)).kind).toBe("finalized");
|
|
await finalizePluginWorkspace(db, f.workers, workspace, context);
|
|
await finalizePluginWorkspace(db, f.workers, workspace, { ...context, runId: randomUUID() });
|
|
expect(completionIds[1]).toBe(completionIds[0]);
|
|
expect(completionIds[2]).not.toBe(completionIds[0]);
|
|
expect((await inspectPluginWorkspace(db, f.workers, workspace)).kind).toBe("inspection");
|
|
f.workers.call = async () => { throw new Error("sync failed"); };
|
|
await expect(finalizePluginWorkspace(db, f.workers, workspace, context)).rejects.toThrow("incomplete");
|
|
});
|
|
it("keeps the single-connection pool responsive while provider preparation holds its lifecycle lock", async () => {
|
|
const f = await fixture();
|
|
let entered!: () => void, release!: () => void;
|
|
const started = new Promise<void>((resolve) => {
|
|
entered = resolve;
|
|
});
|
|
const resume = new Promise<void>((resolve) => {
|
|
release = resolve;
|
|
});
|
|
const call = f.workers.call.bind(f.workers);
|
|
f.workers.call = async (...args) => {
|
|
if ((args[2] as any).operation.kind === "prepare") {
|
|
entered();
|
|
await resume;
|
|
}
|
|
return call(...args);
|
|
};
|
|
const preparing = preparePluginWorkspace(db, f.workers, f.input);
|
|
await started;
|
|
try {
|
|
await expect(
|
|
projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoRef: "next" },
|
|
),
|
|
).rejects.toThrow("Workspace lifecycle is busy");
|
|
expect(await db.execute(sql`select 1 as usable`)).toMatchObject([
|
|
{ usable: 1 },
|
|
]);
|
|
} finally {
|
|
release();
|
|
}
|
|
const workspace = await preparing;
|
|
await expect(
|
|
inspectPluginWorkspace(db, f.workers, workspace!),
|
|
).resolves.toMatchObject({ kind: "inspection" });
|
|
});
|
|
it("preserves a scheduled selection through the first provider binding and next admission", async () => {
|
|
const f = await fixture();
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
});
|
|
const actor = {
|
|
type: "board" as const,
|
|
source: "local_implicit" as const,
|
|
isInstanceAdmin: true,
|
|
};
|
|
const request = {
|
|
companyId: f.companyId,
|
|
issueId: f.issueId,
|
|
actor,
|
|
selection: { kind: "task_directory" as const },
|
|
expectedBindingRevision: 0,
|
|
requestKey: "during-prepare",
|
|
};
|
|
await svc.selectTaskWorkspace(request);
|
|
await preparePluginWorkspace(db, f.workers, f.input);
|
|
expect(
|
|
await svc.inspectTaskWorkspace(f.companyId, f.issueId, actor),
|
|
).toMatchObject({
|
|
bindingRevision: 2,
|
|
pendingSelection: {
|
|
expectedBindingRevision: 2,
|
|
intent: { request: { expectedBindingRevision: 0 } },
|
|
},
|
|
});
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
expect(
|
|
await svc.applyPendingTaskWorkspaceSelection({
|
|
companyId: f.companyId,
|
|
issueId: f.issueId,
|
|
runId: randomUUID(),
|
|
actor,
|
|
}),
|
|
).toBe(true);
|
|
expect(await svc.selectTaskWorkspace(request)).toMatchObject({
|
|
kind: "applied",
|
|
});
|
|
});
|
|
it("does not retire reusable storage for rejected or normalized unchanged source edits", async () => {
|
|
const f = await fixture(),
|
|
workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "archived", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
|
|
expect(
|
|
await projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoRef: " " },
|
|
),
|
|
).toMatchObject({ repoRef: null });
|
|
expect(
|
|
await projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoUrl: null },
|
|
),
|
|
).toBeNull();
|
|
const [row] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(row.metadata?.workspaceProviderRetired).not.toBe(true);
|
|
expect(
|
|
await executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
}).reopenClosedIsolatedExecutionWorkspaceForIssue({
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board",
|
|
source: "local_implicit",
|
|
isInstanceAdmin: true,
|
|
},
|
|
}),
|
|
).toMatchObject({ ok: true, reopened: true });
|
|
});
|
|
it("uses new inspection identities even when the provider caches operation results", async () => {
|
|
const f = await fixture(),
|
|
workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
const call = f.workers.call.bind(f.workers),
|
|
cache = new Map<string, unknown>();
|
|
f.workers.call = async (...args) => {
|
|
const params = args[2] as any;
|
|
if (cache.has(params.operationId))
|
|
return cache.get(params.operationId) as any;
|
|
const result = await call(...args);
|
|
cache.set(params.operationId, result);
|
|
return result;
|
|
};
|
|
expect(
|
|
await inspectPluginWorkspace(db, f.workers, workspace!),
|
|
).toMatchObject({ hasDirtyTrackedFiles: false });
|
|
f.dirty();
|
|
expect(
|
|
await inspectPluginWorkspace(db, f.workers, workspace!),
|
|
).toMatchObject({ hasDirtyTrackedFiles: true });
|
|
expect(cache.size).toBe(2);
|
|
});
|
|
it("retries closed storage cleanup when the pre-delete inspection failed before recording removal", async () => {
|
|
const f = await fixture(),
|
|
workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done", completedAt: new Date() })
|
|
.where(eq(issues.id, f.issueId));
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
const call = f.workers.call.bind(f.workers);
|
|
let inspections = 0;
|
|
const workers = {
|
|
getWorker: (id: string) =>
|
|
id === f.pluginId
|
|
? { supportedMethods: ["workspaceOperation"] }
|
|
: undefined,
|
|
call: async (...args: Parameters<WorkspaceWorkers["call"]>) => {
|
|
if (
|
|
(args[2] as any).operation.kind === "inspect" &&
|
|
++inspections === 2
|
|
)
|
|
throw new Error("temporary inspection outage");
|
|
return call(...args);
|
|
},
|
|
} as unknown as WorkspaceWorkers;
|
|
let clock = Date.now();
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: workers,
|
|
workspaceReaperCooldownDays: 0,
|
|
now: () => new Date(clock),
|
|
});
|
|
const sweep = () => svc.sweepTerminalWorkspaces(100);
|
|
expect(await sweep()).toMatchObject({ cleanupFailed: 1 });
|
|
const [failed] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(failed.status).toBe("cleanup_failed");
|
|
expect(failed.metadata?.workspaceProviderRemoval).toBeUndefined();
|
|
expect(failed.metadata?.terminalCleanupRetry).toMatchObject({ failures: 1, failure: { kind: "transport" },
|
|
retryAt: new Date(clock + 300_000).toISOString() });
|
|
expect(await sweep()).toMatchObject({ skippedBackoff: 1 });
|
|
clock += 300_000;
|
|
await sweep();
|
|
const [removed] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(removed.metadata?.workspaceProviderRemoved).toBe(true);
|
|
expect(removed.status).toBe("archived");
|
|
expect(removed.metadata).not.toHaveProperty("terminalCleanupRetry");
|
|
});
|
|
it("prepares a durable reference without a local checkout or agent allocation", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
expect(workspace).toMatchObject({
|
|
providerType: "plugin",
|
|
cwd: null,
|
|
status: "active",
|
|
});
|
|
expect(f.calls[0].operation.source).not.toHaveProperty("cwd");
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, f.issueId));
|
|
expect(task.executionWorkspaceId).toBe(workspace!.id);
|
|
expect(
|
|
await inspectPluginWorkspace(db, f.workers, workspace!),
|
|
).toMatchObject({ remotelyRecoverable: true });
|
|
});
|
|
it("retains the reservation and retries the same operation after a lost response", async () => {
|
|
const f = await fixture();
|
|
f.lose();
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, f.input),
|
|
).rejects.toThrow("incomplete");
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, f.issueId));
|
|
const result = await preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
existingWorkspaceId: task.executionWorkspaceId,
|
|
});
|
|
expect(result!.id).toBe(task.executionWorkspaceId);
|
|
expect(f.calls[1].operationId).toBe(f.calls[0].operationId);
|
|
});
|
|
it("reconciles a lost preparation result after the task run stops", async () => {
|
|
const f = await fixture();
|
|
f.lose();
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, f.input),
|
|
).rejects.toThrow("incomplete");
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "cancelled" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, f.issueId));
|
|
const [reservation] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
|
|
expect(reservation.providerRef).toBeNull();
|
|
await expect(
|
|
inspectPluginWorkspace(db, f.workers, reservation),
|
|
).resolves.toMatchObject({ remotelyRecoverable: true });
|
|
expect(f.calls[1]).toMatchObject({
|
|
runId: f.runId,
|
|
agentId: f.input.agentId,
|
|
operation: { kind: "prepare" },
|
|
});
|
|
expect(f.calls[1].operationId).toBe(f.calls[0].operationId);
|
|
const [recorded] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, reservation.id));
|
|
expect(recorded.providerRef).not.toBeNull();
|
|
});
|
|
it.each(["inspect", "retry", "replacement"] as const)("recovers a lost Launcher prepare through %s", async (recovery) => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
capabilities: [...plugin.manifestJson.capabilities, "environment.drivers.register"],
|
|
environmentDrivers: [{ driverKey: "tasks", displayName: "Tasks", configSchema: { type: "object" },
|
|
supportsTasks: true, sessionHandoffVersion: 1 }],
|
|
} }).where(eq(plugins.id, f.pluginId));
|
|
const [environment] = await db.insert(environments).values({ name: `Workspace Launcher ${f.pluginId}`, driver: "plugin",
|
|
status: "active", config: { pluginKey: plugin.pluginKey, driverKey: "tasks" } }).returning();
|
|
const taskLease = { environmentId: environment.id, driverKey: "tasks", providerLeaseId: randomUUID() };
|
|
const leaseId = randomUUID();
|
|
const input = { ...f.input, environment: { ...environment, driver: "plugin" as const },
|
|
acquireTaskLease: async () => ({ leaseId, taskLease }) };
|
|
let deleted = false;
|
|
const replacementTaskLease = { ...taskLease, providerLeaseId: randomUUID() };
|
|
const replacementWorkers = { ...f.workers, call: vi.fn<WorkspaceWorkers["call"]>(async (...args) => {
|
|
const request = args[2] as (typeof f.calls)[number];
|
|
if (deleted && request.taskLease?.providerLeaseId === taskLease.providerLeaseId)
|
|
throw new Error("Previous Launcher allocation is gone");
|
|
const result = await f.workers.call(...args) as { kind: string };
|
|
if (!deleted) throw new Error("Lost committed preparation result");
|
|
return { ...result, created: false } as never;
|
|
}) };
|
|
if (recovery !== "replacement") f.lose();
|
|
await expect(preparePluginWorkspace(db, recovery === "replacement" ? replacementWorkers : f.workers, input)).rejects.toThrow("incomplete");
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
const [reservation] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
|
|
expect(reservation.providerRef).toBeNull();
|
|
expect(reservation.metadata?.workspaceProviderPreparationPlacement).toMatchObject({
|
|
generation: 0, runId: f.runId, agentId: f.input.agentId, taskLease,
|
|
});
|
|
if (recovery === "inspect") {
|
|
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.runId));
|
|
const workers = { ...f.workers, call: vi.fn<WorkspaceWorkers["call"]>(async (...args) => {
|
|
const request = args[2] as (typeof f.calls)[number];
|
|
expect(isPluginWorkspaceReplay({
|
|
pluginId: f.pluginId, companyId: f.companyId, projectId: f.input.projectId,
|
|
projectWorkspaceId: f.input.projectWorkspaceId, executionWorkspaceId: reservation.id,
|
|
})).toBe(request.operation.kind === "prepare");
|
|
return f.workers.call(...args);
|
|
}) };
|
|
await expect(inspectPluginWorkspace(db, workers, reservation)).resolves.toMatchObject({ remotelyRecoverable: true });
|
|
} else if (recovery === "retry") {
|
|
await expect(preparePluginWorkspace(db, f.workers, { ...input, existingWorkspaceId: reservation.id })).resolves.toMatchObject({ id: reservation.id });
|
|
} else {
|
|
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, f.runId));
|
|
const [nextRun] = await db.insert(heartbeatRuns).values({
|
|
companyId: f.companyId, issueId: f.issueId, agentId: f.input.agentId, status: "running",
|
|
}).returning();
|
|
deleted = true;
|
|
await expect(preparePluginWorkspace(db, replacementWorkers, {
|
|
...input, runId: nextRun.id, existingWorkspaceId: reservation.id,
|
|
acquireTaskLease: async () => ({ leaseId: randomUUID(), taskLease: replacementTaskLease }),
|
|
})).resolves.toMatchObject({ id: reservation.id });
|
|
expect(replacementWorkers.call).toHaveBeenCalledTimes(2);
|
|
expect(f.calls[1]).toMatchObject({ runId: nextRun.id, agentId: f.input.agentId, taskLease: replacementTaskLease });
|
|
expect(f.calls[1].operationId).not.toBe(f.calls[0].operationId);
|
|
}
|
|
if (recovery !== "replacement") expect(f.calls[1]).toMatchObject({
|
|
runId: f.runId, agentId: f.input.agentId, taskLease, operationId: f.calls[0].operationId,
|
|
});
|
|
expect(f.calls[1].operation).toEqual(f.calls[0].operation);
|
|
const [prepared] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, reservation.id));
|
|
expect(prepared.providerRef).not.toBeNull();
|
|
expect(prepared.metadata?.workspaceProviderPreparationPlacement).toBeUndefined();
|
|
});
|
|
it("reconciles an unfinished reservation before applying PR-only cleanup eligibility", async () => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
|
|
f.lose();
|
|
await expect(preparePluginWorkspace(db, f.workers, f.input)).rejects.toThrow("incomplete");
|
|
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.runId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
f.dirty();
|
|
await executionWorkspaceService(db, { pluginWorkerManager: f.workers, workspaceReaperCooldownDays: 0 }).sweepTerminalWorkspaces(100);
|
|
expect(f.calls.filter(call => call.operation.kind === "prepare")).toHaveLength(2);
|
|
expect(f.calls.filter(call => call.operation.kind === "inspect")).toHaveLength(1);
|
|
expect(f.calls.some(call => call.operation.kind === "remove")).toBe(false);
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
|
|
expect(workspace.providerRef).not.toBeNull();
|
|
});
|
|
it("fails closed for a disabled plugin, stopped run, and cross-company source", async () => {
|
|
const f = await fixture();
|
|
await db.insert(pluginCompanySettings).values({
|
|
pluginId: f.pluginId,
|
|
companyId: f.companyId,
|
|
enabled: false,
|
|
});
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, f.input),
|
|
).rejects.toThrow("unavailable");
|
|
await db
|
|
.update(pluginCompanySettings)
|
|
.set({ enabled: true })
|
|
.where(eq(pluginCompanySettings.pluginId, f.pluginId));
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "cancelled" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, f.input),
|
|
).rejects.toThrow();
|
|
expect(f.calls).toHaveLength(0);
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
companyId: randomUUID(),
|
|
}),
|
|
).resolves.toBeNull();
|
|
});
|
|
it("keeps source ownership immutable and preserves bindings through metadata edits", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
const svc = projectService(db);
|
|
await expect(
|
|
svc.update(f.input.projectId, {
|
|
executionWorkspacePolicy: { storageProviderPluginId: null },
|
|
}),
|
|
).rejects.toMatchObject({ status: 409, details: { code: "project_storage_locked" } });
|
|
await expect(
|
|
svc.updateWorkspace(f.input.projectId, f.input.projectWorkspaceId, {
|
|
repoUrl: "https://github.com/example/other.git",
|
|
}),
|
|
).rejects.toThrow("repository source");
|
|
await expect(
|
|
svc.removeWorkspace(f.input.projectId, f.input.projectWorkspaceId),
|
|
).rejects.toThrow("repository source");
|
|
const placement = { generation: 0, runId: f.runId, agentId: f.input.agentId,
|
|
taskLease: { environmentId: randomUUID(), driverKey: "tasks", providerLeaseId: randomUUID() } };
|
|
await db.update(executionWorkspaces).set({
|
|
metadata: { ...workspace!.metadata, workspaceProviderPreparationPlacement: placement },
|
|
}).where(eq(executionWorkspaces.id, workspace!.id));
|
|
const execution = executionWorkspaceService(db);
|
|
const updated = await execution.update(workspace!.id, {
|
|
metadata: {
|
|
note: "caller edit",
|
|
workspaceProvider: { workspaceRef: "foreign" },
|
|
workspaceProviderPreparationPlacement: { ...placement, runId: randomUUID() },
|
|
},
|
|
});
|
|
expect(updated!.metadata?.workspaceProviderPreparationPlacement).toBeUndefined();
|
|
const [saved] = await db.select({ metadata: executionWorkspaces.metadata })
|
|
.from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(saved.metadata?.workspaceProviderPreparationPlacement).toEqual(placement);
|
|
expect(updated!.metadata?.workspaceProvider).toEqual(
|
|
workspace!.metadata?.workspaceProvider,
|
|
);
|
|
await expect(
|
|
execution.update(workspace!.id, { providerType: "local_fs" }),
|
|
).rejects.toThrow("storage plugin");
|
|
});
|
|
it("describes provider removal without promising host cleanup commands", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done" })
|
|
.where(eq(issues.id, f.issueId));
|
|
const readiness = await executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
}).getCloseReadiness(workspace!.id);
|
|
expect(readiness!.plannedActions).toContainEqual(
|
|
expect.objectContaining({
|
|
kind: "remove_provider_workspace",
|
|
command: null,
|
|
}),
|
|
);
|
|
expect(
|
|
readiness!.plannedActions.some(
|
|
(action) => action.kind === "git_branch_delete",
|
|
),
|
|
).toBe(false);
|
|
});
|
|
it("reopens through the same provider without requiring a live run or a host checkout", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "archived", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
const result = await executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
}).reopenClosedIsolatedExecutionWorkspaceForIssue({
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board",
|
|
source: "local_implicit",
|
|
isInstanceAdmin: true,
|
|
},
|
|
});
|
|
expect(result).toMatchObject({
|
|
ok: true,
|
|
reopened: true,
|
|
generation: 1,
|
|
workspace: { cwd: null, providerType: "plugin" },
|
|
});
|
|
expect(f.calls.at(-1)).toMatchObject({
|
|
runId: null,
|
|
agentId: null,
|
|
operation: { kind: "prepare", branchMode: "existing" },
|
|
});
|
|
expect(f.calls.at(-1).operation.existing.workspaceRef).toBe(
|
|
workspace!.providerRef,
|
|
);
|
|
});
|
|
it("keeps a relocated restoration reference consistent through subsequent removal", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "archived", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
|
|
f.relocate();
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
});
|
|
const result = await svc.reopenClosedIsolatedExecutionWorkspaceForIssue({
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board",
|
|
source: "local_implicit",
|
|
isInstanceAdmin: true,
|
|
},
|
|
});
|
|
expect(result).toMatchObject({ ok: true, reopened: true });
|
|
const [restored] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(restored.providerRef).toBe(`${workspace!.providerRef}:restored`);
|
|
expect(restored.metadata?.workspaceProvider).toMatchObject({
|
|
workspaceRef: restored.providerRef,
|
|
});
|
|
await removePluginWorkspace(db, f.workers, restored, "a".repeat(40));
|
|
expect(f.calls.at(-1).operation).toMatchObject({
|
|
kind: "remove",
|
|
binding: { workspaceRef: restored.providerRef },
|
|
});
|
|
});
|
|
it("reads workspace metadata without provider inspection", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done" })
|
|
.where(eq(issues.id, f.issueId));
|
|
f.workers.call = async () => { throw new Error("Storage unavailable"); };
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
inspectGitCloseReadiness: async () => {
|
|
throw new Error("Host inspection must not run for provider storage");
|
|
},
|
|
});
|
|
expect(await svc.getById(workspace!.id)).toMatchObject({
|
|
deliveryState: "unknown",
|
|
});
|
|
expect(await svc.getById(workspace!.id)).toMatchObject({
|
|
deliveryState: "unknown",
|
|
});
|
|
expect(
|
|
f.calls.filter((call) => call.operation.kind === "inspect"),
|
|
).toHaveLength(0);
|
|
});
|
|
it("replays a durable removal after a lost result without inspecting the missing checkout", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
f.loseRemoval();
|
|
await expect(
|
|
removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40)),
|
|
).rejects.toThrow("incomplete");
|
|
const inspections = f.calls.filter(
|
|
(call) => call.operation.kind === "inspect",
|
|
).length;
|
|
await removePluginWorkspace(db, f.workers, workspace!, null);
|
|
expect(
|
|
f.calls.filter((call) => call.operation.kind === "inspect"),
|
|
).toHaveLength(inspections);
|
|
const removals = f.calls.filter(
|
|
(call) => call.operation.kind === "remove",
|
|
);
|
|
expect(removals).toHaveLength(2);
|
|
expect(removals[0].operationId).toBe(removals[1].operationId);
|
|
const [row] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(row.metadata).toMatchObject({
|
|
workspaceProviderRemoved: true,
|
|
workspaceProviderRemoval: null,
|
|
});
|
|
});
|
|
it("retires cleaned historical bindings when the storage source changes and refuses later reopen", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "archived", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
|
|
await expect(
|
|
projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoUrl: "https://github.com/example/new.git" },
|
|
),
|
|
).resolves.not.toBeNull();
|
|
const before = f.calls.length;
|
|
const result = await executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
}).reopenClosedIsolatedExecutionWorkspaceForIssue({
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board",
|
|
source: "local_implicit",
|
|
isInstanceAdmin: true,
|
|
},
|
|
});
|
|
expect(result).toMatchObject({ ok: false, code: "not_reopenable" });
|
|
expect(f.calls).toHaveLength(before);
|
|
await expect(
|
|
projectService(db).update(f.input.projectId, {
|
|
executionWorkspacePolicy: {},
|
|
}),
|
|
).resolves.not.toBeNull();
|
|
await expect(
|
|
projectService(db).removeWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
),
|
|
).resolves.not.toBeNull();
|
|
});
|
|
it("keeps sweeping later workspaces when one storage plugin is unavailable", async () => {
|
|
const first = await fixture(),
|
|
second = await fixture();
|
|
await preparePluginWorkspace(db, first.workers, first.input);
|
|
await preparePluginWorkspace(db, second.workers, second.input);
|
|
await db.insert(pluginCompanySettings).values({
|
|
pluginId: first.pluginId,
|
|
companyId: first.companyId,
|
|
enabled: false,
|
|
});
|
|
for (const f of [first, second])
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done", completedAt: new Date() })
|
|
.where(eq(issues.id, f.issueId));
|
|
const workers = {
|
|
getWorker: (id: string) =>
|
|
([first.pluginId, second.pluginId] as string[]).includes(id)
|
|
? { supportedMethods: ["workspaceOperation"] }
|
|
: undefined,
|
|
call: (id: string, ...args: unknown[]) =>
|
|
(id === first.pluginId ? first.workers : second.workers).call(
|
|
id,
|
|
...(args as [any, any, any]),
|
|
),
|
|
} as unknown as WorkspaceWorkers;
|
|
await expect(
|
|
executionWorkspaceService(db, {
|
|
pluginWorkerManager: workers,
|
|
workspaceReaperCooldownDays: 0,
|
|
}).sweepTerminalWorkspaces(100),
|
|
).resolves.toBeDefined();
|
|
expect(
|
|
second.calls.some((call) => call.operation.kind === "inspect"),
|
|
).toBe(true);
|
|
});
|
|
it("retries an ambiguous deletion through the existing terminal sweep", async () => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
f.loseRemoval();
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "cleanup_failed", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done", completedAt: new Date() })
|
|
.where(eq(issues.id, f.issueId));
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "cancelled" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await expect(
|
|
removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40)),
|
|
).rejects.toThrow("incomplete");
|
|
const workers = {
|
|
getWorker: (id: string) =>
|
|
id === f.pluginId
|
|
? { supportedMethods: ["workspaceOperation"] }
|
|
: undefined,
|
|
call: f.workers.call.bind(f.workers),
|
|
} as unknown as WorkspaceWorkers;
|
|
await executionWorkspaceService(db, {
|
|
pluginWorkerManager: workers,
|
|
workspaceReaperCooldownDays: 0,
|
|
}).sweepTerminalWorkspaces(100);
|
|
const [row] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(row).toMatchObject({
|
|
status: "archived",
|
|
metadata: {
|
|
workspaceProviderRemoved: true,
|
|
workspaceProviderRemoval: null,
|
|
},
|
|
});
|
|
expect(
|
|
f.calls.filter((call) => call.operation.kind === "remove"),
|
|
).toHaveLength(2);
|
|
});
|
|
it("keeps restoration intent durable when reopening loses the provider response", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "archived", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
|
|
f.lose();
|
|
const request = {
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board" as const,
|
|
source: "local_implicit" as const,
|
|
isInstanceAdmin: true,
|
|
},
|
|
};
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
});
|
|
expect(
|
|
await svc.reopenClosedIsolatedExecutionWorkspaceForIssue(request),
|
|
).toMatchObject({ ok: false, code: "rebuild_failed" });
|
|
const [pending] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(pending.metadata).toMatchObject({
|
|
workspaceProviderRemoved: false,
|
|
workspaceProviderRestoration: { generation: 1 },
|
|
});
|
|
await expect(
|
|
projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoRef: "new" },
|
|
),
|
|
).rejects.toThrow("storage owner");
|
|
expect(
|
|
await svc.reopenClosedIsolatedExecutionWorkspaceForIssue(request),
|
|
).toMatchObject({ ok: true, generation: 1 });
|
|
});
|
|
it("reconciles a remotely completed restoration before reaping and never reuses an older removal receipt", async () => {
|
|
const f = await fixture();
|
|
const call = f.workers.call.bind(f.workers),
|
|
cache = new Map<string, unknown>();
|
|
let loseRestoreReply = false;
|
|
f.workers.call = async (...args) => {
|
|
const params = args[2] as any;
|
|
if (params.operation.kind === "inspect") return call(...args);
|
|
if (cache.has(params.operationId))
|
|
return cache.get(params.operationId) as any;
|
|
const result = await call(...args);
|
|
cache.set(params.operationId, result);
|
|
if (
|
|
loseRestoreReply &&
|
|
params.operation.kind === "prepare" &&
|
|
params.operation.existing
|
|
) {
|
|
loseRestoreReply = false;
|
|
throw new Error("lost completed restoration reply");
|
|
}
|
|
return result;
|
|
};
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done", completedAt: new Date() })
|
|
.where(eq(issues.id, f.issueId));
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
workspaceReaperCooldownDays: 0,
|
|
});
|
|
await svc.archiveWorkspaceUnderLifecycleLock({
|
|
id: workspace!.id,
|
|
patch: {},
|
|
closedAt: new Date(),
|
|
});
|
|
const read = () =>
|
|
db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id))
|
|
.then((rows) => rows[0]!);
|
|
await removePluginWorkspace(db, f.workers, await read(), "a".repeat(40));
|
|
const firstRemoval = f.calls.find(
|
|
(call) => call.operation.kind === "remove",
|
|
);
|
|
loseRestoreReply = true;
|
|
const restoreEvents: string[] = [];
|
|
const stopRestoreEvents = subscribeDatabaseWork(
|
|
db,
|
|
DELIVERY_QUEUES.terminalWorkspaces,
|
|
(event, hints) => {
|
|
if (hints?.has(`workspace:${workspace!.id}`))
|
|
restoreEvents.push(event);
|
|
},
|
|
);
|
|
try {
|
|
expect(
|
|
await svc.reopenClosedIsolatedExecutionWorkspaceForIssue({
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board",
|
|
source: "local_implicit",
|
|
isInstanceAdmin: true,
|
|
},
|
|
}),
|
|
).toMatchObject({ ok: false, code: "rebuild_failed" });
|
|
} finally {
|
|
stopRestoreEvents();
|
|
}
|
|
expect(restoreEvents).toEqual(["intent", "settled"]);
|
|
const pending = await read();
|
|
expect(pending.metadata?.workspaceProviderRestoration).toMatchObject({
|
|
generation: firstRemoval.generation + 1,
|
|
});
|
|
await expect(
|
|
removePluginWorkspace(db, f.workers, pending, "a".repeat(40)),
|
|
).rejects.toThrow("restoration");
|
|
const settled: Array<ReturnType<typeof read>> = [];
|
|
const events: string[] = [];
|
|
const unsubscribe = subscribeDatabaseWork(
|
|
db,
|
|
DELIVERY_QUEUES.terminalWorkspaces,
|
|
(event, hints) => {
|
|
if (!hints?.has(`workspace:${workspace!.id}`)) return;
|
|
events.push(event);
|
|
if (event === "settled") settled.push(read());
|
|
},
|
|
);
|
|
try {
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
} finally {
|
|
unsubscribe();
|
|
}
|
|
expect(events).toEqual(["intent", "settled"]);
|
|
expect(await settled[0]).toMatchObject({
|
|
metadata: {
|
|
workspaceProviderRestoration: null,
|
|
lifecycleGeneration: firstRemoval.generation + 1,
|
|
workspaceProviderRemoved: false,
|
|
},
|
|
});
|
|
const recovered = await read();
|
|
expect(recovered.metadata).toMatchObject({
|
|
workspaceProviderRemoved: false,
|
|
workspaceProviderRestoration: null,
|
|
lifecycleGeneration: firstRemoval.generation + 1,
|
|
});
|
|
await expect(
|
|
projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoRef: "after-recovery" },
|
|
),
|
|
).rejects.toThrow("storage owner");
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
expect((await read()).metadata?.workspaceProviderRemoved).toBe(true);
|
|
const removals = f.calls.filter(
|
|
(call) => call.operation.kind === "remove",
|
|
);
|
|
expect(removals).toHaveLength(2);
|
|
expect(removals[1].generation).toBe(firstRemoval.generation + 1);
|
|
expect(removals[1].operationId).not.toBe(firstRemoval.operationId);
|
|
});
|
|
it("rejects unsupported provider modes before making a reservation", async () => {
|
|
const f = await fixture();
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
mode: "shared_workspace",
|
|
strategy: "project_primary",
|
|
}),
|
|
).rejects.toThrow("does not support");
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, f.issueId));
|
|
expect(task.executionWorkspaceId).toBeNull();
|
|
expect(f.calls).toHaveLength(0);
|
|
});
|
|
it("checks the saved mode when project defaults change on a reused workspace", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
const reused = await preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
existingWorkspaceId: workspace!.id,
|
|
mode: "shared_workspace",
|
|
strategy: "project_primary",
|
|
});
|
|
expect(reused!.id).toBe(workspace!.id);
|
|
expect(f.calls.at(-1).operation).toMatchObject({
|
|
mode: "isolated_workspace",
|
|
strategy: "git_worktree",
|
|
});
|
|
});
|
|
it("preserves a saved null branch when reuse defaults change", async () => {
|
|
const f = await fixture();
|
|
const [plugin] = await db
|
|
.select()
|
|
.from(plugins)
|
|
.where(eq(plugins.id, f.pluginId));
|
|
await db
|
|
.update(plugins)
|
|
.set({
|
|
manifestJson: {
|
|
...plugin.manifestJson,
|
|
workspaceProvider: {
|
|
modes: ["shared_workspace", "isolated_workspace"],
|
|
strategies: ["project_primary", "git_worktree"],
|
|
},
|
|
},
|
|
})
|
|
.where(eq(plugins.id, f.pluginId));
|
|
const workspace = await preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
mode: "shared_workspace",
|
|
strategy: "project_primary",
|
|
branchName: null,
|
|
});
|
|
expect(workspace!.branchName).toBeNull();
|
|
await preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
existingWorkspaceId: workspace!.id,
|
|
branchName: "new-default",
|
|
});
|
|
expect(f.calls.at(-1).operation).toMatchObject({
|
|
mode: "shared_workspace",
|
|
strategy: "project_primary",
|
|
branchName: null,
|
|
});
|
|
});
|
|
it("does not share inspection results across opaque references owned by different providers", async () => {
|
|
const first = await fixture(),
|
|
second = await fixture(first.companyId);
|
|
for (const f of [first, second]) {
|
|
const workers = {
|
|
getWorker: f.workers.getWorker.bind(f.workers),
|
|
call: async (...args: any[]) => {
|
|
const result = await (f.workers.call as any)(...args);
|
|
return result.kind === "prepared"
|
|
? {
|
|
...result,
|
|
binding: { ...result.binding, workspaceRef: "opaque" },
|
|
}
|
|
: result;
|
|
},
|
|
} as unknown as WorkspaceWorkers;
|
|
await preparePluginWorkspace(db, workers, f.input);
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done", completedAt: new Date() })
|
|
.where(eq(issues.id, f.issueId));
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "cancelled" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
}
|
|
second.dirty();
|
|
const workers = {
|
|
getWorker: (id: string) =>
|
|
([first.pluginId, second.pluginId] as string[]).includes(id)
|
|
? { supportedMethods: ["workspaceOperation"] }
|
|
: undefined,
|
|
call: (id: string, ...args: any[]) =>
|
|
(
|
|
(id === first.pluginId ? first.workers : second.workers).call as any
|
|
)(id, ...args),
|
|
} as unknown as WorkspaceWorkers;
|
|
await executionWorkspaceService(db, {
|
|
pluginWorkerManager: workers,
|
|
workspaceReaperCooldownDays: 0,
|
|
}).sweepTerminalWorkspaces(100);
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, second.issueId));
|
|
const [retained] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
|
|
expect(retained.status).toBe("active");
|
|
expect(
|
|
second.calls.some((call) => call.operation.kind === "inspect"),
|
|
).toBe(true);
|
|
});
|
|
it("never removes a checkout with local-only changes", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
f.dirty();
|
|
await expect(
|
|
removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40)),
|
|
).rejects.toThrow("local changes");
|
|
expect(f.calls.some((call) => call.operation.kind === "remove")).toBe(
|
|
false,
|
|
);
|
|
});
|
|
},
|
|
);
|