mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core owns run leases and delegates remote execution to environment plugins. > - A retry must verify its old checkout on the placement that will run the task. > - A completed task must still release its provider resources. > - This change uses the existing lease for workspace verification and always calls the release hook after settlement. > - Saved settlement proof lets cleanup retry without repeating execution work. ## Linked Issues or Issue Description Refs #21237. That PR records stop proof before lease release. This PR keeps provider cleanup after settlement and changes workspace verification sequencing. Both touch task completion; preserve both changes if the merge order requires conflict resolution. **What happened?** Workspace verification ran before a portable task had a lease, so a provider could not verify the checkout on its selected placement. Core also skipped the provider release hook after successful task completion. This could leave resources allocated after the run ended. **Expected behavior** Verify the old checkout through the selected run lease. A confirmed refusal can use the authorized fresh-workspace fallback. A transport failure must remain retryable. Call the provider release hook after settlement and retry failed cleanup without changing the run result. **Steps to reproduce** 1. Retry a task whose prior checkout uses plugin-managed storage and portable task placement. 2. Verify that the plugin receives the same task lease for verification and preparation. 3. Complete a task, fail the provider release once, and reconstruct the cleanup controller. 4. Confirm that cleanup calls only the release hook and retains the completed run outcome. ## What Changed - Acquire the logical run lease before provider workspace verification and reuse it for preparation and execution. - Preserve the existing session and workspace choices. Recompute workspace-dependent inputs only after a definite refusal; acquire credentials and managed runtime inputs once. - Save the exact refusal and lease before cleanup. Use a workspace-scoped creation key only for the resulting fresh attempt. - Replay interrupted verification on the same unlaunched lease. Keep transport errors retryable and reject replay after execution starts. - Always call the provider release hook after task settlement. Pass refreshed settlement proof and route failed releases through the existing cleanup reaper. - Add database regressions for call order, restart, retained writers, refusal, transport errors, and settlement-only cleanup replay. `plugin-workspaces.ts` grows from 997 to 1,001 lines. The four added lines extend the existing workspace transaction and lease callbacks. Keeping the checks there preserves the ownership assertion, provider result validation, and binding update in one operation. Extracting only the verification branch would split that transaction without removing a separate responsibility. ## Verification - Server TypeScript check passed: `tsc -p server/tsconfig.json --noEmit`. The complete `server/scripts/typecheck-layers.mjs` check also passed, including all test-tree layers. - The first CI typecheck caught a readonly array in a test fixture that the narrower source check did not include. Commit8d9ec19e7acorrects that fixture; all layered checks pass afterward. - Added a public heartbeat refusal regression in7c598a4da3. All ten session-preparation tests pass; independent execution of the refusal case also passes. It checks successful completion, the saved fresh workspace, a new conversation, and one credential acquisition. - Nine database/service suites passed 335 tests: plugin warm tasks, plugin runner tasks, startup cleanup recovery, credential writeback, plugin workspaces, session preparation, plugin environments, retry workspace continuity, and heartbeat workspace preparation. - After the final restart change, both targeted refusal/restart tests passed. The primary agent independently reran those cases and the final server typecheck. After integrating current master, the primary agent's independent rerun passed all 237 tests across the warm-task, plugin-workspace, session-preparation, and heartbeat-workspace suites. - `pnpm verify` returned exit 2 and requested broad CI coverage because the plugin execution contract documentation changed. No full local Core-suite pass is claimed. - Previous-revision CI was green on8d9ec19e7a: 59 checks passed and 3 were skipped. The first run failed the unchanged legacy-continuation browser assertion and credential-cleanup server shard. Each passed on one targeted retry with no code changes. The credential-cleanup integration suite also passed all 21 tests locally on Node 22. The browser failure concerned an existing continuation metadata marker; the server failure cascaded after a cleanup error left a test spy installed. The successful retries do not establish the cause of the first failures. - Local database fixtures use simulated provider workers. No live provider or deployed environment was changed. - Additional verification for59c7a1dae8: the layered server typecheck passed. Independent execution passed all 265 tests across durable task retries, workspace preparation, and public heartbeat preparation. The new shutdown regression preserves the fresh Retry choice, waits for parent cleanup, and prevents the successor from inheriting the parent lease reference. Same-run recovery retains its receipt. Required CI is green on59c7a1dae8: 58 checks passed and 3 were skipped. Standard review scored 5/5 with no unresolved comments. The separate Superagent Security Scan is neutral/inconclusive because its workflow failed; a direct rerun request returned HTTP 404. That scan is not claimed as passed. ## Risks - Providers must use the supplied task lease for verification and return a definite `binding_mismatch` only when the checkout is unconfirmed. Transport errors retain the retry decision. - A refused checkout waits for confirmed release before a fresh allocation. Failed deletion remains owned by the existing cleanup reaper. - Task settlement and physical resource release stay separate. Retained handoff leases do not enter this release path. - No migration or SDK schema change is required. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) implemented the changes and tests. The primary Codex agent independently reviewed the connected Core and plugin paths and ran checks. A separate GPT-6.1 Sol agent reviewed lifecycle and restart boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue using the bug template labels - [x] I have not referenced internal or private project details - [x] My branch name describes the change - [x] I have added relevant tests and documentation - [x] I have considered and documented risks - [x] All required CI gates are green - [x] Greptile is 5/5 with no unresolved comments - [x] I will address reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1438 lines
67 KiB
TypeScript
1438 lines
67 KiB
TypeScript
import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, PluginWorkspaceOperationError, workspaceOperationErrorData } from "@paperclipai/plugin-sdk";
|
|
import { isPluginWorkspaceReplay } from "../services/plugin-workspace-replay-context.js";
|
|
import { createPluginWorkspaceFixture } from "./helpers/plugin-workspaces.js";
|
|
import { randomUUID } from "node:crypto";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import {
|
|
createDb,
|
|
subscribeDatabaseWork,
|
|
environmentLeases,
|
|
environments,
|
|
executionWorkspaces,
|
|
heartbeatRuns,
|
|
issues,
|
|
issueWorkProducts,
|
|
plugins,
|
|
pluginCompanySettings,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
preparePluginWorkspace,
|
|
inspectPluginWorkspace,
|
|
finalizePluginWorkspace,
|
|
removePluginWorkspace,
|
|
type WorkspaceWorkers,
|
|
} from "../services/plugin-workspaces.js";
|
|
import { verifyRetryWorkspace } from "../services/fresh-workspace-continuation.js";
|
|
import { executionWorkspaceService } from "../services/execution-workspaces.js";
|
|
import { WorkspaceProviderFailureError } from "../services/workspace-provider-failure.js";
|
|
import { findPluginWorkerUnavailableError, PluginWorkerUnavailableError } from "../services/plugin-worker-unavailable-error.js";
|
|
import { pluginRegistryService } from "../services/plugin-registry.js";
|
|
import { projectService } from "../services/projects.js";
|
|
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
|
import {
|
|
getEmbeddedPostgresTestSupport,
|
|
startEmbeddedPostgresTestDatabase,
|
|
} from "./helpers/embedded-postgres.js";
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
(support.supported ? describe : describe.skip)(
|
|
"plugin workspace ownership",
|
|
() => {
|
|
let db: ReturnType<typeof createDb>,
|
|
stopDb: (() => Promise<void>) | undefined;
|
|
beforeAll(async () => {
|
|
const started =
|
|
await startEmbeddedPostgresTestDatabase("plugin-workspaces");
|
|
db = createDb(started.connectionString, { maxConnections: 1 });
|
|
stopDb = started.cleanup;
|
|
});
|
|
afterAll(async () => {
|
|
await db?.$client.end();
|
|
await stopDb?.();
|
|
});
|
|
const fixture = (companyId?: string) =>
|
|
createPluginWorkspaceFixture(db, companyId);
|
|
it.each(["ready", "lost_reply", "unconfirmed", "transport", "worker_lookup", "live_writer", "older_retained", "unsupported"] as const)(
|
|
"verifies the same checkout before choosing a fresh retry (%s)", async (state) => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
if (state !== "unsupported") await db.update(plugins).set({ manifestJson: {
|
|
...plugin.manifestJson, workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, workspaceVerificationVersion: 1 },
|
|
} }).where(eq(plugins.id, f.pluginId));
|
|
if (state === "lost_reply") {
|
|
f.lose();
|
|
await expect(preparePluginWorkspace(db, f.workers, f.input)).rejects.toThrow();
|
|
} else {
|
|
await preparePluginWorkspace(db, f.workers, f.input);
|
|
}
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
const workspaceId = task.executionWorkspaceId!;
|
|
await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
|
|
if (state !== "lost_reply") await db.insert(environmentLeases).values({
|
|
companyId: f.companyId, issueId: f.issueId, heartbeatRunId: f.runId,
|
|
provider: "plugin:test:tasks", executionWorkspaceId: workspaceId, leasePolicy: "ephemeral",
|
|
status: state === "live_writer" ? "active" : "released",
|
|
releasedAt: state === "live_writer" ? null : new Date(),
|
|
});
|
|
if (state === "older_retained") {
|
|
const [older] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.input.agentId,
|
|
issueId: f.issueId, status: "succeeded", finishedAt: new Date() }).returning();
|
|
await db.insert(environmentLeases).values({
|
|
companyId: f.companyId, issueId: f.issueId, heartbeatRunId: older.id,
|
|
provider: "plugin:test:tasks", executionWorkspaceId: workspaceId, leasePolicy: "reuse_by_environment",
|
|
status: "retained", releasedAt: null,
|
|
});
|
|
}
|
|
const retryIntentId = randomUUID();
|
|
const [retry] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.input.agentId,
|
|
issueId: f.issueId, status: "running", contextSnapshot: { taskRetryRequestId: retryIntentId },
|
|
}).returning();
|
|
const fresh = { mode: "fresh_isolated_workspace" as const, retryIntentId, sourceRunId: f.runId, retainedSourceIds: [] };
|
|
const call = vi.fn<WorkspaceWorkers["call"]>(async (...args) => {
|
|
const result = await f.workers.call(...args) as { kind: string };
|
|
if (state === "transport") throw new Error("connection closed");
|
|
if (state === "unconfirmed") throw new JsonRpcCallError({
|
|
code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
|
|
message: "checkout incomplete",
|
|
data: workspaceOperationErrorData(new PluginWorkspaceOperationError("binding_mismatch")),
|
|
});
|
|
return { ...result, created: false };
|
|
});
|
|
const workers: WorkspaceWorkers = {
|
|
getWorker: (pluginId) => {
|
|
if (state === "worker_lookup") throw new Error("worker lookup timed out");
|
|
const worker = f.workers.getWorker(pluginId);
|
|
return worker && { ...worker, supportedMethods: ["workspaceOperation", "workspacePrepare"] };
|
|
},
|
|
call,
|
|
};
|
|
const verificationStarted = Date.now();
|
|
const result = verifyRetryWorkspace(db, workers, retry, f.issueId, null, fresh);
|
|
if (state === "transport" || state === "worker_lookup") {
|
|
if (state === "transport") await expect(result).rejects.toMatchObject({ kind: "transport", retryable: true });
|
|
else await expect(result).rejects.toThrow("worker lookup timed out");
|
|
return;
|
|
}
|
|
const verified = state === "ready" || state === "lost_reply";
|
|
expect(await result).toEqual(verified
|
|
? { mode: "reuse_issue_workspace", retryIntentId, sourceRunId: f.runId, workspaceId, cwd: null }
|
|
: fresh);
|
|
expect(f.calls.slice(1)).toHaveLength(["live_writer", "older_retained", "unsupported"].includes(state) ? 0 : 1);
|
|
if (f.calls.length > 1) {
|
|
expect(call).toHaveBeenCalledExactlyOnceWith(f.pluginId, "workspaceOperation", expect.objectContaining({
|
|
executionWorkspaceId: workspaceId, runId: retry.id, operation: expect.objectContaining({ verifyExisting: true }),
|
|
}), 12_000);
|
|
const deadline = Date.parse(f.calls[1].deadline);
|
|
expect(deadline).toBeGreaterThanOrEqual(verificationStarted + 10_000);
|
|
expect(deadline).toBeLessThanOrEqual(Date.now() + 10_000);
|
|
}
|
|
const [retained] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
|
|
expect(retained.providerRef).toBe(`checkout:${workspaceId}`);
|
|
});
|
|
|
|
it("polls bounded requests using the provider's advertised preparation deadline", async () => {
|
|
const f = await fixture();
|
|
const deadline = new Date(Date.now() + 30_000).toISOString();
|
|
const phases: string[] = [];
|
|
const workers = {
|
|
getWorker: () => ({ supportedMethods: ["workspaceOperation", "workspacePrepare"] }),
|
|
call: async (pluginId: string, method: string, params: any, timeout: number) => {
|
|
expect(method).toBe("workspacePrepare");
|
|
expect(timeout).toBeLessThanOrEqual(15_000);
|
|
phases.push(params.phase);
|
|
if (params.phase === "submit") return { kind: "pending", deadline };
|
|
expect(params.deadline).toBe(deadline);
|
|
if (phases.length === 2) throw new Error("lost status reply");
|
|
const { phase: _phase, ...original } = params;
|
|
return f.workers.call(pluginId, "workspaceOperation", original);
|
|
},
|
|
} as unknown as WorkspaceWorkers;
|
|
expect(await preparePluginWorkspace(db, workers, f.input)).toBeTruthy();
|
|
expect(phases).toEqual(["submit", "poll", "poll"]);
|
|
expect(f.calls).toHaveLength(1);
|
|
});
|
|
|
|
it("stops preparation at the provider deadline before issuing a poll", async () => {
|
|
const f = await fixture();
|
|
const workers = {
|
|
getWorker: () => ({ supportedMethods: ["workspaceOperation", "workspacePrepare"] }),
|
|
call: async (_pluginId: string, method: string, params: { phase: string }) => {
|
|
expect(method).toBe("workspacePrepare");
|
|
expect(params.phase).toBe("submit");
|
|
return { kind: "pending", deadline: new Date(Date.now() - 1).toISOString() };
|
|
},
|
|
} as unknown as WorkspaceWorkers;
|
|
await expect(preparePluginWorkspace(db, workers, f.input)).rejects.toMatchObject({
|
|
kind: "provider", reason: "deadline_exceeded", retryable: true,
|
|
});
|
|
});
|
|
|
|
it("rejects an oversized provider deadline before it can retain the workspace lifecycle lock", async () => {
|
|
const f = await fixture();
|
|
const workers = {
|
|
getWorker: () => ({ supportedMethods: ["workspaceOperation", "workspacePrepare"] }),
|
|
call: async (_pluginId: string, method: string, params: { phase: string }) => {
|
|
expect(method).toBe("workspacePrepare");
|
|
expect(params.phase).toBe("submit");
|
|
return { kind: "pending", deadline: "2099-01-01T00:00:00Z" };
|
|
},
|
|
} as unknown as WorkspaceWorkers;
|
|
await expect(preparePluginWorkspace(db, workers, f.input)).rejects.toMatchObject({
|
|
kind: "provider", reason: "unsupported_operation", retryable: false,
|
|
});
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
expect(task?.executionWorkspaceId).toBeTruthy();
|
|
expect(await preparePluginWorkspace(db, f.workers, { ...f.input, existingWorkspaceId: task!.executionWorkspaceId })).toBeTruthy();
|
|
});
|
|
|
|
it.each([
|
|
["missing", undefined],
|
|
["stopped", { status: "stopped", supportedMethods: [] }],
|
|
["starting", { status: "starting", supportedMethods: [] }],
|
|
] as const)("retries preparation when the ready provider's worker is %s, before sending a request", async (_state, worker) => {
|
|
const f = await fixture();
|
|
const workers = { getWorker: () => worker, call: f.workers.call.bind(f.workers) } as unknown as WorkspaceWorkers;
|
|
const failure = await preparePluginWorkspace(db, workers, f.input).then(() => null, (error: unknown) => error);
|
|
expect(failure).toBeInstanceOf(WorkspaceProviderFailureError);
|
|
expect((failure as WorkspaceProviderFailureError).summary()).toEqual({ kind: "worker_unavailable", reason: null, retryable: true });
|
|
expect(findPluginWorkerUnavailableError(failure)).toBeInstanceOf(PluginWorkerUnavailableError);
|
|
expect(f.calls).toHaveLength(0);
|
|
});
|
|
it("keeps a running worker without workspace operations an unavailable provider", async () => {
|
|
const f = await fixture();
|
|
const workers = { getWorker: () => ({ status: "running", supportedMethods: [] }), call: f.workers.call.bind(f.workers) } as unknown as WorkspaceWorkers;
|
|
const failure = await preparePluginWorkspace(db, workers, f.input).then(() => null, (error: unknown) => error);
|
|
expect(failure).not.toBeInstanceOf(WorkspaceProviderFailureError);
|
|
expect((failure as Error).message).toBe("The workspace storage plugin is unavailable");
|
|
expect(f.calls).toHaveLength(0);
|
|
});
|
|
it.each([1, undefined])("gates recovery hints by provider capability (%s)", async (version) => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, recoveryPullRequestVersion: version as 1 | undefined } } }).where(eq(plugins.id, f.pluginId));
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
const recoveryPullRequest = { number: 42, headSha: "a".repeat(40) };
|
|
await inspectPluginWorkspace(db, f.workers, workspace!, recoveryPullRequest);
|
|
expect(f.calls.at(-1).operation.recoveryPullRequest).toEqual(version === 1 ? recoveryPullRequest : undefined);
|
|
let freshInspectionCompleted = false;
|
|
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), recoveryPullRequest, async () => {
|
|
freshInspectionCompleted = f.calls.at(-1).operation.kind === "inspect";
|
|
throw new Error("delivery changed");
|
|
})).rejects.toThrow("delivery changed");
|
|
expect(freshInspectionCompleted).toBe(true);
|
|
const [unchanged] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(unchanged.metadata?.workspaceProviderRemoval).toBeUndefined();
|
|
f.loseRemoval();
|
|
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), recoveryPullRequest)).rejects.toThrow();
|
|
const [pending] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect((pending.metadata?.workspaceProviderRemoval as { operation: { recoveryPullRequest?: unknown } }).operation.recoveryPullRequest)
|
|
.toEqual(version === 1 ? recoveryPullRequest : undefined);
|
|
await removePluginWorkspace(db, f.workers, pending, "a".repeat(40), recoveryPullRequest, async () => { throw new Error("must not reauthorize pending removal"); });
|
|
expect(f.calls.at(-1).operation.recoveryPullRequest).toEqual(version === 1 ? recoveryPullRequest : undefined);
|
|
});
|
|
it.each([1, undefined])("replays a lost removal acknowledgement unchanged after capability flip (%s)", async (version) => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
const setCapability = (recoveryPullRequestVersion: 1 | undefined) => db.update(plugins).set({ manifestJson: {
|
|
...plugin.manifestJson, workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, recoveryPullRequestVersion },
|
|
} }).where(eq(plugins.id, f.pluginId));
|
|
await setCapability(version as 1 | undefined);
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
f.loseRemoval();
|
|
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), { number: 42, headSha: "a".repeat(40) })).rejects.toThrow();
|
|
const first = f.calls.at(-1);
|
|
await db.update(plugins).set({ status: "disabled" }).where(eq(plugins.id, f.pluginId));
|
|
const count = f.calls.length;
|
|
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40))).rejects.toThrow("unavailable");
|
|
expect(f.calls).toHaveLength(count);
|
|
await db.update(plugins).set({ status: "ready" }).where(eq(plugins.id, f.pluginId));
|
|
await setCapability(version === 1 ? undefined : 1);
|
|
const [pending] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
await removePluginWorkspace(db, f.workers, pending, "a".repeat(40));
|
|
expect(f.calls.at(-1).operation).toEqual(first.operation);
|
|
expect(f.calls.at(-1).operationId).toBe(first.operationId);
|
|
expect(pending.metadata?.workspaceProviderRemoval).toMatchObject({ operation: first.operation, operationId: first.operationId });
|
|
const [removed] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(removed.metadata?.workspaceProviderRemoved).toBe(true);
|
|
});
|
|
it.each(["legacy", "ambiguous", "mismatched"] as const)("fails closed for uncertain removal identities (%s)", async (identity) => {
|
|
const hint = identity !== "legacy";
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, recoveryPullRequestVersion: 1 } } }).where(eq(plugins.id, f.pluginId));
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
f.loseRemoval();
|
|
await expect(removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40), hint ? { number: 42, headSha: "a".repeat(40) } : undefined)).rejects.toThrow();
|
|
const first = f.calls.at(-1);
|
|
const [pending] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
const intent = pending.metadata!.workspaceProviderRemoval as Record<string, unknown>;
|
|
if (identity === "mismatched") intent.operationId = "0".repeat(64);
|
|
else delete intent.operationId;
|
|
await db.update(executionWorkspaces).set({ metadata: { ...pending.metadata, workspaceProviderRemoval: intent } }).where(eq(executionWorkspaces.id, pending.id));
|
|
if (hint) {
|
|
const count = f.calls.length;
|
|
await expect(removePluginWorkspace(db, f.workers, pending, "a".repeat(40))).rejects.toThrow(identity === "ambiguous" ? "identity is ambiguous" : "identity does not match");
|
|
expect(f.calls).toHaveLength(count);
|
|
} else {
|
|
await removePluginWorkspace(db, f.workers, pending, "a".repeat(40));
|
|
expect(f.calls.at(-1).operationId).toBe(first.operationId);
|
|
expect(f.calls.at(-1).operation).toEqual(first.operation);
|
|
}
|
|
});
|
|
it.each(["refs/heads/main", null, undefined])("persists the actual prepared destination (%s)", async (destinationRef) => {
|
|
const f = await fixture();
|
|
const workers: WorkspaceWorkers = { ...f.workers, call: async (...args: Parameters<typeof f.workers.call>) => {
|
|
const result = await f.workers.call(...args) as { kind?: string };
|
|
return result.kind === "prepared" ? { ...result, ...(destinationRef !== undefined ? { destinationRef } : {}) } : result;
|
|
} };
|
|
const workspace = await preparePluginWorkspace(db, workers, f.input);
|
|
expect(workspace!.metadata?.workspaceProviderDestinationRef).toBe(destinationRef);
|
|
});
|
|
it.each([true, undefined])("wakes parked cleanup when a provider upgrade enables ancestry (%s)", async (supportsMergeAncestry) => {
|
|
const f = await fixture();
|
|
await preparePluginWorkspace(db, f.workers, f.input);
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId));
|
|
const svc = executionWorkspaceService(db, { pluginWorkerManager: f.workers, workspaceReaperCooldownDays: 0 });
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
expect(await svc.nextTerminalWorkspaceSweepAt()).toBeNull();
|
|
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(false);
|
|
const events: string[] = [];
|
|
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.terminalWorkspaces, (event, hints) => {
|
|
events.push(event);
|
|
if (event === "settled") svc.terminalWorkspaceWorkSettled(hints);
|
|
});
|
|
try {
|
|
await pluginRegistryService(db).update(f.pluginId, { manifest: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry } } });
|
|
expect(events).toEqual(["intent", "settled"]);
|
|
} finally { unsubscribe(); }
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(true);
|
|
expect(f.calls.some(call => call.operation.kind === "remove")).toBe(true);
|
|
});
|
|
it("wakes cleanup only for provider contract changes and activation", async () => {
|
|
const f = await fixture();
|
|
const registry = pluginRegistryService(db);
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
const events: string[] = [];
|
|
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.terminalWorkspaces, event => events.push(event));
|
|
try {
|
|
await registry.update(f.pluginId, { manifest: { ...plugin.manifestJson, description: "Updated description" } });
|
|
await registry.updateStatus(f.pluginId, { status: "disabled" });
|
|
expect(events).toEqual([]);
|
|
await registry.updateStatus(f.pluginId, { status: "ready" });
|
|
expect(events.splice(0)).toEqual(["intent", "settled"]);
|
|
await registry.updateStatus(f.pluginId, { status: "ready" });
|
|
await registry.upsertCompanySettings(f.pluginId, f.companyId, { enabled: false, settingsJson: {} });
|
|
expect(events).toEqual([]);
|
|
await registry.upsertCompanySettings(f.pluginId, f.companyId, { enabled: true, settingsJson: {} });
|
|
expect(events.splice(0)).toEqual(["intent", "settled"]);
|
|
await registry.upsertCompanySettings(f.pluginId, f.companyId, { enabled: true, settingsJson: {} });
|
|
expect(events).toEqual([]);
|
|
await db.transaction(async tx => {
|
|
await pluginRegistryService(tx as unknown as typeof db).update(f.pluginId, {
|
|
manifest: { ...plugin.manifestJson, workspaceProvider: undefined },
|
|
});
|
|
expect(events).toEqual(["intent"]);
|
|
});
|
|
expect(events.splice(0)).toEqual(["intent", "settled"]);
|
|
await registry.updateStatus(f.pluginId, { status: "disabled" });
|
|
await registry.updateStatus(f.pluginId, { status: "ready" });
|
|
await registry.update(f.pluginId, { manifest: { ...plugin.manifestJson, workspaceProvider: undefined, description: "Other plugin" } });
|
|
expect(events).toEqual([]);
|
|
} finally { unsubscribe(); }
|
|
});
|
|
it("keeps storage idle without a settled completion HEAD, even with a recorded PR", async () => {
|
|
const f = await fixture();
|
|
await preparePluginWorkspace(db, f.workers, f.input);
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId));
|
|
await db.insert(issueWorkProducts).values({ companyId: f.companyId, issueId: f.issueId,
|
|
type: "pull_request", provider: "github", title: "Work", status: "active",
|
|
url: "https://github.com/example/repo/pull/1" });
|
|
const options = { pluginWorkerManager: f.workers, workspaceReaperCooldownDays: 0,
|
|
resolveWorkspaceDelivery: async () => { throw new Error("No delivery lookup without a completion HEAD"); } };
|
|
const svc = executionWorkspaceService(db, options);
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(false);
|
|
expect(await svc.nextTerminalWorkspaceSweepAt()).toBeNull();
|
|
svc.terminalWorkspaceWorkSettled(new Set([`issue:${randomUUID()}`]));
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
expect(await svc.nextTerminalWorkspaceSweepAt()).toBeNull();
|
|
await executionWorkspaceService(db, options).sweepTerminalWorkspaces(100);
|
|
expect(f.calls.some(call => call.operation.kind === "inspect")).toBe(false);
|
|
});
|
|
it("uses bounded finalization only for opted-in providers, preserving maintenance inspection", async () => {
|
|
const f = await fixture();
|
|
const workspace = (await preparePluginWorkspace(db, f.workers, f.input))!;
|
|
const context = { runId: f.runId, agentId: f.input.agentId };
|
|
expect((await finalizePluginWorkspace(db, f.workers, workspace, context)).kind).toBe("inspection");
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, finalizationVersion: 1 } } }).where(eq(plugins.id, f.pluginId));
|
|
const completionIds: string[] = [];
|
|
const call = f.workers.call.bind(f.workers);
|
|
f.workers.call = async (...args) => {
|
|
const p = args[2] as any;
|
|
if (p.operation.kind !== "finalize") return call(...args);
|
|
expect(p.agentId).toBe(context.agentId);
|
|
completionIds.push(p.operationId);
|
|
expect(Date.parse(p.deadline) - Date.now()).toBeLessThanOrEqual(30_000);
|
|
expect(args[3]).toBe(40_000);
|
|
return { kind: "finalized", branchName: workspace.branchName, headSha: "a".repeat(40) };
|
|
};
|
|
expect((await finalizePluginWorkspace(db, f.workers, workspace, context)).kind).toBe("finalized");
|
|
await finalizePluginWorkspace(db, f.workers, workspace, context);
|
|
await finalizePluginWorkspace(db, f.workers, workspace, { ...context, runId: randomUUID() });
|
|
expect(completionIds[1]).toBe(completionIds[0]);
|
|
expect(completionIds[2]).not.toBe(completionIds[0]);
|
|
expect((await inspectPluginWorkspace(db, f.workers, workspace)).kind).toBe("inspection");
|
|
f.workers.call = async () => { throw new Error("sync failed"); };
|
|
await expect(finalizePluginWorkspace(db, f.workers, workspace, context)).rejects.toThrow("incomplete");
|
|
});
|
|
it("keeps the single-connection pool responsive while provider preparation holds its lifecycle lock", async () => {
|
|
const f = await fixture();
|
|
let entered!: () => void, release!: () => void;
|
|
const started = new Promise<void>((resolve) => {
|
|
entered = resolve;
|
|
});
|
|
const resume = new Promise<void>((resolve) => {
|
|
release = resolve;
|
|
});
|
|
const call = f.workers.call.bind(f.workers);
|
|
f.workers.call = async (...args) => {
|
|
if ((args[2] as any).operation.kind === "prepare") {
|
|
entered();
|
|
await resume;
|
|
}
|
|
return call(...args);
|
|
};
|
|
const preparing = preparePluginWorkspace(db, f.workers, f.input);
|
|
await started;
|
|
try {
|
|
await expect(
|
|
projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoRef: "next" },
|
|
),
|
|
).rejects.toThrow("Workspace lifecycle is busy");
|
|
expect(await db.execute(sql`select 1 as usable`)).toMatchObject([
|
|
{ usable: 1 },
|
|
]);
|
|
} finally {
|
|
release();
|
|
}
|
|
const workspace = await preparing;
|
|
await expect(
|
|
inspectPluginWorkspace(db, f.workers, workspace!),
|
|
).resolves.toMatchObject({ kind: "inspection" });
|
|
});
|
|
it("preserves a scheduled selection through the first provider binding and next admission", async () => {
|
|
const f = await fixture();
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
});
|
|
const actor = {
|
|
type: "board" as const,
|
|
source: "local_implicit" as const,
|
|
isInstanceAdmin: true,
|
|
};
|
|
const request = {
|
|
companyId: f.companyId,
|
|
issueId: f.issueId,
|
|
actor,
|
|
selection: { kind: "task_directory" as const },
|
|
expectedBindingRevision: 0,
|
|
requestKey: "during-prepare",
|
|
};
|
|
await svc.selectTaskWorkspace(request);
|
|
await preparePluginWorkspace(db, f.workers, f.input);
|
|
expect(
|
|
await svc.inspectTaskWorkspace(f.companyId, f.issueId, actor),
|
|
).toMatchObject({
|
|
bindingRevision: 2,
|
|
pendingSelection: {
|
|
expectedBindingRevision: 2,
|
|
intent: { request: { expectedBindingRevision: 0 } },
|
|
},
|
|
});
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
expect(
|
|
await svc.applyPendingTaskWorkspaceSelection({
|
|
companyId: f.companyId,
|
|
issueId: f.issueId,
|
|
runId: randomUUID(),
|
|
actor,
|
|
}),
|
|
).toBe(true);
|
|
expect(await svc.selectTaskWorkspace(request)).toMatchObject({
|
|
kind: "applied",
|
|
});
|
|
});
|
|
it("does not retire reusable storage for rejected or normalized unchanged source edits", async () => {
|
|
const f = await fixture(),
|
|
workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "archived", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
|
|
expect(
|
|
await projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoRef: " " },
|
|
),
|
|
).toMatchObject({ repoRef: null });
|
|
expect(
|
|
await projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoUrl: null },
|
|
),
|
|
).toBeNull();
|
|
const [row] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(row.metadata?.workspaceProviderRetired).not.toBe(true);
|
|
expect(
|
|
await executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
}).reopenClosedIsolatedExecutionWorkspaceForIssue({
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board",
|
|
source: "local_implicit",
|
|
isInstanceAdmin: true,
|
|
},
|
|
}),
|
|
).toMatchObject({ ok: true, reopened: true });
|
|
});
|
|
it("uses new inspection identities even when the provider caches operation results", async () => {
|
|
const f = await fixture(),
|
|
workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
const call = f.workers.call.bind(f.workers),
|
|
cache = new Map<string, unknown>();
|
|
f.workers.call = async (...args) => {
|
|
const params = args[2] as any;
|
|
if (cache.has(params.operationId))
|
|
return cache.get(params.operationId) as any;
|
|
const result = await call(...args);
|
|
cache.set(params.operationId, result);
|
|
return result;
|
|
};
|
|
expect(
|
|
await inspectPluginWorkspace(db, f.workers, workspace!),
|
|
).toMatchObject({ hasDirtyTrackedFiles: false });
|
|
f.dirty();
|
|
expect(
|
|
await inspectPluginWorkspace(db, f.workers, workspace!),
|
|
).toMatchObject({ hasDirtyTrackedFiles: true });
|
|
expect(cache.size).toBe(2);
|
|
});
|
|
it("retries closed storage cleanup when the pre-delete inspection failed before recording removal", async () => {
|
|
const f = await fixture(),
|
|
workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done", completedAt: new Date() })
|
|
.where(eq(issues.id, f.issueId));
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
const call = f.workers.call.bind(f.workers);
|
|
let inspections = 0;
|
|
const workers = {
|
|
getWorker: (id: string) =>
|
|
id === f.pluginId
|
|
? { supportedMethods: ["workspaceOperation"] }
|
|
: undefined,
|
|
call: async (...args: Parameters<WorkspaceWorkers["call"]>) => {
|
|
if (
|
|
(args[2] as any).operation.kind === "inspect" &&
|
|
++inspections === 2
|
|
)
|
|
throw new Error("temporary inspection outage");
|
|
return call(...args);
|
|
},
|
|
} as unknown as WorkspaceWorkers;
|
|
let clock = Date.now();
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: workers,
|
|
workspaceReaperCooldownDays: 0,
|
|
now: () => new Date(clock),
|
|
});
|
|
const sweep = () => svc.sweepTerminalWorkspaces(100);
|
|
expect(await sweep()).toMatchObject({ cleanupFailed: 1 });
|
|
const [failed] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(failed.status).toBe("cleanup_failed");
|
|
expect(failed.metadata?.workspaceProviderRemoval).toBeUndefined();
|
|
expect(failed.metadata?.terminalCleanupRetry).toMatchObject({ failures: 1, failure: { kind: "transport" },
|
|
retryAt: new Date(clock + 300_000).toISOString() });
|
|
expect(await sweep()).toMatchObject({ skippedBackoff: 1 });
|
|
clock += 300_000;
|
|
await sweep();
|
|
const [removed] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(removed.metadata?.workspaceProviderRemoved).toBe(true);
|
|
expect(removed.status).toBe("archived");
|
|
expect(removed.metadata).not.toHaveProperty("terminalCleanupRetry");
|
|
});
|
|
it("prepares a durable reference without a local checkout or agent allocation", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
expect(workspace).toMatchObject({
|
|
providerType: "plugin",
|
|
cwd: null,
|
|
status: "active",
|
|
});
|
|
expect(f.calls[0].operation.source).not.toHaveProperty("cwd");
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, f.issueId));
|
|
expect(task.executionWorkspaceId).toBe(workspace!.id);
|
|
expect(
|
|
await inspectPluginWorkspace(db, f.workers, workspace!),
|
|
).toMatchObject({ remotelyRecoverable: true });
|
|
});
|
|
it("retains the reservation and retries the same operation after a lost response", async () => {
|
|
const f = await fixture();
|
|
f.lose();
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, f.input),
|
|
).rejects.toThrow("incomplete");
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, f.issueId));
|
|
const result = await preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
existingWorkspaceId: task.executionWorkspaceId,
|
|
});
|
|
expect(result!.id).toBe(task.executionWorkspaceId);
|
|
expect(f.calls[1].operationId).toBe(f.calls[0].operationId);
|
|
});
|
|
it("reconciles a lost preparation result after the task run stops", async () => {
|
|
const f = await fixture();
|
|
f.lose();
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, f.input),
|
|
).rejects.toThrow("incomplete");
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "cancelled" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, f.issueId));
|
|
const [reservation] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
|
|
expect(reservation.providerRef).toBeNull();
|
|
await expect(
|
|
inspectPluginWorkspace(db, f.workers, reservation),
|
|
).resolves.toMatchObject({ remotelyRecoverable: true });
|
|
expect(f.calls[1]).toMatchObject({
|
|
runId: f.runId,
|
|
agentId: f.input.agentId,
|
|
operation: { kind: "prepare" },
|
|
});
|
|
expect(f.calls[1].operationId).toBe(f.calls[0].operationId);
|
|
const [recorded] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, reservation.id));
|
|
expect(recorded.providerRef).not.toBeNull();
|
|
});
|
|
it.each(["inspect", "retry", "replacement"] as const)("recovers a lost Launcher prepare through %s", async (recovery) => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
capabilities: [...plugin.manifestJson.capabilities, "environment.drivers.register"],
|
|
environmentDrivers: [{ driverKey: "tasks", displayName: "Tasks", configSchema: { type: "object" },
|
|
supportsTasks: true, sessionHandoffVersion: 1 }],
|
|
} }).where(eq(plugins.id, f.pluginId));
|
|
const [environment] = await db.insert(environments).values({ name: `Workspace Launcher ${f.pluginId}`, driver: "plugin",
|
|
status: "active", config: { pluginKey: plugin.pluginKey, driverKey: "tasks" } }).returning();
|
|
const taskLease = { environmentId: environment.id, driverKey: "tasks", providerLeaseId: randomUUID() };
|
|
const leaseId = randomUUID();
|
|
const input = { ...f.input, environment: { ...environment, driver: "plugin" as const },
|
|
acquireTaskLease: async () => ({ leaseId, taskLease }) };
|
|
let deleted = false;
|
|
const replacementTaskLease = { ...taskLease, providerLeaseId: randomUUID() };
|
|
const replacementWorkers = { ...f.workers, call: vi.fn<WorkspaceWorkers["call"]>(async (...args) => {
|
|
const request = args[2] as (typeof f.calls)[number];
|
|
if (deleted && request.taskLease?.providerLeaseId === taskLease.providerLeaseId)
|
|
throw new Error("Previous Launcher allocation is gone");
|
|
const result = await f.workers.call(...args) as { kind: string };
|
|
if (!deleted) throw new Error("Lost committed preparation result");
|
|
return { ...result, created: false } as never;
|
|
}) };
|
|
if (recovery !== "replacement") f.lose();
|
|
await expect(preparePluginWorkspace(db, recovery === "replacement" ? replacementWorkers : f.workers, input)).rejects.toThrow("incomplete");
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
const [reservation] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
|
|
expect(reservation.providerRef).toBeNull();
|
|
expect(reservation.metadata?.workspaceProviderPreparationPlacement).toMatchObject({
|
|
generation: 0, runId: f.runId, agentId: f.input.agentId, taskLease,
|
|
});
|
|
if (recovery === "inspect") {
|
|
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.runId));
|
|
const workers = { ...f.workers, call: vi.fn<WorkspaceWorkers["call"]>(async (...args) => {
|
|
const request = args[2] as (typeof f.calls)[number];
|
|
expect(isPluginWorkspaceReplay({
|
|
pluginId: f.pluginId, companyId: f.companyId, projectId: f.input.projectId,
|
|
projectWorkspaceId: f.input.projectWorkspaceId, executionWorkspaceId: reservation.id,
|
|
})).toBe(request.operation.kind === "prepare");
|
|
return f.workers.call(...args);
|
|
}) };
|
|
await expect(inspectPluginWorkspace(db, workers, reservation)).resolves.toMatchObject({ remotelyRecoverable: true });
|
|
} else if (recovery === "retry") {
|
|
await expect(preparePluginWorkspace(db, f.workers, { ...input, existingWorkspaceId: reservation.id })).resolves.toMatchObject({ id: reservation.id });
|
|
} else {
|
|
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, f.runId));
|
|
const [nextRun] = await db.insert(heartbeatRuns).values({
|
|
companyId: f.companyId, issueId: f.issueId, agentId: f.input.agentId, status: "running",
|
|
}).returning();
|
|
deleted = true;
|
|
await expect(preparePluginWorkspace(db, replacementWorkers, {
|
|
...input, runId: nextRun.id, existingWorkspaceId: reservation.id,
|
|
acquireTaskLease: async () => ({ leaseId: randomUUID(), taskLease: replacementTaskLease }),
|
|
})).resolves.toMatchObject({ id: reservation.id });
|
|
expect(replacementWorkers.call).toHaveBeenCalledTimes(2);
|
|
expect(f.calls[1]).toMatchObject({ runId: nextRun.id, agentId: f.input.agentId, taskLease: replacementTaskLease });
|
|
expect(f.calls[1].operationId).not.toBe(f.calls[0].operationId);
|
|
}
|
|
if (recovery !== "replacement") expect(f.calls[1]).toMatchObject({
|
|
runId: f.runId, agentId: f.input.agentId, taskLease, operationId: f.calls[0].operationId,
|
|
});
|
|
expect(f.calls[1].operation).toEqual(f.calls[0].operation);
|
|
const [prepared] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, reservation.id));
|
|
expect(prepared.providerRef).not.toBeNull();
|
|
expect(prepared.metadata?.workspaceProviderPreparationPlacement).toBeUndefined();
|
|
});
|
|
it("reconciles an unfinished reservation before applying PR-only cleanup eligibility", async () => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
|
|
f.lose();
|
|
await expect(preparePluginWorkspace(db, f.workers, f.input)).rejects.toThrow("incomplete");
|
|
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.runId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
f.dirty();
|
|
await executionWorkspaceService(db, { pluginWorkerManager: f.workers, workspaceReaperCooldownDays: 0 }).sweepTerminalWorkspaces(100);
|
|
expect(f.calls.filter(call => call.operation.kind === "prepare")).toHaveLength(2);
|
|
expect(f.calls.filter(call => call.operation.kind === "inspect")).toHaveLength(1);
|
|
expect(f.calls.some(call => call.operation.kind === "remove")).toBe(false);
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
|
|
expect(workspace.providerRef).not.toBeNull();
|
|
});
|
|
it("fails closed for a disabled plugin, stopped run, and cross-company source", async () => {
|
|
const f = await fixture();
|
|
await db.insert(pluginCompanySettings).values({
|
|
pluginId: f.pluginId,
|
|
companyId: f.companyId,
|
|
enabled: false,
|
|
});
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, f.input),
|
|
).rejects.toThrow("unavailable");
|
|
await db
|
|
.update(pluginCompanySettings)
|
|
.set({ enabled: true })
|
|
.where(eq(pluginCompanySettings.pluginId, f.pluginId));
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "cancelled" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, f.input),
|
|
).rejects.toThrow();
|
|
expect(f.calls).toHaveLength(0);
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
companyId: randomUUID(),
|
|
}),
|
|
).resolves.toBeNull();
|
|
});
|
|
it("keeps source ownership immutable and preserves bindings through metadata edits", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
const svc = projectService(db);
|
|
await expect(
|
|
svc.update(f.input.projectId, {
|
|
executionWorkspacePolicy: { storageProviderPluginId: null },
|
|
}),
|
|
).rejects.toMatchObject({ status: 409, details: { code: "project_storage_locked" } });
|
|
await expect(
|
|
svc.updateWorkspace(f.input.projectId, f.input.projectWorkspaceId, {
|
|
repoUrl: "https://github.com/example/other.git",
|
|
}),
|
|
).rejects.toThrow("repository source");
|
|
await expect(
|
|
svc.removeWorkspace(f.input.projectId, f.input.projectWorkspaceId),
|
|
).rejects.toThrow("repository source");
|
|
const placement = { generation: 0, runId: f.runId, agentId: f.input.agentId,
|
|
taskLease: { environmentId: randomUUID(), driverKey: "tasks", providerLeaseId: randomUUID() } };
|
|
await db.update(executionWorkspaces).set({
|
|
metadata: { ...workspace!.metadata, workspaceProviderPreparationPlacement: placement },
|
|
}).where(eq(executionWorkspaces.id, workspace!.id));
|
|
const execution = executionWorkspaceService(db);
|
|
const updated = await execution.update(workspace!.id, {
|
|
metadata: {
|
|
note: "caller edit",
|
|
workspaceProvider: { workspaceRef: "foreign" },
|
|
workspaceProviderPreparationPlacement: { ...placement, runId: randomUUID() },
|
|
},
|
|
});
|
|
expect(updated!.metadata?.workspaceProviderPreparationPlacement).toBeUndefined();
|
|
const [saved] = await db.select({ metadata: executionWorkspaces.metadata })
|
|
.from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(saved.metadata?.workspaceProviderPreparationPlacement).toEqual(placement);
|
|
expect(updated!.metadata?.workspaceProvider).toEqual(
|
|
workspace!.metadata?.workspaceProvider,
|
|
);
|
|
await expect(
|
|
execution.update(workspace!.id, { providerType: "local_fs" }),
|
|
).rejects.toThrow("storage plugin");
|
|
});
|
|
it("describes provider removal without promising host cleanup commands", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done" })
|
|
.where(eq(issues.id, f.issueId));
|
|
const readiness = await executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
}).getCloseReadiness(workspace!.id);
|
|
expect(readiness!.plannedActions).toContainEqual(
|
|
expect.objectContaining({
|
|
kind: "remove_provider_workspace",
|
|
command: null,
|
|
}),
|
|
);
|
|
expect(
|
|
readiness!.plannedActions.some(
|
|
(action) => action.kind === "git_branch_delete",
|
|
),
|
|
).toBe(false);
|
|
});
|
|
it("reopens through the same provider without requiring a live run or a host checkout", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "archived", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
const result = await executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
}).reopenClosedIsolatedExecutionWorkspaceForIssue({
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board",
|
|
source: "local_implicit",
|
|
isInstanceAdmin: true,
|
|
},
|
|
});
|
|
expect(result).toMatchObject({
|
|
ok: true,
|
|
reopened: true,
|
|
generation: 1,
|
|
workspace: { cwd: null, providerType: "plugin" },
|
|
});
|
|
expect(f.calls.at(-1)).toMatchObject({
|
|
runId: null,
|
|
agentId: null,
|
|
operation: { kind: "prepare", branchMode: "existing" },
|
|
});
|
|
expect(f.calls.at(-1).operation.existing.workspaceRef).toBe(
|
|
workspace!.providerRef,
|
|
);
|
|
});
|
|
it("keeps a relocated restoration reference consistent through subsequent removal", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "archived", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
|
|
f.relocate();
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
});
|
|
const result = await svc.reopenClosedIsolatedExecutionWorkspaceForIssue({
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board",
|
|
source: "local_implicit",
|
|
isInstanceAdmin: true,
|
|
},
|
|
});
|
|
expect(result).toMatchObject({ ok: true, reopened: true });
|
|
const [restored] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(restored.providerRef).toBe(`${workspace!.providerRef}:restored`);
|
|
expect(restored.metadata?.workspaceProvider).toMatchObject({
|
|
workspaceRef: restored.providerRef,
|
|
});
|
|
await removePluginWorkspace(db, f.workers, restored, "a".repeat(40));
|
|
expect(f.calls.at(-1).operation).toMatchObject({
|
|
kind: "remove",
|
|
binding: { workspaceRef: restored.providerRef },
|
|
});
|
|
});
|
|
it("reads workspace metadata without provider inspection", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done" })
|
|
.where(eq(issues.id, f.issueId));
|
|
f.workers.call = async () => { throw new Error("Storage unavailable"); };
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
inspectGitCloseReadiness: async () => {
|
|
throw new Error("Host inspection must not run for provider storage");
|
|
},
|
|
});
|
|
expect(await svc.getById(workspace!.id)).toMatchObject({
|
|
deliveryState: "unknown",
|
|
});
|
|
expect(await svc.getById(workspace!.id)).toMatchObject({
|
|
deliveryState: "unknown",
|
|
});
|
|
expect(
|
|
f.calls.filter((call) => call.operation.kind === "inspect"),
|
|
).toHaveLength(0);
|
|
});
|
|
it("replays a durable removal after a lost result without inspecting the missing checkout", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
f.loseRemoval();
|
|
await expect(
|
|
removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40)),
|
|
).rejects.toThrow("incomplete");
|
|
const inspections = f.calls.filter(
|
|
(call) => call.operation.kind === "inspect",
|
|
).length;
|
|
await removePluginWorkspace(db, f.workers, workspace!, null);
|
|
expect(
|
|
f.calls.filter((call) => call.operation.kind === "inspect"),
|
|
).toHaveLength(inspections);
|
|
const removals = f.calls.filter(
|
|
(call) => call.operation.kind === "remove",
|
|
);
|
|
expect(removals).toHaveLength(2);
|
|
expect(removals[0].operationId).toBe(removals[1].operationId);
|
|
const [row] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(row.metadata).toMatchObject({
|
|
workspaceProviderRemoved: true,
|
|
workspaceProviderRemoval: null,
|
|
});
|
|
});
|
|
it("retires cleaned historical bindings when the storage source changes and refuses later reopen", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "archived", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
|
|
await expect(
|
|
projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoUrl: "https://github.com/example/new.git" },
|
|
),
|
|
).resolves.not.toBeNull();
|
|
const before = f.calls.length;
|
|
const result = await executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
}).reopenClosedIsolatedExecutionWorkspaceForIssue({
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board",
|
|
source: "local_implicit",
|
|
isInstanceAdmin: true,
|
|
},
|
|
});
|
|
expect(result).toMatchObject({ ok: false, code: "not_reopenable" });
|
|
expect(f.calls).toHaveLength(before);
|
|
await expect(
|
|
projectService(db).update(f.input.projectId, {
|
|
executionWorkspacePolicy: {},
|
|
}),
|
|
).resolves.not.toBeNull();
|
|
await expect(
|
|
projectService(db).removeWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
),
|
|
).resolves.not.toBeNull();
|
|
});
|
|
it("keeps sweeping later workspaces when one storage plugin is unavailable", async () => {
|
|
const first = await fixture(),
|
|
second = await fixture();
|
|
await preparePluginWorkspace(db, first.workers, first.input);
|
|
await preparePluginWorkspace(db, second.workers, second.input);
|
|
await db.insert(pluginCompanySettings).values({
|
|
pluginId: first.pluginId,
|
|
companyId: first.companyId,
|
|
enabled: false,
|
|
});
|
|
for (const f of [first, second])
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done", completedAt: new Date() })
|
|
.where(eq(issues.id, f.issueId));
|
|
const workers = {
|
|
getWorker: (id: string) =>
|
|
([first.pluginId, second.pluginId] as string[]).includes(id)
|
|
? { supportedMethods: ["workspaceOperation"] }
|
|
: undefined,
|
|
call: (id: string, ...args: unknown[]) =>
|
|
(id === first.pluginId ? first.workers : second.workers).call(
|
|
id,
|
|
...(args as [any, any, any]),
|
|
),
|
|
} as unknown as WorkspaceWorkers;
|
|
await expect(
|
|
executionWorkspaceService(db, {
|
|
pluginWorkerManager: workers,
|
|
workspaceReaperCooldownDays: 0,
|
|
}).sweepTerminalWorkspaces(100),
|
|
).resolves.toBeDefined();
|
|
expect(
|
|
second.calls.some((call) => call.operation.kind === "inspect"),
|
|
).toBe(true);
|
|
});
|
|
it("retries an ambiguous deletion through the existing terminal sweep", async () => {
|
|
const f = await fixture();
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, f.pluginId));
|
|
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson,
|
|
workspaceProvider: { ...plugin.manifestJson.workspaceProvider!, supportsMergeAncestry: false } } }).where(eq(plugins.id, f.pluginId));
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
f.loseRemoval();
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "cleanup_failed", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done", completedAt: new Date() })
|
|
.where(eq(issues.id, f.issueId));
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "cancelled" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await expect(
|
|
removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40)),
|
|
).rejects.toThrow("incomplete");
|
|
const workers = {
|
|
getWorker: (id: string) =>
|
|
id === f.pluginId
|
|
? { supportedMethods: ["workspaceOperation"] }
|
|
: undefined,
|
|
call: f.workers.call.bind(f.workers),
|
|
} as unknown as WorkspaceWorkers;
|
|
await executionWorkspaceService(db, {
|
|
pluginWorkerManager: workers,
|
|
workspaceReaperCooldownDays: 0,
|
|
}).sweepTerminalWorkspaces(100);
|
|
const [row] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(row).toMatchObject({
|
|
status: "archived",
|
|
metadata: {
|
|
workspaceProviderRemoved: true,
|
|
workspaceProviderRemoval: null,
|
|
},
|
|
});
|
|
expect(
|
|
f.calls.filter((call) => call.operation.kind === "remove"),
|
|
).toHaveLength(2);
|
|
});
|
|
it("keeps restoration intent durable when reopening loses the provider response", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({ status: "archived", closedAt: new Date() })
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
await removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40));
|
|
f.lose();
|
|
const request = {
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board" as const,
|
|
source: "local_implicit" as const,
|
|
isInstanceAdmin: true,
|
|
},
|
|
};
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
});
|
|
expect(
|
|
await svc.reopenClosedIsolatedExecutionWorkspaceForIssue(request),
|
|
).toMatchObject({ ok: false, code: "rebuild_failed" });
|
|
const [pending] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(pending.metadata).toMatchObject({
|
|
workspaceProviderRemoved: false,
|
|
workspaceProviderRestoration: { generation: 1 },
|
|
});
|
|
await expect(
|
|
projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoRef: "new" },
|
|
),
|
|
).rejects.toThrow("storage owner");
|
|
expect(
|
|
await svc.reopenClosedIsolatedExecutionWorkspaceForIssue(request),
|
|
).toMatchObject({ ok: true, generation: 1 });
|
|
});
|
|
it("reconciles a remotely completed restoration before reaping and never reuses an older removal receipt", async () => {
|
|
const f = await fixture();
|
|
const call = f.workers.call.bind(f.workers),
|
|
cache = new Map<string, unknown>();
|
|
let loseRestoreReply = false;
|
|
f.workers.call = async (...args) => {
|
|
const params = args[2] as any;
|
|
if (params.operation.kind === "inspect") return call(...args);
|
|
if (cache.has(params.operationId))
|
|
return cache.get(params.operationId) as any;
|
|
const result = await call(...args);
|
|
cache.set(params.operationId, result);
|
|
if (
|
|
loseRestoreReply &&
|
|
params.operation.kind === "prepare" &&
|
|
params.operation.existing
|
|
) {
|
|
loseRestoreReply = false;
|
|
throw new Error("lost completed restoration reply");
|
|
}
|
|
return result;
|
|
};
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done", completedAt: new Date() })
|
|
.where(eq(issues.id, f.issueId));
|
|
const svc = executionWorkspaceService(db, {
|
|
pluginWorkerManager: f.workers,
|
|
workspaceReaperCooldownDays: 0,
|
|
});
|
|
await svc.archiveWorkspaceUnderLifecycleLock({
|
|
id: workspace!.id,
|
|
patch: {},
|
|
closedAt: new Date(),
|
|
});
|
|
const read = () =>
|
|
db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace!.id))
|
|
.then((rows) => rows[0]!);
|
|
await removePluginWorkspace(db, f.workers, await read(), "a".repeat(40));
|
|
const firstRemoval = f.calls.find(
|
|
(call) => call.operation.kind === "remove",
|
|
);
|
|
loseRestoreReply = true;
|
|
const restoreEvents: string[] = [];
|
|
const stopRestoreEvents = subscribeDatabaseWork(
|
|
db,
|
|
DELIVERY_QUEUES.terminalWorkspaces,
|
|
(event, hints) => {
|
|
if (hints?.has(`workspace:${workspace!.id}`))
|
|
restoreEvents.push(event);
|
|
},
|
|
);
|
|
try {
|
|
expect(
|
|
await svc.reopenClosedIsolatedExecutionWorkspaceForIssue({
|
|
workspaceId: workspace!.id,
|
|
issue: {
|
|
id: f.issueId,
|
|
companyId: f.companyId,
|
|
projectId: f.input.projectId,
|
|
},
|
|
actor: { actorType: "user", agentId: null },
|
|
authorizationActor: {
|
|
type: "board",
|
|
source: "local_implicit",
|
|
isInstanceAdmin: true,
|
|
},
|
|
}),
|
|
).toMatchObject({ ok: false, code: "rebuild_failed" });
|
|
} finally {
|
|
stopRestoreEvents();
|
|
}
|
|
expect(restoreEvents).toEqual(["intent", "settled"]);
|
|
const pending = await read();
|
|
expect(pending.metadata?.workspaceProviderRestoration).toMatchObject({
|
|
generation: firstRemoval.generation + 1,
|
|
});
|
|
await expect(
|
|
removePluginWorkspace(db, f.workers, pending, "a".repeat(40)),
|
|
).rejects.toThrow("restoration");
|
|
const settled: Array<ReturnType<typeof read>> = [];
|
|
const events: string[] = [];
|
|
const unsubscribe = subscribeDatabaseWork(
|
|
db,
|
|
DELIVERY_QUEUES.terminalWorkspaces,
|
|
(event, hints) => {
|
|
if (!hints?.has(`workspace:${workspace!.id}`)) return;
|
|
events.push(event);
|
|
if (event === "settled") settled.push(read());
|
|
},
|
|
);
|
|
try {
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
} finally {
|
|
unsubscribe();
|
|
}
|
|
expect(events).toEqual(["intent", "settled"]);
|
|
expect(await settled[0]).toMatchObject({
|
|
metadata: {
|
|
workspaceProviderRestoration: null,
|
|
lifecycleGeneration: firstRemoval.generation + 1,
|
|
workspaceProviderRemoved: false,
|
|
},
|
|
});
|
|
const recovered = await read();
|
|
expect(recovered.metadata).toMatchObject({
|
|
workspaceProviderRemoved: false,
|
|
workspaceProviderRestoration: null,
|
|
lifecycleGeneration: firstRemoval.generation + 1,
|
|
});
|
|
await expect(
|
|
projectService(db).updateWorkspace(
|
|
f.input.projectId,
|
|
f.input.projectWorkspaceId,
|
|
{ repoRef: "after-recovery" },
|
|
),
|
|
).rejects.toThrow("storage owner");
|
|
await svc.sweepTerminalWorkspaces(100);
|
|
expect((await read()).metadata?.workspaceProviderRemoved).toBe(true);
|
|
const removals = f.calls.filter(
|
|
(call) => call.operation.kind === "remove",
|
|
);
|
|
expect(removals).toHaveLength(2);
|
|
expect(removals[1].generation).toBe(firstRemoval.generation + 1);
|
|
expect(removals[1].operationId).not.toBe(firstRemoval.operationId);
|
|
});
|
|
it("rejects unsupported provider modes before making a reservation", async () => {
|
|
const f = await fixture();
|
|
await expect(
|
|
preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
mode: "shared_workspace",
|
|
strategy: "project_primary",
|
|
}),
|
|
).rejects.toThrow("does not support");
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, f.issueId));
|
|
expect(task.executionWorkspaceId).toBeNull();
|
|
expect(f.calls).toHaveLength(0);
|
|
});
|
|
it("checks the saved mode when project defaults change on a reused workspace", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
const reused = await preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
existingWorkspaceId: workspace!.id,
|
|
mode: "shared_workspace",
|
|
strategy: "project_primary",
|
|
});
|
|
expect(reused!.id).toBe(workspace!.id);
|
|
expect(f.calls.at(-1).operation).toMatchObject({
|
|
mode: "isolated_workspace",
|
|
strategy: "git_worktree",
|
|
});
|
|
});
|
|
it("preserves a saved null branch when reuse defaults change", async () => {
|
|
const f = await fixture();
|
|
const [plugin] = await db
|
|
.select()
|
|
.from(plugins)
|
|
.where(eq(plugins.id, f.pluginId));
|
|
await db
|
|
.update(plugins)
|
|
.set({
|
|
manifestJson: {
|
|
...plugin.manifestJson,
|
|
workspaceProvider: {
|
|
modes: ["shared_workspace", "isolated_workspace"],
|
|
strategies: ["project_primary", "git_worktree"],
|
|
},
|
|
},
|
|
})
|
|
.where(eq(plugins.id, f.pluginId));
|
|
const workspace = await preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
mode: "shared_workspace",
|
|
strategy: "project_primary",
|
|
branchName: null,
|
|
});
|
|
expect(workspace!.branchName).toBeNull();
|
|
await preparePluginWorkspace(db, f.workers, {
|
|
...f.input,
|
|
existingWorkspaceId: workspace!.id,
|
|
branchName: "new-default",
|
|
});
|
|
expect(f.calls.at(-1).operation).toMatchObject({
|
|
mode: "shared_workspace",
|
|
strategy: "project_primary",
|
|
branchName: null,
|
|
});
|
|
});
|
|
it("does not share inspection results across opaque references owned by different providers", async () => {
|
|
const first = await fixture(),
|
|
second = await fixture(first.companyId);
|
|
for (const f of [first, second]) {
|
|
const workers = {
|
|
getWorker: f.workers.getWorker.bind(f.workers),
|
|
call: async (...args: any[]) => {
|
|
const result = await (f.workers.call as any)(...args);
|
|
return result.kind === "prepared"
|
|
? {
|
|
...result,
|
|
binding: { ...result.binding, workspaceRef: "opaque" },
|
|
}
|
|
: result;
|
|
},
|
|
} as unknown as WorkspaceWorkers;
|
|
await preparePluginWorkspace(db, workers, f.input);
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done", completedAt: new Date() })
|
|
.where(eq(issues.id, f.issueId));
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "cancelled" })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
}
|
|
second.dirty();
|
|
const workers = {
|
|
getWorker: (id: string) =>
|
|
([first.pluginId, second.pluginId] as string[]).includes(id)
|
|
? { supportedMethods: ["workspaceOperation"] }
|
|
: undefined,
|
|
call: (id: string, ...args: any[]) =>
|
|
(
|
|
(id === first.pluginId ? first.workers : second.workers).call as any
|
|
)(id, ...args),
|
|
} as unknown as WorkspaceWorkers;
|
|
await executionWorkspaceService(db, {
|
|
pluginWorkerManager: workers,
|
|
workspaceReaperCooldownDays: 0,
|
|
}).sweepTerminalWorkspaces(100);
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, second.issueId));
|
|
const [retained] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
|
|
expect(retained.status).toBe("active");
|
|
expect(
|
|
second.calls.some((call) => call.operation.kind === "inspect"),
|
|
).toBe(true);
|
|
});
|
|
it("never removes a checkout with local-only changes", async () => {
|
|
const f = await fixture();
|
|
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
|
|
f.dirty();
|
|
await expect(
|
|
removePluginWorkspace(db, f.workers, workspace!, "a".repeat(40)),
|
|
).rejects.toThrow("local changes");
|
|
expect(f.calls.some((call) => call.operation.kind === "remove")).toBe(
|
|
false,
|
|
);
|
|
});
|
|
},
|
|
);
|