Files
PaperClipAI/server/src/__tests__/plugin-fresh-workspaces.integration.test.ts
T
c2e7df2dcc fix(workspaces): use project storage for a fresh retry after unowned task storage (#20756)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A project can use a storage provider plugin. That plugin prepares
each task's git worktree
> - A board Retry after a failed run can start a fresh attempt. The
fresh attempt starts from the project source
> - `preparePluginWorkspace` returns `null` when the task is still bound
to a workspace that no storage provider owns. It does this also for a
fresh attempt
> - On a plugin task environment, the host then reserves an empty
adapter-managed "Fresh task workspace" with no repository. The runner
fails in it. The next Retry finds that empty workspace bound to the
task, returns `null` again and reserves another empty workspace. The
task can never start again
> - This pull request lets a fresh attempt use the project storage
provider even when the previous workspace is not provider-owned
> - The benefit is that one Retry gives the task a normal provider
worktree again

## Linked Issues or Issue Description

**What happened?**
Tasks moved off an older execution environment kept a binding to a
workspace that no storage provider owned. A board Retry selected
`fresh_isolated_workspace`. `preparePluginWorkspace` returned `null`
because the bound workspace was not plugin-owned. The host fell back to
`reserveFreshPluginTaskWorkspace`, which made an empty `adapter_managed`
workspace (strategy `project_primary`, no repository, no recovery
snapshots). The run failed in about one second with "Plugin-provided
Runner execution operation unavailable; reconcile the same task before
retrying". Each later Retry made another empty workspace, chained to the
last one.

**Steps to reproduce**
1. Use a project with a storage provider plugin that supports fresh
workspaces.
2. Bind a task to a workspace that the provider does not own (for
example an adapter-managed workspace).
3. Call `preparePluginWorkspace` for a fresh attempt with that workspace
as the previous workspace.

**Expected behavior**
The storage provider prepares a new worktree for the fresh attempt.

**Actual behavior**
`preparePluginWorkspace` returns `null` and never calls the provider.

## What Changed

- `server/src/services/plugin-workspaces.ts`: the early `return null`
for a previous workspace that no provider owns no longer applies to a
fresh attempt. The fresh attempt then uses the project's storage
provider, as for a task with no bound workspace.
- `server/src/__tests__/plugin-fresh-workspaces.integration.test.ts`:
new test "prepares provider storage for a fresh retry that follows a
workspace the provider does not own".

## Verification

- The first commit adds only the test. CI on that commit (`d09ad235`,
[failed
job](https://github.com/paperclipai/paperclip/actions/runs/42976154239/job/130393285731))
shows only this test fail on the current code (`preparePluginWorkspace`
returns `null`).
- The second commit adds the fix. CI shows the test pass.
- The other tests in the file keep the fresh-attempt guarantees: an
older provider still fails closed, a changed binding revision still
stops the call, and a lost reply still reconciles to the same operation.
- I did not run the suite locally. The test needs embedded Postgres, and
this environment does not install packages. CI runs it.

## Risks

- Low to medium risk. A fresh attempt in a project with a storage
provider now uses that provider even when the previous workspace was
local or adapter-managed. Before, such an attempt used the fallback
path. This matches the project's storage policy, and a fresh attempt
does not reuse the previous workspace's files.
- Projects without a storage provider are not affected: the function
still returns `null` for them.
- Non-fresh preparation is not affected.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
(shell, file edits). Standard reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: lucktastic <13757243+lucktastic@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:10:25 -08:00

143 lines
10 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { eq, sql } from "drizzle-orm";
import { createDb, executionWorkspaces, heartbeatRuns, issues, plugins } from "@paperclipai/db";
import { preparePluginWorkspace, removePluginWorkspace } from "../services/plugin-workspaces.js";
import { executionWorkspaceService } from "../services/execution-workspaces.js";
import { reserveFreshPluginTaskWorkspace } from "../services/provider-fresh-workspace.js";
import { createPluginWorkspaceFixture } from "./helpers/plugin-workspaces.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
const cache = vi.hoisted(() => ({ release: vi.fn() }));
vi.mock("../services/provider-fresh-workspace.js", async importOriginal => ({
...await importOriginal<typeof import("../services/provider-fresh-workspace.js")>(),
releaseProviderRecoveryTransfers: cache.release,
}));
const support = await getEmbeddedPostgresTestSupport();
describe.skipIf(!support.supported)("provider-owned fresh retry reservations", () => {
let db: ReturnType<typeof createDb>, stopDb: (() => Promise<void>) | undefined;
beforeAll(async () => {
const started = await startEmbeddedPostgresTestDatabase("plugin-fresh-workspaces");
db = createDb(started.connectionString); stopDb = started.cleanup;
});
afterAll(async () => { await db?.$client.end(); await stopDb?.(); });
beforeEach(() => { cache.release.mockReset().mockResolvedValue(undefined); });
async function fixture() {
const f = await createPluginWorkspaceFixture(db);
const previous = await preparePluginWorkspace(db, f.workers, f.input);
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, f.runId));
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.input.agentId,
issueId: f.issueId, status: "running" }).returning();
const fresh = { attemptId: randomUUID(), sourceRunId: f.runId, previousWorkspaceId: previous!.id, recoverySnapshots: [] };
const input = { ...f.input, runId: run.id, existingWorkspaceId: previous!.id, fresh, expectedBindingRevision: 1 };
return { ...f, previous: previous!, fresh, input };
}
const enable = async (id: string) => db.update(plugins).set({ manifestJson:
sql`jsonb_set(${plugins.manifestJson}, '{workspaceProvider,freshWorkspaceVersion}', '1')` }).where(eq(plugins.id, id));
it("fails closed for an older provider without replacing the old task binding", async () => {
const f = await fixture();
await expect(preparePluginWorkspace(db, f.workers, f.input)).rejects.toThrow("fresh attempt is unavailable");
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
expect(task.executionWorkspaceId).toBe(f.previous.id);
expect(f.calls).toHaveLength(1);
});
it("reserves a new binding once and reconciles the same operation after a lost provider reply", async () => {
const f = await fixture(); await enable(f.pluginId);
f.lose();
await expect(preparePluginWorkspace(db, f.workers, f.input)).rejects.toThrow("incomplete");
const ambiguous = f.calls.at(-1);
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
expect(workspace!.id).not.toBe(f.previous.id);
expect(f.calls.at(-1).executionWorkspaceId).toBe(ambiguous.executionWorkspaceId);
expect(f.calls.at(-1).operationId).toBe(ambiguous.operationId);
expect(f.calls.at(-1).operation).toMatchObject({ fresh: f.fresh, branchMode: "create", branchName: null, existing: null });
const [retained] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.previous.id));
expect(retained.providerRef).toBe(f.previous.providerRef);
expect(retained.metadata).toEqual(f.previous.metadata);
const rows = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId));
expect(rows).toHaveLength(2);
});
it("keeps fresh directory authority server-owned through metadata updates", async () => {
const f = await fixture(); await enable(f.pluginId);
const service = executionWorkspaceService(db);
const original = await service.update(f.previous.id, { metadata: {
note: "ordinary metadata", workspaceProviderFresh: f.fresh, runnerTaskFreshWorkspace: f.fresh,
runnerTaskWorkspaceOwner: { pluginKey: "forged" },
} });
expect(original!.metadata).toMatchObject({ note: "ordinary metadata" });
for (const key of ["workspaceProviderFresh", "runnerTaskFreshWorkspace", "runnerTaskWorkspaceOwner"]) {
expect(original!.metadata?.[key]).toBeUndefined();
}
const prepared = await preparePluginWorkspace(db, f.workers, f.input);
const updated = await service.update(prepared!.id, { metadata: { note: "caller edit", workspaceProviderFresh: null } });
expect(updated!.metadata).toMatchObject({ note: "caller edit", workspaceProviderFresh: f.fresh });
await db.update(issues).set({ projectId: null, projectWorkspaceId: null }).where(eq(issues.id, f.issueId));
const owner = { agentId: f.input.agentId, environmentId: randomUUID(), pluginKey: "test.tasks", driverKey: "tasks" };
const next = { ...f.fresh, attemptId: randomUUID(), previousWorkspaceId: prepared!.id };
const id = await reserveFreshPluginTaskWorkspace({ db, companyId: f.companyId, issueId: f.issueId,
runId: f.input.runId, agentId: f.input.agentId, responsibleUserId: null, fresh: next, owner,
expectedBindingRevision: 2 });
const directory = await service.update(id, { metadata: { runnerTaskFreshWorkspace: f.fresh,
runnerTaskWorkspaceOwner: { pluginKey: "forged" } } });
expect(directory!.metadata).toMatchObject({ runnerTaskFreshWorkspace: next, runnerTaskWorkspaceOwner: owner });
const cleared = await service.update(id, { metadata: null });
expect(cleared!.metadata).toMatchObject({ runnerTaskFreshWorkspace: next, runnerTaskWorkspaceOwner: owner });
});
it("retains the durable removal intent if transfer cache cleanup fails", async () => {
const f = await fixture(); await enable(f.pluginId);
const workspace = await preparePluginWorkspace(db, f.workers, f.input);
const [closed] = await db.update(executionWorkspaces).set({ status: "archived" })
.where(eq(executionWorkspaces.id, workspace!.id)).returning();
cache.release.mockRejectedValueOnce(new Error("temporary cache cleanup failure"));
await expect(removePluginWorkspace(db, f.workers, closed, "a".repeat(40))).rejects.toThrow("cache cleanup failure");
const [pending] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
expect(pending.metadata?.workspaceProviderRemoval).toBeTruthy();
expect(pending.metadata?.workspaceProviderRemoved).not.toBe(true);
await removePluginWorkspace(db, f.workers, pending, "a".repeat(40));
const [settled] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
expect(settled.metadata?.workspaceProviderRemoved).toBe(true);
expect(cache.release).toHaveBeenCalledTimes(2);
});
it("prepares provider storage for a fresh retry that follows a workspace the provider does not own", async () => {
const f = await createPluginWorkspaceFixture(db); await enable(f.pluginId);
const [previous] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, projectId: f.input.projectId,
projectWorkspaceId: f.input.projectWorkspaceId, sourceIssueId: f.issueId, mode: "isolated_workspace",
strategyType: "project_primary", name: "Fresh task workspace", status: "active", providerType: "adapter_managed" }).returning();
await db.update(issues).set({ executionWorkspaceId: previous.id, workspaceBindingRevision: 1 }).where(eq(issues.id, f.issueId));
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, f.runId));
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.input.agentId,
issueId: f.issueId, status: "running" }).returning();
const fresh = { attemptId: randomUUID(), sourceRunId: f.runId, previousWorkspaceId: previous.id, recoverySnapshots: [] };
const workspace = await preparePluginWorkspace(db, f.workers, { ...f.input, runId: run.id,
existingWorkspaceId: previous.id, fresh, expectedBindingRevision: 1 });
expect(workspace).not.toBeNull();
expect(workspace!.id).not.toBe(previous.id);
expect(workspace!.providerType).toBe("plugin");
expect(f.calls).toHaveLength(1);
expect(f.calls[0].operation).toMatchObject({ kind: "prepare", fresh, branchName: null });
});
it("rejects a changed task revision before calling the provider", async () => {
const f = await fixture(); await enable(f.pluginId);
await db.update(issues).set({ workspaceBindingRevision: 2 }).where(eq(issues.id, f.issueId));
await expect(preparePluginWorkspace(db, f.workers, f.input)).rejects.toThrow("binding changed");
expect(f.calls).toHaveLength(1);
});
it("keeps one projectless attempt directory identity across reconciliation and later reuse", async () => {
const f = await fixture();
await db.update(issues).set({ projectId: null, projectWorkspaceId: null }).where(eq(issues.id, f.issueId));
const owner = { agentId: f.input.agentId, environmentId: randomUUID(), pluginKey: "test.tasks", driverKey: "tasks" };
const input = { db, companyId: f.companyId, issueId: f.issueId, runId: f.input.runId,
agentId: f.input.agentId, responsibleUserId: null, fresh: f.fresh, owner, expectedBindingRevision: 1 };
const id = await reserveFreshPluginTaskWorkspace(input);
expect(await reserveFreshPluginTaskWorkspace(input)).toBe(id);
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, id));
expect(workspace.metadata).toMatchObject({ runnerTaskFreshWorkspace: f.fresh, runnerTaskWorkspaceOwner: owner });
expect(workspace.cwd).toBe(`/workspaces/${f.fresh.attemptId}`);
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
expect(task.executionWorkspaceId).toBe(id);
expect(task.workspaceBindingRevision).toBe(2);
await db.update(heartbeatRuns).set({ responsibleUserId: "changed-user" }).where(eq(heartbeatRuns.id, f.input.runId));
await expect(reserveFreshPluginTaskWorkspace(input)).rejects.toThrow("responsible user changed");
});
});