Files
PaperClipAI/server/src/__tests__/workspace-operations-reconciliation.test.ts
T
DottaandClaude Opus 5.5 87fe4d9793 fix(workspaces): retry a workspace_finalize record write that loses a deadlock (#20971)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run ends, Paperclip records a `workspace_finalize` workspace
operation. Dependent work reads that record to know if the workspace is
safe to use.
> - The record insert has foreign keys to `execution_workspaces`,
`companies`, `heartbeat_runs` and `issues`. Postgres takes a key-share
lock on each parent row, in trigger order: workspace, company, run,
issue.
> - The native finalizer and other writers lock the issue row and then
the run row (the order that #20819 documents). When such a transaction
holds the issue and then asks for the run, while the finalize insert
holds the run and waits for the issue, Postgres detects a deadlock and
aborts one side.
> - In restart-chaos wave `w-20261111T205259-s6301` (task t01, run
`e7129ec7`) the insert lost. The log says "failed to record
workspace_finalize=failed operation; dependents may remain gated". The
failed-finalize record was lost.
> - The insert's lock order comes from the foreign keys, so the code
cannot reorder it without a new explicit lock order that other writers
(for example the remote-workspace release path, workspace then run)
could invert again. A deadlock victim is safe to run again, because its
transaction rolled back completely.
> - This pull request retries the finalize record write on lock timeout,
deadlock and serialization failure (55P03, 40P01, 40001), with the same
bounded policy that #20850 added for the provider-pack admission write.
> - The benefit is that a short lock conflict can no longer delete the
record that releases dependent work.

## Linked Issues or Issue Description

**What happened?**
Run `e7129ec7` failed its native session recovery
(`runner_harness_state_mismatch: backup_without_reusable_lease`). The
failure path called `recordWorkspaceFinalize("failed")`. The insert into
`workspace_operations` failed with `PostgresError: deadlock detected`
(40P01). The server logged that dependents may remain gated. The run
then settled as failed with `native_session_interrupted`; the next
`finalizeNativeRun` call logged `native_finalization_missing` because
that run had no proposed result.

**Expected behavior**
A transient deadlock does not lose the finalize record. The write runs
again and the record is saved.

**Steps to reproduce**
1. Transaction A: lock the issue row `FOR UPDATE`.
2. Start a `workspace_finalize` record for a run on that issue. Its
insert takes key-share on the workspace, company and run rows, then
waits for the issue.
3. Transaction A: lock the run row `FOR UPDATE`.
4. Postgres reports a deadlock. Before this change the insert is the
victim and the record is lost.

**Additional context**
The new test reproduces this with two real transactions on embedded
Postgres. It gives transaction A a longer `deadlock_timeout`, so the
insert is always the victim, as on the chaos run.

Refs #20850
Refs #20819

## What Changed

- New `server/src/services/transient-lock-contention.ts`:
`postgresErrorCode`, `isTransientLockContention` and
`retryTransientLockContention` (5 attempts, 100 ms × attempt backoff,
only 55P03, 40P01 and 40001).
- `workspaceOperationService.createRecorder().recordOperation` wraps its
`workspace_finalize` state writes (the running-row insert and the
terminal update) in that retry. Each attempt is a new transaction, or a
new savepoint when the service runs inside a caller's transaction.
- `native-provider-pack-admission.ts` uses the shared helper. Its
behavior and its exported `nativeProviderPackAdmissionFailureCode` name
do not change.
- New test in
`server/src/__tests__/workspace-operations-reconciliation.test.ts`: a
real two-transaction deadlock between an issue-then-run locker and the
finalize insert.

## Verification

- Red on master: `cd server && npx vitest run
src/__tests__/workspace-operations-reconciliation.test.ts -t "loses a
deadlock"` failed with `deadlock detected`, code `40P01`, on the same
`insert into "workspace_operations"` query as the chaos log.
- Green: the same test passes; I ran it 3 times. The full file and the
provider-pack admission tests pass (49 tests).
`native-result-commit-work.test.ts` passes.
- `npx tsc --noEmit -p server` passes.
- I stopped `pnpm verify` after it selected 2533 test files; required CI
supplies broad coverage.
- Chaos: no new cell. The existing W2 × K1 cell (Codex, Daytona warm)
reached this path when the restart fault broke native session recovery.
I did not rerun a wave; both wave slots were busy.

## Risks

- Low risk. Only the `workspace_finalize` state writes retry, and only
on the three contention codes. Other errors throw as before.
- A deadlock victim's transaction is fully rolled back, so the retried
insert (same operation id) and the retried update are idempotent.
- The worst added delay is about 1 s of backoff across 5 attempts, on
the failure path only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:09:57 -06:00

470 lines
19 KiB
TypeScript

import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { randomUUID } from "node:crypto";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
agents,
companies,
createDb,
executionWorkspaces,
heartbeatRuns,
issues,
projects,
workspaceOperations,
} from "@paperclipai/db";
import { eq, sql } from "drizzle-orm";
import { subscribeDatabaseWork } from "@paperclipai/db/work-signals";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { getWorkspaceOperationLogStore } from "../services/workspace-operation-log-store.js";
import {
RUNTIME_CONTROL_STALE_AFTER_MS,
resetWorkspaceRuntimeControlLocksForTests,
runExclusiveWorkspaceRuntimeControl,
workspaceOperationService,
} from "../services/workspace-operations.js";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./helpers/embedded-postgres.js";
describe("workspace runtime control serialization", () => {
afterEach(() => {
resetWorkspaceRuntimeControlLocksForTests();
});
it("deterministically rejects an overlapping control operation and releases the claim", async () => {
let releaseFirst!: () => void;
const firstGate = new Promise<void>((resolve) => {
releaseFirst = resolve;
});
const first = runExclusiveWorkspaceRuntimeControl({
executionWorkspaceId: "workspace-1",
action: "start",
run: async () => {
await firstGate;
return "started";
},
});
await expect(runExclusiveWorkspaceRuntimeControl({
executionWorkspaceId: "workspace-1",
action: "repair",
run: async () => "repaired",
})).rejects.toMatchObject({
status: 409,
details: {
code: "workspace_runtime_control_in_progress",
activeAction: "start",
requestedAction: "repair",
},
});
releaseFirst();
await expect(first).resolves.toBe("started");
await expect(runExclusiveWorkspaceRuntimeControl({
executionWorkspaceId: "workspace-1",
action: "stop",
run: async () => "stopped",
})).resolves.toBe("stopped");
});
});
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_OPERATIONS_TEST_DATABASE_URL");
const embeddedPostgresSupport = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
if (!embeddedPostgresSupport.supported) {
console.warn(
`Skipping workspace-operation reconciliation tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
);
}
describeEmbeddedPostgres("workspace operation reconciliation", () => {
let db!: ReturnType<typeof createDb>;
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
let logRoot = "";
let previousLogRoot: string | undefined;
beforeAll(async () => {
if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-workspace-operation-reconcile-");
db = createDb(externalUrl ?? tempDb!.connectionString);
logRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-operation-logs-"));
previousLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = logRoot;
}, 20_000);
afterEach(async () => {
await db.delete(workspaceOperations);
await db.delete(executionWorkspaces);
await db.delete(projects);
await db.delete(companies);
});
afterAll(async () => {
if (previousLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = previousLogRoot;
await fs.rm(logRoot, { recursive: true, force: true });
await db?.$client.end({ timeout: 1 });
await tempDb?.cleanup();
});
async function seedFinalizationWorkspace() {
const companyId = randomUUID(), projectId = randomUUID(), executionWorkspaceId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Delivery receipt wake",
issuePrefix: `W${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}` });
await db.insert(projects).values({ id: projectId, companyId, name: "Delivery", status: "in_progress" });
await db.insert(executionWorkspaces).values({ id: executionWorkspaceId, companyId, projectId,
mode: "isolated_workspace", strategyType: "git_worktree", name: "Finalization", status: "active" });
return { companyId, executionWorkspaceId };
}
it.each(["succeeded", "failed"] as const)("wakes cleanup after %s finalization commits, never at a nested savepoint", async status => {
const scope = await seedFinalizationWorkspace();
const settled: string[][] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.terminalWorkspaces, (event, hints) => {
if (event === "settled") settled.push([...(hints ?? [])]);
});
try {
await db.transaction(async tx => {
const recorder = workspaceOperationService(tx as unknown as typeof db).createRecorder(scope);
const operation = await recorder.recordOperation({ phase: "workspace_finalize", run: async () => ({ status }) });
expect(operation.status).toBe(status);
expect(settled).toEqual([]);
});
expect(settled).toEqual([[`workspace:${scope.executionWorkspaceId}`]]);
const rows = await db.select().from(workspaceOperations);
expect(rows).toHaveLength(1);
expect(rows[0]?.status).toBe(status);
} finally { unsubscribe(); }
});
it("wakes the attached workspace when an already settled finalization gains its identity", async () => {
const scope = await seedFinalizationWorkspace();
const recorder = workspaceOperationService(db).createRecorder({ companyId: scope.companyId });
const operation = await recorder.recordOperation({ phase: "workspace_finalize", run: async () => ({}) });
const settled: string[][] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.terminalWorkspaces, (event, hints) => {
if (event === "settled") settled.push([...(hints ?? [])]);
});
try {
await recorder.attachExecutionWorkspaceId(scope.executionWorkspaceId);
expect(settled).toEqual([[`workspace:${scope.executionWorkspaceId}`]]);
expect((await workspaceOperationService(db).getById(operation.id))?.executionWorkspaceId).toBe(scope.executionWorkspaceId);
} finally { unsubscribe(); }
});
it("keeps the finalization record when its write loses a deadlock to an issue-then-run locker", async () => {
const scope = await seedFinalizationWorkspace();
const agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
await db.insert(agents).values({ id: agentId, companyId: scope.companyId, name: "Finalizer", adapterType: "paperclip_runner" });
await db.insert(issues).values({ id: issueId, companyId: scope.companyId, title: "Finalize", status: "in_progress",
assigneeAgentId: agentId });
await db.insert(heartbeatRuns).values({ id: runId, companyId: scope.companyId, agentId, status: "running",
contextSnapshot: { issueId } });
const locker = createDb(externalUrl ?? tempDb!.connectionString);
let lockedIssue!: () => void, releaseLocker!: () => void;
const issueHeld = new Promise<void>(resolve => { lockedIssue = resolve; });
const lockerGate = new Promise<void>(resolve => { releaseLocker = resolve; });
const lockerDone = locker.transaction(async tx => {
await tx.execute(sql`set local deadlock_timeout = '20s'`);
await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, issueId)).for("update");
lockedIssue();
await lockerGate;
await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
});
try {
await Promise.race([issueHeld, lockerDone.then(() => { throw new Error("locker ended before it held the issue"); })]);
const recording = workspaceOperationService(db)
.createRecorder({ ...scope, heartbeatRunId: runId, issueId })
.recordOperation({ phase: "workspace_finalize", run: async () => ({ status: "failed" }) })
.then(value => ({ value }), error => ({ error }));
await vi.waitFor(async () => {
const [waiting] = await locker.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity
where wait_event_type = 'Lock' and query like 'insert into "workspace_operations"%'`);
expect(waiting?.count).toBe(1);
}, { timeout: 10_000, interval: 50 });
releaseLocker();
await lockerDone;
const outcome = await recording;
expect(outcome).not.toHaveProperty("error");
expect((await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, runId)))
.map(row => [row.phase, row.status])).toEqual([["workspace_finalize", "failed"]]);
} finally {
releaseLocker();
await lockerDone.catch(() => undefined);
await locker.$client.end({ timeout: 1 });
await db.delete(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, runId));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
await db.delete(issues).where(eq(issues.id, issueId));
await db.delete(agents).where(eq(agents.id, agentId));
}
}, 30_000);
it("does not publish a finalization receipt rolled back by the outer transaction", async () => {
const scope = await seedFinalizationWorkspace();
await expect(db.transaction(async tx => {
await workspaceOperationService(tx as unknown as typeof db).createRecorder(scope)
.recordOperation({ phase: "workspace_finalize", run: async () => ({}) });
throw new Error("rollback receipt");
})).rejects.toThrow("rollback receipt");
expect(await db.select().from(workspaceOperations)).toEqual([]);
});
it("terminalizes an orphaned runtime start with an inspectable reconciliation log", async () => {
const companyId = randomUUID();
const projectId = randomUUID();
const executionWorkspaceId = randomUUID();
const staleOperationId = randomUUID();
const unrelatedOperationId = randomUUID();
await db.insert(companies).values({
id: companyId,
name: "Workspace operation reconciliation",
issuePrefix: `W${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}`,
requireBoardApprovalForNewAgents: false,
});
await db.insert(projects).values({
id: projectId,
companyId,
name: "Runtime reconciliation",
status: "in_progress",
});
await db.insert(executionWorkspaces).values({
id: executionWorkspaceId,
companyId,
projectId,
mode: "isolated_workspace",
strategyType: "git_worktree",
name: "Reconciliation workspace",
status: "active",
cwd: "/tmp/reconciliation-workspace",
});
const logHandle = await getWorkspaceOperationLogStore().begin({
companyId,
operationId: staleOperationId,
});
await db.insert(workspaceOperations).values([
{
id: staleOperationId,
companyId,
executionWorkspaceId,
phase: "workspace_provision",
command: "workspace command start",
cwd: "/tmp/reconciliation-workspace",
status: "running",
logStore: logHandle.store,
logRef: logHandle.logRef,
metadata: { action: "start", executionWorkspaceId },
},
{
id: unrelatedOperationId,
companyId,
executionWorkspaceId,
phase: "workspace_provision",
command: "bash ./scripts/provision-worktree.sh",
cwd: "/tmp/reconciliation-workspace",
status: "running",
metadata: { created: true },
},
]);
const service = workspaceOperationService(db);
// Recovery is bounded and cannot steal a live operation (PAP-17249): a `running` control
// that has signalled recently is left alone even when it carries no ownership stamp, which
// is the shape a control recorded by a pre-stamp build has.
await expect(service.reconcileStaleRuntimeControlOperations(executionWorkspaceId)).resolves.toEqual({
reconciled: 0,
operationIds: [],
});
expect((await service.getById(staleOperationId))?.status).toBe("running");
// Once it has gone quiet for longer than the staleness window, it is terminalized.
await expect(
service.reconcileStaleRuntimeControlOperations(executionWorkspaceId, {
now: new Date(Date.now() + RUNTIME_CONTROL_STALE_AFTER_MS + 5_000),
}),
).resolves.toEqual({
reconciled: 1,
operationIds: [staleOperationId],
});
const stale = await service.getById(staleOperationId);
expect(stale).toMatchObject({
status: "failed",
metadata: {
action: "start",
reconciled: true,
reconciliationReason: "orphaned_runtime_control",
},
});
expect(stale?.finishedAt).toBeInstanceOf(Date);
expect(stale?.stderrExcerpt).toContain("no live owner has reported progress");
expect((await service.readLog(staleOperationId)).content).toContain(
"no live owner has reported progress",
);
const unrelated = await db
.select({ status: workspaceOperations.status })
.from(workspaceOperations)
.where(eq(workspaceOperations.id, unrelatedOperationId))
.then((rows) => rows[0]);
expect(unrelated?.status).toBe("running");
});
it("persists bounded repair phase progress on the operation row and log", async () => {
const companyId = randomUUID();
const projectId = randomUUID();
const executionWorkspaceId = randomUUID();
await db.insert(companies).values({
id: companyId,
name: "Workspace repair diagnostics",
issuePrefix: `R${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}`,
requireBoardApprovalForNewAgents: false,
});
await db.insert(projects).values({
id: projectId,
companyId,
name: "Repair diagnostics",
status: "in_progress",
});
await db.insert(executionWorkspaces).values({
id: executionWorkspaceId,
companyId,
projectId,
mode: "isolated_workspace",
strategyType: "git_worktree",
name: "Repair diagnostics workspace",
status: "active",
cwd: "/tmp/repair-diagnostics-workspace",
});
const operation = await workspaceOperationService(db)
.createRecorder({ companyId, executionWorkspaceId })
.recordOperation({
phase: "workspace_repair",
command: "workspace command repair",
cwd: "/tmp/repair-diagnostics-workspace",
metadata: { action: "repair" },
run: async (reportProgress) => {
await reportProgress({
metadata: {
repairPhase: "target_backup",
repairDiagnostics: [{
phase: "target_backup",
status: "succeeded",
at: "2026-08-18T00:00:00.000Z",
}],
databaseOnly: true,
worktreePreserved: true,
},
system: "Workspace repair target_backup: succeeded.\n",
});
return { status: "succeeded", metadata: { backupRetained: true } };
},
});
expect(operation).toMatchObject({
status: "succeeded",
metadata: {
action: "repair",
repairPhase: "target_backup",
databaseOnly: true,
worktreePreserved: true,
backupRetained: true,
},
});
expect((await workspaceOperationService(db).readLog(operation.id)).content).toContain(
"Workspace repair target_backup: succeeded.",
);
});
describe("when the operation log file cannot be written", () => {
// The log file is created, then every append hits a full disk (PAPERCLIP-BACKEND-4H).
function failLogAppends() {
const enospc = Object.assign(new Error("ENOSPC: no space left on device, write"), {
code: "ENOSPC",
errno: -28,
syscall: "write",
});
const appendFile = fs.appendFile;
const spy = vi.spyOn(fs, "appendFile").mockImplementation(async (file, ...rest) => {
if (typeof file === "string" && file.startsWith(logRoot)) throw enospc;
return appendFile(file, ...rest);
});
return { enospc, spy };
}
async function seedCompany() {
const companyId = randomUUID();
await db.insert(companies).values({
id: companyId,
name: "Workspace operation full disk",
issuePrefix: `F${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}`,
requireBoardApprovalForNewAgents: false,
});
return companyId;
}
async function operationRows(companyId: string) {
return db.select().from(workspaceOperations).where(eq(workspaceOperations.companyId, companyId));
}
it("terminalizes the operation and surfaces the write error", async () => {
const companyId = await seedCompany();
const { enospc, spy } = failLogAppends();
try {
await expect(
workspaceOperationService(db)
.createRecorder({ companyId })
.recordOperation({
phase: "worktree_prepare",
command: "git worktree add",
cwd: "/tmp/full-disk-workspace",
run: async () => ({ status: "succeeded", exitCode: 0, stdout: "Preparing worktree\n" }),
}),
).rejects.toBe(enospc);
} finally {
spy.mockRestore();
}
const rows = await operationRows(companyId);
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({ status: "failed", phase: "worktree_prepare" });
expect(rows[0]?.finishedAt).toBeInstanceOf(Date);
expect(rows[0]?.stderrExcerpt).toContain("ENOSPC: no space left on device, write");
});
it("keeps the operation's own error instead of the log write error", async () => {
const companyId = await seedCompany();
const { spy } = failLogAppends();
const gitError = new Error("fatal: could not create work tree dir");
try {
await expect(
workspaceOperationService(db)
.createRecorder({ companyId })
.recordOperation({
phase: "worktree_prepare",
command: "git worktree add",
cwd: "/tmp/full-disk-workspace",
run: async () => {
throw gitError;
},
}),
).rejects.toBe(gitError);
} finally {
spy.mockRestore();
}
const rows = await operationRows(companyId);
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({ status: "failed" });
expect(rows[0]?.finishedAt).toBeInstanceOf(Date);
expect(rows[0]?.stderrExcerpt).toContain("fatal: could not create work tree dir");
});
});
});