mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a run ends, Paperclip records a `workspace_finalize` workspace operation. Dependent work reads that record to know if the workspace is safe to use. > - The record insert has foreign keys to `execution_workspaces`, `companies`, `heartbeat_runs` and `issues`. Postgres takes a key-share lock on each parent row, in trigger order: workspace, company, run, issue. > - The native finalizer and other writers lock the issue row and then the run row (the order that #20819 documents). When such a transaction holds the issue and then asks for the run, while the finalize insert holds the run and waits for the issue, Postgres detects a deadlock and aborts one side. > - In restart-chaos wave `w-20261111T205259-s6301` (task t01, run `e7129ec7`) the insert lost. The log says "failed to record workspace_finalize=failed operation; dependents may remain gated". The failed-finalize record was lost. > - The insert's lock order comes from the foreign keys, so the code cannot reorder it without a new explicit lock order that other writers (for example the remote-workspace release path, workspace then run) could invert again. A deadlock victim is safe to run again, because its transaction rolled back completely. > - This pull request retries the finalize record write on lock timeout, deadlock and serialization failure (55P03, 40P01, 40001), with the same bounded policy that #20850 added for the provider-pack admission write. > - The benefit is that a short lock conflict can no longer delete the record that releases dependent work. ## Linked Issues or Issue Description **What happened?** Run `e7129ec7` failed its native session recovery (`runner_harness_state_mismatch: backup_without_reusable_lease`). The failure path called `recordWorkspaceFinalize("failed")`. The insert into `workspace_operations` failed with `PostgresError: deadlock detected` (40P01). The server logged that dependents may remain gated. The run then settled as failed with `native_session_interrupted`; the next `finalizeNativeRun` call logged `native_finalization_missing` because that run had no proposed result. **Expected behavior** A transient deadlock does not lose the finalize record. The write runs again and the record is saved. **Steps to reproduce** 1. Transaction A: lock the issue row `FOR UPDATE`. 2. Start a `workspace_finalize` record for a run on that issue. Its insert takes key-share on the workspace, company and run rows, then waits for the issue. 3. Transaction A: lock the run row `FOR UPDATE`. 4. Postgres reports a deadlock. Before this change the insert is the victim and the record is lost. **Additional context** The new test reproduces this with two real transactions on embedded Postgres. It gives transaction A a longer `deadlock_timeout`, so the insert is always the victim, as on the chaos run. Refs #20850 Refs #20819 ## What Changed - New `server/src/services/transient-lock-contention.ts`: `postgresErrorCode`, `isTransientLockContention` and `retryTransientLockContention` (5 attempts, 100 ms × attempt backoff, only 55P03, 40P01 and 40001). - `workspaceOperationService.createRecorder().recordOperation` wraps its `workspace_finalize` state writes (the running-row insert and the terminal update) in that retry. Each attempt is a new transaction, or a new savepoint when the service runs inside a caller's transaction. - `native-provider-pack-admission.ts` uses the shared helper. Its behavior and its exported `nativeProviderPackAdmissionFailureCode` name do not change. - New test in `server/src/__tests__/workspace-operations-reconciliation.test.ts`: a real two-transaction deadlock between an issue-then-run locker and the finalize insert. ## Verification - Red on master: `cd server && npx vitest run src/__tests__/workspace-operations-reconciliation.test.ts -t "loses a deadlock"` failed with `deadlock detected`, code `40P01`, on the same `insert into "workspace_operations"` query as the chaos log. - Green: the same test passes; I ran it 3 times. The full file and the provider-pack admission tests pass (49 tests). `native-result-commit-work.test.ts` passes. - `npx tsc --noEmit -p server` passes. - I stopped `pnpm verify` after it selected 2533 test files; required CI supplies broad coverage. - Chaos: no new cell. The existing W2 × K1 cell (Codex, Daytona warm) reached this path when the restart fault broke native session recovery. I did not rerun a wave; both wave slots were busy. ## Risks - Low risk. Only the `workspace_finalize` state writes retry, and only on the three contention codes. Other errors throw as before. - A deadlock victim's transaction is fully rolled back, so the retried insert (same operation id) and the retried update are idempotent. - The worst added delay is about 1 s of backoff across 5 attempts, on the failure path only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
470 lines
19 KiB
TypeScript
470 lines
19 KiB
TypeScript
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
|
import { randomUUID } from "node:crypto";
|
|
import fs from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
agents,
|
|
companies,
|
|
createDb,
|
|
executionWorkspaces,
|
|
heartbeatRuns,
|
|
issues,
|
|
projects,
|
|
workspaceOperations,
|
|
} from "@paperclipai/db";
|
|
import { eq, sql } from "drizzle-orm";
|
|
import { subscribeDatabaseWork } from "@paperclipai/db/work-signals";
|
|
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
|
import { getWorkspaceOperationLogStore } from "../services/workspace-operation-log-store.js";
|
|
import {
|
|
RUNTIME_CONTROL_STALE_AFTER_MS,
|
|
resetWorkspaceRuntimeControlLocksForTests,
|
|
runExclusiveWorkspaceRuntimeControl,
|
|
workspaceOperationService,
|
|
} from "../services/workspace-operations.js";
|
|
import {
|
|
getEmbeddedPostgresTestSupport,
|
|
startEmbeddedPostgresTestDatabase,
|
|
} from "./helpers/embedded-postgres.js";
|
|
|
|
describe("workspace runtime control serialization", () => {
|
|
afterEach(() => {
|
|
resetWorkspaceRuntimeControlLocksForTests();
|
|
});
|
|
|
|
it("deterministically rejects an overlapping control operation and releases the claim", async () => {
|
|
let releaseFirst!: () => void;
|
|
const firstGate = new Promise<void>((resolve) => {
|
|
releaseFirst = resolve;
|
|
});
|
|
const first = runExclusiveWorkspaceRuntimeControl({
|
|
executionWorkspaceId: "workspace-1",
|
|
action: "start",
|
|
run: async () => {
|
|
await firstGate;
|
|
return "started";
|
|
},
|
|
});
|
|
|
|
await expect(runExclusiveWorkspaceRuntimeControl({
|
|
executionWorkspaceId: "workspace-1",
|
|
action: "repair",
|
|
run: async () => "repaired",
|
|
})).rejects.toMatchObject({
|
|
status: 409,
|
|
details: {
|
|
code: "workspace_runtime_control_in_progress",
|
|
activeAction: "start",
|
|
requestedAction: "repair",
|
|
},
|
|
});
|
|
|
|
releaseFirst();
|
|
await expect(first).resolves.toBe("started");
|
|
await expect(runExclusiveWorkspaceRuntimeControl({
|
|
executionWorkspaceId: "workspace-1",
|
|
action: "stop",
|
|
run: async () => "stopped",
|
|
})).resolves.toBe("stopped");
|
|
});
|
|
});
|
|
|
|
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_OPERATIONS_TEST_DATABASE_URL");
|
|
const embeddedPostgresSupport = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
|
|
|
if (!embeddedPostgresSupport.supported) {
|
|
console.warn(
|
|
`Skipping workspace-operation reconciliation tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
|
);
|
|
}
|
|
|
|
describeEmbeddedPostgres("workspace operation reconciliation", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
let logRoot = "";
|
|
let previousLogRoot: string | undefined;
|
|
|
|
beforeAll(async () => {
|
|
if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-workspace-operation-reconcile-");
|
|
db = createDb(externalUrl ?? tempDb!.connectionString);
|
|
logRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-operation-logs-"));
|
|
previousLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
|
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = logRoot;
|
|
}, 20_000);
|
|
|
|
afterEach(async () => {
|
|
await db.delete(workspaceOperations);
|
|
await db.delete(executionWorkspaces);
|
|
await db.delete(projects);
|
|
await db.delete(companies);
|
|
});
|
|
|
|
afterAll(async () => {
|
|
if (previousLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
|
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = previousLogRoot;
|
|
await fs.rm(logRoot, { recursive: true, force: true });
|
|
await db?.$client.end({ timeout: 1 });
|
|
await tempDb?.cleanup();
|
|
});
|
|
|
|
async function seedFinalizationWorkspace() {
|
|
const companyId = randomUUID(), projectId = randomUUID(), executionWorkspaceId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Delivery receipt wake",
|
|
issuePrefix: `W${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}` });
|
|
await db.insert(projects).values({ id: projectId, companyId, name: "Delivery", status: "in_progress" });
|
|
await db.insert(executionWorkspaces).values({ id: executionWorkspaceId, companyId, projectId,
|
|
mode: "isolated_workspace", strategyType: "git_worktree", name: "Finalization", status: "active" });
|
|
return { companyId, executionWorkspaceId };
|
|
}
|
|
|
|
it.each(["succeeded", "failed"] as const)("wakes cleanup after %s finalization commits, never at a nested savepoint", async status => {
|
|
const scope = await seedFinalizationWorkspace();
|
|
const settled: string[][] = [];
|
|
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.terminalWorkspaces, (event, hints) => {
|
|
if (event === "settled") settled.push([...(hints ?? [])]);
|
|
});
|
|
try {
|
|
await db.transaction(async tx => {
|
|
const recorder = workspaceOperationService(tx as unknown as typeof db).createRecorder(scope);
|
|
const operation = await recorder.recordOperation({ phase: "workspace_finalize", run: async () => ({ status }) });
|
|
expect(operation.status).toBe(status);
|
|
expect(settled).toEqual([]);
|
|
});
|
|
expect(settled).toEqual([[`workspace:${scope.executionWorkspaceId}`]]);
|
|
const rows = await db.select().from(workspaceOperations);
|
|
expect(rows).toHaveLength(1);
|
|
expect(rows[0]?.status).toBe(status);
|
|
} finally { unsubscribe(); }
|
|
});
|
|
|
|
it("wakes the attached workspace when an already settled finalization gains its identity", async () => {
|
|
const scope = await seedFinalizationWorkspace();
|
|
const recorder = workspaceOperationService(db).createRecorder({ companyId: scope.companyId });
|
|
const operation = await recorder.recordOperation({ phase: "workspace_finalize", run: async () => ({}) });
|
|
const settled: string[][] = [];
|
|
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.terminalWorkspaces, (event, hints) => {
|
|
if (event === "settled") settled.push([...(hints ?? [])]);
|
|
});
|
|
try {
|
|
await recorder.attachExecutionWorkspaceId(scope.executionWorkspaceId);
|
|
expect(settled).toEqual([[`workspace:${scope.executionWorkspaceId}`]]);
|
|
expect((await workspaceOperationService(db).getById(operation.id))?.executionWorkspaceId).toBe(scope.executionWorkspaceId);
|
|
} finally { unsubscribe(); }
|
|
});
|
|
|
|
it("keeps the finalization record when its write loses a deadlock to an issue-then-run locker", async () => {
|
|
const scope = await seedFinalizationWorkspace();
|
|
const agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
|
|
await db.insert(agents).values({ id: agentId, companyId: scope.companyId, name: "Finalizer", adapterType: "paperclip_runner" });
|
|
await db.insert(issues).values({ id: issueId, companyId: scope.companyId, title: "Finalize", status: "in_progress",
|
|
assigneeAgentId: agentId });
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId: scope.companyId, agentId, status: "running",
|
|
contextSnapshot: { issueId } });
|
|
const locker = createDb(externalUrl ?? tempDb!.connectionString);
|
|
let lockedIssue!: () => void, releaseLocker!: () => void;
|
|
const issueHeld = new Promise<void>(resolve => { lockedIssue = resolve; });
|
|
const lockerGate = new Promise<void>(resolve => { releaseLocker = resolve; });
|
|
const lockerDone = locker.transaction(async tx => {
|
|
await tx.execute(sql`set local deadlock_timeout = '20s'`);
|
|
await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, issueId)).for("update");
|
|
lockedIssue();
|
|
await lockerGate;
|
|
await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
|
|
});
|
|
try {
|
|
await Promise.race([issueHeld, lockerDone.then(() => { throw new Error("locker ended before it held the issue"); })]);
|
|
const recording = workspaceOperationService(db)
|
|
.createRecorder({ ...scope, heartbeatRunId: runId, issueId })
|
|
.recordOperation({ phase: "workspace_finalize", run: async () => ({ status: "failed" }) })
|
|
.then(value => ({ value }), error => ({ error }));
|
|
await vi.waitFor(async () => {
|
|
const [waiting] = await locker.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity
|
|
where wait_event_type = 'Lock' and query like 'insert into "workspace_operations"%'`);
|
|
expect(waiting?.count).toBe(1);
|
|
}, { timeout: 10_000, interval: 50 });
|
|
releaseLocker();
|
|
await lockerDone;
|
|
const outcome = await recording;
|
|
expect(outcome).not.toHaveProperty("error");
|
|
expect((await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, runId)))
|
|
.map(row => [row.phase, row.status])).toEqual([["workspace_finalize", "failed"]]);
|
|
} finally {
|
|
releaseLocker();
|
|
await lockerDone.catch(() => undefined);
|
|
await locker.$client.end({ timeout: 1 });
|
|
await db.delete(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, runId));
|
|
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
await db.delete(issues).where(eq(issues.id, issueId));
|
|
await db.delete(agents).where(eq(agents.id, agentId));
|
|
}
|
|
}, 30_000);
|
|
|
|
it("does not publish a finalization receipt rolled back by the outer transaction", async () => {
|
|
const scope = await seedFinalizationWorkspace();
|
|
await expect(db.transaction(async tx => {
|
|
await workspaceOperationService(tx as unknown as typeof db).createRecorder(scope)
|
|
.recordOperation({ phase: "workspace_finalize", run: async () => ({}) });
|
|
throw new Error("rollback receipt");
|
|
})).rejects.toThrow("rollback receipt");
|
|
expect(await db.select().from(workspaceOperations)).toEqual([]);
|
|
});
|
|
|
|
it("terminalizes an orphaned runtime start with an inspectable reconciliation log", async () => {
|
|
const companyId = randomUUID();
|
|
const projectId = randomUUID();
|
|
const executionWorkspaceId = randomUUID();
|
|
const staleOperationId = randomUUID();
|
|
const unrelatedOperationId = randomUUID();
|
|
|
|
await db.insert(companies).values({
|
|
id: companyId,
|
|
name: "Workspace operation reconciliation",
|
|
issuePrefix: `W${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}`,
|
|
requireBoardApprovalForNewAgents: false,
|
|
});
|
|
await db.insert(projects).values({
|
|
id: projectId,
|
|
companyId,
|
|
name: "Runtime reconciliation",
|
|
status: "in_progress",
|
|
});
|
|
await db.insert(executionWorkspaces).values({
|
|
id: executionWorkspaceId,
|
|
companyId,
|
|
projectId,
|
|
mode: "isolated_workspace",
|
|
strategyType: "git_worktree",
|
|
name: "Reconciliation workspace",
|
|
status: "active",
|
|
cwd: "/tmp/reconciliation-workspace",
|
|
});
|
|
|
|
const logHandle = await getWorkspaceOperationLogStore().begin({
|
|
companyId,
|
|
operationId: staleOperationId,
|
|
});
|
|
await db.insert(workspaceOperations).values([
|
|
{
|
|
id: staleOperationId,
|
|
companyId,
|
|
executionWorkspaceId,
|
|
phase: "workspace_provision",
|
|
command: "workspace command start",
|
|
cwd: "/tmp/reconciliation-workspace",
|
|
status: "running",
|
|
logStore: logHandle.store,
|
|
logRef: logHandle.logRef,
|
|
metadata: { action: "start", executionWorkspaceId },
|
|
},
|
|
{
|
|
id: unrelatedOperationId,
|
|
companyId,
|
|
executionWorkspaceId,
|
|
phase: "workspace_provision",
|
|
command: "bash ./scripts/provision-worktree.sh",
|
|
cwd: "/tmp/reconciliation-workspace",
|
|
status: "running",
|
|
metadata: { created: true },
|
|
},
|
|
]);
|
|
|
|
const service = workspaceOperationService(db);
|
|
|
|
// Recovery is bounded and cannot steal a live operation (PAP-17249): a `running` control
|
|
// that has signalled recently is left alone even when it carries no ownership stamp, which
|
|
// is the shape a control recorded by a pre-stamp build has.
|
|
await expect(service.reconcileStaleRuntimeControlOperations(executionWorkspaceId)).resolves.toEqual({
|
|
reconciled: 0,
|
|
operationIds: [],
|
|
});
|
|
expect((await service.getById(staleOperationId))?.status).toBe("running");
|
|
|
|
// Once it has gone quiet for longer than the staleness window, it is terminalized.
|
|
await expect(
|
|
service.reconcileStaleRuntimeControlOperations(executionWorkspaceId, {
|
|
now: new Date(Date.now() + RUNTIME_CONTROL_STALE_AFTER_MS + 5_000),
|
|
}),
|
|
).resolves.toEqual({
|
|
reconciled: 1,
|
|
operationIds: [staleOperationId],
|
|
});
|
|
|
|
const stale = await service.getById(staleOperationId);
|
|
expect(stale).toMatchObject({
|
|
status: "failed",
|
|
metadata: {
|
|
action: "start",
|
|
reconciled: true,
|
|
reconciliationReason: "orphaned_runtime_control",
|
|
},
|
|
});
|
|
expect(stale?.finishedAt).toBeInstanceOf(Date);
|
|
expect(stale?.stderrExcerpt).toContain("no live owner has reported progress");
|
|
expect((await service.readLog(staleOperationId)).content).toContain(
|
|
"no live owner has reported progress",
|
|
);
|
|
|
|
const unrelated = await db
|
|
.select({ status: workspaceOperations.status })
|
|
.from(workspaceOperations)
|
|
.where(eq(workspaceOperations.id, unrelatedOperationId))
|
|
.then((rows) => rows[0]);
|
|
expect(unrelated?.status).toBe("running");
|
|
});
|
|
|
|
it("persists bounded repair phase progress on the operation row and log", async () => {
|
|
const companyId = randomUUID();
|
|
const projectId = randomUUID();
|
|
const executionWorkspaceId = randomUUID();
|
|
await db.insert(companies).values({
|
|
id: companyId,
|
|
name: "Workspace repair diagnostics",
|
|
issuePrefix: `R${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}`,
|
|
requireBoardApprovalForNewAgents: false,
|
|
});
|
|
await db.insert(projects).values({
|
|
id: projectId,
|
|
companyId,
|
|
name: "Repair diagnostics",
|
|
status: "in_progress",
|
|
});
|
|
await db.insert(executionWorkspaces).values({
|
|
id: executionWorkspaceId,
|
|
companyId,
|
|
projectId,
|
|
mode: "isolated_workspace",
|
|
strategyType: "git_worktree",
|
|
name: "Repair diagnostics workspace",
|
|
status: "active",
|
|
cwd: "/tmp/repair-diagnostics-workspace",
|
|
});
|
|
|
|
const operation = await workspaceOperationService(db)
|
|
.createRecorder({ companyId, executionWorkspaceId })
|
|
.recordOperation({
|
|
phase: "workspace_repair",
|
|
command: "workspace command repair",
|
|
cwd: "/tmp/repair-diagnostics-workspace",
|
|
metadata: { action: "repair" },
|
|
run: async (reportProgress) => {
|
|
await reportProgress({
|
|
metadata: {
|
|
repairPhase: "target_backup",
|
|
repairDiagnostics: [{
|
|
phase: "target_backup",
|
|
status: "succeeded",
|
|
at: "2026-08-18T00:00:00.000Z",
|
|
}],
|
|
databaseOnly: true,
|
|
worktreePreserved: true,
|
|
},
|
|
system: "Workspace repair target_backup: succeeded.\n",
|
|
});
|
|
return { status: "succeeded", metadata: { backupRetained: true } };
|
|
},
|
|
});
|
|
|
|
expect(operation).toMatchObject({
|
|
status: "succeeded",
|
|
metadata: {
|
|
action: "repair",
|
|
repairPhase: "target_backup",
|
|
databaseOnly: true,
|
|
worktreePreserved: true,
|
|
backupRetained: true,
|
|
},
|
|
});
|
|
expect((await workspaceOperationService(db).readLog(operation.id)).content).toContain(
|
|
"Workspace repair target_backup: succeeded.",
|
|
);
|
|
});
|
|
|
|
describe("when the operation log file cannot be written", () => {
|
|
// The log file is created, then every append hits a full disk (PAPERCLIP-BACKEND-4H).
|
|
function failLogAppends() {
|
|
const enospc = Object.assign(new Error("ENOSPC: no space left on device, write"), {
|
|
code: "ENOSPC",
|
|
errno: -28,
|
|
syscall: "write",
|
|
});
|
|
const appendFile = fs.appendFile;
|
|
const spy = vi.spyOn(fs, "appendFile").mockImplementation(async (file, ...rest) => {
|
|
if (typeof file === "string" && file.startsWith(logRoot)) throw enospc;
|
|
return appendFile(file, ...rest);
|
|
});
|
|
return { enospc, spy };
|
|
}
|
|
|
|
async function seedCompany() {
|
|
const companyId = randomUUID();
|
|
await db.insert(companies).values({
|
|
id: companyId,
|
|
name: "Workspace operation full disk",
|
|
issuePrefix: `F${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}`,
|
|
requireBoardApprovalForNewAgents: false,
|
|
});
|
|
return companyId;
|
|
}
|
|
|
|
async function operationRows(companyId: string) {
|
|
return db.select().from(workspaceOperations).where(eq(workspaceOperations.companyId, companyId));
|
|
}
|
|
|
|
it("terminalizes the operation and surfaces the write error", async () => {
|
|
const companyId = await seedCompany();
|
|
const { enospc, spy } = failLogAppends();
|
|
try {
|
|
await expect(
|
|
workspaceOperationService(db)
|
|
.createRecorder({ companyId })
|
|
.recordOperation({
|
|
phase: "worktree_prepare",
|
|
command: "git worktree add",
|
|
cwd: "/tmp/full-disk-workspace",
|
|
run: async () => ({ status: "succeeded", exitCode: 0, stdout: "Preparing worktree\n" }),
|
|
}),
|
|
).rejects.toBe(enospc);
|
|
} finally {
|
|
spy.mockRestore();
|
|
}
|
|
|
|
const rows = await operationRows(companyId);
|
|
expect(rows).toHaveLength(1);
|
|
expect(rows[0]).toMatchObject({ status: "failed", phase: "worktree_prepare" });
|
|
expect(rows[0]?.finishedAt).toBeInstanceOf(Date);
|
|
expect(rows[0]?.stderrExcerpt).toContain("ENOSPC: no space left on device, write");
|
|
});
|
|
|
|
it("keeps the operation's own error instead of the log write error", async () => {
|
|
const companyId = await seedCompany();
|
|
const { spy } = failLogAppends();
|
|
const gitError = new Error("fatal: could not create work tree dir");
|
|
try {
|
|
await expect(
|
|
workspaceOperationService(db)
|
|
.createRecorder({ companyId })
|
|
.recordOperation({
|
|
phase: "worktree_prepare",
|
|
command: "git worktree add",
|
|
cwd: "/tmp/full-disk-workspace",
|
|
run: async () => {
|
|
throw gitError;
|
|
},
|
|
}),
|
|
).rejects.toBe(gitError);
|
|
} finally {
|
|
spy.mockRestore();
|
|
}
|
|
|
|
const rows = await operationRows(companyId);
|
|
expect(rows).toHaveLength(1);
|
|
expect(rows[0]).toMatchObject({ status: "failed" });
|
|
expect(rows[0]?.finishedAt).toBeInstanceOf(Date);
|
|
expect(rows[0]?.stderrExcerpt).toContain("fatal: could not create work tree dir");
|
|
});
|
|
});
|
|
});
|