Files
PaperClipAI/server/src/services/isolated-remote-workspace-source-lock.test.ts
T
DottaandClaude Opus 5.5 c97c0dd248 fix(workspaces): give run preparation its own source lock scope (#19734)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run prepares its task workspace under per-source advisory locks
(`workspace-source-lock.ts`). The locks live in a transaction on a
dedicated connection. An `AsyncLocalStorage` scope lets nested calls
reuse that transaction.
> - Timers and detached promises keep the async context that created
them. A native recovery timer or dispatch can start a run from inside
another operation's source scope.
> - When that run prepares its workspace, it sees the other scope as
"held". If that scope already committed, its dedicated connection is
closed. The lock query then fails with `Failed query: select
pg_try_advisory_xact_lock(...)` and cause `CONNECTION_CLOSED`. If that
scope is still open, the run's lock is tied to the other transaction and
is released early.
> - In a restart chaos wave, a reattached follow-up run failed setup
this way 50 ms after "Recovering the existing native runner process".
The setup-failure log did not show the driver cause.
> - This pull request makes only a live scope count as held, and makes
run workspace preparation take its locks in an independent scope.
> - The benefit is that a recovered or dispatched run gets its own
source lock, and a lost-connection failure is diagnosable from the log.

## Linked Issues or Issue Description

**What happened?**
After a control-plane restart, a native run was reattached
(`reattach_remote_runner`). Its workspace preparation failed at once
with `Failed query: select
pg_try_advisory_xact_lock(hashtextextended($1, 0)) as acquired params:
isolated_repository_source:<company>:<issue>`. The run ended
`failed`/`setup_failed`. The stack showed the "already held scope"
branch of `withWorkspaceSourceLocks`, not a new acquisition. The same
failure occurred earlier in the same wave on a different issue without a
restart.

**Steps to reproduce**
1. Call `withIsolatedRepositorySourceLock` for issue A. In its action,
start a detached promise that later calls
`withIsolatedRepositorySourceLock` for issue B.
2. Let the outer action return so its transaction commits.
3. Let the detached call run. It fails with `Failed query ...
pg_try_advisory_xact_lock`, cause `CONNECTION_CLOSED`.

**Expected behavior**
Work that starts after a source scope ends, or independent work such as
a run's preparation, acquires its own locks on its own connection.

**Environment**
Restart chaos campaign on Daytona (OpenCode, workload C1, fault K1,
comment trigger), embedded Postgres. Reproduced locally on embedded
Postgres by the new tests.

Related: #14910 (deferred comment wake reopens a closed issue). This PR
does not change wake promotion.

## What Changed

- `server/src/services/workspace-source-lock.ts`: a scope has an
`active` flag. The flag is cleared when its transaction ends.
`withWorkspaceSourceLocks`, `retainManagedProjectSource`,
`retainCurrentWorkspaceSource`, and `assertCurrentWorkspaceSourceHeld`
use only a live scope.
- `server/src/services/workspace-source-lock.ts`: new
`withIndependentWorkspaceSourceLocks`. It acquires locks in a new scope
and ignores an inherited one.
- `server/src/services/heartbeat/workspace-preparation.ts`: run
workspace preparation uses `withIndependentWorkspaceSourceLocks`.
- `server/src/services/heartbeat/run-completion.ts`: the "heartbeat
execution setup failed" log records a bounded driver `causeCode` (for
example `CONNECTION_CLOSED` or a SQLSTATE). It does not log the cause
text.
- Tests: two embedded-Postgres tests in
`isolated-remote-workspace-source-lock.test.ts`, one unit test in
`run-completion.test.ts`.

## Verification

- `cd server && npx vitest run
src/services/isolated-remote-workspace-source-lock.test.ts` — 4 passed.
- Red (old scope rule): "gives work detached from a finished scope its
own source lock" failed with `Failed query: select
pg_try_advisory_xact_lock(...)`, cause `CONNECTION_CLOSED`. This is the
production error.
- Red: "keeps an independent preparation's lock after the scope that
started it commits" failed because the lock was released with the other
transaction.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts` — 95 passed.
- `cd server && npx vitest run
src/services/remote-workspace-online-source.test.ts
src/services/heartbeat/workspace-preparation.test.ts
src/services/isolated-repository-rehome.test.ts
src/services/isolated-task-directory-retention.test.ts
src/services/isolated-task-directory.test.ts
src/__tests__/execution-workspace-reopen.test.ts` — all passed.
- `cd server && npx tsc --noEmit` — clean.
- No live Daytona replay was run.

## Risks

- Low. A nested call inside a live scope behaves as before.
- A call that inherited an ended scope now opens its own dedicated
connection and lock transaction. Before, it failed.
- Run preparation never joins another operation's lock scope. No caller
relies on that today. If one did, it would now take a second,
independent lock and could see `workspace_source_busy` instead of
silently sharing.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, embedded Postgres tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 23:08:57 -06:00

125 lines
5.9 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { setTimeout as delay } from "node:timers/promises";
import { sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { createDb } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { IsolatedRepositorySourceBusy, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js";
import { withIndependentWorkspaceSourceLocks } from "./workspace-source-lock.js";
const support = await getEmbeddedPostgresTestSupport();
const describePostgres = support.supported ? describe : describe.skip;
if (!support.supported) console.warn(`Workspace source lock tests require Postgres: ${support.reason}`);
describePostgres("workspace source ownership during preparation", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("workspace-source-lock-");
db = createDb(database.connectionString, { maxConnections: 1 });
const rows = await db.execute(sql`select current_user as name`);
const role = String(rows[0].name);
await db.execute(sql`alter role ${sql.identifier(role)} set idle_in_transaction_session_timeout = '1s'`);
});
afterAll(async () => {
try {
await db?.$client.end({ timeout: 1 });
} finally {
await database?.cleanup();
}
});
it("keeps preparation exclusive beyond the idle timeout while the ordinary pool stays usable", async () => {
const companyId = randomUUID(), issueId = randomUUID();
let entered!: () => void, release!: () => void;
const started = new Promise<void>(resolve => { entered = resolve; });
const pending = new Promise<void>(resolve => { release = resolve; });
const preparation = withIsolatedRepositorySourceLock(db, companyId, issueId, async () => {
await db.execute(sql`select 1`);
entered();
await pending;
return withIsolatedRepositorySourceLock(db, companyId, issueId, async () => "prepared");
}).then(value => ({ value }), error => ({ error }));
try {
await started;
await delay(1400);
await expect(db.execute(sql`select 1 as usable`)).resolves.toMatchObject([{ usable: 1 }]);
await expect(withIsolatedRepositorySourceLock(db, companyId, issueId, async () => "overlap", true))
.rejects.toBeInstanceOf(IsolatedRepositorySourceBusy);
await expect(withIsolatedRepositorySourceLock(db, randomUUID(), issueId, async () => "independent", true))
.resolves.toBe("independent");
} finally {
release();
await preparation;
}
expect(await preparation).toEqual({ value: "prepared" });
await expect(withIsolatedRepositorySourceLock(db, companyId, issueId, async () => "next", true))
.resolves.toBe("next");
});
it("releases a failed preparation and preserves the original failure", async () => {
const companyId = randomUUID(), issueId = randomUUID();
const failure = new Error("source preparation failed");
await expect(withIsolatedRepositorySourceLock(db, companyId, issueId, async () => {
await delay(1400);
throw failure;
})).rejects.toBe(failure);
await expect(withIsolatedRepositorySourceLock(db, companyId, issueId, async () => "recovered", true))
.resolves.toBe("recovered");
});
// RC-53: a recovery timer armed inside one source scope later dispatched a
// reattached run. Its preparation joined the committed scope and failed with
// `Failed query: select pg_try_advisory_xact_lock(...)` on a closed connection.
it("gives work detached from a finished scope its own source lock", async () => {
const companyId = randomUUID(), outerIssueId = randomUUID(), laterIssueId = randomUUID();
let fire!: () => void, entered!: () => void, release!: () => void;
const fired = new Promise<void>(resolve => { fire = resolve; });
const started = new Promise<void>(resolve => { entered = resolve; });
const pending = new Promise<void>(resolve => { release = resolve; });
let detached!: Promise<unknown>;
await withIsolatedRepositorySourceLock(db, companyId, outerIssueId, async () => {
detached = fired.then(() => withIsolatedRepositorySourceLock(db, companyId, laterIssueId, async () => {
entered();
await pending;
return "prepared";
})).then(value => ({ value }), error => ({ error: String(error) }));
});
fire();
expect(await Promise.race([started.then(() => "entered"), detached])).toBe("entered");
try {
await expect(withIsolatedRepositorySourceLock(db, companyId, laterIssueId, async () => "overlap", true))
.rejects.toBeInstanceOf(IsolatedRepositorySourceBusy);
} finally {
release();
}
expect(await detached).toEqual({ value: "prepared" });
});
it("keeps an independent preparation's lock after the scope that started it commits", async () => {
const companyId = randomUUID(), outerIssueId = randomUUID(), runIssueId = randomUUID();
let entered!: () => void, release!: () => void;
const started = new Promise<void>(resolve => { entered = resolve; });
const pending = new Promise<void>(resolve => { release = resolve; });
let preparation!: Promise<unknown>;
await withIsolatedRepositorySourceLock(db, companyId, outerIssueId, async () => {
preparation = withIndependentWorkspaceSourceLocks(db, { companyId, issueIds: [runIssueId] }, async () => {
entered();
await pending;
return "prepared";
}).then(value => ({ value }), error => ({ error: String(error) }));
await started;
});
try {
await expect(withIsolatedRepositorySourceLock(db, companyId, runIssueId, async () => "overlap", true))
.rejects.toBeInstanceOf(IsolatedRepositorySourceBusy);
} finally {
release();
}
expect(await preparation).toEqual({ value: "prepared" });
});
});