mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run prepares its task workspace under per-source advisory locks (`workspace-source-lock.ts`). The locks live in a transaction on a dedicated connection. An `AsyncLocalStorage` scope lets nested calls reuse that transaction. > - Timers and detached promises keep the async context that created them. A native recovery timer or dispatch can start a run from inside another operation's source scope. > - When that run prepares its workspace, it sees the other scope as "held". If that scope already committed, its dedicated connection is closed. The lock query then fails with `Failed query: select pg_try_advisory_xact_lock(...)` and cause `CONNECTION_CLOSED`. If that scope is still open, the run's lock is tied to the other transaction and is released early. > - In a restart chaos wave, a reattached follow-up run failed setup this way 50 ms after "Recovering the existing native runner process". The setup-failure log did not show the driver cause. > - This pull request makes only a live scope count as held, and makes run workspace preparation take its locks in an independent scope. > - The benefit is that a recovered or dispatched run gets its own source lock, and a lost-connection failure is diagnosable from the log. ## Linked Issues or Issue Description **What happened?** After a control-plane restart, a native run was reattached (`reattach_remote_runner`). Its workspace preparation failed at once with `Failed query: select pg_try_advisory_xact_lock(hashtextextended($1, 0)) as acquired params: isolated_repository_source:<company>:<issue>`. The run ended `failed`/`setup_failed`. The stack showed the "already held scope" branch of `withWorkspaceSourceLocks`, not a new acquisition. The same failure occurred earlier in the same wave on a different issue without a restart. **Steps to reproduce** 1. Call `withIsolatedRepositorySourceLock` for issue A. In its action, start a detached promise that later calls `withIsolatedRepositorySourceLock` for issue B. 2. Let the outer action return so its transaction commits. 3. Let the detached call run. It fails with `Failed query ... pg_try_advisory_xact_lock`, cause `CONNECTION_CLOSED`. **Expected behavior** Work that starts after a source scope ends, or independent work such as a run's preparation, acquires its own locks on its own connection. **Environment** Restart chaos campaign on Daytona (OpenCode, workload C1, fault K1, comment trigger), embedded Postgres. Reproduced locally on embedded Postgres by the new tests. Related: #14910 (deferred comment wake reopens a closed issue). This PR does not change wake promotion. ## What Changed - `server/src/services/workspace-source-lock.ts`: a scope has an `active` flag. The flag is cleared when its transaction ends. `withWorkspaceSourceLocks`, `retainManagedProjectSource`, `retainCurrentWorkspaceSource`, and `assertCurrentWorkspaceSourceHeld` use only a live scope. - `server/src/services/workspace-source-lock.ts`: new `withIndependentWorkspaceSourceLocks`. It acquires locks in a new scope and ignores an inherited one. - `server/src/services/heartbeat/workspace-preparation.ts`: run workspace preparation uses `withIndependentWorkspaceSourceLocks`. - `server/src/services/heartbeat/run-completion.ts`: the "heartbeat execution setup failed" log records a bounded driver `causeCode` (for example `CONNECTION_CLOSED` or a SQLSTATE). It does not log the cause text. - Tests: two embedded-Postgres tests in `isolated-remote-workspace-source-lock.test.ts`, one unit test in `run-completion.test.ts`. ## Verification - `cd server && npx vitest run src/services/isolated-remote-workspace-source-lock.test.ts` — 4 passed. - Red (old scope rule): "gives work detached from a finished scope its own source lock" failed with `Failed query: select pg_try_advisory_xact_lock(...)`, cause `CONNECTION_CLOSED`. This is the production error. - Red: "keeps an independent preparation's lock after the scope that started it commits" failed because the lock was released with the other transaction. - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts` — 95 passed. - `cd server && npx vitest run src/services/remote-workspace-online-source.test.ts src/services/heartbeat/workspace-preparation.test.ts src/services/isolated-repository-rehome.test.ts src/services/isolated-task-directory-retention.test.ts src/services/isolated-task-directory.test.ts src/__tests__/execution-workspace-reopen.test.ts` — all passed. - `cd server && npx tsc --noEmit` — clean. - No live Daytona replay was run. ## Risks - Low. A nested call inside a live scope behaves as before. - A call that inherited an ended scope now opens its own dedicated connection and lock transaction. Before, it failed. - Run preparation never joins another operation's lock scope. No caller relies on that today. If one did, it would now take a second, independent lock and could see `workspace_source_busy` instead of silently sharing. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
125 lines
5.9 KiB
TypeScript
125 lines
5.9 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { setTimeout as delay } from "node:timers/promises";
|
|
import { sql } from "drizzle-orm";
|
|
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
|
import { createDb } from "@paperclipai/db";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
|
import { IsolatedRepositorySourceBusy, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js";
|
|
import { withIndependentWorkspaceSourceLocks } from "./workspace-source-lock.js";
|
|
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
const describePostgres = support.supported ? describe : describe.skip;
|
|
if (!support.supported) console.warn(`Workspace source lock tests require Postgres: ${support.reason}`);
|
|
|
|
describePostgres("workspace source ownership during preparation", () => {
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
|
|
beforeAll(async () => {
|
|
database = await startEmbeddedPostgresTestDatabase("workspace-source-lock-");
|
|
db = createDb(database.connectionString, { maxConnections: 1 });
|
|
const rows = await db.execute(sql`select current_user as name`);
|
|
const role = String(rows[0].name);
|
|
await db.execute(sql`alter role ${sql.identifier(role)} set idle_in_transaction_session_timeout = '1s'`);
|
|
});
|
|
|
|
afterAll(async () => {
|
|
try {
|
|
await db?.$client.end({ timeout: 1 });
|
|
} finally {
|
|
await database?.cleanup();
|
|
}
|
|
});
|
|
|
|
it("keeps preparation exclusive beyond the idle timeout while the ordinary pool stays usable", async () => {
|
|
const companyId = randomUUID(), issueId = randomUUID();
|
|
let entered!: () => void, release!: () => void;
|
|
const started = new Promise<void>(resolve => { entered = resolve; });
|
|
const pending = new Promise<void>(resolve => { release = resolve; });
|
|
const preparation = withIsolatedRepositorySourceLock(db, companyId, issueId, async () => {
|
|
await db.execute(sql`select 1`);
|
|
entered();
|
|
await pending;
|
|
return withIsolatedRepositorySourceLock(db, companyId, issueId, async () => "prepared");
|
|
}).then(value => ({ value }), error => ({ error }));
|
|
|
|
try {
|
|
await started;
|
|
await delay(1400);
|
|
await expect(db.execute(sql`select 1 as usable`)).resolves.toMatchObject([{ usable: 1 }]);
|
|
await expect(withIsolatedRepositorySourceLock(db, companyId, issueId, async () => "overlap", true))
|
|
.rejects.toBeInstanceOf(IsolatedRepositorySourceBusy);
|
|
await expect(withIsolatedRepositorySourceLock(db, randomUUID(), issueId, async () => "independent", true))
|
|
.resolves.toBe("independent");
|
|
} finally {
|
|
release();
|
|
await preparation;
|
|
}
|
|
expect(await preparation).toEqual({ value: "prepared" });
|
|
await expect(withIsolatedRepositorySourceLock(db, companyId, issueId, async () => "next", true))
|
|
.resolves.toBe("next");
|
|
});
|
|
|
|
it("releases a failed preparation and preserves the original failure", async () => {
|
|
const companyId = randomUUID(), issueId = randomUUID();
|
|
const failure = new Error("source preparation failed");
|
|
await expect(withIsolatedRepositorySourceLock(db, companyId, issueId, async () => {
|
|
await delay(1400);
|
|
throw failure;
|
|
})).rejects.toBe(failure);
|
|
await expect(withIsolatedRepositorySourceLock(db, companyId, issueId, async () => "recovered", true))
|
|
.resolves.toBe("recovered");
|
|
});
|
|
|
|
// RC-53: a recovery timer armed inside one source scope later dispatched a
|
|
// reattached run. Its preparation joined the committed scope and failed with
|
|
// `Failed query: select pg_try_advisory_xact_lock(...)` on a closed connection.
|
|
it("gives work detached from a finished scope its own source lock", async () => {
|
|
const companyId = randomUUID(), outerIssueId = randomUUID(), laterIssueId = randomUUID();
|
|
let fire!: () => void, entered!: () => void, release!: () => void;
|
|
const fired = new Promise<void>(resolve => { fire = resolve; });
|
|
const started = new Promise<void>(resolve => { entered = resolve; });
|
|
const pending = new Promise<void>(resolve => { release = resolve; });
|
|
let detached!: Promise<unknown>;
|
|
await withIsolatedRepositorySourceLock(db, companyId, outerIssueId, async () => {
|
|
detached = fired.then(() => withIsolatedRepositorySourceLock(db, companyId, laterIssueId, async () => {
|
|
entered();
|
|
await pending;
|
|
return "prepared";
|
|
})).then(value => ({ value }), error => ({ error: String(error) }));
|
|
});
|
|
fire();
|
|
expect(await Promise.race([started.then(() => "entered"), detached])).toBe("entered");
|
|
try {
|
|
await expect(withIsolatedRepositorySourceLock(db, companyId, laterIssueId, async () => "overlap", true))
|
|
.rejects.toBeInstanceOf(IsolatedRepositorySourceBusy);
|
|
} finally {
|
|
release();
|
|
}
|
|
expect(await detached).toEqual({ value: "prepared" });
|
|
});
|
|
|
|
it("keeps an independent preparation's lock after the scope that started it commits", async () => {
|
|
const companyId = randomUUID(), outerIssueId = randomUUID(), runIssueId = randomUUID();
|
|
let entered!: () => void, release!: () => void;
|
|
const started = new Promise<void>(resolve => { entered = resolve; });
|
|
const pending = new Promise<void>(resolve => { release = resolve; });
|
|
let preparation!: Promise<unknown>;
|
|
await withIsolatedRepositorySourceLock(db, companyId, outerIssueId, async () => {
|
|
preparation = withIndependentWorkspaceSourceLocks(db, { companyId, issueIds: [runIssueId] }, async () => {
|
|
entered();
|
|
await pending;
|
|
return "prepared";
|
|
}).then(value => ({ value }), error => ({ error: String(error) }));
|
|
await started;
|
|
});
|
|
try {
|
|
await expect(withIsolatedRepositorySourceLock(db, companyId, runIssueId, async () => "overlap", true))
|
|
.rejects.toBeInstanceOf(IsolatedRepositorySourceBusy);
|
|
} finally {
|
|
release();
|
|
}
|
|
expect(await preparation).toEqual({ value: "prepared" });
|
|
});
|
|
});
|