perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every native event that announces work to another part of the server
(chat progress, a final response, an evidence outcome, recovery)
registers a wake in its transaction before it writes the event
> - Such an event takes six statements: BEGIN, the run lock read, a
replay probe, a read of the transaction id for the wake, the append, and
COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock,
the transaction id, the write, COMMIT
> - In the agent scaling load model these two paths are now the largest
per-agent cost after #21511 and #21520, and other events fall behind
first at 600 agents
> - This pull request returns the transaction id from the lock read that
each path already sends, and moves the replay probe into the run lock
read when it is safe
> - The benefit is four statements for such an event and four for a
checkpoint save, with the same locks, the same wake order and the same
replay rule

## Linked Issues or Issue Description

Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent
scaling plan and load model).

**What existing behavior does this improve?**
The statement count of native events that register a wake
(`appendNativeSourceEvent`) and of controller checkpoint saves
(`nativeRunnerCheckpointWriter`).

**Subsystem affected**
`packages/db/src/work-signals.ts` (one new export),
`server/src/services/heartbeat-run-events.ts`,
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Current behavior**
`signalDatabaseWork` sends `select pg_current_xact_id()` for the first
wake of each transaction. A native event with a wake reads the run lock,
then probes for its replay in a separate statement, because a statement
that waits for the lock keeps its earlier snapshot.

**Proposed behavior**
- The run lock read and the checkpoint owner lock read also return
`pg_current_xact_id()::text`. Both already lock a row, so the
transaction has an id at that point. `primeDatabaseWorkTransactionId`
gives it to the transaction's wake state, and the wake sends no
statement for it.
- For an event that may register a progress, response, reassessment or
retention wake, the run lock read also reads `next_event_seq` from its
own snapshot and checks for a replay in that snapshot. Every event
append advances `next_event_seq` in its transaction. So when the locked
row has the same `next_event_seq` as the snapshot, no append committed
in between, and the probe result is current. Otherwise, or when the
probe finds a replay, the separate probe runs as before.

**Reason and benefit**
Fewer round trips while a pool connection is held in a transaction. In
the load model these paths hold the main pool and the three-connection
checkpoint pool longest per event.

**Breaking changes**
None. No schema change.

## What Changed

- `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`,
exported from `@paperclipai/db`): inside a transaction whose id is not
yet known, it records the id. Outside a transaction it does nothing. An
uncertain commit still resolves with `pg_xact_status` on the same id.
- `appendNativeSourceEvent`: the lock read returns the transaction id
when the event has any wake. For a probing wake it is one statement: a
CTE that reads `next_event_seq` from the snapshot, a CTE that locks the
run row (`for no key update`, unchanged), and `exists (...)` for the
replay. The wake is still registered after the replay check and before
the append.
- `lockNativeRunnerCheckpointOwner` returns the transaction id with its
owner lock and primes the wake state. Its return value and every caller
are unchanged.

## Verification

- Red/green on embedded PostgreSQL; each new assertion failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "registers a native
event's wakes before its write and only when the event is new": four
statements (before: six), the wake's intent and settlement are
published, and a replay publishes no wake.
- New: "finds a replay that another writer committed while the lock read
waited, and registers no wake for it". Another connection holds the run
row (the test waits until PostgreSQL reports the append waiting for that
lock), stores the same event and moves `next_event_seq`; the waiting
append returns `duplicate`, runs the separate probe, and publishes no
wake. With the `next_event_seq` comparison removed, this test fails.
- New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the
owner lock with the transaction id, the write, COMMIT (before: five
statements), and its recovery wake is published; a save after the lease
expired writes nothing and fails with
`native_runner_checkpoint_owner_lost`.
- `work-signals.test.ts`: a primed transaction sends no id statement for
its wakes; a primed transaction whose connection is killed after its
write stays pending and then resolves through `reconcileDatabaseWork`,
with no row stored.
- Added after merging `origin/master` (with #21511, #21520 and #21436)
into this branch:
- "gives the wake the transaction id of the transaction that took the
run lock": inside a caller's transaction, an event with a progress wake
sends no separate `pg_current_xact_id()` statement; the transaction's
connection is then killed, and `reconcileDatabaseWork` checks
`pg_xact_status` with exactly the id that `pg_current_xact_id()` returns
on that same transaction. Priming a wrong id makes it fail.
- "primes nothing when the lock read fails, so the transaction is not
left uncertain": a lock read that fails on `lock_timeout` leaves no
pending work; after the same failure inside a savepoint, the next signal
on that transaction still sends its own `pg_current_xact_id()`. Priming
before the lock read makes it fail.
- Both call sites pass only the value returned by the statement they
just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)`
then `primeDatabaseWorkTransactionId(tx, run.xid)`) and
`lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then
`primeDatabaseWorkTransactionId(tx, xid)`).
- Retest on the combined tree, one file at a time: atomic-append 28
(including the equivalence test and both race tests), the port suite
with conformance 105, checkpoint-save 2, `live-recovery-work` 51,
`work-signals` 25, runner `native-session-runtime` 297; server typecheck
passes. No code change was needed for the combination.
- Other focused suites pass, one file at a time: the port suite with its
conformance suite (100), `chat-native-progress-work` (35),
`native-observer-replay` (1), `heartbeat-run-event-sequencing` (5),
`native-event-boundary-golden` (1), `native-recovered-owner-followup`
(97), `native-restart-recovery-isolation` (1),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51),
`work-signals` (25).
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). Stacked: this pull request merged locally with #21511, and the
model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model
applies the port's batching rules itself; that mode is not in any pull
request). Default pool of 10 connections; embedded PostgreSQL on the
same, shared machine; runs one after the other with a pause between
them. No run had a failed operation, a failed checkpoint save or a lost
lease. "Before" rows are `origin/master` from the same session.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 /
12,000 | 1 / 70 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 /
24,000 | 2 / 1,274 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 /
6,335 | 61,158 / 72,000 | 11,678 / 21,212 |
| 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 /
72,000 | 1,136 / 4,980 |
| 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 |
71,958 / 72,000 | 1,319 / 5,712 |
| 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 /
48,000 | 1,974 / 3,835 |

With the three changes, one process finishes 99.9% of the offered work
at 600 agents, but its event-loop utilization is 0.90–0.93, above the
0.75 target, and p99 lateness is about 5 s. It saturates between 600 and
800 agents.
- Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5
where commit-time notifications are on).

## Risks

- The transaction id is read in a statement that already assigns one (a
row lock), so no transaction gets an id it did not have before.
- The probe in the lock read is used only when `next_event_seq` did not
change between the snapshot and the lock. Every writer of
`heartbeat_run_events` moves `next_event_seq` in the same transaction
(native and ordinary appends, the run coordinator store, the retry
queue). A future writer that inserts events without it would make this
probe miss a concurrent replay; the append statement then still finds
the replay and stores nothing, but the wake would already be registered
(a spurious wake, not a lost or duplicate event).
- This pull request and #21511 both edit `heartbeat-run-events.ts`, in
different functions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 20:48:17 -06:00
1 parent dc018f9bc0
commit 1dee321844
7 files changed
+292 -34

No files matched your search

+1 -1
View File
@@ -54,5 +54,5 @@ export { issueRelations } from "./schema/issue_relations.js";
export { issueReferenceMentions } from "./schema/issue_reference_mentions.js";
export * from "./schema/index.js";
export { databaseWorkScopeKey, signalDatabaseWork, subscribeDatabaseWork, databaseWorkPending, reconcileDatabaseWork } from "./work-signals.js";
export { databaseWorkScopeKey, primeDatabaseWorkTransactionId, signalDatabaseWork, subscribeDatabaseWork, databaseWorkPending, reconcileDatabaseWork } from "./work-signals.js";
export { withDatabaseOperation } from "./response-capture.js";
+28 -1
View File
@@ -3,7 +3,7 @@ import { drizzle } from "drizzle-orm/postgres-js";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { createDb, withDedicatedDbConnection } from "./client.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js";
import { DATABASE_WORK_CHANNEL, databaseWorkScopeKey, databaseWorkPending, enableDatabaseWorkNotifications, installDatabaseWorkSignals, notifyDatabaseWorkOnCommit, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js";
import { DATABASE_WORK_CHANNEL, databaseWorkScopeKey, databaseWorkPending, enableDatabaseWorkNotifications, installDatabaseWorkSignals, notifyDatabaseWorkOnCommit, primeDatabaseWorkTransactionId, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js";
const support = await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("database work signals", () => {
@@ -145,6 +145,22 @@ const support = await getEmbeddedPostgresTestSupport();
} finally { release(); await serverDone?.catch(() => {}); unsubscribe(); }
});
it("resolves an uncertain transaction with a transaction id that a statement returned", async () => {
await expect(db.transaction(async tx => {
const [{ xid }] = await tx.execute(sql`select pg_current_xact_id()::text as xid`);
primeDatabaseWorkTransactionId(tx, xid as string);
await signalDatabaseWork(tx, "primed-disconnect");
await tx.execute(sql`insert into work_signal_test values ('primed-disconnect')`);
const [{ pid }] = await tx.execute(sql`select pg_backend_pid() as pid`);
await db.execute(sql`select pg_terminate_backend(${pid as number})`);
await tx.execute(sql`select 1`);
})).rejects.toThrow();
expect(databaseWorkPending(db, "primed-disconnect")).toBe(true);
await reconcileDatabaseWork(db, "primed-disconnect");
expect(databaseWorkPending(db, "primed-disconnect")).toBe(false);
expect(await db.execute(sql`select * from work_signal_test where id = 'primed-disconnect'`)).toHaveLength(0);
});
it("recovers after a real backend disconnect without replaying the queue write", async () => {
await expect(db.transaction(async tx => {
await signalDatabaseWork(tx, "disconnect");
@@ -196,6 +212,17 @@ describe("work signal query boundaries", () => {
expect(write).not.toHaveBeenCalled();
expect(databaseWorkPending(connection, "failed")).toBe(false);
});
it("reads no transaction id when a statement of the transaction already returned it", async () => {
const execute = vi.fn();
const connection = installDatabaseWorkSignals({ execute, transaction: async (callback: any) => callback({ execute, transaction: async () => {} }) });
primeDatabaseWorkTransactionId(connection, "outside");
await connection.transaction(async (tx: any) => {
primeDatabaseWorkTransactionId(tx, "42");
await signalDatabaseWork(tx, "primed");
await signalDatabaseWork(tx, "primed-again");
});
expect(execute).not.toHaveBeenCalled();
});
it("rejects registration on an autocommit root", async () => {
const connection = installDatabaseWorkSignals({ execute: vi.fn(), transaction: vi.fn() });
await expect(signalDatabaseWork(connection, "bad")).rejects.toThrow("inside a transaction");
+5
View File
@@ -88,6 +88,11 @@ export async function signalDatabaseWork(connection: object, topic: string, hint
if (context.scope.notify) notifyAtCommit(context, notification(context.scope, topic));
}
export function primeDatabaseWorkTransactionId(connection: object, xid: string): void {
const state = contexts.get(connection)?.transaction;
if (state && !state.xid) state.xid = Promise.resolve(xid);
}
function notifyAtCommit(context: Context, payload: string): void {
if (context.ended) throw new Error("Database work was signalled after its transaction or savepoint ended");
const key = `${DATABASE_WORK_CHANNEL}\u0000${payload}`;
@@ -1,9 +1,11 @@
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { randomBytes } from "node:crypto";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { eq, inArray, sql } from "drizzle-orm";
import { drizzle } from "drizzle-orm/postgres-js";
import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issues, nativeRunnerCheckpoints } from "@paperclipai/db";
import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issues, nativeRunnerCheckpoints, databaseWorkPending,
reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { appendHeartbeatRunEvent, appendHeartbeatRunEventBatch } from "../services/heartbeat-run-events.js";
import { nativeSha256 } from "../services/native-runtime/canonical.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
@@ -293,6 +295,7 @@ describeEmbeddedPostgres("atomic ordinary run events", () => {
locked();
await released;
});
void holder.catch(() => undefined);
await lockedPromise;
return { release: () => { release(); return holder; } };
};
@@ -485,17 +488,143 @@ describeEmbeddedPostgres("atomic ordinary run events", () => {
const runId = await makeRun();
const [issue] = await db.insert(issues).values({ companyId, title: "Progress task" }).returning();
await db.update(heartbeatRuns).set({ nativeIssueId: issue!.id, runtimeMode: "native" }).where(eq(heartbeatRuns.id, runId));
const first = await sentStatements(db.$client, () => appendHeartbeatRunEvent(db, nativeEvent(runId, 1, "item.completed")));
// BEGIN, lock, duplicate probe, the wake's transaction id, the append, COMMIT.
expect(first.statements).toHaveLength(6);
const wake = first.statements.findIndex(text => text.includes("pg_current_xact_id"));
expect(wake).toBeGreaterThan(0);
expect(wake).toBeLessThan(first.statements.findIndex(text => text.includes("insert into heartbeat_run_events")));
expect(first.result).toMatchObject({ disposition: "committed", highestContiguousSourceSeq: 1 });
const replay = await sentStatements(db.$client, () => appendHeartbeatRunEvent(db, nativeEvent(runId, 1, "item.completed")));
expect(replay.statements.some(text => text.includes("pg_current_xact_id"))).toBe(false);
expect(replay.result).toEqual({ row: first.result.row, disposition: "duplicate", highestContiguousSourceSeq: 1 });
expect(await readEvents(runId)).toEqual([first.result.row]);
const wakes: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatNativeProgress, event => { wakes.push(event); });
try {
const first = await sentStatements(db.$client, () => appendHeartbeatRunEvent(db, nativeEvent(runId, 1, "item.completed")));
expect(first.statements).toHaveLength(4);
expect(first.statements[0]).toMatch(/^begin/i);
expect(first.statements[1]).toMatch(/^with "snapshot_run" as .*pg_current_xact_id\(\).* for no key update.*"replay_found"/);
expect(first.statements[2]).toMatch(/insert into heartbeat_run_events/);
expect(first.statements[3]).toMatch(/^commit/i);
expect(wakes).toEqual(["intent", "settled"]);
expect(first.result).toMatchObject({ disposition: "committed", highestContiguousSourceSeq: 1 });
wakes.length = 0;
const replay = await sentStatements(db.$client, () => appendHeartbeatRunEvent(db, nativeEvent(runId, 1, "item.completed")));
expect(wakes).toEqual([]);
expect(replay.statements).toHaveLength(4);
expect(replay.result).toEqual({ row: first.result.row, disposition: "duplicate", highestContiguousSourceSeq: 1 });
expect(await readEvents(runId)).toEqual([first.result.row]);
} finally { unsubscribe(); }
});
it("gives the wake the transaction id of the transaction that took the run lock", async () => {
const runId = await makeRun();
const [issue] = await db.insert(issues).values({ companyId, title: "Primed wake" }).returning();
await db.update(heartbeatRuns).set({ nativeIssueId: issue!.id, runtimeMode: "native" }).where(eq(heartbeatRuns.id, runId));
const topic = DELIVERY_QUEUES.chatNativeProgress;
const options = db.$client.options as { debug?: unknown };
const sent: Array<{ text: string; parameters: unknown[] }> = [];
let ownXid: string | undefined;
options.debug = (_connection: number, text: string, parameters: unknown[]) => {
sent.push({ text: text.trim().replace(/\s+/g, " "), parameters });
};
try {
await expect(db.transaction(async tx => {
await appendHeartbeatRunEvent(tx as unknown as typeof db, nativeEvent(runId, 1, "item.completed"));
expect(sent.filter(statement => /^select pg_current_xact_id\(\)/.test(statement.text))).toEqual([]);
expect(sent.filter(statement => statement.text.includes("pg_current_xact_id()"))).toHaveLength(1);
const [own] = await tx.execute<{ xid: string; pid: number }>(sql`select pg_current_xact_id()::text as xid, pg_backend_pid() as pid`);
ownXid = own!.xid;
await otherDb.execute(sql`select pg_terminate_backend(${own!.pid})`);
await tx.execute(sql`select 1`);
})).rejects.toThrow();
expect(databaseWorkPending(db, topic)).toBe(true);
sent.length = 0;
await reconcileDatabaseWork(db, topic);
} finally { options.debug = undefined; }
const statusRead = sent.find(statement => statement.text.includes("pg_xact_status"));
expect(ownXid).toMatch(/^\d+$/);
expect(statusRead?.parameters).toEqual([ownXid]);
expect(databaseWorkPending(db, topic)).toBe(false);
expect(await readEvents(runId)).toEqual([]);
});
it("primes nothing when the lock read fails, so the transaction is not left uncertain", async () => {
const runId = await makeRun();
const [issue] = await db.insert(issues).values({ companyId, title: "Failed lock read" }).returning();
await db.update(heartbeatRuns).set({ nativeIssueId: issue!.id, runtimeMode: "native" }).where(eq(heartbeatRuns.id, runId));
let release!: () => void, locked!: () => void;
const released = new Promise<void>(resolve => { release = resolve; });
const lockedRun = new Promise<void>(resolve => { locked = resolve; });
const holder = otherDb.transaction(async tx => {
await tx.execute(sql`select id from heartbeat_runs where id = ${runId} for update`);
locked();
await released;
});
void holder.catch(() => undefined);
await lockedRun;
const wakes: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatNativeProgress, wake => { wakes.push(wake); });
const options = db.$client.options as { debug?: unknown };
const sent: string[] = [];
try {
await expect(db.transaction(async tx => {
await tx.execute(sql`set local lock_timeout = '100ms'`);
await appendHeartbeatRunEvent(tx as unknown as typeof db, nativeEvent(runId, 1, "item.completed"));
})).rejects.toThrow();
expect(databaseWorkPending(db, DELIVERY_QUEUES.chatNativeProgress)).toBe(false);
await expect(db.transaction(async tx => {
await tx.execute(sql`set local lock_timeout = '100ms'`);
await expect(tx.transaction(savepoint =>
appendHeartbeatRunEvent(savepoint as unknown as typeof db, nativeEvent(runId, 1, "item.completed")))).rejects.toThrow();
options.debug = (_connection: number, text: string) => { sent.push(text.trim().replace(/\s+/g, " ")); };
await signalDatabaseWork(tx, "lock-read-failed");
options.debug = undefined;
throw new Error("roll back");
})).rejects.toThrow("roll back");
} finally {
options.debug = undefined;
unsubscribe();
release();
await holder;
}
expect(sent.filter(text => /^select pg_current_xact_id\(\)/.test(text))).toHaveLength(1);
await reconcileDatabaseWork(db, "lock-read-failed");
expect(databaseWorkPending(db, "lock-read-failed")).toBe(false);
expect(wakes).toEqual([]);
expect(databaseWorkPending(db, DELIVERY_QUEUES.chatNativeProgress)).toBe(false);
expect(await readEvents(runId)).toEqual([]);
});
it("finds a replay that another writer committed while the lock read waited, and registers no wake for it", async () => {
const runId = await makeRun();
const [issue] = await db.insert(issues).values({ companyId, title: "Progress race" }).returning();
await db.update(heartbeatRuns).set({ nativeIssueId: issue!.id, runtimeMode: "native" }).where(eq(heartbeatRuns.id, runId));
const event = nativeEvent(runId, 1, "item.completed");
let release!: () => void, locked!: () => void;
const released = new Promise<void>(resolve => { release = resolve; });
const lockedRun = new Promise<void>(resolve => { locked = resolve; });
const holder = otherDb.transaction(async tx => {
await tx.execute(sql`select id from heartbeat_runs where id = ${runId} for update`);
await tx.insert(heartbeatRunEvents).values({ companyId, agentId, runId, seq: 1, eventType: event.eventType, stream: "system",
level: "info", payload: event.payload, sourceInstanceId: "native-writer", sourceEventId: event.nativeSource.sourceEventId,
sourceSeq: 1, sourcePayloadSha256: nativeSha256(event.nativeSource.canonicalPayload), protocolSchemaVersion: 1 });
await tx.update(heartbeatRuns).set({ nextEventSeq: 2 }).where(eq(heartbeatRuns.id, runId));
locked();
await released;
});
void holder.catch(() => undefined);
await lockedRun;
const wakes: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatNativeProgress, wake => { wakes.push(wake); });
try {
const appended = sentStatements(db.$client, () => appendHeartbeatRunEvent(db, event));
void appended.catch(() => undefined);
await vi.waitFor(async () => {
const [waiting] = await otherDb.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity
where datname = current_database() and wait_event_type = 'Lock' and query like '%"locked_run"%'`);
expect(waiting?.count).toBe(1);
}, { timeout: 10_000, interval: 20 });
release();
await holder;
const { result, statements } = await appended;
expect(result).toMatchObject({ disposition: "duplicate", row: { seq: 1, sourceSeq: 1 } });
expect(statements.some(text => /^select \* from heartbeat_run_events/.test(text))).toBe(true);
expect(wakes).toEqual([]);
} finally { unsubscribe(); }
expect((await readEvents(runId)).map(row => row.seq)).toEqual([1]);
expect((await readRun(runId)).nextEventSeq).toBe(2);
});
it("keeps one ordered row per source event when two servers append the same stream", async () => {
@@ -601,11 +730,11 @@ describeEmbeddedPostgres("atomic ordinary run events", () => {
expect(replay.highestContiguousSourceSeq).toBe(kind === "native" ? 1 : 0);
expect((await readRun(runId)).nextEventSeq).toBe(2);
expect(await readEvents(runId)).toEqual([committed.row]);
const lockReads = statements.filter(query => query.startsWith("select ") &&
query.includes('from "heartbeat_runs"') && query.endsWith("for no key update"));
const lockReads = statements.filter(query => /^(select|with) /.test(query) &&
query.includes('from "heartbeat_runs"') && query.includes("for no key update"));
expect(lockReads).toHaveLength(2);
for (const query of lockReads) {
expect(query).toMatch(/^select "company_id", "agent_id"(, \("native_source_cursors" ->> \$1\)::bigint as "source_cursor")? from "heartbeat_runs" /);
expect(query).toMatch(/select "company_id", "agent_id"(, \("native_source_cursors" ->> \$\d+\)::bigint as "source_cursor")?(, pg_current_xact_id\(\)::text as "xid")?(, "next_event_seq")? from "heartbeat_runs" /);
expect(query).not.toMatch(/context_snapshot|runner_profile_json|result_json|stdout_excerpt/);
}
});
@@ -631,9 +760,11 @@ describeEmbeddedPostgres("atomic ordinary run events", () => {
holderLocked();
await held;
});
void holder.catch(() => undefined);
await locked;
let secondSettled = false;
const second = appendHeartbeatRunEvent(db, native(2)).finally(() => { secondSettled = true; });
void second.catch(() => undefined);
try {
// A controller checkpoint row references the run row by foreign key.
// Its insert must not wait for the open append.
+22 -12
View File
@@ -9,7 +9,7 @@ import { boundRunIssueIdSql } from "./run-task-binding.js";
import { and, asc, eq, getTableColumns, is, sql } from "drizzle-orm";
import { PgTransaction } from "drizzle-orm/pg-core";
import type { Db } from "@paperclipai/db";
import { heartbeatRunEvents, heartbeatRuns } from "@paperclipai/db";
import { heartbeatRunEvents, heartbeatRuns, primeDatabaseWorkTransactionId } from "@paperclipai/db";
import { nativeSha256 } from "./native-runtime/canonical.js";
import { isUniqueViolation } from "../db-errors.js";
@@ -345,9 +345,9 @@ function decodeRunEventRow(raw: Record<string, unknown>): typeof heartbeatRunEve
/**
* A native source event: one lock read, then one statement that finds a
* replayed event or reserves the sequence and inserts, and reads how much of
* the source is retained. The lock must stay a statement of its own. A
* statement that waits for the run lock keeps the snapshot it took before
* waiting, so it would not see an event the lock holder had just committed.
* the source is retained. A statement that waits for the run lock keeps the
* snapshot it took before waiting, so it would not see an event the lock
* holder had just committed.
*/
async function appendNativeSourceEvent(
tx: Db,
@@ -361,15 +361,26 @@ async function appendNativeSourceEvent(
// checkpoints, that are written for the same run while events stream.
// Only progress needs the task binding. Other native/recovery events
// must not read a potentially large context snapshot to append a log.
const replayed = sql`run_id = ${input.runId} and (source_event_id = ${source.sourceEventId}
or (source_instance_id = ${source.sourceInstanceId} and source_seq = ${source.sourceSeq}))`;
const probes = wakes.reassessment || wakes.progress || wakes.response || wakes.retention;
const columns = sql`"company_id", "agent_id", ("native_source_cursors" ->> ${source.sourceInstanceId})::bigint as "source_cursor"${
wakes.progress || wakes.response || wakes.reassessment ? sql`, "runtime_mode"` : sql.empty()}${
wakes.progress ? sql`, ${boundRunIssueIdSql()} as "issue_id"` : sql.empty()}${
Object.values(wakes).some(Boolean) ? sql`, pg_current_xact_id()::text as "xid"` : sql.empty()}`;
const [run] = await tx.execute<{ company_id: string; agent_id: string; source_cursor: string | number | null;
runtime_mode?: string | null; issue_id?: string | null }>(
sql`select "company_id", "agent_id", ("native_source_cursors" ->> ${source.sourceInstanceId})::bigint as "source_cursor"${
wakes.progress || wakes.response || wakes.reassessment ? sql`, "runtime_mode"` : sql.empty()}${
wakes.progress ? sql`, ${boundRunIssueIdSql()} as "issue_id"` : sql.empty()
} from "heartbeat_runs" where "id" = ${input.runId} for no key update`);
runtime_mode?: string | null; issue_id?: string | null; xid?: string; probe_current?: boolean; replay_found?: boolean }>(probes
? sql`with "snapshot_run" as (select "next_event_seq" from "heartbeat_runs" where "id" = ${input.runId}),
"locked_run" as (select ${columns}, "next_event_seq" from "heartbeat_runs" where "id" = ${input.runId} for no key update)
select "locked_run".*, "locked_run"."next_event_seq" = (select "next_event_seq" from "snapshot_run") as "probe_current",
exists (select 1 from heartbeat_run_events where ${replayed}) as "replay_found"
from "locked_run"`
: sql`select ${columns} from "heartbeat_runs" where "id" = ${input.runId} for no key update`);
if (!run || run.company_id !== input.companyId || run.agent_id !== input.agentId) {
throw new Error("heartbeat_run_event_binding_mismatch");
}
if (run.xid) primeDatabaseWorkTransactionId(tx, run.xid);
const knownNew = run.probe_current === true && run.replay_found === false;
// The stored cursor says the run holds every event of this source up to it.
// An event at or just past it moves the cursor without counting the run's
// events; any other event, or a run without a cursor, recounts them.
@@ -380,15 +391,14 @@ async function appendNativeSourceEvent(
if (wakes.recovery) await notifyLiveRunRecovery(tx, input);
const sourceHash = nativeSha256(source.canonicalPayload);
const replayed = sql`run_id = ${input.runId} and (source_event_id = ${source.sourceEventId}
or (source_instance_id = ${source.sourceInstanceId} and source_seq = ${source.sourceSeq}))`;
const native = run.runtime_mode === "native";
const reassessment = wakes.reassessment && native, progress = wakes.progress && native && run.issue_id;
const response = wakes.response && native;
if (reassessment || progress || response || wakes.retention) {
// A wake is registered before the write it announces, and a replayed
// event registers none. So these events look for their replay first.
const [existing] = await tx.execute<Record<string, unknown>>(sql`select * from heartbeat_run_events where ${replayed} limit 1`);
const [existing] = knownNew ? []
: await tx.execute<Record<string, unknown>>(sql`select * from heartbeat_run_events where ${replayed} limit 1`);
if (existing) {
const row = decodeRunEventRow(existing);
if (row.sourcePayloadSha256 !== sourceHash) throw new HeartbeatRunEventConflictError();
@@ -0,0 +1,82 @@
import { randomBytes, randomUUID } from "node:crypto";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { eq } from "drizzle-orm";
import { agents, checkpointDb, closeCheckpointDb, companies, createDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints,
subscribeDatabaseWork } from "@paperclipai/db";
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import { nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("native runner checkpoint save", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
process.env.PAPERCLIP_SECRETS_MASTER_KEY ??= randomBytes(32).toString("base64");
temporary = await startEmbeddedPostgresTestDatabase("paperclip-checkpoint-save-");
db = createDb(temporary.connectionString);
}, 60_000);
afterAll(async () => {
if (db) await closeCheckpointDb(db, 1);
await db?.$client.end({ timeout: 1 });
await temporary?.cleanup();
});
const fixture = async (leaseExpiresAt: Date) => {
const [company] = await db.insert(companies).values({ name: "Checkpoint save", issuePrefix: `CS${randomUUID().slice(0, 4)}` }).returning();
const [agent] = await db.insert(agents).values({ companyId: company!.id, name: "Checkpoint agent", adapterType: "process" }).returning();
const [issue] = await db.insert(issues).values({ companyId: company!.id, title: "Checkpoint task" }).returning();
const runId = randomUUID(), leaseOwner = randomUUID(), bootId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: company!.id, agentId: agent!.id, status: "running",
runtimeMode: "native", nativeIssueId: issue!.id, issueId: issue!.id });
await db.insert(nativeRunFinalizations).values({ runId, companyId: company!.id, issueId: issue!.id, phase: "observed", attempt: 0,
controllerBootId: bootId, controllerPid: process.pid, controllerProcessStartedAt: new Date(), controllerGeneration: 1,
leaseOwner, leaseExpiresAt });
const write = nativeRunnerCheckpointWriter(db, { companyId: company!.id, issueId: issue!.id, runId, authorityRunId: runId,
bootId, generation: 1, leaseOwner });
const checkpoint = (changed: Map<string, string>, reset: boolean) => ({
runId, records: new Map(changed), changedRecords: changed, removedRecords: [], reset, workCommandIds: [], predecessor: null,
runnerVersion: "test", runnerDigest: `sha256:${"a".repeat(64)}`,
recoveryEvidence: { runnerInstanceId: "test", normalizedSessionId: "test", authenticatedConnectionCount: 1, highestRecordedSourceSeq: 0 },
restoreSessionDirectory() {},
}) as unknown as Parameters<typeof write>[0];
return { runId, write, checkpoint };
};
const sentStatements = async (action: () => Promise<unknown>) => {
const statements: string[] = [];
const options = checkpointDb(db).$client.options as { debug?: unknown };
options.debug = (_connection: number, text: string) => { statements.push(text.trim().replace(/\s+/g, " ")); };
try { await action(); return statements; } finally { options.debug = undefined; }
};
it("saves a checkpoint with the owner lock, the write and the commit, and registers its wake before the write", async () => {
const f = await fixture(new Date(Date.now() + 3_600_000));
await f.write(f.checkpoint(new Map([["header", "{}"], ["cursor", "{\"ackedSourceSeq\":0}"]]), true));
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeProcessSessions, event => { events.push(event); });
let statements: string[];
try {
statements = await sentStatements(() => f.write(f.checkpoint(new Map([["event:1", "{}"], ["cursor", "{\"ackedSourceSeq\":1}"]]), false)));
} finally { unsubscribe(); }
expect(statements).toHaveLength(4);
expect(statements[0]).toMatch(/^begin/i);
expect(statements[1]).toMatch(/pg_current_xact_id\(\)/);
expect(statements[2]).toMatch(/^insert into "native_runner_checkpoints"/);
expect(statements[3]).toMatch(/^commit/i);
expect(events).toEqual(["intent", "settled"]);
expect((await db.select({ part: nativeRunnerCheckpoints.part }).from(nativeRunnerCheckpoints)
.where(eq(nativeRunnerCheckpoints.runId, f.runId))).map(row => row.part).sort()).toEqual(["cursor", "event:1", "header"]);
});
it("refuses a save after the owner's lease expired and writes nothing", async () => {
const f = await fixture(new Date(Date.now() + 3_600_000));
await f.write(f.checkpoint(new Map([["header", "{}"]]), true));
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) })
.where(eq(nativeRunFinalizations.runId, f.runId));
await expect(f.write(f.checkpoint(new Map([["event:1", "{}"]]), false))).rejects.toThrow("native_runner_checkpoint_owner_lost");
expect((await db.select({ part: nativeRunnerCheckpoints.part }).from(nativeRunnerCheckpoints)
.where(eq(nativeRunnerCheckpoints.runId, f.runId))).map(row => row.part)).toEqual(["header"]);
});
});
@@ -3,7 +3,7 @@ import { NATIVE_PREPARATION_FENCING_VERSION } from "./native-active-runs.js";
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
import { createHash } from "node:crypto";
import { and, eq, exists, gt, isNull, lte, inArray, notExists, or, sql, type SQL } from "drizzle-orm";
import { checkpointDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
import { checkpointDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, primeDatabaseWorkTransactionId, type Db } from "@paperclipai/db";
import { inspectDurablePrpCheckpointHeader, inspectUnauthenticatedBootstrapCheckpoint, inspectUnclaimedBootstrapCheckpoint, decodeDurablePrpCheckpoint, type DurablePrpCheckpoint, type DurablePrpControlPlane, type DurableRecoveryIdentity } from "../../vendor/paperclip-runner/index.js";
import { createLocalEncryptedCipher, deriveLocalEncryptedInstanceSecret } from "../../secrets/local-encrypted-provider.js";
import { legacyControllerBootId } from "../legacy-controller-lease.js";
@@ -76,9 +76,12 @@ export async function lockNativeRunnerCheckpointOwner(
// wall time. Sampling in the locking SELECT can precede its lock wait.
// Keep both steps in one statement without another database round trip.
const locked = nativeRunnerCheckpointOwnerLock(tx, owner);
const [owned] = await tx.with(locked).select({ leaseOwner: locked.leaseOwner,
leaseExpiresAt: locked.leaseExpiresAt, resultId: locked.resultId, phase: locked.phase, now: sql<Date>`clock_timestamp()` }).from(locked);
if (!owned) throw new Error("native_runner_checkpoint_owner_lost");
const [row] = await tx.with(locked).select({ leaseOwner: locked.leaseOwner,
leaseExpiresAt: locked.leaseExpiresAt, resultId: locked.resultId, phase: locked.phase, now: sql<Date>`clock_timestamp()`,
xid: sql<string>`pg_current_xact_id()::text` }).from(locked);
if (!row) throw new Error("native_runner_checkpoint_owner_lost");
const { xid, ...owned } = row;
primeDatabaseWorkTransactionId(tx, xid);
return owned;
}