mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote runner's controller saves a checkpoint after each runner event it commits, on a 3-connection pool, before it acknowledges the event > - Each save is a transaction of 4 statements (5 where commit notifications are on): BEGIN, the owner lock with the transaction id, the write, COMMIT > - What forces the transaction is the live-run recovery wake that every save registers. Recovery does not react to a routine save > - This pull request writes a routine save (new or changed event, command and cursor records) with one statement that checks the owner and writes, and keeps today's transaction for every other save > - The benefit is one statement instead of four for most checkpoint saves, on the pool that limits one process in a busy minute ## Linked Issues or Issue Description Refs #21528 (owner lock returns the transaction id). Refs #21549 (load model at recorded rates; the plan records the checkpoint pool as a per-process limit). **What existing behavior does this improve?** `nativeRunnerCheckpointWriter` in `server/src/services/native-runtime/native-runner-checkpoint.ts`. **Subsystem affected** Controller checkpoint persistence for remote native runs. **Current behavior** Every save locks the coordinator row (`native_run_finalizations` `for update`), checks the owner, calls `notifyLiveRunRecovery` (6 recovery queues), and upserts the changed parts, in one transaction. **Proposed behavior** A routine save is one statement: a CTE locks the coordinator row only while the lease owner, boot id, generation and an unexpired lease (`clock_timestamp()`) all match, and the upsert writes only through that row. It registers no recovery wake. If the owner does not match, it writes nothing and the save takes today's full path, which runs the complete owner check (successor rules included) and fails with `native_runner_checkpoint_owner_lost` when it should. A save is routine when it has changed parts, no reset, no removed parts, no header change, no predecessor, and the run is its own authority. Every other save (the first save of a controller, which is a reset, a save after a failed save, header or retirement changes, removals, source-run consumers) is unchanged. **Reason and benefit** At recorded rates about a third of one process's statements are checkpoint saves. In a busy minute the 3-connection pool cannot keep up at 500 agents (#21549). **Breaking changes** None. No schema change. ## What Changed - `ROUTINE_CHECKPOINT_SAVE_SQL` and the routine branch in `nativeRunnerCheckpointWriter`. ## Why a routine save needs no recovery wake I read the six consumers of `notifyLiveRunRecovery`: - `nativeProcessSessions`, `nativeQuestionCancellations`, `activeExecutionRecovery`, `settledNativeProcesses` (`live-recovery-work.ts`): each selects candidates from `heartbeat_runs` status, runtime mode, `native_run_finalizations` phase, lease and retry times, the context snapshot and process identity. None reads checkpoint rows to decide eligibility. - `strandedAgentStatus` (same file, `agents` rows) and `activeRunSilence` (`active-run-watchdog/adapters/deadline-work.ts`, `heartbeat_runs.last_output_at`): neither reads checkpoint rows. - The checkpoint rows matter to recovery in two places: `listExpiredNativeRunnerAuthorities` requires that a checkpoint exists, and the remote bootstrap proof reads its parts. A checkpoint first appears with a reset save, which keeps its wake. A routine save only adds event and command parts, which can only take a run further from the bootstrap proof; recovery reads the current rows when its own deadline (lease expiry) fires. - The intent of the wake also fenced a recovery pass against a save in flight. The routine save is one atomic statement and holds the coordinator row lock while it writes, so a recovery claim that takes that row waits for it, and a reader sees the save whole or not at all. ## Verification - Red/green on embedded PostgreSQL (`native-runner-checkpoint-save.test.ts`); the new tests failed against `origin/master` first: - A routine save is one statement (`with owner as ... for update ... insert into native_runner_checkpoints`) and registers no recovery wake (before: 4 statements and a wake). - A reset save keeps BEGIN, the owner lock with the transaction id, and COMMIT, and still registers its recovery wake. - A routine save that waits while another connection takes over the lease (new owner and generation) falls back to the full path and fails with `native_runner_checkpoint_owner_lost`; the checkpoint is unchanged. - The same race where the new owner also resets the checkpoint (deletes every part and writes its own header): the save fails the same way and the checkpoint holds only the new owner's header. - A lease that expires while the routine save waits for the coordinator row lock (the holder does not change the row) writes nothing: the statement judges the lease after the lock, as `lockNativeRunnerCheckpointOwner` does. My first version checked the lease in the locking CTE itself, and PostgreSQL does not re-check a condition after a lock wait on an unchanged row; the existing `native-runner-restart-recovery.integration.test.ts` test "fences checkpoint delivery when ownership expires during an unchanged row-lock wait" failed on it, and so does the new test. Both pass now. - The existing test "refuses a save after the owner's lease expired and writes nothing" passes through the new fallback. - Mutation checks: with the owner conditions removed from the statement, both race tests and the expired-lease test fail; with the lease check moved back into the locking CTE, the lock-wait expiry test fails. - Other focused suites, one file at a time: `native-runner-restart-recovery.integration` (231 passed, 14 skipped), `native-runner-restart-checkpoint-cost.integration` (1; it greps for `insert into "native_runner_checkpoints"`, so the statement quotes the table name), `native-recovered-owner-followup` (97), `native-controller-recovery.integration` (30), `native-runner-checkpoint-authority.integration` (1), `native-restart-recovery-isolation` (1), the port suite with conformance (105), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51 tests pass; on this busy machine its teardown hook timed out, as it has before on master). `pnpm --filter @paperclipai/server run typecheck` passes. - Statements per routine save: 4 → 1 (5 → 1 where commit notifications are on). I did not rerun the load model for this change: the machine is too busy for a fair timing comparison today, and the statement count is exact. - CPU per operation (a local script, not committed: 600 routine saves of one event part and the cursor for one run on embedded PostgreSQL, one connection, three rounds, `origin/master` code against this branch): main-thread CPU per save 0.38–0.56 ms before and 0.10–0.12 ms after (encryption included in both); wall time per save 0.45–0.73 ms before and 0.22–0.30 ms after. ## Risks - A routine save no longer wakes the six recovery queues. Their own deadlines and the 30 s reconstruction still run; see the reading above for why no decision depends on a routine save. - The owner check of the routine statement accepts only the exact active owner. Any other state goes to the unchanged full path, so a successor or a finished result keeps today's rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
172 lines
10 KiB
TypeScript
172 lines
10 KiB
TypeScript
import { randomBytes, randomUUID } from "node:crypto";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { eq, sql } from "drizzle-orm";
|
|
import { agents, checkpointDb, closeCheckpointDb, companies, createDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints,
|
|
subscribeDatabaseWork } from "@paperclipai/db";
|
|
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
|
import { nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
|
|
(support.supported ? describe : describe.skip)("native runner checkpoint save", () => {
|
|
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
let otherDb: ReturnType<typeof createDb>;
|
|
beforeAll(async () => {
|
|
process.env.PAPERCLIP_SECRETS_MASTER_KEY ??= randomBytes(32).toString("base64");
|
|
temporary = await startEmbeddedPostgresTestDatabase("paperclip-checkpoint-save-");
|
|
db = createDb(temporary.connectionString);
|
|
otherDb = createDb(temporary.connectionString);
|
|
}, 60_000);
|
|
afterAll(async () => {
|
|
if (db) await closeCheckpointDb(db, 1);
|
|
await otherDb?.$client.end({ timeout: 1 });
|
|
await db?.$client.end({ timeout: 1 });
|
|
await temporary?.cleanup();
|
|
});
|
|
|
|
const fixture = async (leaseExpiresAt: Date) => {
|
|
const [company] = await db.insert(companies).values({ name: "Checkpoint save", issuePrefix: `CS${randomUUID().slice(0, 4)}` }).returning();
|
|
const [agent] = await db.insert(agents).values({ companyId: company!.id, name: "Checkpoint agent", adapterType: "process" }).returning();
|
|
const [issue] = await db.insert(issues).values({ companyId: company!.id, title: "Checkpoint task" }).returning();
|
|
const runId = randomUUID(), leaseOwner = randomUUID(), bootId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId: company!.id, agentId: agent!.id, status: "running",
|
|
runtimeMode: "native", nativeIssueId: issue!.id, issueId: issue!.id });
|
|
await db.insert(nativeRunFinalizations).values({ runId, companyId: company!.id, issueId: issue!.id, phase: "observed", attempt: 0,
|
|
controllerBootId: bootId, controllerPid: process.pid, controllerProcessStartedAt: new Date(), controllerGeneration: 1,
|
|
leaseOwner, leaseExpiresAt });
|
|
const write = nativeRunnerCheckpointWriter(db, { companyId: company!.id, issueId: issue!.id, runId, authorityRunId: runId,
|
|
bootId, generation: 1, leaseOwner });
|
|
const checkpoint = (changed: Map<string, string>, reset: boolean) => ({
|
|
runId, records: new Map(changed), changedRecords: changed, removedRecords: [], reset, workCommandIds: [], predecessor: null,
|
|
runnerVersion: "test", runnerDigest: `sha256:${"a".repeat(64)}`,
|
|
recoveryEvidence: { runnerInstanceId: "test", normalizedSessionId: "test", authenticatedConnectionCount: 1, highestRecordedSourceSeq: 0 },
|
|
restoreSessionDirectory() {},
|
|
}) as unknown as Parameters<typeof write>[0];
|
|
return { runId, companyId: company!.id, issueId: issue!.id, write, checkpoint };
|
|
};
|
|
|
|
const sentStatements = async (action: () => Promise<unknown>) => {
|
|
const statements: string[] = [];
|
|
const options = checkpointDb(db).$client.options as { debug?: unknown };
|
|
options.debug = (_connection: number, text: string) => { statements.push(text.trim().replace(/\s+/g, " ")); };
|
|
try { await action(); return statements; } finally { options.debug = undefined; }
|
|
};
|
|
|
|
const parts = async (runId: string) => (await db.select({ part: nativeRunnerCheckpoints.part })
|
|
.from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, runId))).map(row => row.part).sort();
|
|
const recoveryWakes = () => {
|
|
const events: string[] = [];
|
|
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeProcessSessions, event => { events.push(event); });
|
|
return { events, unsubscribe };
|
|
};
|
|
|
|
it("saves a routine checkpoint in one statement that checks the owner, and registers no recovery wake", async () => {
|
|
const f = await fixture(new Date(Date.now() + 3_600_000));
|
|
await f.write(f.checkpoint(new Map([["header", "{}"], ["cursor", "{\"ackedSourceSeq\":0}"]]), true));
|
|
const wakes = recoveryWakes();
|
|
let statements: string[];
|
|
try {
|
|
statements = await sentStatements(() => f.write(f.checkpoint(new Map([["event:1", "{}"], ["cursor", "{\"ackedSourceSeq\":1}"]]), false)));
|
|
} finally { wakes.unsubscribe(); }
|
|
expect(statements).toHaveLength(1);
|
|
expect(statements[0]).toMatch(/^with locked_owner as .* for update .* lease_expires_at > clock_timestamp\(\) .* insert into "native_runner_checkpoints"/);
|
|
expect(wakes.events).toEqual([]);
|
|
expect(await parts(f.runId)).toEqual(["cursor", "event:1", "header"]);
|
|
});
|
|
|
|
it("saves a reset with the owner lock, the write and the commit, and registers its recovery wake before the write", async () => {
|
|
const f = await fixture(new Date(Date.now() + 3_600_000));
|
|
await f.write(f.checkpoint(new Map([["header", "{}"], ["cursor", "{\"ackedSourceSeq\":0}"]]), true));
|
|
const wakes = recoveryWakes();
|
|
let statements: string[];
|
|
try {
|
|
statements = await sentStatements(() => f.write(f.checkpoint(new Map([["header", "{}"], ["cursor", "{\"ackedSourceSeq\":2}"]]), true)));
|
|
} finally { wakes.unsubscribe(); }
|
|
expect(statements[0]).toMatch(/^begin/i);
|
|
expect(statements[1]).toMatch(/pg_current_xact_id\(\)/);
|
|
expect(statements.at(-1)).toMatch(/^commit/i);
|
|
expect(wakes.events).toEqual(["intent", "settled"]);
|
|
expect(await parts(f.runId)).toEqual(["cursor", "header"]);
|
|
});
|
|
|
|
it.each(["takeover", "takeover with a reset"])("falls back to the full owner check when a routine save races a lease %s", async (race) => {
|
|
const f = await fixture(new Date(Date.now() + 3_600_000));
|
|
await f.write(f.checkpoint(new Map([["header", "{}"], ["cursor", "{\"ackedSourceSeq\":0}"]]), true));
|
|
let release!: () => void, locked!: () => void;
|
|
const released = new Promise<void>(resolve => { release = resolve; });
|
|
const lockedOwner = new Promise<void>(resolve => { locked = resolve; });
|
|
const holder = otherDb.transaction(async tx => {
|
|
await tx.execute(sql`select run_id from native_run_finalizations where run_id = ${f.runId} for update`);
|
|
await tx.update(nativeRunFinalizations).set({ leaseOwner: randomUUID(), controllerGeneration: 2 })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
if (race === "takeover with a reset") {
|
|
await tx.delete(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId));
|
|
await tx.insert(nativeRunnerCheckpoints).values({ companyId: f.companyId, issueId: f.issueId, runId: f.runId,
|
|
part: "header", material: { successor: true } });
|
|
}
|
|
locked();
|
|
await released;
|
|
});
|
|
void holder.catch(() => undefined);
|
|
await lockedOwner;
|
|
const saved = f.write(f.checkpoint(new Map([["event:1", "{}"], ["cursor", "{\"ackedSourceSeq\":1}"]]), false))
|
|
.then(() => "saved", (error: Error) => error.message);
|
|
try {
|
|
await vi.waitFor(async () => {
|
|
const [waiting] = await otherDb.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity
|
|
where datname = current_database() and wait_event_type = 'Lock' and query like '%with locked_owner as%'`);
|
|
expect(waiting?.count).toBe(1);
|
|
}, { timeout: 10_000, interval: 20 });
|
|
} finally {
|
|
release();
|
|
await holder;
|
|
}
|
|
expect(await saved).toBe("native_runner_checkpoint_owner_lost");
|
|
expect(await parts(f.runId)).toEqual(race === "takeover" ? ["cursor", "header"] : ["header"]);
|
|
});
|
|
|
|
it("judges the lease after the owner lock, so a lease that expires while a routine save waits writes nothing", async () => {
|
|
const f = await fixture(new Date(Date.now() + 3_600_000));
|
|
await f.write(f.checkpoint(new Map([["header", "{}"], ["cursor", "{\"ackedSourceSeq\":0}"]]), true));
|
|
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: sql`clock_timestamp() + interval '1 second'` })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
let release!: () => void, locked!: () => void;
|
|
const released = new Promise<void>(resolve => { release = resolve; });
|
|
const lockedOwner = new Promise<void>(resolve => { locked = resolve; });
|
|
const holder = otherDb.transaction(async tx => {
|
|
await tx.execute(sql`select run_id from native_run_finalizations where run_id = ${f.runId} for update`);
|
|
locked();
|
|
await released;
|
|
});
|
|
void holder.catch(() => undefined);
|
|
await lockedOwner;
|
|
const saved = f.write(f.checkpoint(new Map([["event:1", "{}"], ["cursor", "{\"ackedSourceSeq\":1}"]]), false))
|
|
.then(() => "saved", (error: Error) => error.message);
|
|
try {
|
|
await vi.waitFor(async () => {
|
|
const [waiting] = await otherDb.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity
|
|
where datname = current_database() and wait_event_type = 'Lock' and query like '%with locked_owner as%'`);
|
|
expect(waiting?.count).toBe(1);
|
|
}, { timeout: 10_000, interval: 20 });
|
|
await otherDb.execute(sql`select pg_sleep(1.2)`);
|
|
} finally {
|
|
release();
|
|
await holder;
|
|
}
|
|
expect(await saved).toBe("native_runner_checkpoint_owner_lost");
|
|
expect(await parts(f.runId)).toEqual(["cursor", "header"]);
|
|
});
|
|
|
|
it("refuses a save after the owner's lease expired and writes nothing", async () => {
|
|
const f = await fixture(new Date(Date.now() + 3_600_000));
|
|
await f.write(f.checkpoint(new Map([["header", "{}"]]), true));
|
|
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
await expect(f.write(f.checkpoint(new Map([["event:1", "{}"]]), false))).rejects.toThrow("native_runner_checkpoint_owner_lost");
|
|
expect((await db.select({ part: nativeRunnerCheckpoints.part }).from(nativeRunnerCheckpoints)
|
|
.where(eq(nativeRunnerCheckpoints.runId, f.runId))).map(row => row.part)).toEqual(["header"]);
|
|
});
|
|
});
|