perf(runner): keep at most one checkpoint waiting behind a slow write (#21411)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner control plane saves a checkpoint of each run's
controller state before it acknowledges an event or sends a command
> - Each new state revision captured a full snapshot and queued it
behind the write in progress
> - Inbound events wait for their own checkpoint, so they do not pile
up. Commands that the server queues do not wait, so they can
> - When the database or the event loop is slow, the queue of full
snapshots grows without limit, and events wait behind the whole queue
> - This pull request keeps at most one snapshot waiting per run. A
newer flush replaces the waiting snapshot of the same run
> - The benefit is bounded memory and fewer database writes under slow
persistence, with the same durability guarantee

## Linked Issues or Issue Description

**What happened?**

A hosted server ran out of JavaScript heap at intervals of 47 to 105
minutes. A retained-memory sample attributed about 16% of the memory
that stayed alive to `captureCheckpoint` (from `flushCheckpoint`) and to
`#capturePublishedState` in
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`.

**Expected behavior**

Memory for checkpoints stays proportional to the number of live runs,
not to the number of state revisions that wait for a slow write.

**Steps to reproduce**

1. Create control planes with checkpoint persistence that takes 1 second
per write.
2. Commit events at 13 per second and queue commands at 2 per second for
each run.
3. Measure the heap in the middle of the run.

`node --import ./server/node_modules/tsx/dist/loader.mjs
scripts/benchmark-runner-checkpoint-backlog.ts` does these steps.

**Paperclip version or commit**

`master` at 2659ff1d22.

## What Changed

- `DurablePrpControlPlane.flushCheckpoint` keeps one snapshot that waits
for the write in progress. A newer flush for the same run replaces that
snapshot and returns the same promise. A flush for a different run (a
warm handoff) still queues its own write.
- New tests in `durable-prp-control-plane.test.ts`:
- 40 commands queued behind a slow write produce one more write, not 40.
Each caller resolves only after the stored records contain its command.
The stored records decode to the last written checkpoint.
- When the combined write fails, every caller that waited on it rejects,
and the next flush writes a full reset that contains the commands.
- New `scripts/benchmark-runner-checkpoint-backlog.ts` measures retained
heap per run, checkpoint writes, and whether finished control planes are
collected.

Why this is safe:

- A waiting snapshot has not started its write. No caller has received a
result for it, so no acknowledgement or command send depends on it.
- The replacement is captured from a later state of the same store. It
contains every change that the replaced snapshot contained.
- Every caller of the replaced snapshot waits on the same promise. The
promise resolves only after the newer snapshot is durable. A failure
rejects all of them, and the existing retry writes a full reset.
- Writes stay on one chain, in order. The server fence, the encryption,
and the stored checkpoint format do not change.
- A flush for a different run ID never replaces a waiting snapshot, so
the earlier run still gets its last state.

## Verification

- Red/green: the two new tests fail on `master` (42 writes instead of 3;
the waiting callers resolve instead of rejecting) and pass with the fix.
- `npx vitest run src/control-plane/durable-prp-control-plane.test.ts`
in `packages/paperclip-runner`: 219 of 219 pass.
- `node scripts/check-module-boundaries.mjs` passes.
- Benchmark, 8 runs, 30 s, 13 events/s and 2 commands/s per run, 1 s
writes, 300 events of 2 KB already in each run:

| | `master` | this PR |
| --- | --- | --- |
| Retained heap per run in the middle of the load | 15,009 KB | 2,447 KB
|
| Retained heap per idle live run | 1,296 KB | 1,415 KB |
| Checkpoint writes | 515 (one per revision) | 248 (for 720 revisions) |
| New events committed per run in 30 s | 4 | 30 |
| Control planes collected after stop | 8 of 8 | 8 of 8 |

- With commands at 0 per second, `master` has no backlog: an inbound
event waits for its own checkpoint before the next frame is read. The
backlog comes from flushes that do not wait.

## Risks

- Behavioral shift: when a combined write fails, callers whose own
earlier revision would have been written alone now see that failure. A
checkpoint failure already disconnects the runner and the next flush
writes a full reset, so recovery is the same.
- Fewer intermediate states reach the database. A crash can only lose a
state that no caller has seen as saved, which is the same as before.
- Not covered by a test: a warm handoff whose run ID changes while a
snapshot waits. The code does not combine those; the existing
warm-transition tests pass.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 11:08:15 -06:00
1 parent 85f4a91e70
commit 594f9d3ec2
3 files changed
+260

No files matched your search

@@ -787,6 +787,134 @@ it.each([false, true])("recovers checkpoint persistence and the same connection
}
});
it("keeps one waiting checkpoint behind a slow write and saves every queued command before resolving", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-coalesce-"));
const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest });
const stored = new Map<string, string>();
const writes: DurablePrpCheckpoint[] = [];
let release!: () => void;
let gate: Promise<void> = Promise.resolve();
core.setCheckpointPersistence(async snapshot => {
writes.push(snapshot);
await gate;
if (snapshot.reset) stored.clear();
for (const key of snapshot.removedRecords) stored.delete(key);
for (const [key, value] of snapshot.changedRecords) stored.set(key, value);
});
try {
await core.flushCheckpoint();
gate = new Promise<void>(resolveWrite => { release = resolveWrite; });
core.queueCommand("session.snapshot", {}, "in-flight-command");
const inFlight = core.flushCheckpoint();
await vi.waitFor(() => expect(writes).toHaveLength(2));
const settled: string[] = [];
const waiting = Array.from({ length: 40 }, (_, index) => {
const commandId = `waiting-command-${index}`;
core.queueCommand("session.snapshot", {}, commandId);
return core.flushCheckpoint().then(() => {
expect([...stored.values()].join("\n")).toContain(commandId);
settled.push(commandId);
});
});
await new Promise<void>(resolveTurn => setImmediate(resolveTurn));
expect(writes).toHaveLength(2);
expect(settled).toEqual([]);
release();
await inFlight;
await Promise.all(waiting);
expect(writes).toHaveLength(3);
expect(settled).toHaveLength(40);
expect(stored).toEqual(writes.at(-1)!.records);
expect(decodeDurablePrpCheckpoint(stored).records).toEqual(writes.at(-1)!.records);
await core.flushCheckpoint();
expect(writes).toHaveLength(3);
} finally {
release?.();
await core.stop();
rmSync(root, { recursive: true, force: true });
}
});
it("writes a waiting checkpoint for each run in order when the run changes behind a slow write", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-coalesce-handoff-"));
const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" };
const template = { provider: { provider: "codex", command: "fixture" }, authorizedTools: {} };
const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest });
const writes: DurablePrpCheckpoint[] = [];
let release!: () => void;
let gate: Promise<void> = Promise.resolve();
core.setCheckpointPersistence(async snapshot => { writes.push(snapshot); await gate; });
const contents = (checkpoint: DurablePrpCheckpoint) => [...checkpoint.records.values()].join("\n");
try {
await core.flushCheckpoint();
gate = new Promise<void>(resolveWrite => { release = resolveWrite; });
core.queueCommand("session.snapshot", {}, "source-in-flight");
const inFlight = core.flushCheckpoint();
await vi.waitFor(() => expect(writes).toHaveLength(2));
core.queueCommand("session.snapshot", {}, "source-waiting");
const sourceWaiting = core.flushCheckpoint();
core.stageRunAttachment(target, template);
const targetWaiting = core.flushCheckpoint();
core.queueCommand("session.snapshot", {}, "target-later");
const targetLater = core.flushCheckpoint();
expect(targetLater).toBe(targetWaiting);
expect(sourceWaiting).not.toBe(targetWaiting);
release();
await Promise.all([inFlight, sourceWaiting, targetWaiting, targetLater]);
expect(writes.map(write => write.runId)).toEqual([identity.runId, identity.runId, identity.runId, target.runId]);
expect(contents(writes[2]!)).toContain("source-waiting");
expect(contents(writes[2]!)).not.toContain("target-later");
expect(writes[2]!.records.has("staged-attachment-template")).toBe(false);
expect(writes[3]).toMatchObject({ reset: true, predecessor: { runId: identity.runId } });
expect(contents(writes[3]!)).toContain("target-later");
expect(writes[3]!.records.has("staged-attachment-template")).toBe(true);
await core.flushCheckpoint();
expect(writes).toHaveLength(4);
} finally {
release?.();
await core.stop();
rmSync(root, { recursive: true, force: true });
}
});
it("rejects every caller waiting on a coalesced checkpoint whose write fails", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-coalesce-failure-"));
const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest });
const writes: DurablePrpCheckpoint[] = [];
let release!: () => void;
let gate: Promise<void> = Promise.resolve();
let fail = true;
core.setCheckpointPersistence(async snapshot => {
writes.push(snapshot);
await gate;
if (fail && [...snapshot.records.values()].join("\n").includes("second-waiting")) throw new Error("coalesced write failed");
});
try {
await core.flushCheckpoint();
gate = new Promise<void>(resolveWrite => { release = resolveWrite; });
core.queueCommand("session.snapshot", {}, "in-flight-command");
const inFlight = core.flushCheckpoint();
await vi.waitFor(() => expect(writes).toHaveLength(2));
const waiting = ["first-waiting", "second-waiting"].map(commandId => {
core.queueCommand("session.snapshot", {}, commandId);
return core.flushCheckpoint();
});
const rejected = waiting.map(promise => expect(promise).rejects.toThrow("coalesced write failed"));
release();
await inFlight;
await Promise.all(rejected);
expect(writes).toHaveLength(3);
fail = false;
await core.flushCheckpoint();
expect(writes.at(-1)!.reset).toBe(true);
expect([...writes.at(-1)!.records.values()].join("\n")).toContain("second-waiting");
} finally {
release?.();
await core.stop();
rmSync(root, { recursive: true, force: true });
}
});
it("rebuilds a queued checkpoint after the preceding write fails", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-queued-retry-"));
const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest });
@@ -2131,6 +2131,7 @@ export class DurablePrpControlPlane {
#visibleStateCache?: { working: StoredCoreState; published: StoredCoreState; revision: number; state: StoredCoreState };
#checkpointPersistenceFailed = false;
#persistenceTail = Promise.resolve();
#waitingCheckpoint: { snapshot: DurablePrpCheckpoint; published: StoredCoreState; revision: number } | null = null;
#sendQueues = new Map<AuthorityConnection, Promise<void>>();
#recoveredCursor: { runId: string; sourceSeq: number } | null = null;
#readableStateListeners = new Set<() => void>();
@@ -2297,8 +2298,19 @@ export class DurablePrpControlPlane {
const published = this.#capturePublishedState(snapshot);
const revision = this.#store.revision;
this.#queuedRevision = revision;
const waiting = this.#waitingCheckpoint;
if (waiting !== null && waiting.snapshot.runId === snapshot.runId) {
waiting.snapshot = snapshot;
waiting.published = published;
waiting.revision = revision;
return this.#persistenceTail;
}
const job = { snapshot, published, revision };
this.#waitingCheckpoint = job;
const persist = this.#persistCheckpoint;
const pending = this.#persistenceTail.catch(() => undefined).then(async () => {
if (this.#waitingCheckpoint === job) this.#waitingCheckpoint = null;
const { snapshot, published, revision } = job;
// The previous write may have failed after committing. Rebuild the delta
// at execution time, and reset after uncertainty instead of assuming it failed.
const checkpoint = { ...snapshot, ...checkpointChanges(snapshot.records, snapshot.runId,
@@ -0,0 +1,120 @@
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { resolve } from "node:path";
import { setFlagsFromString } from "node:v8";
import { runInNewContext } from "node:vm";
import { DurablePrpControlPlane } from "../packages/paperclip-runner/src/control-plane/durable-prp-control-plane.js";
const numberOption = (name: string, fallback: number) => {
const value = process.argv.find(arg => arg.startsWith(`--${name}=`))?.slice(name.length + 3);
return value === undefined ? fallback : Number(value);
};
const runs = numberOption("runs", 8);
const seconds = numberOption("seconds", 30);
const eventsPerSecond = numberOption("events-per-second", 13);
const commandsPerSecond = numberOption("commands-per-second", 2);
const writeMs = numberOption("write-ms", 1000);
const eventBytes = numberOption("event-bytes", 2048);
const seedEvents = numberOption("seed-events", 300);
setFlagsFromString("--expose-gc");
const gc = runInNewContext("gc") as () => void;
const pause = (ms: number) => new Promise(resolveTimer => setTimeout(resolveTimer, ms));
const settledHeap = async () => {
for (let index = 0; index < 6; index++) {
gc();
await new Promise(resolveTurn => setImmediate(resolveTurn));
}
return process.memoryUsage().heapUsed;
};
type WorkingStore = {
revision: number;
state: DurablePrpControlPlane["store"]["state"];
commit(candidate: DurablePrpControlPlane["store"]["state"], checkpointRequired?: boolean): void;
};
const identity = (index: number) => ({
runnerInstanceId: `runner-bench-${index}`, environmentLeaseId: `environment-bench-${index}`,
runId: `00000000-0000-4000-8000-${String(index + 1).padStart(12, "0")}`,
normalizedSessionId: `session-bench-${index}`, turnId: `turn-bench-${index}`, itemId: `item-bench-${index}`,
});
const collected = { count: 0 };
const registry = new FinalizationRegistry<number>(() => { collected.count += 1; });
const roots: string[] = [];
const baseline = await settledHeap();
let writesStarted = 0, revisionsRequested = 0;
function benchEvent(index: number, sourceSeq: number) {
return { sourceSeq, sourceEventId: `bench-${index}-${sourceSeq}`, eventType: "item.completed", priority: 1 as const,
envelope: { kind: "event", payload: { sourceSeq, eventType: "item.completed", payload: { text: "x".repeat(eventBytes) } } },
deliveryCount: 1, logicalEffectCount: 1 as const };
}
async function startRun(index: number) {
const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-backlog-"));
roots.push(root);
const core = new DurablePrpControlPlane({ stateDirectory: root, identity: identity(index),
expectedRunnerVersion: "0.3.0", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
const working = core.store as unknown as WorkingStore;
const seeded = Array.from({ length: seedEvents }, (_, offset) => benchEvent(index, offset + 1));
working.commit({ ...working.state, ackedSourceSeq: seedEvents, committedEvents: seeded });
let lastRevision = -1;
core.setCheckpointPersistence(async () => {
writesStarted += 1;
await pause(writeMs);
});
const flush = core.flushCheckpoint.bind(core);
core.flushCheckpoint = () => {
if (working.revision !== lastRevision) { lastRevision = working.revision; revisionsRequested += 1; }
return flush();
};
registry.register(core, index);
return { core, working };
}
async function driveEvents(run: Awaited<ReturnType<typeof startRun>>, until: number, index: number) {
let sourceSeq = run.working.state.ackedSourceSeq;
while (Date.now() < until) {
sourceSeq += 1;
const state = run.working.state;
run.working.commit({ ...state, ackedSourceSeq: sourceSeq,
committedEvents: [...state.committedEvents, benchEvent(index, sourceSeq)].slice(-4096) });
await Promise.all([run.core.flushCheckpoint(), pause(1000 / eventsPerSecond)]);
}
}
async function driveCommands(run: Awaited<ReturnType<typeof startRun>>, until: number, index: number) {
const pending: Promise<void>[] = [];
for (let count = 0; commandsPerSecond > 0 && Date.now() < until && count < 400; count++) {
run.core.queueCommand("session.snapshot", {}, `bench-command-${index}-${count}`);
pending.push(run.core.flushCheckpoint());
await pause(1000 / commandsPerSecond);
}
await Promise.all(pending);
}
let active = await Promise.all(Array.from({ length: runs }, (_, index) => startRun(index)));
const until = Date.now() + seconds * 1000;
const driving = Promise.all(active.flatMap((run, index) => [driveEvents(run, until, index), driveCommands(run, until, index)]));
await pause(seconds * 500);
const midRun = await settledHeap();
await driving;
await Promise.all(active.map(run => run.core.flushCheckpoint()));
const live = await settledHeap();
const committedEvents = active[0]!.working.state.committedEvents.length;
await Promise.all(active.map(run => run.core.stop()));
active = [];
for (let attempt = 0; attempt < 40 && collected.count < runs; attempt++) {
await settledHeap();
await pause(25);
}
const finished = await settledHeap();
for (const root of roots) rmSync(root, { recursive: true, force: true });
const kilobytes = (bytes: number) => Math.round(bytes / runs / 1024);
console.log(JSON.stringify({
settings: { runs, seconds, eventsPerSecond, commandsPerSecond, writeMs, eventBytes, seedEvents },
newEventsCommittedPerRun: committedEvents - seedEvents, revisionsFlushed: revisionsRequested, checkpointWrites: writesStarted,
midRunRetainedKBPerRun: kilobytes(midRun - baseline), idleLiveRetainedKBPerRun: kilobytes(live - baseline),
finishedRetainedKBPerRun: kilobytes(finished - baseline), controlPlanesCollected: `${collected.count}/${runs}`,
}));