From 594f9d3ec2120fb8f14d513a4b640131fe5baab1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 11:08:15 -0600 Subject: [PATCH] perf(runner): keep at most one checkpoint waiting behind a slow write (#21411) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native Runner control plane saves a checkpoint of each run's controller state before it acknowledges an event or sends a command > - Each new state revision captured a full snapshot and queued it behind the write in progress > - Inbound events wait for their own checkpoint, so they do not pile up. Commands that the server queues do not wait, so they can > - When the database or the event loop is slow, the queue of full snapshots grows without limit, and events wait behind the whole queue > - This pull request keeps at most one snapshot waiting per run. A newer flush replaces the waiting snapshot of the same run > - The benefit is bounded memory and fewer database writes under slow persistence, with the same durability guarantee ## Linked Issues or Issue Description **What happened?** A hosted server ran out of JavaScript heap at intervals of 47 to 105 minutes. A retained-memory sample attributed about 16% of the memory that stayed alive to `captureCheckpoint` (from `flushCheckpoint`) and to `#capturePublishedState` in `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`. **Expected behavior** Memory for checkpoints stays proportional to the number of live runs, not to the number of state revisions that wait for a slow write. **Steps to reproduce** 1. Create control planes with checkpoint persistence that takes 1 second per write. 2. Commit events at 13 per second and queue commands at 2 per second for each run. 3. Measure the heap in the middle of the run. `node --import ./server/node_modules/tsx/dist/loader.mjs scripts/benchmark-runner-checkpoint-backlog.ts` does these steps. **Paperclip version or commit** `master` at 2659ff1d22. ## What Changed - `DurablePrpControlPlane.flushCheckpoint` keeps one snapshot that waits for the write in progress. A newer flush for the same run replaces that snapshot and returns the same promise. A flush for a different run (a warm handoff) still queues its own write. - New tests in `durable-prp-control-plane.test.ts`: - 40 commands queued behind a slow write produce one more write, not 40. Each caller resolves only after the stored records contain its command. The stored records decode to the last written checkpoint. - When the combined write fails, every caller that waited on it rejects, and the next flush writes a full reset that contains the commands. - New `scripts/benchmark-runner-checkpoint-backlog.ts` measures retained heap per run, checkpoint writes, and whether finished control planes are collected. Why this is safe: - A waiting snapshot has not started its write. No caller has received a result for it, so no acknowledgement or command send depends on it. - The replacement is captured from a later state of the same store. It contains every change that the replaced snapshot contained. - Every caller of the replaced snapshot waits on the same promise. The promise resolves only after the newer snapshot is durable. A failure rejects all of them, and the existing retry writes a full reset. - Writes stay on one chain, in order. The server fence, the encryption, and the stored checkpoint format do not change. - A flush for a different run ID never replaces a waiting snapshot, so the earlier run still gets its last state. ## Verification - Red/green: the two new tests fail on `master` (42 writes instead of 3; the waiting callers resolve instead of rejecting) and pass with the fix. - `npx vitest run src/control-plane/durable-prp-control-plane.test.ts` in `packages/paperclip-runner`: 219 of 219 pass. - `node scripts/check-module-boundaries.mjs` passes. - Benchmark, 8 runs, 30 s, 13 events/s and 2 commands/s per run, 1 s writes, 300 events of 2 KB already in each run: | | `master` | this PR | | --- | --- | --- | | Retained heap per run in the middle of the load | 15,009 KB | 2,447 KB | | Retained heap per idle live run | 1,296 KB | 1,415 KB | | Checkpoint writes | 515 (one per revision) | 248 (for 720 revisions) | | New events committed per run in 30 s | 4 | 30 | | Control planes collected after stop | 8 of 8 | 8 of 8 | - With commands at 0 per second, `master` has no backlog: an inbound event waits for its own checkpoint before the next frame is read. The backlog comes from flushes that do not wait. ## Risks - Behavioral shift: when a combined write fails, callers whose own earlier revision would have been written alone now see that failure. A checkpoint failure already disconnects the runner and the next flush writes a full reset, so recovery is the same. - Fewer intermediate states reach the database. A crash can only lose a state that no caller has seen as saved, which is the same as before. - Not covered by a test: a warm handoff whose run ID changes while a snapshot waits. The code does not combine those; the existing warm-transition tests pass. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../durable-prp-control-plane.test.ts | 128 ++++++++++++++++++ .../durable-prp-control-plane.ts | 12 ++ .../benchmark-runner-checkpoint-backlog.ts | 120 ++++++++++++++++ 3 files changed, 260 insertions(+) create mode 100644 scripts/benchmark-runner-checkpoint-backlog.ts diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts index 918779e5261..bd4f062de21 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts @@ -787,6 +787,134 @@ it.each([false, true])("recovers checkpoint persistence and the same connection } }); +it("keeps one waiting checkpoint behind a slow write and saves every queued command before resolving", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-coalesce-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + const stored = new Map(); + const writes: DurablePrpCheckpoint[] = []; + let release!: () => void; + let gate: Promise = Promise.resolve(); + core.setCheckpointPersistence(async snapshot => { + writes.push(snapshot); + await gate; + if (snapshot.reset) stored.clear(); + for (const key of snapshot.removedRecords) stored.delete(key); + for (const [key, value] of snapshot.changedRecords) stored.set(key, value); + }); + try { + await core.flushCheckpoint(); + gate = new Promise(resolveWrite => { release = resolveWrite; }); + core.queueCommand("session.snapshot", {}, "in-flight-command"); + const inFlight = core.flushCheckpoint(); + await vi.waitFor(() => expect(writes).toHaveLength(2)); + const settled: string[] = []; + const waiting = Array.from({ length: 40 }, (_, index) => { + const commandId = `waiting-command-${index}`; + core.queueCommand("session.snapshot", {}, commandId); + return core.flushCheckpoint().then(() => { + expect([...stored.values()].join("\n")).toContain(commandId); + settled.push(commandId); + }); + }); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + expect(writes).toHaveLength(2); + expect(settled).toEqual([]); + release(); + await inFlight; + await Promise.all(waiting); + expect(writes).toHaveLength(3); + expect(settled).toHaveLength(40); + expect(stored).toEqual(writes.at(-1)!.records); + expect(decodeDurablePrpCheckpoint(stored).records).toEqual(writes.at(-1)!.records); + await core.flushCheckpoint(); + expect(writes).toHaveLength(3); + } finally { + release?.(); + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it("writes a waiting checkpoint for each run in order when the run changes behind a slow write", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-coalesce-handoff-")); + const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" }; + const template = { provider: { provider: "codex", command: "fixture" }, authorizedTools: {} }; + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + const writes: DurablePrpCheckpoint[] = []; + let release!: () => void; + let gate: Promise = Promise.resolve(); + core.setCheckpointPersistence(async snapshot => { writes.push(snapshot); await gate; }); + const contents = (checkpoint: DurablePrpCheckpoint) => [...checkpoint.records.values()].join("\n"); + try { + await core.flushCheckpoint(); + gate = new Promise(resolveWrite => { release = resolveWrite; }); + core.queueCommand("session.snapshot", {}, "source-in-flight"); + const inFlight = core.flushCheckpoint(); + await vi.waitFor(() => expect(writes).toHaveLength(2)); + core.queueCommand("session.snapshot", {}, "source-waiting"); + const sourceWaiting = core.flushCheckpoint(); + core.stageRunAttachment(target, template); + const targetWaiting = core.flushCheckpoint(); + core.queueCommand("session.snapshot", {}, "target-later"); + const targetLater = core.flushCheckpoint(); + expect(targetLater).toBe(targetWaiting); + expect(sourceWaiting).not.toBe(targetWaiting); + release(); + await Promise.all([inFlight, sourceWaiting, targetWaiting, targetLater]); + expect(writes.map(write => write.runId)).toEqual([identity.runId, identity.runId, identity.runId, target.runId]); + expect(contents(writes[2]!)).toContain("source-waiting"); + expect(contents(writes[2]!)).not.toContain("target-later"); + expect(writes[2]!.records.has("staged-attachment-template")).toBe(false); + expect(writes[3]).toMatchObject({ reset: true, predecessor: { runId: identity.runId } }); + expect(contents(writes[3]!)).toContain("target-later"); + expect(writes[3]!.records.has("staged-attachment-template")).toBe(true); + await core.flushCheckpoint(); + expect(writes).toHaveLength(4); + } finally { + release?.(); + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it("rejects every caller waiting on a coalesced checkpoint whose write fails", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-coalesce-failure-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + const writes: DurablePrpCheckpoint[] = []; + let release!: () => void; + let gate: Promise = Promise.resolve(); + let fail = true; + core.setCheckpointPersistence(async snapshot => { + writes.push(snapshot); + await gate; + if (fail && [...snapshot.records.values()].join("\n").includes("second-waiting")) throw new Error("coalesced write failed"); + }); + try { + await core.flushCheckpoint(); + gate = new Promise(resolveWrite => { release = resolveWrite; }); + core.queueCommand("session.snapshot", {}, "in-flight-command"); + const inFlight = core.flushCheckpoint(); + await vi.waitFor(() => expect(writes).toHaveLength(2)); + const waiting = ["first-waiting", "second-waiting"].map(commandId => { + core.queueCommand("session.snapshot", {}, commandId); + return core.flushCheckpoint(); + }); + const rejected = waiting.map(promise => expect(promise).rejects.toThrow("coalesced write failed")); + release(); + await inFlight; + await Promise.all(rejected); + expect(writes).toHaveLength(3); + fail = false; + await core.flushCheckpoint(); + expect(writes.at(-1)!.reset).toBe(true); + expect([...writes.at(-1)!.records.values()].join("\n")).toContain("second-waiting"); + } finally { + release?.(); + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + it("rebuilds a queued checkpoint after the preceding write fails", async () => { const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-queued-retry-")); const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index 6f73fe23383..8e75bf1ff8c 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -2131,6 +2131,7 @@ export class DurablePrpControlPlane { #visibleStateCache?: { working: StoredCoreState; published: StoredCoreState; revision: number; state: StoredCoreState }; #checkpointPersistenceFailed = false; #persistenceTail = Promise.resolve(); + #waitingCheckpoint: { snapshot: DurablePrpCheckpoint; published: StoredCoreState; revision: number } | null = null; #sendQueues = new Map>(); #recoveredCursor: { runId: string; sourceSeq: number } | null = null; #readableStateListeners = new Set<() => void>(); @@ -2297,8 +2298,19 @@ export class DurablePrpControlPlane { const published = this.#capturePublishedState(snapshot); const revision = this.#store.revision; this.#queuedRevision = revision; + const waiting = this.#waitingCheckpoint; + if (waiting !== null && waiting.snapshot.runId === snapshot.runId) { + waiting.snapshot = snapshot; + waiting.published = published; + waiting.revision = revision; + return this.#persistenceTail; + } + const job = { snapshot, published, revision }; + this.#waitingCheckpoint = job; const persist = this.#persistCheckpoint; const pending = this.#persistenceTail.catch(() => undefined).then(async () => { + if (this.#waitingCheckpoint === job) this.#waitingCheckpoint = null; + const { snapshot, published, revision } = job; // The previous write may have failed after committing. Rebuild the delta // at execution time, and reset after uncertainty instead of assuming it failed. const checkpoint = { ...snapshot, ...checkpointChanges(snapshot.records, snapshot.runId, diff --git a/scripts/benchmark-runner-checkpoint-backlog.ts b/scripts/benchmark-runner-checkpoint-backlog.ts new file mode 100644 index 00000000000..3323a7fb089 --- /dev/null +++ b/scripts/benchmark-runner-checkpoint-backlog.ts @@ -0,0 +1,120 @@ +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { resolve } from "node:path"; +import { setFlagsFromString } from "node:v8"; +import { runInNewContext } from "node:vm"; +import { DurablePrpControlPlane } from "../packages/paperclip-runner/src/control-plane/durable-prp-control-plane.js"; + +const numberOption = (name: string, fallback: number) => { + const value = process.argv.find(arg => arg.startsWith(`--${name}=`))?.slice(name.length + 3); + return value === undefined ? fallback : Number(value); +}; +const runs = numberOption("runs", 8); +const seconds = numberOption("seconds", 30); +const eventsPerSecond = numberOption("events-per-second", 13); +const commandsPerSecond = numberOption("commands-per-second", 2); +const writeMs = numberOption("write-ms", 1000); +const eventBytes = numberOption("event-bytes", 2048); +const seedEvents = numberOption("seed-events", 300); + +setFlagsFromString("--expose-gc"); +const gc = runInNewContext("gc") as () => void; +const pause = (ms: number) => new Promise(resolveTimer => setTimeout(resolveTimer, ms)); +const settledHeap = async () => { + for (let index = 0; index < 6; index++) { + gc(); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + } + return process.memoryUsage().heapUsed; +}; + +type WorkingStore = { + revision: number; + state: DurablePrpControlPlane["store"]["state"]; + commit(candidate: DurablePrpControlPlane["store"]["state"], checkpointRequired?: boolean): void; +}; +const identity = (index: number) => ({ + runnerInstanceId: `runner-bench-${index}`, environmentLeaseId: `environment-bench-${index}`, + runId: `00000000-0000-4000-8000-${String(index + 1).padStart(12, "0")}`, + normalizedSessionId: `session-bench-${index}`, turnId: `turn-bench-${index}`, itemId: `item-bench-${index}`, +}); + +const collected = { count: 0 }; +const registry = new FinalizationRegistry(() => { collected.count += 1; }); +const roots: string[] = []; +const baseline = await settledHeap(); +let writesStarted = 0, revisionsRequested = 0; + +function benchEvent(index: number, sourceSeq: number) { + return { sourceSeq, sourceEventId: `bench-${index}-${sourceSeq}`, eventType: "item.completed", priority: 1 as const, + envelope: { kind: "event", payload: { sourceSeq, eventType: "item.completed", payload: { text: "x".repeat(eventBytes) } } }, + deliveryCount: 1, logicalEffectCount: 1 as const }; +} + +async function startRun(index: number) { + const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-backlog-")); + roots.push(root); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity: identity(index), + expectedRunnerVersion: "0.3.0", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + const working = core.store as unknown as WorkingStore; + const seeded = Array.from({ length: seedEvents }, (_, offset) => benchEvent(index, offset + 1)); + working.commit({ ...working.state, ackedSourceSeq: seedEvents, committedEvents: seeded }); + let lastRevision = -1; + core.setCheckpointPersistence(async () => { + writesStarted += 1; + await pause(writeMs); + }); + const flush = core.flushCheckpoint.bind(core); + core.flushCheckpoint = () => { + if (working.revision !== lastRevision) { lastRevision = working.revision; revisionsRequested += 1; } + return flush(); + }; + registry.register(core, index); + return { core, working }; +} + +async function driveEvents(run: Awaited>, until: number, index: number) { + let sourceSeq = run.working.state.ackedSourceSeq; + while (Date.now() < until) { + sourceSeq += 1; + const state = run.working.state; + run.working.commit({ ...state, ackedSourceSeq: sourceSeq, + committedEvents: [...state.committedEvents, benchEvent(index, sourceSeq)].slice(-4096) }); + await Promise.all([run.core.flushCheckpoint(), pause(1000 / eventsPerSecond)]); + } +} + +async function driveCommands(run: Awaited>, until: number, index: number) { + const pending: Promise[] = []; + for (let count = 0; commandsPerSecond > 0 && Date.now() < until && count < 400; count++) { + run.core.queueCommand("session.snapshot", {}, `bench-command-${index}-${count}`); + pending.push(run.core.flushCheckpoint()); + await pause(1000 / commandsPerSecond); + } + await Promise.all(pending); +} + +let active = await Promise.all(Array.from({ length: runs }, (_, index) => startRun(index))); +const until = Date.now() + seconds * 1000; +const driving = Promise.all(active.flatMap((run, index) => [driveEvents(run, until, index), driveCommands(run, until, index)])); +await pause(seconds * 500); +const midRun = await settledHeap(); +await driving; +await Promise.all(active.map(run => run.core.flushCheckpoint())); +const live = await settledHeap(); +const committedEvents = active[0]!.working.state.committedEvents.length; +await Promise.all(active.map(run => run.core.stop())); +active = []; +for (let attempt = 0; attempt < 40 && collected.count < runs; attempt++) { + await settledHeap(); + await pause(25); +} +const finished = await settledHeap(); +for (const root of roots) rmSync(root, { recursive: true, force: true }); +const kilobytes = (bytes: number) => Math.round(bytes / runs / 1024); +console.log(JSON.stringify({ + settings: { runs, seconds, eventsPerSecond, commandsPerSecond, writeMs, eventBytes, seedEvents }, + newEventsCommittedPerRun: committedEvents - seedEvents, revisionsFlushed: revisionsRequested, checkpointWrites: writesStarted, + midRunRetainedKBPerRun: kilobytes(midRun - baseline), idleLiveRetainedKBPerRun: kilobytes(live - baseline), + finishedRetainedKBPerRun: kilobytes(finished - baseline), controlPlanesCollected: `${collected.count}/${runs}`, +}));