mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(runner): keep at most one checkpoint waiting behind a slow write (#21411)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner control plane saves a checkpoint of each run's
controller state before it acknowledges an event or sends a command
> - Each new state revision captured a full snapshot and queued it
behind the write in progress
> - Inbound events wait for their own checkpoint, so they do not pile
up. Commands that the server queues do not wait, so they can
> - When the database or the event loop is slow, the queue of full
snapshots grows without limit, and events wait behind the whole queue
> - This pull request keeps at most one snapshot waiting per run. A
newer flush replaces the waiting snapshot of the same run
> - The benefit is bounded memory and fewer database writes under slow
persistence, with the same durability guarantee
## Linked Issues or Issue Description
**What happened?**
A hosted server ran out of JavaScript heap at intervals of 47 to 105
minutes. A retained-memory sample attributed about 16% of the memory
that stayed alive to `captureCheckpoint` (from `flushCheckpoint`) and to
`#capturePublishedState` in
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`.
**Expected behavior**
Memory for checkpoints stays proportional to the number of live runs,
not to the number of state revisions that wait for a slow write.
**Steps to reproduce**
1. Create control planes with checkpoint persistence that takes 1 second
per write.
2. Commit events at 13 per second and queue commands at 2 per second for
each run.
3. Measure the heap in the middle of the run.
`node --import ./server/node_modules/tsx/dist/loader.mjs
scripts/benchmark-runner-checkpoint-backlog.ts` does these steps.
**Paperclip version or commit**
`master` at 2659ff1d22.
## What Changed
- `DurablePrpControlPlane.flushCheckpoint` keeps one snapshot that waits
for the write in progress. A newer flush for the same run replaces that
snapshot and returns the same promise. A flush for a different run (a
warm handoff) still queues its own write.
- New tests in `durable-prp-control-plane.test.ts`:
- 40 commands queued behind a slow write produce one more write, not 40.
Each caller resolves only after the stored records contain its command.
The stored records decode to the last written checkpoint.
- When the combined write fails, every caller that waited on it rejects,
and the next flush writes a full reset that contains the commands.
- New `scripts/benchmark-runner-checkpoint-backlog.ts` measures retained
heap per run, checkpoint writes, and whether finished control planes are
collected.
Why this is safe:
- A waiting snapshot has not started its write. No caller has received a
result for it, so no acknowledgement or command send depends on it.
- The replacement is captured from a later state of the same store. It
contains every change that the replaced snapshot contained.
- Every caller of the replaced snapshot waits on the same promise. The
promise resolves only after the newer snapshot is durable. A failure
rejects all of them, and the existing retry writes a full reset.
- Writes stay on one chain, in order. The server fence, the encryption,
and the stored checkpoint format do not change.
- A flush for a different run ID never replaces a waiting snapshot, so
the earlier run still gets its last state.
## Verification
- Red/green: the two new tests fail on `master` (42 writes instead of 3;
the waiting callers resolve instead of rejecting) and pass with the fix.
- `npx vitest run src/control-plane/durable-prp-control-plane.test.ts`
in `packages/paperclip-runner`: 219 of 219 pass.
- `node scripts/check-module-boundaries.mjs` passes.
- Benchmark, 8 runs, 30 s, 13 events/s and 2 commands/s per run, 1 s
writes, 300 events of 2 KB already in each run:
| | `master` | this PR |
| --- | --- | --- |
| Retained heap per run in the middle of the load | 15,009 KB | 2,447 KB
|
| Retained heap per idle live run | 1,296 KB | 1,415 KB |
| Checkpoint writes | 515 (one per revision) | 248 (for 720 revisions) |
| New events committed per run in 30 s | 4 | 30 |
| Control planes collected after stop | 8 of 8 | 8 of 8 |
- With commands at 0 per second, `master` has no backlog: an inbound
event waits for its own checkpoint before the next frame is read. The
backlog comes from flushes that do not wait.
## Risks
- Behavioral shift: when a combined write fails, callers whose own
earlier revision would have been written alone now see that failure. A
checkpoint failure already disconnects the runner and the next flush
writes a full reset, so recovery is the same.
- Fewer intermediate states reach the database. A crash can only lose a
state that no caller has seen as saved, which is the same as before.
- Not covered by a test: a warm handoff whose run ID changes while a
snapshot waits. The code does not combine those; the existing
warm-transition tests pass.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
85f4a91e70
commit
594f9d3ec2
3 files changed
+260
No files matched your search
@@ -787,6 +787,134 @@ it.each([false, true])("recovers checkpoint persistence and the same connection
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps one waiting checkpoint behind a slow write and saves every queued command before resolving", async () => {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-coalesce-"));
|
||||
const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest });
|
||||
const stored = new Map<string, string>();
|
||||
const writes: DurablePrpCheckpoint[] = [];
|
||||
let release!: () => void;
|
||||
let gate: Promise<void> = Promise.resolve();
|
||||
core.setCheckpointPersistence(async snapshot => {
|
||||
writes.push(snapshot);
|
||||
await gate;
|
||||
if (snapshot.reset) stored.clear();
|
||||
for (const key of snapshot.removedRecords) stored.delete(key);
|
||||
for (const [key, value] of snapshot.changedRecords) stored.set(key, value);
|
||||
});
|
||||
try {
|
||||
await core.flushCheckpoint();
|
||||
gate = new Promise<void>(resolveWrite => { release = resolveWrite; });
|
||||
core.queueCommand("session.snapshot", {}, "in-flight-command");
|
||||
const inFlight = core.flushCheckpoint();
|
||||
await vi.waitFor(() => expect(writes).toHaveLength(2));
|
||||
const settled: string[] = [];
|
||||
const waiting = Array.from({ length: 40 }, (_, index) => {
|
||||
const commandId = `waiting-command-${index}`;
|
||||
core.queueCommand("session.snapshot", {}, commandId);
|
||||
return core.flushCheckpoint().then(() => {
|
||||
expect([...stored.values()].join("\n")).toContain(commandId);
|
||||
settled.push(commandId);
|
||||
});
|
||||
});
|
||||
await new Promise<void>(resolveTurn => setImmediate(resolveTurn));
|
||||
expect(writes).toHaveLength(2);
|
||||
expect(settled).toEqual([]);
|
||||
release();
|
||||
await inFlight;
|
||||
await Promise.all(waiting);
|
||||
expect(writes).toHaveLength(3);
|
||||
expect(settled).toHaveLength(40);
|
||||
expect(stored).toEqual(writes.at(-1)!.records);
|
||||
expect(decodeDurablePrpCheckpoint(stored).records).toEqual(writes.at(-1)!.records);
|
||||
await core.flushCheckpoint();
|
||||
expect(writes).toHaveLength(3);
|
||||
} finally {
|
||||
release?.();
|
||||
await core.stop();
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("writes a waiting checkpoint for each run in order when the run changes behind a slow write", async () => {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-coalesce-handoff-"));
|
||||
const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" };
|
||||
const template = { provider: { provider: "codex", command: "fixture" }, authorizedTools: {} };
|
||||
const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest });
|
||||
const writes: DurablePrpCheckpoint[] = [];
|
||||
let release!: () => void;
|
||||
let gate: Promise<void> = Promise.resolve();
|
||||
core.setCheckpointPersistence(async snapshot => { writes.push(snapshot); await gate; });
|
||||
const contents = (checkpoint: DurablePrpCheckpoint) => [...checkpoint.records.values()].join("\n");
|
||||
try {
|
||||
await core.flushCheckpoint();
|
||||
gate = new Promise<void>(resolveWrite => { release = resolveWrite; });
|
||||
core.queueCommand("session.snapshot", {}, "source-in-flight");
|
||||
const inFlight = core.flushCheckpoint();
|
||||
await vi.waitFor(() => expect(writes).toHaveLength(2));
|
||||
core.queueCommand("session.snapshot", {}, "source-waiting");
|
||||
const sourceWaiting = core.flushCheckpoint();
|
||||
core.stageRunAttachment(target, template);
|
||||
const targetWaiting = core.flushCheckpoint();
|
||||
core.queueCommand("session.snapshot", {}, "target-later");
|
||||
const targetLater = core.flushCheckpoint();
|
||||
expect(targetLater).toBe(targetWaiting);
|
||||
expect(sourceWaiting).not.toBe(targetWaiting);
|
||||
release();
|
||||
await Promise.all([inFlight, sourceWaiting, targetWaiting, targetLater]);
|
||||
expect(writes.map(write => write.runId)).toEqual([identity.runId, identity.runId, identity.runId, target.runId]);
|
||||
expect(contents(writes[2]!)).toContain("source-waiting");
|
||||
expect(contents(writes[2]!)).not.toContain("target-later");
|
||||
expect(writes[2]!.records.has("staged-attachment-template")).toBe(false);
|
||||
expect(writes[3]).toMatchObject({ reset: true, predecessor: { runId: identity.runId } });
|
||||
expect(contents(writes[3]!)).toContain("target-later");
|
||||
expect(writes[3]!.records.has("staged-attachment-template")).toBe(true);
|
||||
await core.flushCheckpoint();
|
||||
expect(writes).toHaveLength(4);
|
||||
} finally {
|
||||
release?.();
|
||||
await core.stop();
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects every caller waiting on a coalesced checkpoint whose write fails", async () => {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-coalesce-failure-"));
|
||||
const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest });
|
||||
const writes: DurablePrpCheckpoint[] = [];
|
||||
let release!: () => void;
|
||||
let gate: Promise<void> = Promise.resolve();
|
||||
let fail = true;
|
||||
core.setCheckpointPersistence(async snapshot => {
|
||||
writes.push(snapshot);
|
||||
await gate;
|
||||
if (fail && [...snapshot.records.values()].join("\n").includes("second-waiting")) throw new Error("coalesced write failed");
|
||||
});
|
||||
try {
|
||||
await core.flushCheckpoint();
|
||||
gate = new Promise<void>(resolveWrite => { release = resolveWrite; });
|
||||
core.queueCommand("session.snapshot", {}, "in-flight-command");
|
||||
const inFlight = core.flushCheckpoint();
|
||||
await vi.waitFor(() => expect(writes).toHaveLength(2));
|
||||
const waiting = ["first-waiting", "second-waiting"].map(commandId => {
|
||||
core.queueCommand("session.snapshot", {}, commandId);
|
||||
return core.flushCheckpoint();
|
||||
});
|
||||
const rejected = waiting.map(promise => expect(promise).rejects.toThrow("coalesced write failed"));
|
||||
release();
|
||||
await inFlight;
|
||||
await Promise.all(rejected);
|
||||
expect(writes).toHaveLength(3);
|
||||
fail = false;
|
||||
await core.flushCheckpoint();
|
||||
expect(writes.at(-1)!.reset).toBe(true);
|
||||
expect([...writes.at(-1)!.records.values()].join("\n")).toContain("second-waiting");
|
||||
} finally {
|
||||
release?.();
|
||||
await core.stop();
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("rebuilds a queued checkpoint after the preceding write fails", async () => {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-queued-retry-"));
|
||||
const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest });
|
||||
|
||||
@@ -2131,6 +2131,7 @@ export class DurablePrpControlPlane {
|
||||
#visibleStateCache?: { working: StoredCoreState; published: StoredCoreState; revision: number; state: StoredCoreState };
|
||||
#checkpointPersistenceFailed = false;
|
||||
#persistenceTail = Promise.resolve();
|
||||
#waitingCheckpoint: { snapshot: DurablePrpCheckpoint; published: StoredCoreState; revision: number } | null = null;
|
||||
#sendQueues = new Map<AuthorityConnection, Promise<void>>();
|
||||
#recoveredCursor: { runId: string; sourceSeq: number } | null = null;
|
||||
#readableStateListeners = new Set<() => void>();
|
||||
@@ -2297,8 +2298,19 @@ export class DurablePrpControlPlane {
|
||||
const published = this.#capturePublishedState(snapshot);
|
||||
const revision = this.#store.revision;
|
||||
this.#queuedRevision = revision;
|
||||
const waiting = this.#waitingCheckpoint;
|
||||
if (waiting !== null && waiting.snapshot.runId === snapshot.runId) {
|
||||
waiting.snapshot = snapshot;
|
||||
waiting.published = published;
|
||||
waiting.revision = revision;
|
||||
return this.#persistenceTail;
|
||||
}
|
||||
const job = { snapshot, published, revision };
|
||||
this.#waitingCheckpoint = job;
|
||||
const persist = this.#persistCheckpoint;
|
||||
const pending = this.#persistenceTail.catch(() => undefined).then(async () => {
|
||||
if (this.#waitingCheckpoint === job) this.#waitingCheckpoint = null;
|
||||
const { snapshot, published, revision } = job;
|
||||
// The previous write may have failed after committing. Rebuild the delta
|
||||
// at execution time, and reset after uncertainty instead of assuming it failed.
|
||||
const checkpoint = { ...snapshot, ...checkpointChanges(snapshot.records, snapshot.runId,
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { resolve } from "node:path";
|
||||
import { setFlagsFromString } from "node:v8";
|
||||
import { runInNewContext } from "node:vm";
|
||||
import { DurablePrpControlPlane } from "../packages/paperclip-runner/src/control-plane/durable-prp-control-plane.js";
|
||||
|
||||
const numberOption = (name: string, fallback: number) => {
|
||||
const value = process.argv.find(arg => arg.startsWith(`--${name}=`))?.slice(name.length + 3);
|
||||
return value === undefined ? fallback : Number(value);
|
||||
};
|
||||
const runs = numberOption("runs", 8);
|
||||
const seconds = numberOption("seconds", 30);
|
||||
const eventsPerSecond = numberOption("events-per-second", 13);
|
||||
const commandsPerSecond = numberOption("commands-per-second", 2);
|
||||
const writeMs = numberOption("write-ms", 1000);
|
||||
const eventBytes = numberOption("event-bytes", 2048);
|
||||
const seedEvents = numberOption("seed-events", 300);
|
||||
|
||||
setFlagsFromString("--expose-gc");
|
||||
const gc = runInNewContext("gc") as () => void;
|
||||
const pause = (ms: number) => new Promise(resolveTimer => setTimeout(resolveTimer, ms));
|
||||
const settledHeap = async () => {
|
||||
for (let index = 0; index < 6; index++) {
|
||||
gc();
|
||||
await new Promise(resolveTurn => setImmediate(resolveTurn));
|
||||
}
|
||||
return process.memoryUsage().heapUsed;
|
||||
};
|
||||
|
||||
type WorkingStore = {
|
||||
revision: number;
|
||||
state: DurablePrpControlPlane["store"]["state"];
|
||||
commit(candidate: DurablePrpControlPlane["store"]["state"], checkpointRequired?: boolean): void;
|
||||
};
|
||||
const identity = (index: number) => ({
|
||||
runnerInstanceId: `runner-bench-${index}`, environmentLeaseId: `environment-bench-${index}`,
|
||||
runId: `00000000-0000-4000-8000-${String(index + 1).padStart(12, "0")}`,
|
||||
normalizedSessionId: `session-bench-${index}`, turnId: `turn-bench-${index}`, itemId: `item-bench-${index}`,
|
||||
});
|
||||
|
||||
const collected = { count: 0 };
|
||||
const registry = new FinalizationRegistry<number>(() => { collected.count += 1; });
|
||||
const roots: string[] = [];
|
||||
const baseline = await settledHeap();
|
||||
let writesStarted = 0, revisionsRequested = 0;
|
||||
|
||||
function benchEvent(index: number, sourceSeq: number) {
|
||||
return { sourceSeq, sourceEventId: `bench-${index}-${sourceSeq}`, eventType: "item.completed", priority: 1 as const,
|
||||
envelope: { kind: "event", payload: { sourceSeq, eventType: "item.completed", payload: { text: "x".repeat(eventBytes) } } },
|
||||
deliveryCount: 1, logicalEffectCount: 1 as const };
|
||||
}
|
||||
|
||||
async function startRun(index: number) {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-backlog-"));
|
||||
roots.push(root);
|
||||
const core = new DurablePrpControlPlane({ stateDirectory: root, identity: identity(index),
|
||||
expectedRunnerVersion: "0.3.0", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
|
||||
const working = core.store as unknown as WorkingStore;
|
||||
const seeded = Array.from({ length: seedEvents }, (_, offset) => benchEvent(index, offset + 1));
|
||||
working.commit({ ...working.state, ackedSourceSeq: seedEvents, committedEvents: seeded });
|
||||
let lastRevision = -1;
|
||||
core.setCheckpointPersistence(async () => {
|
||||
writesStarted += 1;
|
||||
await pause(writeMs);
|
||||
});
|
||||
const flush = core.flushCheckpoint.bind(core);
|
||||
core.flushCheckpoint = () => {
|
||||
if (working.revision !== lastRevision) { lastRevision = working.revision; revisionsRequested += 1; }
|
||||
return flush();
|
||||
};
|
||||
registry.register(core, index);
|
||||
return { core, working };
|
||||
}
|
||||
|
||||
async function driveEvents(run: Awaited<ReturnType<typeof startRun>>, until: number, index: number) {
|
||||
let sourceSeq = run.working.state.ackedSourceSeq;
|
||||
while (Date.now() < until) {
|
||||
sourceSeq += 1;
|
||||
const state = run.working.state;
|
||||
run.working.commit({ ...state, ackedSourceSeq: sourceSeq,
|
||||
committedEvents: [...state.committedEvents, benchEvent(index, sourceSeq)].slice(-4096) });
|
||||
await Promise.all([run.core.flushCheckpoint(), pause(1000 / eventsPerSecond)]);
|
||||
}
|
||||
}
|
||||
|
||||
async function driveCommands(run: Awaited<ReturnType<typeof startRun>>, until: number, index: number) {
|
||||
const pending: Promise<void>[] = [];
|
||||
for (let count = 0; commandsPerSecond > 0 && Date.now() < until && count < 400; count++) {
|
||||
run.core.queueCommand("session.snapshot", {}, `bench-command-${index}-${count}`);
|
||||
pending.push(run.core.flushCheckpoint());
|
||||
await pause(1000 / commandsPerSecond);
|
||||
}
|
||||
await Promise.all(pending);
|
||||
}
|
||||
|
||||
let active = await Promise.all(Array.from({ length: runs }, (_, index) => startRun(index)));
|
||||
const until = Date.now() + seconds * 1000;
|
||||
const driving = Promise.all(active.flatMap((run, index) => [driveEvents(run, until, index), driveCommands(run, until, index)]));
|
||||
await pause(seconds * 500);
|
||||
const midRun = await settledHeap();
|
||||
await driving;
|
||||
await Promise.all(active.map(run => run.core.flushCheckpoint()));
|
||||
const live = await settledHeap();
|
||||
const committedEvents = active[0]!.working.state.committedEvents.length;
|
||||
await Promise.all(active.map(run => run.core.stop()));
|
||||
active = [];
|
||||
for (let attempt = 0; attempt < 40 && collected.count < runs; attempt++) {
|
||||
await settledHeap();
|
||||
await pause(25);
|
||||
}
|
||||
const finished = await settledHeap();
|
||||
for (const root of roots) rmSync(root, { recursive: true, force: true });
|
||||
const kilobytes = (bytes: number) => Math.round(bytes / runs / 1024);
|
||||
console.log(JSON.stringify({
|
||||
settings: { runs, seconds, eventsPerSecond, commandsPerSecond, writeMs, eventBytes, seedEvents },
|
||||
newEventsCommittedPerRun: committedEvents - seedEvents, revisionsFlushed: revisionsRequested, checkpointWrites: writesStarted,
|
||||
midRunRetainedKBPerRun: kilobytes(midRun - baseline), idleLiveRetainedKBPerRun: kilobytes(live - baseline),
|
||||
finishedRetainedKBPerRun: kilobytes(finished - baseline), controlPlanesCollected: `${collected.count}/${runs}`,
|
||||
}));
|
||||
Reference in new issue
Block a user