mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner control plane saves a checkpoint of each run's
controller state before it acknowledges an event or sends a command
> - Each new state revision captured a full snapshot and queued it
behind the write in progress
> - Inbound events wait for their own checkpoint, so they do not pile
up. Commands that the server queues do not wait, so they can
> - When the database or the event loop is slow, the queue of full
snapshots grows without limit, and events wait behind the whole queue
> - This pull request keeps at most one snapshot waiting per run. A
newer flush replaces the waiting snapshot of the same run
> - The benefit is bounded memory and fewer database writes under slow
persistence, with the same durability guarantee
## Linked Issues or Issue Description
**What happened?**
A hosted server ran out of JavaScript heap at intervals of 47 to 105
minutes. A retained-memory sample attributed about 16% of the memory
that stayed alive to `captureCheckpoint` (from `flushCheckpoint`) and to
`#capturePublishedState` in
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`.
**Expected behavior**
Memory for checkpoints stays proportional to the number of live runs,
not to the number of state revisions that wait for a slow write.
**Steps to reproduce**
1. Create control planes with checkpoint persistence that takes 1 second
per write.
2. Commit events at 13 per second and queue commands at 2 per second for
each run.
3. Measure the heap in the middle of the run.
`node --import ./server/node_modules/tsx/dist/loader.mjs
scripts/benchmark-runner-checkpoint-backlog.ts` does these steps.
**Paperclip version or commit**
`master` at 2659ff1d22.
## What Changed
- `DurablePrpControlPlane.flushCheckpoint` keeps one snapshot that waits
for the write in progress. A newer flush for the same run replaces that
snapshot and returns the same promise. A flush for a different run (a
warm handoff) still queues its own write.
- New tests in `durable-prp-control-plane.test.ts`:
- 40 commands queued behind a slow write produce one more write, not 40.
Each caller resolves only after the stored records contain its command.
The stored records decode to the last written checkpoint.
- When the combined write fails, every caller that waited on it rejects,
and the next flush writes a full reset that contains the commands.
- New `scripts/benchmark-runner-checkpoint-backlog.ts` measures retained
heap per run, checkpoint writes, and whether finished control planes are
collected.
Why this is safe:
- A waiting snapshot has not started its write. No caller has received a
result for it, so no acknowledgement or command send depends on it.
- The replacement is captured from a later state of the same store. It
contains every change that the replaced snapshot contained.
- Every caller of the replaced snapshot waits on the same promise. The
promise resolves only after the newer snapshot is durable. A failure
rejects all of them, and the existing retry writes a full reset.
- Writes stay on one chain, in order. The server fence, the encryption,
and the stored checkpoint format do not change.
- A flush for a different run ID never replaces a waiting snapshot, so
the earlier run still gets its last state.
## Verification
- Red/green: the two new tests fail on `master` (42 writes instead of 3;
the waiting callers resolve instead of rejecting) and pass with the fix.
- `npx vitest run src/control-plane/durable-prp-control-plane.test.ts`
in `packages/paperclip-runner`: 219 of 219 pass.
- `node scripts/check-module-boundaries.mjs` passes.
- Benchmark, 8 runs, 30 s, 13 events/s and 2 commands/s per run, 1 s
writes, 300 events of 2 KB already in each run:
| | `master` | this PR |
| --- | --- | --- |
| Retained heap per run in the middle of the load | 15,009 KB | 2,447 KB
|
| Retained heap per idle live run | 1,296 KB | 1,415 KB |
| Checkpoint writes | 515 (one per revision) | 248 (for 720 revisions) |
| New events committed per run in 30 s | 4 | 30 |
| Control planes collected after stop | 8 of 8 | 8 of 8 |
- With commands at 0 per second, `master` has no backlog: an inbound
event waits for its own checkpoint before the next frame is read. The
backlog comes from flushes that do not wait.
## Risks
- Behavioral shift: when a combined write fails, callers whose own
earlier revision would have been written alone now see that failure. A
checkpoint failure already disconnects the runner and the next flush
writes a full reset, so recovery is the same.
- Fewer intermediate states reach the database. A crash can only lose a
state that no caller has seen as saved, which is the same as before.
- Not covered by a test: a warm handoff whose run ID changes while a
snapshot waits. The code does not combine those; the existing
warm-transition tests pass.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
121 lines
5.5 KiB
TypeScript
121 lines
5.5 KiB
TypeScript
import { mkdtempSync, rmSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import { resolve } from "node:path";
|
|
import { setFlagsFromString } from "node:v8";
|
|
import { runInNewContext } from "node:vm";
|
|
import { DurablePrpControlPlane } from "../packages/paperclip-runner/src/control-plane/durable-prp-control-plane.js";
|
|
|
|
const numberOption = (name: string, fallback: number) => {
|
|
const value = process.argv.find(arg => arg.startsWith(`--${name}=`))?.slice(name.length + 3);
|
|
return value === undefined ? fallback : Number(value);
|
|
};
|
|
const runs = numberOption("runs", 8);
|
|
const seconds = numberOption("seconds", 30);
|
|
const eventsPerSecond = numberOption("events-per-second", 13);
|
|
const commandsPerSecond = numberOption("commands-per-second", 2);
|
|
const writeMs = numberOption("write-ms", 1000);
|
|
const eventBytes = numberOption("event-bytes", 2048);
|
|
const seedEvents = numberOption("seed-events", 300);
|
|
|
|
setFlagsFromString("--expose-gc");
|
|
const gc = runInNewContext("gc") as () => void;
|
|
const pause = (ms: number) => new Promise(resolveTimer => setTimeout(resolveTimer, ms));
|
|
const settledHeap = async () => {
|
|
for (let index = 0; index < 6; index++) {
|
|
gc();
|
|
await new Promise(resolveTurn => setImmediate(resolveTurn));
|
|
}
|
|
return process.memoryUsage().heapUsed;
|
|
};
|
|
|
|
type WorkingStore = {
|
|
revision: number;
|
|
state: DurablePrpControlPlane["store"]["state"];
|
|
commit(candidate: DurablePrpControlPlane["store"]["state"], checkpointRequired?: boolean): void;
|
|
};
|
|
const identity = (index: number) => ({
|
|
runnerInstanceId: `runner-bench-${index}`, environmentLeaseId: `environment-bench-${index}`,
|
|
runId: `00000000-0000-4000-8000-${String(index + 1).padStart(12, "0")}`,
|
|
normalizedSessionId: `session-bench-${index}`, turnId: `turn-bench-${index}`, itemId: `item-bench-${index}`,
|
|
});
|
|
|
|
const collected = { count: 0 };
|
|
const registry = new FinalizationRegistry<number>(() => { collected.count += 1; });
|
|
const roots: string[] = [];
|
|
const baseline = await settledHeap();
|
|
let writesStarted = 0, revisionsRequested = 0;
|
|
|
|
function benchEvent(index: number, sourceSeq: number) {
|
|
return { sourceSeq, sourceEventId: `bench-${index}-${sourceSeq}`, eventType: "item.completed", priority: 1 as const,
|
|
envelope: { kind: "event", payload: { sourceSeq, eventType: "item.completed", payload: { text: "x".repeat(eventBytes) } } },
|
|
deliveryCount: 1, logicalEffectCount: 1 as const };
|
|
}
|
|
|
|
async function startRun(index: number) {
|
|
const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-backlog-"));
|
|
roots.push(root);
|
|
const core = new DurablePrpControlPlane({ stateDirectory: root, identity: identity(index),
|
|
expectedRunnerVersion: "0.3.0", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
|
|
const working = core.store as unknown as WorkingStore;
|
|
const seeded = Array.from({ length: seedEvents }, (_, offset) => benchEvent(index, offset + 1));
|
|
working.commit({ ...working.state, ackedSourceSeq: seedEvents, committedEvents: seeded });
|
|
let lastRevision = -1;
|
|
core.setCheckpointPersistence(async () => {
|
|
writesStarted += 1;
|
|
await pause(writeMs);
|
|
});
|
|
const flush = core.flushCheckpoint.bind(core);
|
|
core.flushCheckpoint = () => {
|
|
if (working.revision !== lastRevision) { lastRevision = working.revision; revisionsRequested += 1; }
|
|
return flush();
|
|
};
|
|
registry.register(core, index);
|
|
return { core, working };
|
|
}
|
|
|
|
async function driveEvents(run: Awaited<ReturnType<typeof startRun>>, until: number, index: number) {
|
|
let sourceSeq = run.working.state.ackedSourceSeq;
|
|
while (Date.now() < until) {
|
|
sourceSeq += 1;
|
|
const state = run.working.state;
|
|
run.working.commit({ ...state, ackedSourceSeq: sourceSeq,
|
|
committedEvents: [...state.committedEvents, benchEvent(index, sourceSeq)].slice(-4096) });
|
|
await Promise.all([run.core.flushCheckpoint(), pause(1000 / eventsPerSecond)]);
|
|
}
|
|
}
|
|
|
|
async function driveCommands(run: Awaited<ReturnType<typeof startRun>>, until: number, index: number) {
|
|
const pending: Promise<void>[] = [];
|
|
for (let count = 0; commandsPerSecond > 0 && Date.now() < until && count < 400; count++) {
|
|
run.core.queueCommand("session.snapshot", {}, `bench-command-${index}-${count}`);
|
|
pending.push(run.core.flushCheckpoint());
|
|
await pause(1000 / commandsPerSecond);
|
|
}
|
|
await Promise.all(pending);
|
|
}
|
|
|
|
let active = await Promise.all(Array.from({ length: runs }, (_, index) => startRun(index)));
|
|
const until = Date.now() + seconds * 1000;
|
|
const driving = Promise.all(active.flatMap((run, index) => [driveEvents(run, until, index), driveCommands(run, until, index)]));
|
|
await pause(seconds * 500);
|
|
const midRun = await settledHeap();
|
|
await driving;
|
|
await Promise.all(active.map(run => run.core.flushCheckpoint()));
|
|
const live = await settledHeap();
|
|
const committedEvents = active[0]!.working.state.committedEvents.length;
|
|
await Promise.all(active.map(run => run.core.stop()));
|
|
active = [];
|
|
for (let attempt = 0; attempt < 40 && collected.count < runs; attempt++) {
|
|
await settledHeap();
|
|
await pause(25);
|
|
}
|
|
const finished = await settledHeap();
|
|
for (const root of roots) rmSync(root, { recursive: true, force: true });
|
|
const kilobytes = (bytes: number) => Math.round(bytes / runs / 1024);
|
|
console.log(JSON.stringify({
|
|
settings: { runs, seconds, eventsPerSecond, commandsPerSecond, writeMs, eventBytes, seedEvents },
|
|
newEventsCommittedPerRun: committedEvents - seedEvents, revisionsFlushed: revisionsRequested, checkpointWrites: writesStarted,
|
|
midRunRetainedKBPerRun: kilobytes(midRun - baseline), idleLiveRetainedKBPerRun: kilobytes(live - baseline),
|
|
finishedRetainedKBPerRun: kilobytes(finished - baseline), controlPlanesCollected: `${collected.count}/${runs}`,
|
|
}));
|