Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 351b69941c perf(checkpoint): write a routine controller checkpoint save in one statement (#21553)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote runner's controller saves a checkpoint after each runner
event it commits, on a 3-connection pool, before it acknowledges the
event
> - Each save is a transaction of 4 statements (5 where commit
notifications are on): BEGIN, the owner lock with the transaction id,
the write, COMMIT
> - What forces the transaction is the live-run recovery wake that every
save registers. Recovery does not react to a routine save
> - This pull request writes a routine save (new or changed event,
command and cursor records) with one statement that checks the owner and
writes, and keeps today's transaction for every other save
> - The benefit is one statement instead of four for most checkpoint
saves, on the pool that limits one process in a busy minute

## Linked Issues or Issue Description

Refs #21528 (owner lock returns the transaction id). Refs #21549 (load
model at recorded rates; the plan records the checkpoint pool as a
per-process limit).

**What existing behavior does this improve?**
`nativeRunnerCheckpointWriter` in
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Subsystem affected**
Controller checkpoint persistence for remote native runs.

**Current behavior**
Every save locks the coordinator row (`native_run_finalizations` `for
update`), checks the owner, calls `notifyLiveRunRecovery` (6 recovery
queues), and upserts the changed parts, in one transaction.

**Proposed behavior**
A routine save is one statement: a CTE locks the coordinator row only
while the lease owner, boot id, generation and an unexpired lease
(`clock_timestamp()`) all match, and the upsert writes only through that
row. It registers no recovery wake. If the owner does not match, it
writes nothing and the save takes today's full path, which runs the
complete owner check (successor rules included) and fails with
`native_runner_checkpoint_owner_lost` when it should.

A save is routine when it has changed parts, no reset, no removed parts,
no header change, no predecessor, and the run is its own authority.
Every other save (the first save of a controller, which is a reset, a
save after a failed save, header or retirement changes, removals,
source-run consumers) is unchanged.

**Reason and benefit**
At recorded rates about a third of one process's statements are
checkpoint saves. In a busy minute the 3-connection pool cannot keep up
at 500 agents (#21549).

**Breaking changes**
None. No schema change.

## What Changed

- `ROUTINE_CHECKPOINT_SAVE_SQL` and the routine branch in
`nativeRunnerCheckpointWriter`.

## Why a routine save needs no recovery wake

I read the six consumers of `notifyLiveRunRecovery`:
- `nativeProcessSessions`, `nativeQuestionCancellations`,
`activeExecutionRecovery`, `settledNativeProcesses`
(`live-recovery-work.ts`): each selects candidates from `heartbeat_runs`
status, runtime mode, `native_run_finalizations` phase, lease and retry
times, the context snapshot and process identity. None reads checkpoint
rows to decide eligibility.
- `strandedAgentStatus` (same file, `agents` rows) and
`activeRunSilence` (`active-run-watchdog/adapters/deadline-work.ts`,
`heartbeat_runs.last_output_at`): neither reads checkpoint rows.
- The checkpoint rows matter to recovery in two places:
`listExpiredNativeRunnerAuthorities` requires that a checkpoint exists,
and the remote bootstrap proof reads its parts. A checkpoint first
appears with a reset save, which keeps its wake. A routine save only
adds event and command parts, which can only take a run further from the
bootstrap proof; recovery reads the current rows when its own deadline
(lease expiry) fires.
- The intent of the wake also fenced a recovery pass against a save in
flight. The routine save is one atomic statement and holds the
coordinator row lock while it writes, so a recovery claim that takes
that row waits for it, and a reader sees the save whole or not at all.

## Verification

- Red/green on embedded PostgreSQL
(`native-runner-checkpoint-save.test.ts`); the new tests failed against
`origin/master` first:
- A routine save is one statement (`with owner as ... for update ...
insert into native_runner_checkpoints`) and registers no recovery wake
(before: 4 statements and a wake).
- A reset save keeps BEGIN, the owner lock with the transaction id, and
COMMIT, and still registers its recovery wake.
- A routine save that waits while another connection takes over the
lease (new owner and generation) falls back to the full path and fails
with `native_runner_checkpoint_owner_lost`; the checkpoint is unchanged.
- The same race where the new owner also resets the checkpoint (deletes
every part and writes its own header): the save fails the same way and
the checkpoint holds only the new owner's header.
- A lease that expires while the routine save waits for the coordinator
row lock (the holder does not change the row) writes nothing: the
statement judges the lease after the lock, as
`lockNativeRunnerCheckpointOwner` does. My first version checked the
lease in the locking CTE itself, and PostgreSQL does not re-check a
condition after a lock wait on an unchanged row; the existing
`native-runner-restart-recovery.integration.test.ts` test "fences
checkpoint delivery when ownership expires during an unchanged row-lock
wait" failed on it, and so does the new test. Both pass now.
- The existing test "refuses a save after the owner's lease expired and
writes nothing" passes through the new fallback.
- Mutation checks: with the owner conditions removed from the statement,
both race tests and the expired-lease test fail; with the lease check
moved back into the locking CTE, the lock-wait expiry test fails.
- Other focused suites, one file at a time:
`native-runner-restart-recovery.integration` (231 passed, 14 skipped),
`native-runner-restart-checkpoint-cost.integration` (1; it greps for
`insert into "native_runner_checkpoints"`, so the statement quotes the
table name), `native-recovered-owner-followup` (97),
`native-controller-recovery.integration` (30),
`native-runner-checkpoint-authority.integration` (1),
`native-restart-recovery-isolation` (1), the port suite with conformance
(105), `heartbeat-native-restart-suspension` (115), `live-recovery-work`
(51 tests pass; on this busy machine its teardown hook timed out, as it
has before on master). `pnpm --filter @paperclipai/server run typecheck`
passes.
- Statements per routine save: 4 → 1 (5 → 1 where commit notifications
are on). I did not rerun the load model for this change: the machine is
too busy for a fair timing comparison today, and the statement count is
exact.

- CPU per operation (a local script, not committed: 600 routine saves of
one event part and the cursor for one run on embedded PostgreSQL, one
connection, three rounds, `origin/master` code against this branch):
main-thread CPU per save 0.38–0.56 ms before and 0.10–0.12 ms after
(encryption included in both); wall time per save 0.45–0.73 ms before
and 0.22–0.30 ms after.

## Risks

- A routine save no longer wakes the six recovery queues. Their own
deadlines and the 30 s reconstruction still run; see the reading above
for why no decision depends on a routine save.
- The owner check of the routine statement accepts only the exact active
owner. Any other state goes to the unchanged full path, so a successor
or a finished result keeps today's rules.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:58:06 -06:00
DottaandClaude Opus 5.5 1dee321844 perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every native event that announces work to another part of the server
(chat progress, a final response, an evidence outcome, recovery)
registers a wake in its transaction before it writes the event
> - Such an event takes six statements: BEGIN, the run lock read, a
replay probe, a read of the transaction id for the wake, the append, and
COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock,
the transaction id, the write, COMMIT
> - In the agent scaling load model these two paths are now the largest
per-agent cost after #21511 and #21520, and other events fall behind
first at 600 agents
> - This pull request returns the transaction id from the lock read that
each path already sends, and moves the replay probe into the run lock
read when it is safe
> - The benefit is four statements for such an event and four for a
checkpoint save, with the same locks, the same wake order and the same
replay rule

## Linked Issues or Issue Description

Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent
scaling plan and load model).

**What existing behavior does this improve?**
The statement count of native events that register a wake
(`appendNativeSourceEvent`) and of controller checkpoint saves
(`nativeRunnerCheckpointWriter`).

**Subsystem affected**
`packages/db/src/work-signals.ts` (one new export),
`server/src/services/heartbeat-run-events.ts`,
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Current behavior**
`signalDatabaseWork` sends `select pg_current_xact_id()` for the first
wake of each transaction. A native event with a wake reads the run lock,
then probes for its replay in a separate statement, because a statement
that waits for the lock keeps its earlier snapshot.

**Proposed behavior**
- The run lock read and the checkpoint owner lock read also return
`pg_current_xact_id()::text`. Both already lock a row, so the
transaction has an id at that point. `primeDatabaseWorkTransactionId`
gives it to the transaction's wake state, and the wake sends no
statement for it.
- For an event that may register a progress, response, reassessment or
retention wake, the run lock read also reads `next_event_seq` from its
own snapshot and checks for a replay in that snapshot. Every event
append advances `next_event_seq` in its transaction. So when the locked
row has the same `next_event_seq` as the snapshot, no append committed
in between, and the probe result is current. Otherwise, or when the
probe finds a replay, the separate probe runs as before.

**Reason and benefit**
Fewer round trips while a pool connection is held in a transaction. In
the load model these paths hold the main pool and the three-connection
checkpoint pool longest per event.

**Breaking changes**
None. No schema change.

## What Changed

- `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`,
exported from `@paperclipai/db`): inside a transaction whose id is not
yet known, it records the id. Outside a transaction it does nothing. An
uncertain commit still resolves with `pg_xact_status` on the same id.
- `appendNativeSourceEvent`: the lock read returns the transaction id
when the event has any wake. For a probing wake it is one statement: a
CTE that reads `next_event_seq` from the snapshot, a CTE that locks the
run row (`for no key update`, unchanged), and `exists (...)` for the
replay. The wake is still registered after the replay check and before
the append.
- `lockNativeRunnerCheckpointOwner` returns the transaction id with its
owner lock and primes the wake state. Its return value and every caller
are unchanged.

## Verification

- Red/green on embedded PostgreSQL; each new assertion failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "registers a native
event's wakes before its write and only when the event is new": four
statements (before: six), the wake's intent and settlement are
published, and a replay publishes no wake.
- New: "finds a replay that another writer committed while the lock read
waited, and registers no wake for it". Another connection holds the run
row (the test waits until PostgreSQL reports the append waiting for that
lock), stores the same event and moves `next_event_seq`; the waiting
append returns `duplicate`, runs the separate probe, and publishes no
wake. With the `next_event_seq` comparison removed, this test fails.
- New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the
owner lock with the transaction id, the write, COMMIT (before: five
statements), and its recovery wake is published; a save after the lease
expired writes nothing and fails with
`native_runner_checkpoint_owner_lost`.
- `work-signals.test.ts`: a primed transaction sends no id statement for
its wakes; a primed transaction whose connection is killed after its
write stays pending and then resolves through `reconcileDatabaseWork`,
with no row stored.
- Added after merging `origin/master` (with #21511, #21520 and #21436)
into this branch:
- "gives the wake the transaction id of the transaction that took the
run lock": inside a caller's transaction, an event with a progress wake
sends no separate `pg_current_xact_id()` statement; the transaction's
connection is then killed, and `reconcileDatabaseWork` checks
`pg_xact_status` with exactly the id that `pg_current_xact_id()` returns
on that same transaction. Priming a wrong id makes it fail.
- "primes nothing when the lock read fails, so the transaction is not
left uncertain": a lock read that fails on `lock_timeout` leaves no
pending work; after the same failure inside a savepoint, the next signal
on that transaction still sends its own `pg_current_xact_id()`. Priming
before the lock read makes it fail.
- Both call sites pass only the value returned by the statement they
just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)`
then `primeDatabaseWorkTransactionId(tx, run.xid)`) and
`lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then
`primeDatabaseWorkTransactionId(tx, xid)`).
- Retest on the combined tree, one file at a time: atomic-append 28
(including the equivalence test and both race tests), the port suite
with conformance 105, checkpoint-save 2, `live-recovery-work` 51,
`work-signals` 25, runner `native-session-runtime` 297; server typecheck
passes. No code change was needed for the combination.
- Other focused suites pass, one file at a time: the port suite with its
conformance suite (100), `chat-native-progress-work` (35),
`native-observer-replay` (1), `heartbeat-run-event-sequencing` (5),
`native-event-boundary-golden` (1), `native-recovered-owner-followup`
(97), `native-restart-recovery-isolation` (1),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51),
`work-signals` (25).
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). Stacked: this pull request merged locally with #21511, and the
model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model
applies the port's batching rules itself; that mode is not in any pull
request). Default pool of 10 connections; embedded PostgreSQL on the
same, shared machine; runs one after the other with a pause between
them. No run had a failed operation, a failed checkpoint save or a lost
lease. "Before" rows are `origin/master` from the same session.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 /
12,000 | 1 / 70 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 /
24,000 | 2 / 1,274 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 /
6,335 | 61,158 / 72,000 | 11,678 / 21,212 |
| 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 /
72,000 | 1,136 / 4,980 |
| 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 |
71,958 / 72,000 | 1,319 / 5,712 |
| 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 /
48,000 | 1,974 / 3,835 |

With the three changes, one process finishes 99.9% of the offered work
at 600 agents, but its event-loop utilization is 0.90–0.93, above the
0.75 target, and p99 lateness is about 5 s. It saturates between 600 and
800 agents.
- Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5
where commit-time notifications are on).

## Risks

- The transaction id is read in a statement that already assigns one (a
row lock), so no transaction gets an id it did not have before.
- The probe in the lock read is used only when `next_event_seq` did not
change between the snapshot and the lock. Every writer of
`heartbeat_run_events` moves `next_event_seq` in the same transaction
(native and ordinary appends, the run coordinator store, the retry
queue). A future writer that inserts events without it would make this
probe miss a concurrent replay; the append statement then still finds
the replay and stores nothing, but the wake would already be registered
(a spurious wake, not a lost or duplicate event).
- This pull request and #21511 both edit `heartbeat-run-events.ts`, in
different functions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:17 -06:00