mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs all background recovery on one Node main thread, and
each database statement costs time on it
> - The ownership reconciliation pass settles in-progress records every
30 s while it has records to watch
> - Two of its kinds listed records that can never change again: dirty
remote workspaces whose writer reached runtime, and snapshot ledger rows
that are uploaded or still held by a stale writer's epoch
> - Each such record cost a transaction, a row lock and up to four reads
on every pass, forever, and kept the 30 s watch armed on an idle
instance
> - This pull request lists only the records that the settle step can
still act on, locks the writer run only when the settle will write, and
skips one stranded-task read that can never match a native run
> - The benefit is that recovery work follows activity, not history
## Linked Issues or Issue Description
**Describe the problem**
On a hosted instance with 7 native runs, the ownership pass ran about
7.6 + 3.2 + 3.5 + 2.8 statements per second over records that cannot
change, and took a row lock on `heartbeat_runs` for each stale dirty
workspace on every pass. At 1,000 agents this becomes thousands of
statements per second.
**Steps to reproduce**
Seed about 100 dirty remote workspaces whose writer run reached runtime
and ended, 50 pending ledger rows still held by those writers, and 100
uploaded ledger rows. Run the ownership reconciliation worker. The new
census test
`server/src/__tests__/recovery-settled-records-census.test.ts` does this
on the embedded database.
**Expected behavior**
A settled record costs zero statements on later passes. An idle instance
runs no SQL from this pass.
**Actual behavior**
On master, one pass costs 714 statements and 150 row locks. An idle half
hour costs 40,800 statements and 9,000 row locks (the settle watch
re-checks every 30 s).
Refs #21765 (the stranded-assignment hint scoping that this rebases on)
## What Changed
- `remote_workspace_unsaved_files` lists a dirty workspace only when its
writer run can still be a preparation-only writer: stage `preparing`, no
`runtime_mode_resolved_at`, no process, session or runner id, no
finalization row and no work-started event (or no writer at all). The
SQL lives with the settle step in
`remote-workspace-unsaved-files-settlement.ts`.
- The unsaved-files settle reads the writer run without a lock. It takes
`for update` only after every check passes, just before it writes, and
checks the locked row again.
- `remote_snapshot_ledger` lists only `pending` rows with no object key,
whose operation is a workspace save, and whose workspace epoch has moved
past the row's writer. Uploaded rows belong to snapshot retention, as
before.
- `terminalRunSettlingUntil` in `recovery/service.ts` skips its
`heartbeat_runs` read when the latest run already in memory is not a
legacy run of another server boot. `getLatestIssueRun` now also selects
`runtime_mode` and `controller_boot_id`. The live-execution check still
runs first.
- New census test with a staging-shaped fixture, a statement counter
grouped by statement shape and caller, and prompt-pickup checks. New
red/green test for the stranded-task lease read.
Census (embedded PostgreSQL, same fixture, base 531db3c10b vs this
branch):
| Window | Master statements | Master row locks | Branch statements |
Branch row locks |
| --- | --- | --- | --- | --- |
| One full ownership pass | 714 | 150 | 62 | 0 |
| One settle-watch re-check | 681 | 150 | 29 | 0 |
| Idle 30 min (real worker) | 40,800 | 9,000 | 0 | 0 |
| Active, 7 live runs hinting every 10 s | 47,257 per 30 min | 9,000 |
2,763 per 10 min (about 8,300 per 30 min, of which about 4,200 are the
test's own hint writers) | 0 |
Confirmed callers on master (per full pass): `select
execution_workspace_snapshots` full row at
`remote-snapshot-ledger-settlement.ts:20` (150), `select heartbeat_runs
... for update` at `remote-workspace-unsaved-files-settlement.ts:20`
(100), `select workspace_operations ... for update` at
`remote-snapshot-ledger-settlement.ts:25` (50), `select
execution_workspaces.metadata` at
`remote-snapshot-ledger-settlement.ts:36` (50). The
`execution_replacement` queue costs 2 statements per wake over the same
history (discovery and the deadline read), so it is not the source of
the snapshot or lease reads in this shape.
## Verification
- `cd server && npx vitest run
src/__tests__/recovery-settled-records-census.test.ts
src/__tests__/issue-recovery-terminal-run-settling.test.ts` (fails on
master: 714/40,800 statements and the native lease read; passes here)
- `cd server && npx vitest run
src/__tests__/remote-snapshot-ledger-settlement.test.ts
src/__tests__/ownership-reconciliation-report.test.ts` (51 passed,
unchanged)
- `cd server && npx vitest run
src/__tests__/recovery-deadline-acceptance.test.ts
src/__tests__/legacy-continuation-authority.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (502 passed)
- The census test also checks that a preparation-only workspace is still
settled in the first pass, an abandoned pending ledger row is still
retired, a ledger row held by a stale writer is retired in the next pass
once the workspace epoch moves, and an `active` run lease whose run
ended is still listed and released.
## Risks
- The report shows fewer in-progress records for these two kinds.
Records that the settle step can never change (a dirty flag whose writer
reached runtime, an uploaded ledger row) are no longer listed, so they
no longer appear as `I2_settled` violations in the activity log. On
master each process wrote one such violation per record.
- A pending ledger row held by its stale writer's epoch is no longer
re-checked every 30 s. It is found by the next full pass after the
workspace epoch moves. A new writer or an epoch abandonment both come
with run signals that wake the pass, but on an instance with no other
activity this can now wait until the next wake instead of at most 30 s.
- A dirty workspace whose writer run row is missing is no longer listed.
On master it was re-checked forever and the settle refused it every
time.
- No schema, API or UI change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>