perf(server): skip settled committed finalizations in unscoped passes (#16485)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On restart, `/api/health` reports `starting` until startup recovery
finishes, and deploys wait for it
> - Native recovery and the orphan reap each run an unscoped native
finalization reconciliation pass, and readiness waits for both
> - The pass selects every committed finalization with a result,
including runs that ended long ago, and spends dozens of sequential
queries on each one
> - On a busy instance this is about one minute per pass, so about two
minutes of every restart, and the periodic recovery repeats it on every
tick
> - This pull request skips settled committed finalizations in unscoped
passes, and keeps a small rotating upkeep page so their upkeep still
runs
> - The benefit is faster restarts and less steady database load, with
no change for unsettled work

## Linked Issues or Issue Description

Related: #16330 and #16329 (startup and reconciliation timing).

**What existing behavior does this improve?**
Startup recovery time and steady-state recovery load.

**Current behavior**
- The unscoped candidate scan in `reconcileNativeFinalizations` returns
every native finalization with a result and phase other than
`terminal_failure`. Nothing excludes rows that are already settled.
- Each committed row then takes the full per-row path: pending-effect,
assessment, evidence and decision reads, presentation repair (twice),
the workspace barrier read, a `finalizeNativeRun` re-read of the
committed coordinator, and workspace settlement (lease release check and
agent status update). That's about 40–60 sequential round trips per row.
- Startup runs the unscoped pass twice (native recovery, then the orphan
reap), and readiness waits for both. The periodic orphan reap runs it
again on every tick.
- On one instance, all 268 candidates were committed finalizations whose
runs had ended. Native recovery took 50–80 s and the orphan reap 47–80 s
on every restart, with database round trips of about 3.4 ms.

**Proposed behavior**
An unscoped pass skips a committed finalization when:
- its run is `succeeded`, `failed`, `cancelled` or `timed_out`, and
- the run finished more than an hour ago, and
- its decision has no `pending` or `failed` status-decision effect.

Each unscoped pass also includes up to 10 such settled rows in a
rotating order, so the presentation and evidence upkeep still reaches
every row over time. Scoped passes (explicit run IDs) are unchanged.

**Reason and benefit**
These settled rows have nothing left to finish. Recent rows, unfinished
runs and rows with undelivered effects stay in every pass, so
crash-window repairs and effect delivery behave as before. Startup and
periodic passes no longer scale with the full history of committed runs.

**Breaking changes**
None. A late repair on a settled row can be delayed until its turn in
the upkeep rotation instead of the next pass.

## What Changed

-
`server/src/services/native-runtime/native-finalization-reconciler.ts`:
- `settledCommittedFinalization(now)`: the settled predicate above,
including a `NOT EXISTS` on undelivered `status_decision_effects` for
the same company, issue and decision.
- The unscoped candidate scan adds `NOT settled OR run in the upkeep
page`.
- `nextSettledCommittedUpkeep`: up to
`NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS` (10) settled run IDs after an
in-process cursor, ordered by run ID, wrapping at the end.
- New constants `NATIVE_SETTLED_COMMITTED_GRACE_MS` (1 hour) and
`NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS`, plus
`resetSettledCommittedUpkeepForTests`.
- New
`server/src/__tests__/native-finalization-settled-candidates.test.ts`.
- `doc/execution-semantics.md`: describes the settled rule and the
upkeep rotation.

## Verification

- `pnpm exec vitest run
src/__tests__/native-finalization-settled-candidates.test.ts` (in
`server/`): 4 passed. The test mocks the timing hook to capture the
candidate count and stop the pass after the scan:
- 25 settled rows give 10, 10, 5, then 10 again over four passes
(rotation and wrap).
- A row finished 10 minutes ago, a running row, and a terminal row
without `finished_at` are all candidates.
  - A settled row whose decision has a `failed` effect is a candidate.
  - A scoped pass with an explicit run ID still selects a settled row.
- `pnpm exec vitest run` for `native-finalization-recovery`,
`legacy-finalization-regression`, `native-run-finalizer-telemetry`,
`native-workspace-finalizer`, `native-finalization-reconciler-timing`,
`external-chat-wait.integration` and `native-status-arbiter-corpus`: 316
passed.
- `heartbeat/recovery.test.ts`: the 8 tests that do not need a database
passed. The 20 database tests were skipped locally because embedded
PostgreSQL could not start (the host's shared-memory segments were
exhausted by other processes). CI runs them.
- `pnpm typecheck` in `server/`: passed.

## Risks

- A settled row that needs a late repair (for example a missing chat
presentation) waits for its turn in the upkeep rotation, up to
`ceil(settled rows / 10)` passes, instead of the next pass.
- The upkeep cursor is in memory. After a restart the rotation starts
from the beginning; the work is idempotent.
- A run left `running` with a committed finalization is never considered
settled, so it is still projected before the reaper's active-run loop.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, test runs, database and log analysis).
A separate Claude agent mapped the recovery call tree.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-10-15 16:42:15 -07:00
1 parent c3ffa3f0dd
commit c7ea5c0216
3 files changed
+250 -1

No files matched your search

+8
View File
@@ -859,6 +859,14 @@ On startup and on the periodic recovery loop, Paperclip performs the following r
The stranded-work pass closes the gap where issue state survives a crash but the wake/run path does not. The silent-run scan covers the separate case where a live process exists but has stopped producing observable output.
Native finalization reconciliation (part of native recovery and the orphan
reap) skips settled committed finalizations in unscoped passes: the run is
succeeded, failed, cancelled or timed out, it finished more than an hour ago,
and its decision has no pending or failed effect. Each unscoped pass still
visits up to 10 settled rows in a rotating order, so presentation and evidence
upkeep reaches every row over time. A pass scoped to explicit runs reconciles
those runs whatever their state.
Startup health (`/api/health`) reports `starting` until these recovery passes and
the other startup steps that serving requests depends on finish. Housekeeping
that every scheduler tick repeats does not delay it: the custom-image setup