mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-19 20:44:16 +02:00
Devin FoleyandPaperclip
c7ea5c0216
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On restart, `/api/health` reports `starting` until startup recovery finishes, and deploys wait for it > - Native recovery and the orphan reap each run an unscoped native finalization reconciliation pass, and readiness waits for both > - The pass selects every committed finalization with a result, including runs that ended long ago, and spends dozens of sequential queries on each one > - On a busy instance this is about one minute per pass, so about two minutes of every restart, and the periodic recovery repeats it on every tick > - This pull request skips settled committed finalizations in unscoped passes, and keeps a small rotating upkeep page so their upkeep still runs > - The benefit is faster restarts and less steady database load, with no change for unsettled work ## Linked Issues or Issue Description Related: #16330 and #16329 (startup and reconciliation timing). **What existing behavior does this improve?** Startup recovery time and steady-state recovery load. **Current behavior** - The unscoped candidate scan in `reconcileNativeFinalizations` returns every native finalization with a result and phase other than `terminal_failure`. Nothing excludes rows that are already settled. - Each committed row then takes the full per-row path: pending-effect, assessment, evidence and decision reads, presentation repair (twice), the workspace barrier read, a `finalizeNativeRun` re-read of the committed coordinator, and workspace settlement (lease release check and agent status update). That's about 40–60 sequential round trips per row. - Startup runs the unscoped pass twice (native recovery, then the orphan reap), and readiness waits for both. The periodic orphan reap runs it again on every tick. - On one instance, all 268 candidates were committed finalizations whose runs had ended. Native recovery took 50–80 s and the orphan reap 47–80 s on every restart, with database round trips of about 3.4 ms. **Proposed behavior** An unscoped pass skips a committed finalization when: - its run is `succeeded`, `failed`, `cancelled` or `timed_out`, and - the run finished more than an hour ago, and - its decision has no `pending` or `failed` status-decision effect. Each unscoped pass also includes up to 10 such settled rows in a rotating order, so the presentation and evidence upkeep still reaches every row over time. Scoped passes (explicit run IDs) are unchanged. **Reason and benefit** These settled rows have nothing left to finish. Recent rows, unfinished runs and rows with undelivered effects stay in every pass, so crash-window repairs and effect delivery behave as before. Startup and periodic passes no longer scale with the full history of committed runs. **Breaking changes** None. A late repair on a settled row can be delayed until its turn in the upkeep rotation instead of the next pass. ## What Changed - `server/src/services/native-runtime/native-finalization-reconciler.ts`: - `settledCommittedFinalization(now)`: the settled predicate above, including a `NOT EXISTS` on undelivered `status_decision_effects` for the same company, issue and decision. - The unscoped candidate scan adds `NOT settled OR run in the upkeep page`. - `nextSettledCommittedUpkeep`: up to `NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS` (10) settled run IDs after an in-process cursor, ordered by run ID, wrapping at the end. - New constants `NATIVE_SETTLED_COMMITTED_GRACE_MS` (1 hour) and `NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS`, plus `resetSettledCommittedUpkeepForTests`. - New `server/src/__tests__/native-finalization-settled-candidates.test.ts`. - `doc/execution-semantics.md`: describes the settled rule and the upkeep rotation. ## Verification - `pnpm exec vitest run src/__tests__/native-finalization-settled-candidates.test.ts` (in `server/`): 4 passed. The test mocks the timing hook to capture the candidate count and stop the pass after the scan: - 25 settled rows give 10, 10, 5, then 10 again over four passes (rotation and wrap). - A row finished 10 minutes ago, a running row, and a terminal row without `finished_at` are all candidates. - A settled row whose decision has a `failed` effect is a candidate. - A scoped pass with an explicit run ID still selects a settled row. - `pnpm exec vitest run` for `native-finalization-recovery`, `legacy-finalization-regression`, `native-run-finalizer-telemetry`, `native-workspace-finalizer`, `native-finalization-reconciler-timing`, `external-chat-wait.integration` and `native-status-arbiter-corpus`: 316 passed. - `heartbeat/recovery.test.ts`: the 8 tests that do not need a database passed. The 20 database tests were skipped locally because embedded PostgreSQL could not start (the host's shared-memory segments were exhausted by other processes). CI runs them. - `pnpm typecheck` in `server/`: passed. ## Risks - A settled row that needs a late repair (for example a missing chat presentation) waits for its turn in the upkeep rotation, up to `ceil(settled rows / 10)` passes, instead of the next pass. - The upkeep cursor is in memory. After a restart the rotation starts from the beginning; the work is idempotent. - A run left `running` with a committed finalization is never considered settled, so it is still projected before the reaper's active-run loop. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, test runs, database and log analysis). A separate Claude agent mapped the recovery call tree. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>