mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues on one Node main thread. `stranded-assignments` and `assigned-work-invariant` find assigned tasks that have no live run and restart them. > - #21552 scoped run hints to the run's task graph and kept a company "backstop" pass to catch a missed hint. Any graph hint arms the backstop, and the backstop runs at most every 5 minutes after the last full pass. > - The backstop reloads and re-checks every open assigned task of the company. With any trickle of activity it runs every 5 minutes. On the hosted staging instance that is about 850 tasks every 5 minutes, about 2.8 checks per second at idle, close to the measured 3.2 per second. That is work proportional to history, not to activity. > - This pull request makes the 5-minute backstop re-check only tasks whose own rows changed since the previous look, with one set-based query. A full company pass runs at most once an hour, still armed only by activity. > - The benefit is no re-checks between hourly passes when nothing changed: idle statements drop from 59 per second to 5.5 per second in a staging-shaped fixture. A task whose own state changed without a hint is still recovered within 5 minutes. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The missed-hint backstop of the `stranded-assignments` and `assigned-work-invariant` recovery queues (`server/src/services/recovery/work.ts`). **Subsystem affected** Server: issue recovery background work. **Current behavior** Each graph hint arms a `{companyId, backstop: true}` scope. When it opens (5 minutes after the last full pass), the queue loads every open assigned task of the company and checks each one again: about 22 statements per task for the stranded check. With a hint at least every 5 minutes, this repeats every 5 minutes, whatever changed. **Proposed behavior** Between full passes, the backstop loads only the tasks whose own rows changed since the previous look (minus a 5-minute overlap for late commits): the task row, its runs, its recovery actions, its wake requests, its thread interactions, its comments, or its assignee agent. A full company pass runs when the last one is an hour old. A real change still sends its own hint and gets an immediate targeted check. **Reason and benefit** On staging at idle, the stranded check runs about 3.2 times per second with zero runs. A fixture with staging's population shape attributes all of it to the backstop. ## What Changed - `server/src/services/recovery/work.ts`: - `fullPasses()` keeps the time of the last full pass and of the last changed-tasks pass per company. The backstop gate is 5 minutes after the later of the two. - A backstop scan decides on its first page: a full load when the last full pass is at least `ISSUE_RECOVERY_FULL_BACKSTOP_MS` (1 hour) old, otherwise a changed-tasks load since the previous look minus `ISSUE_RECOVERY_BACKSTOP_MS`. All pages of one scan use the same decision. - Each scan records its start time on its first page. A completed scan advances the saved full or changed time only to that start, so a change behind the cursor of a slow scan falls in the next window. - The stranded queue adds `changedTaskCondition(since)` to its load query. The assigned-work queue passes `changedSince` to `listSuspects`. Both return `keepUnseen: true` for a changed-tasks load. - A new `fullBackstopMs` option. When a caller sets only `backstopMs` (tests), every backstop stays a full pass, as before. - With `PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments`, each `check` event now names why the task was loaded: `deadline`, `reload` (unscoped reconstruction), `backstop`, `company` (company-wide hint) or `hint` (task or graph hint). - `server/src/services/recovery/changed-tasks.ts` (new): the set-based "changed since" condition. Each part uses an existing index: `issues (company_id, updated_at)`, runs by `(company_id, context issueId)`, wakes by `(company_id, payload issueId)`, interactions and comments by `(company_id, issue_id)`. - `server/src/services/scoped-deadline-work.ts`: a load result may set `keepUnseen`. Then a completed scan does not drop the pending entries it did not load, so their saved deadlines stay. - `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects` accepts `changedSince`. - `server/src/services/issue-recovery-notifications.ts`: `ISSUE_RECOVERY_FULL_BACKSTOP_MS = 60 * 60_000`. - Tests: - `server/src/__tests__/issue-recovery-idle-backstop.test.ts` (new): the staging-shaped fixture, see Verification. - The staging-shaped population helper and the fixture's statement meter come from #21724, now merged into this branch. - `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`: the check event now names its load reason (`hint` or, when the backstop loaded the task last, `backstop`) instead of `load`. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: the backlog helper writes its rows with the test's fake clock, as production writes use the server clock. Without it, rows stamped with the database clock look older than the fake-clock window. ## Verification - Fixture (`issue-recovery-idle-backstop.test.ts`, embedded PostgreSQL, the real recovery, dispatch and retry-cancellation workers, fake clock, statement meter, `PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`): one company, 50 agents, 780 in-review tasks with succeeded native runs and no execution state, 40 in-progress and 30 to-do tasks of a paused agent, 12 tasks with an active agent-owned recovery action, 3 tasks of a terminated agent. After boot, one unrelated graph hint per minute stands in for idle activity. - Wake reasons on master over 30 idle minutes with that trickle: 5,100 stranded checks, all `backstop`; 0 from hints, 0 from company hints, 0 from deadlines. That is 2.83 checks per second and 106,632 statements (59.2 per second). With no hint at all after boot, master still made one backstop pass (850 checks) in 30 minutes, armed by boot activity. - This branch, same fixture: | | master | this branch | | --- | --- | --- | | Stranded checks, first 30 idle minutes | 5,100 | 0 | | Statements, first 30 idle minutes | 106,632 | 1,044 | | Stranded checks, one idle hour (one full pass) | about 10,200 | 850 | | Statements per second over that hour | 59.2 | 5.5 | - Missed hint: a task inserted with no hint is checked and gets a run within 5 minutes plus one pass (`ISSUE_RECOVERY_BACKSTOP_MS + 2 min` asserted). - A change outside the task's own rows (its agent resumed by a raw update that sets no `updated_at`, with no hint) is not seen by the 5-minute pass. It is checked and gets a run after more than 10 minutes and within 62 minutes, through the hourly full pass. - Prompt pickup with the backstop closed: a finished run (`notifyNativeStatusReassessment`), a user comment, a hold release (`pause_lifted`, which a budget raise and an agent resume send) and an agent resume each get a check of the affected task within 60 seconds. The company-wide events re-check the whole company at 25 tasks per pass, as before. - Two regression tests in `issue-recovery-deadlines.test.ts`, red on379dda9e26and green on802a76c153: - "keeps a saved deadline of a task that a changed-tasks backstop pass does not load": a task with a 16-minute saved deadline is checked at that deadline after two changed-tasks passes that did not load it. - "starts the next changed-tasks window from the start of a slow backstop scan": each check advances the clock by 20 s, so the scan spans more than 10 minutes. A task changed behind the cursor early in the scan is read by the next changed-tasks pass. - #21552 backstop tests in `issue-recovery-deadlines.test.ts` pass, including "finds a stranded task outside the affected set within the backstop bound", which now runs with the production defaults. - Regression, embedded PostgreSQL: issue-recovery-idle-backstop, issue-recovery-deadlines, scoped-deadline-work, assigned-work-recheck-idle, live-recovery-work (98 tests; "resolves a run's task graph through index lookups" failed once under 3 parallel workers and passes alone; it does not touch this change); live-recovery-work, assigned-work-recheck-idle, assigned-work-invariant, recovery-deadline-acceptance, issue-recovery-recorded-decision-idle, continuation-delivery-unwakeable-agent-idle. - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - **The missed-hint bound changes for one class of change. The owner must see this.** A change that touches the task's own rows (task, runs, recovery actions, wakes, interactions, comments, or the assignee agent's row) and sends no hint is still recovered within 5 minutes plus one pass. A change elsewhere that sends no hint (company or budget rows, tree holds, an execution-state participant agent, or a write that does not set `updated_at`) is now recovered within 60 minutes plus one pass, where it was 5 minutes. Every such writer that I found sends a company or graph hint, which still triggers an immediate check. - The changed-tasks window trusts `updated_at` stamps. A transaction that commits more than 5 minutes after its stamp can be missed until the next full pass. - The fixture cannot run the full set of workers that `index.ts` registers. It runs the recovery, dispatch and retry-cancellation workers, and stands in for other activity with one graph hint per minute. - Not changed here: a company-wide hint (`{companyId}` with no task) still reloads and re-checks every task, at most every 30 seconds. About 20 writers publish one, for example `documents.ts`, `task-watchdogs.ts`, `issue-tree-control.ts`, `issues.ts`, `heartbeat/queue.ts` and `run-dispatch-notifications.ts` without an issue. In the fixture these came only from boot escalations, not at idle. - At boot, a graph hint that arrives before the first full reconstruction completes still opens one full backstop pass. - master (with #21724) is merged into this branch. The changed-tasks backstop and #21724's `beforeChecks` pass with its shared review-wait read work together: the focused tests of both pull requests pass on the merged tree (107 tests). On the merged tree the hour that holds the full pass reads 7.7 MB instead of 15.0 MB, because of #21724's lean reads. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local PostgreSQL test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>