mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents work on tasks through runs, and recovery code starts the next run when a run stops > - On staging, many open agent tasks show "running" or "blocked", but no run, wake or retry is pending, and no person has to act > - Staging deploys about every 40 minutes. Each deploy stops some runs, and some tasks then stay idle until a person presses Retry > - The stranded sweep has many lanes. Some lanes get "no retry" back and skip the task with no record, and the sweep never looks at `blocked` tasks > - This pull request adds one invariant with one pure predicate, and one event-driven sweep that repairs a violation with one ordinary continuation wake > - The benefit is that no open agent task stays with nothing running, nothing pending and no one else's move ## Linked Issues or Issue Description Refs #21107 and #21046 (the pattern for leaving a run of another live server alone). Related open work: #21135 (fresh continuation from saved work at wake admission), #21124 (sweep calls the bounded scheduler for pre-provider legacy failures), #21127 (run-start hand-over). **What happened?** On a staging snapshot (2026-11-12, about 14:35Z), 1,030 open agent tasks had no live run. The new predicate, run read-only against the staging API, classifies 141 of them as violations: 40 offer a Retry only the system can act on, 38 are blocked with no recorded reason and no hold, 14 are blocked by an automatic recovery outcome with no hold left, 5 have nothing pending, and 44 are blocked under an active recovery hold (left to the hold's owner, see below). 25 more are blocked by recovery for more than 7 days. The other tasks wait on a person or on a pending intent: 542 chat threads, 122 agent-declared blocks, 102 credential waits, 51 dependencies, 28 questions. Nine Retry-started `in_progress` tasks show the main gap. A person pressed Retry, a deploy stopped the run during setup (`server_shutdown_interrupted`, no provider work), and the bounded scheduler then refused with `continuation_user_authorization_missing` ("could not revalidate the original user continuation") on every sweep. The sweep's escalation did not take effect either, so each task stayed `in_progress` with a Retry button and nothing pending, for up to 10 hours. **Expected behavior** Each open task assigned to an agent has a live run, a durable pending intent, or a named reason that is someone else's move. If it has none of these, Paperclip continues the task automatically. **Steps to reproduce** 1. Start a person's Retry on an `in_progress` task with a legacy agent. Stop the server during setup, so the run ends `interrupted` with `server_shutdown_interrupted`. 2. Let the bounded retry scheduler refuse to revalidate the person's continuation. 3. Run the stranded sweep. Nothing is queued. The task stays `in_progress` with nothing pending. 4. The same happens for a native predecessor (`enqueueStrandedIssueRecovery` returns `null` for every native run) and for every `blocked` task (the sweep reads only `todo`, `in_progress` and `in_review`). ## What Changed - `server/src/services/recovery/assigned-work-invariant.ts`: the invariant as one pure predicate. Facts go in. The verdict is `ok` with its clause (live run, pending intent, or one reason from a closed list of waits), `settling` with a re-check time, or `violation` with its reason. An unknown hold, block reason or person action is a violation. It also holds the per-task continuation allowance: at most 6 per hour (the server-restart lane's bound, the same cap as #21135) and, below the cap, a spacing of 2, 4, 8 and then at most 15 minutes. At the cap the next try waits until the oldest continuation leaves the hour, so a task that keeps needing continuation is retried at about 6 per hour, never left for hours. - `server/src/services/recovery/assigned-work-sweep.ts`: an indexed pre-filter, a batched fact loader, the repair and a read-only report. - A running run counts as live unless the shared owner proof (`owner-proof.ts`) proves its server gone. A held controller lease is live, so a run of the other server in a deploy overlap is never touched. A run whose owner is proven gone goes to heartbeat's existing lost-process settle path. Native running runs are always treated as live; the native settle watch judges their owners. - Native active paths match the stranded sweep's check, including #21108's exclusion of a native finalization that a person's Retry replaced. - A run that ended less than 90 seconds ago, or whose controller lease another server still holds, is `settling`. The sweep looks again at that deadline. - A pending question counts as the person's move, unless the person pressed Retry (or sent a message) after the question was asked. - The repair takes the task's row lock, checks again that nothing is pending, applies the allowance and writes a plain-language activity entry. This entry is the claim, so two servers make one wake. For a `blocked` task with no recovery hold, it also puts the task back to the status the block recorded. Then it enqueues one ordinary `issue_continuation_needed` wake. The wake has no retry link, so it does not use the task's failure allowance. With #21135, wake admission turns that wake into a fresh run from saved work when a hold would defer it. - A `blocked` task under an active recovery hold is reported as a violation but not repaired here: the hold's owner (#21135) re-continues it, so the two never continue one task twice. When #21124 escalates once on a permanent scheduler refusal, the task is blocked under a hold, so #21135 continues it; when no hold is left, this sweep continues it. Neither case leaves the task waiting on a person for a reason that is not the person's. - The claim re-runs the full predicate on a fresh read under the task's row lock, so a question, approval, blocker, monitor, wake or run that appeared after the first read stops the repair. - A lifted block is restored to the status the blocking entry recorded (`_previous.status`, `previousStatus` or `fromStatus`). A block that interrupted a review goes back to `in_review` with no agent wake; the review's own recovery wakes its participant. If admission refuses the wake, the block is put back with its original block time. - The settling window is keyed on three events only: the latest run ending (a held lease can extend it, but never past 5 minutes after the run ended), a never-run task's creation, and a blocker finishing. Routine touches (comments, reads, labels) never postpone a repair. A run cut by a deploy's shutdown is repaired about 90 seconds after it ended. - Each process starts at most 10 repairs per minute, newest-updated tasks first. The other tasks wait for a deadline. - Each burst of checks writes one structured log line (`assigned_work_invariant.pass`) with the counts by clause, the repairs and the deferrals. - `server/src/services/recovery/work.ts`, `issue-recovery-notifications.ts`, `delivery-work-notifications.ts`: a new `assigned-work-invariant` queue on the shared delivery-work coordinator. The existing issue-recovery notifications (run state changes, hold resolutions) and server start mark a company dirty. There is no interval polling. Deadlines exist only for a real obligation (settling grace, back-off, rate-limit deferral). - `server/src/services/recovery/service.ts`, `heartbeat.ts`: wire the sweep with the existing wake, live-execution and settle dependencies. - `server/src/routes/agents.ts`, `server/src/routes/openapi.ts`, `docs/api/agents.md`: `GET /api/companies/:companyId/assigned-work-invariant` (board only, company-scoped) and filtered by the caller's issue-privacy condition: private tasks the caller cannot read are absent from every count and sample returns the counts by clause, the violation count, the held subset, stale blocks and up to 20 samples. Use it after a deploy to check for 0 violations. Why there are two sweeps: the stranded sweep decides how a known failure resumes (bounded retries, disposition repair, review recovery, escalation). The invariant decides only that something is pending, and repairs with the ordinary continuation. It does not change the stranded sweep's lanes or the recovery-refusal paths. ## Verification - `server/src/services/recovery/assigned-work-invariant.test.ts` (45 tests): every clause, every wait reason, the unknown cases, settling, the allowance and its long back-off. - `server/src/__tests__/assigned-work-invariant.test.ts` (17 tests, embedded Postgres): a refused or later-declined continuation stays a violation and is retried under the cap; the real wake queue deferring behind a recovery hold; scheduling stops when a task is done, cancelled or reassigned to a person; running vs stopped or blocked session goals; the deploy-interrupted Retry shape and the native reconciliation shape (the stranded sweep does nothing, then exactly one wake); the nine Retry-started tasks whose scheduler refused revalidation, including one with a question asked before the Retry; a block after a legacy run's retries ran out with no hold left (`controller_lease_lost`); a block under an active hold left to its owner; two concurrent sweeps make one wake; another live server's run is not touched; an owner-gone run is settled; approval, question, dependency and declared-unblock tasks are not touched; an unknown block reason is repaired; the per-task cap and the per-process budget; a run that ended seconds ago; old-shaped rows (a board-owned stranded action from an older release, native `awaiting_runner_reattach` and `retryable_failure`, a running run saved without its controller process, a block older than 7 days); query cost at staging size. - Query cost at staging size (800 open tasks, 14,000 runs over 7 days): one suspect page 2 ms, one task check 165 ms (including the repair), a full 800-task report 386 ms. The suspect anti-join uses the run status index (no sequential scan on `heartbeat_runs`, 0.2 ms). - Route test in `agent-live-run-routes.test.ts`: a board operator reads its own company; another company and an agent get 403. - Existing production-wiring suites pass with the new queue: `issue-recovery-deadlines`, `issue-recovery-conversations`, `live-recovery-work`, `issue-recovery-work-lifecycle`, `recovery-deadline-acceptance`, `server-startup-feedback-export`, `heartbeat-held-comment-wake`, `delivery-work-coordinator`. - `node scripts/typecheck-layers.mjs` for the server source layers passes. ## Risks - On the first deploy, the sweep continues about 97 staging tasks over about 10 minutes (10 per minute). Each continuation is an ordinary run. - The stranded sweep can still move a task to `blocked` after its bounded retries. The task is then continued by the hold's owner or by this sweep, within the same cap and back-off. The task gets more activity entries than before. - On current master without #21135: when the wake reaches a task whose last run left a recovery hold, admission defers it (a skipped wake, no pending row). The activity entry then says the continuation did not start, the task stays a violation, and the sweep tries again with spacing (2, 4, 8, then 15 minutes at most; at most 6 per hour). It never loops faster than the cap and never moves the task to blocked for a person. A test with the real wake queue covers this. With #21135, the same wake becomes a fresh run from saved work. - The activity entry is written as an attempt and is published only after admission: as queued, or as not started. A wake that the queue declines later (for example behind the old run's cleanup) leaves the task a violation, and it is retried under the cap. - Chat-thread tasks are left out (the conversation clause), because their next move is a person's message. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>