mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Execution recovery continues a task in a fresh run after a restart or a lost run left it unfinished > - #21234 paced these continuations: at most 1 per agent and 5 per company per pass, and a continuation waits while its agent already has a queued recovery run or a full queue > - A continuation that waited 5 minutes must enter the queue anyway, but that bound applied only to the full-queue check. The "agent already has a queued recovery run" check was in the page query, and it had no bound. > - On staging, an agent had queued recovery runs that did not start for hours. Its pending continuation was never read again, and its task showed "working" with nothing running or queued > - This pull request applies the 5-minute bound to both checks, stops one undeliverable continuation from using its agent's or company's slot, and stops the task view from showing "working" for a continuation that is overdue > - The benefit is that every pending continuation enters the queue within a bounded time, also after a restart, and the pacing still protects people from recovery herds ## Linked Issues or Issue Description Refs #21234, #21135 **What happened?** On staging, a task had a fresh-continuation hold with `continuationDelivery: "pending"` since about 03:30Z. More than 3.5 hours later, no run had started for the task. The task view showed the interruption state `working` with `canRetry: false`, so the user saw nothing wrong. The task's agent had 20 concurrency slots and 11 running runs. It also had queued `issue_continuation_needed` runs from 22:54Z that had not started. The delivery pass reads pending continuations with a query that leaves out every agent that has a queued recovery run (`issue_recovery_action_restored` or `issue_continuation_needed`). That condition had no time bound, so the pass never read this continuation again. The 5-minute admission bound was never reached. Across the company, 203 open tasks showed `working` while the API showed no queued or running run for them. 162 queued runs were more than 1 hour old. **Expected behavior** Every pending continuation enters its agent's queue within a bounded time, also when the agent's queue holds recovery runs that do not start, and also after a server restart. Pacing still limits how many continuations enter a queue at the same time. A task does not show "working" when its continuation is overdue and nothing runs or is queued. **Steps to reproduce** 1. Give an agent a concurrency limit of 1. 2. Put one queued `issue_recovery_action_restored` run on that agent that does not start. 3. Record a pending fresh continuation for another task of the same agent. 4. Wait more than 5 minutes, and let the 15-second continuation pass run. 5. See that the continuation stays `pending`, the task shows `working`, and no run starts. **Paperclip version or commit** master atbbafa78fe2(includes #21234,ade7e0643f) **Deployment mode** Hosted cloud (staging) **Agent adapter(s) involved** paperclip_runner (native runtime). The problem is in the shared continuation delivery pass, so it applies to all adapters. ## What Changed - `deliverReconciledExecutions` in `server/src/services/execution-recovery-resolution.ts`: a continuation held for 5 minutes or more is read even when its agent has a queued recovery run. Before, only the full-queue check let it through. - Each page of the pass now reads at most one candidate per agent (`row_number() over (partition by agent)`), so one agent's backlog cannot fill the page budget and hide other agents' continuations. - A continuation counts against the company's limit only when its wake puts a run in the queue. A continuation that was invalidated or replaced by a newer run does not use its agent's turn. - When an agent refuses a wake (no run, or an error), the pass skips that agent for the rest of the pass and stamps the row with `continuationDeliveryAttemptedAt`. Overdue rows always rank before rows that are not overdue. Among overdue rows, the turn is the later of the hold time and the last attempt, so a refused row goes behind rows held before that attempt and ahead of rows held after it. One refused row cannot pin its agent, and new arrivals cannot keep an older refused row waiting. - New module `server/src/services/recovery-continuation-pacing.ts` holds the pacing constants and the "held since" rule. It adds `RECOVERY_CONTINUATION_DELIVERY_BOUND_MS` (15 minutes). The old constants are still exported from `execution-recovery-resolution.ts`. - `taskInterruptionPresentations` in `server/src/services/task-interruption-presentation.ts` shows a pending continuation as "continuing" only within the delivery bound. After that, the task shows its real state instead of `working`. - No change to the schedule: the continuation duty runs only while pending continuations exist, and it checks the database on start, so a restart does not lose pending rows. ## Verification - `pnpm exec vitest run src/__tests__/recovery-wake-herd.test.ts` (in `server/`): 15 tests pass. New tests: - A continuation held past the bound is delivered while the agent's queue holds recovery runs that never start. - After a restart with a continuation pending past the bound and a full queue, a new schedule is due within one pass, the pass delivers the continuation, and the schedule then goes idle. - Continuations whose wake returns no run do not use the per-company count. Another continuation of the same agent is delivered on the next pass. - An overdue continuation is refused once while the agent's queue is full. On the next pass, it is delivered even though new continuations keep arriving for the same agent. - One agent refuses a backlog of 210 overdue continuations. Another agent's continuation is delivered in the same pass, and the refusing agent gets one wake attempt in that pass. - A task whose continuation is pending past the delivery bound with nothing running does not show `working`. A fresh pending continuation still shows `working` with `canRetry: false`. - Red/green: with the two source files reverted to master (and only the new constant exported), all 4 first new tests fail. The backlog test fails on the first PR head (200 refused wake attempts in one pass). The temporary-refusal test fails on the second PR head (the overdue row stays pending). With this change, all 15 pass. - Related suites pass: fresh-continuation fallback, workspace recovery fallback, native safe replacement, native workspace reconstruction, task interruption presentation, queued-run dispatch order, execution recovery resolution, assigned-work invariant, execution control work, task interruption automatic path and query. - `node scripts/typecheck-layers.mjs` (in `server/`) passes. ## Risks - A continuation that waited 5 minutes now enters a queue that already has recovery runs. The per-agent (1) and per-company (5) limits per pass still apply, and the queue still starts person-initiated work first. - An agent that refuses wakes gets one wake attempt per pass. Its refused rows rotate through `continuationDeliveryAttemptedAt`, and they do not block other agents. - After 15 minutes without delivery, the task no longer shows `working`. With this fix, delivery happens well within that bound, so this is a safety net. - No schema or migration changes. No new polling. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, and read-only staging API requests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>