mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops before it finishes, for example when a server swap reaches a sandbox late, the fresh-continuation fallback records one continuation of the task from its saved work > - The execution-continuation worker turns that record into a queued run. It took one record per agent per pass, and it took the oldest held record first > - After a large swap, the backlog of old records is hours deep. A record for a run that ended a few minutes ago waited behind the whole backlog. The record also stayed outside the queue while the agent's queue was deeper than its concurrency limit > - Meanwhile the task showed "todo" with nothing running, and the assigned-work sweep's wakes for it were folded into the pending record without any trace > - This pull request queues a fresh interruption's continuation on the next pass, keeps a share of every pass for the old backlog, and records a skipped wake row when a system wake is folded into a pending continuation > - The benefit is that an interrupted task gets its next run promptly, it waits in the agent's queue where the dispatcher ranks it, and nothing is lost or invisible ## Linked Issues or Issue Description Refs #21324, #21257, #21234 (earlier pacing and dispatch ranking that this builds on). **What happened?** On a hosted staging instance, five tasks had runs that ended after a server swap ("The runner restored the conversation after process loss, but the previous turn is no longer active."). The fallback recorded a continuation for each task 3 to 6 minutes after the run ended. Each record stayed `continuationDelivery: pending` for 30 to 110 minutes with no delivery attempt. During that time the worker delivered continuations whose source runs ended 4 to 7 hours earlier, one per agent per pass. The tasks read as failed and needing Retry. The sweep tried 3 to 4 times per task, and each wake returned `null` without a wake row. **Expected behavior** A recorded continuation for a recent interruption becomes a queued run within about one delivery pass. Old backlog continues to drain and is not starved. A wake that is not started records why. **Steps to reproduce** 1. Give an agent a queue that is deeper than its concurrency limit. 2. Record pending continuations for older interruptions that are held past the hold bound. 3. Record a continuation for a run that ended a few minutes ago. 4. Run `deliverReconciledExecutions`. Before this change the fresh record is not delivered. The oldest held record is delivered first, one per agent per pass. **Agent adapter(s) involved** Not adapter-specific (core bug). ## What Changed - `deliverReconciledExecutions` now sorts pending continuations into two classes by when the source run ended. "Fresh" means that the source run ended within `RECOVERY_BACKLOG_AGE_MS` (30 minutes), which is the same value the dispatcher uses to tell fresh recovery from backlog. - A fresh continuation is enqueued without the queue-depth room check. It is limited only by the per-agent limit (one fresh continuation per agent per pass) and the per-company cap per pass. Among fresh records, the newest interruption goes first. A record whose wake was refused moves behind records that were not attempted yet. - Backlog continuations keep their earlier rules. These are the queue-depth room check, the bypass after the 5-minute hold bound, oldest held first, and one per agent per pass. - Each pass runs three phases: fresh up to the company cap minus `RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS` (1), then backlog up to the cap, then fresh again up to the cap. When both classes wait, the backlog gets at least 1 of the 5 slots per company per pass. The per-company cap (5) does not change, so a swap that interrupts many runs does not enqueue them all at once. All admissions still go through the one `wake(...)` call. - In `enqueueWakeup`, a system wake that is folded into the task's pending fresh continuation now writes a `skipped` wake row with reason `issue_continuation_pending`. Before this change it returned `null` with no trace. - Tests in `recovery-wake-herd.test.ts`: - New: a fresh continuation is queued on the first pass while the agent's queue is deeper than its limit. - New: a fresh continuation is delivered ahead of an hours-old backlog for the same agent, and the backlog keeps draining. - New: many fresh continuations for one agent are enqueued one per pass, newest first, and none is left behind. - New: the backlog keeps a share of each pass while fresh continuations fill the company cap. - New: a refused fresh wake stays pending and is enqueued on the next pass. - New: a folded system wake leaves a skipped row. - Updated: existing tests that used a recent source run to mean "held because the queue is full" now use backlog-age sources, and assertions that depended on exactly one admission per agent per pass now check the record under test. ## Verification - Red before the fix: the 6 new tests failed. The 18 earlier tests still passed. - Green after the fix: `cd server && npx vitest run src/__tests__/recovery-wake-herd.test.ts` gives 24 passed. - `npx vitest run src/__tests__/execution-control-work.test.ts` gives 27 passed. `src/__tests__/issue-recovery-actions.test.ts` gives 112 passed. `src/services/fresh-continuation-fallback.test.ts` gives 62 passed. `src/services/workspace-recovery-fallback.test.ts` gives 19 passed. `src/services/native-runtime/native-safe-replacement.test.ts` gives 62 passed. - `cd server && npx tsc --noEmit -p tsconfig.json` passes. ## Risks - Behavior shift: an agent with a full queue now gets fresh recovery continuations queued at once. They wait in its queue, where the dispatcher ranking (person-initiated first, recovery share) decides when they start. They are no longer held outside the queue. The per-company cap per pass still limits the burst. - Old backlog drains at a similar pace as before. It always gets at least one slot per company per pass when it waits. - The skipped wake row adds one small row each time the sweep retries a task that has a pending continuation. The sweep spaces its retries, so the volume is low. - No migration. No API or UI contract change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, read-only staging API calls). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>