mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a server restarts or a deploy swaps servers, agent runs are interrupted. The fresh-continuation fallback (#21135) continues each interrupted task in a new run from its saved work. > - The fallback also has a backlog pass. It continues tasks that older releases left held by a recovery record. > - The backlog pass had no age limit and no pacing. It took the oldest records first, 25 per pass. > - On the first hosted release with the fallback, it continued 176 tasks in 30 minutes. 74 of them had a source run that ended more than 7 days ago. The oldest ended four weeks ago. More than 100 runs were queued, and one agent had about 130 runs. > - The user-facing goal of the fallback is "an interruption is invisible". It is not "run last week's failures again". > - This pull request limits the backlog pass to recent interruptions, newest first, at a paced rate. > - The benefit is that a release no longer starts a herd of old, unwanted runs, and recent interrupted work still continues by itself. ## Linked Issues or Issue Description Refs #21135 **What happened?** A hosted instance took its first release with the fresh-continuation fallback. Within 30 minutes, the backlog pass continued 176 held tasks. By source run age: 26 under 1 day, 76 from 1 to 7 days, 74 over 7 days. The source end codes included `adapter_failed` (27), `runner_remote_provider_artifact_incompatible` (14), `native_provider_terminal_failed` (14) and `setup_failed` (8). These are configuration or provider failures, not interruptions. The pass kept firing at 12 to 36 continuations per 5 minutes, and the queued backlog grew past 100 runs. **Expected behavior** The backlog pass continues a task only when the task was recently interrupted by a restart, a deploy, a lost lease, a lost process or a refused recovery. Old records and configuration or provider failures stay as they are, for a person's Retry. The pass starts new runs at a controlled rate, newest first. **Steps to reproduce** 1. Have held recovery records that older releases saved, with source runs that ended days or weeks ago, some with `adapter_failed`. 2. Deploy a release that contains #21135. 3. See that the disposition worker continues all of them, oldest first, 25 per pass, and queues a run for each. ## What Changed - `continueHeldExecutions` and `heldExecutionsAwaitingFreshContinuation` (server/src/services/fresh-continuation-fallback.ts): - Continue a held record only if its source run ended within `BACKLOG_CONTINUATION_MAX_AGE_MS` (24 hours) AND its error code is in `BACKLOG_CONTINUATION_SOURCE_CODES` (restart, deploy, lease, process-loss and recovery-refusal codes). - Do not touch any other held record. Do not settle it and do not modify it. A log line counts these records, at most once every 10 minutes. - Order by the source run's end time, newest first. - Continue at most 5 records per pass and 10 per rolling minute per server. - The rule I chose, and why: 24 hours and interruption codes only. A person who is waiting for a task notices within a day. A run that failed for configuration or provider reasons fails again the same way, so a person must change something and press Retry. The 7-day rule of the assigned-work sweep is too wide for this pass: 76 of the 176 continued tasks were from 1 to 7 days old. - The live path does not change. Fresh interruptions come through restart recovery and the disposition pass (`recovery_refused`), and through continuation wakes. They are not paced and have no age rule. - Upgrade safety: records that a release already continued carry `freshContinuation` and are excluded as before. Queued continuation runs are not changed. - The 10-per-minute limit is per server. During a deploy overlap, two servers can each continue 10 per minute. This is acceptable for a short overlap, and it keeps this change free of shared state. ## Verification - `cd server && npx vitest run src/services/fresh-continuation-fallback.test.ts`: 53 passed. - New test "continues only recent interruptions from the held backlog, newest first, and leaves the rest untouched (staging herd)". It uses mixed-age and mixed-cause records: 1 h and 6 h restart codes, 3-day and 28-day restart codes, and 2 h `runner_remote_provider_artifact_incompatible` and `adapter_failed`. Only the two recent restart-caused records continue, the 1 h record first. The four other records keep the same status, evidence and `updated_at` across repeated passes. - New test "paces the held backlog to ten continuations a minute per server": 12 eligible records give 10 continuations in the first minute and the remaining 2 in the next minute. - RED on origin/master: both new tests fail (`expected [] to have a length of 1`; `expected 12 to be 10`). - The existing tests for the live path (for example "continues tasks that older releases left blocked by these codes", with a source run that ended one minute ago) still pass. - `pnpm --filter @paperclipai/server run typecheck`: clean. ## Risks - A held record from 1 to 7 days ago, caused by an interruption, no longer continues by itself. It stays as it was before #21135: the task shows its recovery state and a person can press Retry. - A record whose cause is not in the list (a future interruption code) is not continued by the backlog pass. The live path still continues fresh interruptions for any code that is not a person's. - No migration. No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>