mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops before it finishes, automatic recovery continues the task, through the fresh-continuation fallback, in a new run that starts from the saved work > - A follow-up run (for example one started by a new comment or by finished subtasks) can fail before any provider work and leave a saved-work hold. The system then queues its own retry, and the dispatch gate cancels that retry at claim with `execution_reconciliation_required` because the hold is still in place > - The fallback takes over the hold when that happens, but it continued only "recent interruptions". That test required the failed run to be a recovery run itself or to fail with a restart code. A first follow-up run that failed with `adapter_failed` passed neither test, even though the system had already decided to retry it. So the fallback answered "a person must press Retry". The backlog pass used the same test, and the assigned-work sweep never listed the task. The task stayed blocked with no run, no pending path and no Retry button > - This pull request treats a system retry refused at dispatch as proof that the system continues the task. It also stops cancelled-at-dispatch attempts from counting against the hourly budget, lets the sweep continue a blocked task whose latest run was refused at dispatch, and keeps recovery notices from asking for a Retry that the task does not offer > - The benefit is that such a task continues by itself in exactly one fresh run from the saved work, whether the saved work was recovered before or after the refusal, and a person's own failed Retry keeps today's behaviour ## Linked Issues or Issue Description Refs #21394, #21348, #21363, #21478 **What happened?** A task's follow-up run, a legacy run that was not a recovery run, failed before any provider work with `adapter_failed`. Its files could not be restored, so it left a saved-work hold (`legacy_execution_requires_reconciliation`, `workspaceRestoreFailure: restore_failed`). The system queued a retry of that run. The dispatch gate cancelled the retry at claim with `execution_reconciliation_required`, and the "older run hold" path handed the hold to the fresh-continuation fallback (`run-dispatch/adapters/postgres.ts` `settleOlderRunHold`). The fallback found the refused run and asked `interruptionEligibility` whether the interruption was recent. That predicate required `continuesAutomaticallySql(source)` (a recovery run, or a restart, deploy or bootstrap code), so it returned `person_owned` / `retry_needed`. The stranded reconciler then blocked the task with "Press Retry", but the saved-work hold reports that Retry is not available. This happened twice in a local restart-chaos wave. In one case the saved work was recovered two minutes before the claim. In the other it was recovered thirty seconds after the claim. **Expected behavior** A task never rests blocked with no live run, no pending path and no Retry. While the saved-work recovery runs, that recovery is the pending path. When it finishes, one continuation starts from the saved work through the fallback (isolation, the shared hourly cap). A run that was cancelled at dispatch before it started does not use up the automatic budget. A person's own failed Retry is still left for that person. **Steps to reproduce** 1. Fail a follow-up run (wake reason `issue_children_completed`) before provider work with `adapter_failed` and `workspaceRestoreFailure: restore_failed`. 2. Queue the system's retry of that run (`retryOfRunId`, wake reason `issue_continuation_needed`) and claim it. The dispatch gate cancels it with `execution_reconciliation_required`. 3. Before this change, no continuation is recorded, the stranded reconciler blocks the task, and nothing ever continues it. **Paperclip version or commit** master at2756be2f32## What Changed - `fresh-continuation-fallback.ts`, `interruptionEligibility`: the interruption counts as one the system continues when the failed run's cause is system-owned and a later run that the dispatch gate refused before it started is a system recovery run (`recoveryRunSql`, which excludes a person's Retry). A failed run that was itself a person's Retry (`task_retry` or `explicitUserContinuation`) never qualifies this way. The 24-hour recency bound is unchanged. The claim-gate hand-off, the backlog pass and its deadline query all use this one predicate. - `recovery/continuation-ledger.ts`, `recovery/task-automatic-attempts.ts`: one shared rule, `startedOrPendingAutomaticRetry`. An automatic retry that the dispatch gate cancelled before it started (`cancelled`, no `startedAt`, `execution_reconciliation_required`) does not count toward the hourly cap in either ledger. - `recovery/assigned-work-sweep.ts`: - When the latest run was refused at dispatch, the sweep continues from the run it was refused for through the fallback. Before, it gave up. - A blocked task with an active hold and a recent refused-at-dispatch latest run is now listed as a suspect. The check hands it to the fallback before it answers `left_to_hold_owner`. A pending recovery becomes `waiting_on_recovery` with its recheck time. - Holds that wait for a person's configuration or confirmation are still excluded. - Review follow-up (`recovery/task-automatic-attempts.ts`, `recovery/assigned-work-sweep.ts`): - The task ledger counts a sweep's fresh-continuation record (`trigger: assigned_work`) itself. It no longer depends on a companion activity entry, and an activity entry marked `continuedFromSavedWork` is not counted a second time. Every sweep continuation counts once, whatever path recorded it. - When the sweep claims a continuation, it marks its claim entry `continuedFromSavedWork` inside the transaction that records the fresh continuation. A restart between the two writes can no longer leave one attempt counted twice. - The sweep's continuation behind a refused run spends the per-minute repair budget (`ASSIGNED_WORK_REPAIRS_PER_MINUTE`). The excess is deferred with a `retryAt`. - `recovery/stranded-notice.ts`, `recovery/service.ts`: when the task's execution blocker offers no Retry, the escalation notice and the recovery action's next action leave out every Retry instruction. They say that Paperclip is holding the task until its saved work is settled. - No change to the dispatch gate (`heartbeat/queue.ts`, the restore-hold predicate in `run-dispatch/adapters/postgres.ts`, or `execution-blocker.ts`). The recorded continuation settles the hold, so the delivered run is not refused again. ## Verification Red before the change, green after (embedded Postgres, real services: `terminalizeLegacyExecution`, the dispatch adapter's `cancelStaleQueuedRun`, `reconcileStrandedAssignedIssues`, `deliverReconciledExecutions`, `continueHeldExecutions`, `recoveryService().assignedWork`): - `server/src/services/fresh-continuation-fallback.test.ts` - "continues a follow-up … when the saved work was recovered before the system retry was refused at claim": red (`continuationDelivery: "invalidated"`, no continuation). Green: the continuation is recorded at the claim, and the stranded reconciler does not block the task. The delivered run is queued and passes the claim gate (`not_stale`). - "… recovered after the system retry was refused at claim": red (no pending check). Green: the claim records `exact_recovery_pending` / `saved_work_recovery_running`. When the recovery completes, the backlog pass continues the task, and the delivered run passes the claim gate. - "continues a follow-up … with its files in place, after its system retry was refused at claim": red, then green. This is the control without a restore failure. - "still leaves a person's failed Retry of a follow-up for that person …": stays `person_owned` / `retry_needed`, before and after. - "counts only automatic retries that started, or still may, against the task's hourly budget": new. - `server/src/__tests__/failed-continuation-continues-again.test.ts`, "the sweep continues a blocked task whose system retry was refused at claim behind a recovered saved-work hold": red (the task was not a suspect). Green: `repaired`, a pending continuation in the hold, the task back in `todo`, and no wake enqueued. The next check reports a pending intent. - Same file, "a continuation the sweep records behind a refused run counts once toward the task's hourly cap": red (5 counted after 5 attempts plus the new continuation), green (6, and the transient-retry cap check refuses the next attempt). - Same file, "continuations behind refused runs spend the sweep's per-minute repair budget": 20 tasks. Red (20 continued in one pass), green (10 continue, 10 are deferred with a `retryAt`, and the 10 continue in the next pass). - Same file, "a sweep continuation interrupted right after it was recorded still counts once": the continuation commits, then the post-commit activity write fails (a simulated server stop). Red: 6 counted for 5 real attempts. Green: 5 counted, and the next automatic retry is still allowed. - `server/src/services/recovery/stranded-notice.test.ts`, "never tells a person to press Retry when the task offers no Retry": new. Local runs on this branch, one file at a time: - `fresh-continuation-fallback.test.ts`: 77 passed. The suite's `afterAll` teardown hit the 30 s hook timeout on a heavily loaded machine. - `failed-continuation-continues-again.test.ts`: 32 passed - `assigned-work-invariant.test.ts`: 31 passed - `stranded-notice.test.ts`: 33 passed - CI on the first head (ce4431410b): every required check passed - `server` `tsc --noEmit`: clean - `pnpm verify --explain` selects most of the repository (2666 files), so required CI supplies the broad coverage. ## Risks - More tasks continue automatically. Each continuation goes through the existing fallback. The 24-hour recency bound, the per-task hourly cap (6), isolation of the next run, and every person-owned gate (person-owned failure codes, a person's Stop, review, a person-owned hold, a person's failed Retry) are unchanged. - The hourly cap now ignores automatic retries that never started. A task whose retries are refused at dispatch again and again is still bounded: the refusal hands the hold to the fallback, and the fallback records one continuation per source run. - Known separate gap, not changed here: after an interrupted preparation's holds are resolved (`settleInterruptedNativeBootstrap`), `scheduleBoundedRetry` can report `exhausted` or `blocked`, and nothing follows that up. The assigned-work sweep may now pick such a task up through the refused-at-dispatch path, but this pull request does not prove that. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>