mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Hosted servers are upgraded often. Each upgrade stops the old server gracefully, and a run that is still preparing ends `interrupted` (`server_shutdown_interrupted`) with bootstrap evidence that provider work did not start. > - #21078 let a run that a person started with Retry continue automatically after such a stop. After it deployed, a read-only audit of a staging server found four groups of Retry-started tasks that still did not continue: some were refused on every sweep, some were never looked at, and one Retry that a person pressed failed and was not retried. > - Product rule: an upgrade never leaves a task that needs a manual Retry, and a Retry a person presses must work. A refusal that cannot succeed later must not repeat forever. > - This pull request fixes each cause found, keeps every authorization check of the continuation gate, and ends the refusal loop with one plain escalation. > - The benefit is that Retry-started tasks continue through upgrades, including tasks earlier releases left waiting, with no manual step. ## Linked Issues or Issue Description Refs #21078, #21046, #21019, #14987, #19384 **What happened?** On a hosted staging server after #21078 deployed (read-only audit, run events and task activity): 1. A Retry-started task was refused every ~2 minutes ("The automatic retry could not revalidate the original user continuation."). While preparing, its Retry had switched to a fresh isolated workspace because the saved files could not be restored (activity `issue.fresh_workspace_selected` by the system, 40 minutes after the Retry). That rewrote the Retry request's strategy from `continue` to `fresh_isolated_workspace`, and `readTaskRetryRetryAuthority` accepts only `continue`. 2. Two tasks whose Retry replaced a native session had no recovery event for hours. The stranded-task sweep skipped them silently. 3. A task whose Retry selected a fresh workspace at admission (`explicitUserContinuation.readiness`) was refused on purpose ("fresh-workspace Retries keep their own contracts"), yet that contract had no path for a shutdown before provider work. The task stayed in progress after every upgrade. 4. A person pressed Retry. The run failed before provider work with `reusable_sandbox_resume_failed`, a transient failure that is already in the bounded bootstrap retry lane. One second later the stranded-task sweep escalated the task to `blocked`, which cleared the claim. Two seconds after that, the run's own failure path scheduled its retry and was refused because the claim no longer named the run. **Expected behavior:** each of these continues automatically once, under the person's original Retry. A refusal that cannot succeed later escalates the task once with a working Retry. **Root causes:** - Cases 1 and 3: the continuation gate refused every fresh-workspace Retry, whether the fresh workspace was chosen at admission or later in preparation. That includes one that a shutdown stopped before it did anything. - Case 2: the sweep's generic skips run before the continuation. Two of them can match only because of state that the replaced session left. An unfinished session goal (`agent_task_sessions.goal_status`) is skipped as "goal recovery owns it". A native finalization row is counted as a live execution path by `hasActiveExecutionPath`. The person's Retry had already started a fresh session, so neither owns the task. The audit could not read the database, so which of the two applied on staging is not proven; the tests cover both. - Case 4: a race. The sweep does not wait while a terminal run's own executor is still settling it, so it escalates before that executor schedules the run's bounded retry. - The refusal loop: after a refusal the sweep left the task `in_progress` and tried again on the next pass. **Paperclip version or commit:** master `482063aeca`. ## What Changed - `explicit-native-continuation.ts`: `readTaskRetryRetryAuthority` renews a fresh-workspace Retry **only** when the parent run was stopped by a graceful shutdown before provider work (`isShutdownInterruptedPreparation`). The admitted `readiness` must name the same Retry and source. A strategy that preparation switched to fresh is rebuilt exactly as preparation records it: this Retry, its source, and the source's leases. An automatic retry of the fresh attempt inherits the selection from its own receipt. The successor's receipt records the selection under `automaticRetry.readiness`. The successor's `explicitUserContinuation` keeps its plain shape. - `fresh-workspace-continuation.ts`: `readFreshWorkspaceReadiness` reads that receipt for an automatic continuation, so the successor prepares the same fresh attempt. A remote fresh workspace is keyed by the Retry, so it is reused rather than created again. `freshReadinessSchema` is exported and reused. - `recovery/service.ts`: - **Shutdown-stopped Retry preparations go first:** at the start of the stranded sweep, every `in_progress` task whose latest run is a legacy `server_shutdown_interrupted` preparation (`providerWorkStarted: false`) carrying `explicitUserContinuation` is offered to the scheduler. It is skipped when the agent is not invokable, the task has a live execution path, a board-owned recovery action or a pause hold, or the run is still settling. The unfinished-goal skip does not apply here, because the person's Retry started a fresh session. - `hasActiveExecutionPath` ignores a native finalization that a person's Retry already replaced (`failureDetail.replacementDenied = explicit_user_continuation` on a terminal run). - The sweep skips a task while its latest terminal run is still being settled: a live execution in this process, or another server's unexpired controller lease. It checks again when that lease expires. - A refused continuation (`continuation_user_authorization_missing`) escalates the task once to `blocked` with the plain notice "Retry needed … Press Retry to continue." Before, the task stayed in progress and was refused again on the next pass. - `heartbeat.ts`: `scheduleRecoveryRetry` reports a refused continuation to the sweep. - `doc/execution-semantics.md`: describes these rules. **Failure classes admitted for a Retry-started run (case 4):** no new class. The run's own bounded retry lanes are unchanged. `reusable_sandbox_resume_failed`, `remote_workspace_restore_failed`, `plugin_worker_unavailable`, `environment_provider_unavailable`, `repository_catalog_unavailable`, runner revision mismatch and the other transient bootstrap failures were already retried before provider work. The fix removes the race that took the claim away from that retry. **Refusals that remain by design:** - The task is reassigned, `done` or `cancelled`. - Another run holds the claim, or newer or live work exists. - The Retry actor, generation or run changed, or it is a chat or queued-message Retry. - The message was edited or deleted. - The receipt is missing or revoked. - The run had another interruption cause, or provider work started. - A fresh-workspace Retry whose run *failed* (not a shutdown) keeps its one-run contract. - A readiness that names another Retry or another source. Each of these now escalates the task once instead of refusing on every sweep. ## Verification - `server/src/services/task-retry.test.ts`, new tests. Red on `origin/master` sources and green on this branch (checked by restoring the master sources): - "continues a fresh-workspace Retry that a shutdown stopped before provider work": - Runs twice: once with the fresh workspace selected at admission, once with it selected at preparation. - Two concurrent sweeps create one successor, which gets the claim, and dispatch accepts it. - `readFreshWorkspaceReadiness(successor)` returns the same fresh attempt. - A second upgrade stops the successor, and the next successor carries the same attempt. - "continues a shutdown-stopped Retry whose replaced session left state behind": two cases, an unfinished session goal and a native finalization that the Retry replaced. - "escalates a refused shutdown continuation once instead of refusing it on every sweep": two sweeps. The task ends `blocked` with one "Press Retry to continue" notice and one refusal event. - "leaves a failed Retry run to its own executor while that executor still settles it": - Production shape: `reusable_sandbox_resume_failed` and a live controller lease from the server that ran the run. - The sweep leaves the task and its claim alone. - The run's bounded retry is then scheduled and dispatch accepts it. - "keeps a fresh-workspace Retry's own contract when it does not match": three refusal cases (other Retry, other source, failed rather than stopped). Green on both. - Existing suites pass: - `task-retry`, `explicit-native-continuation`, `heartbeat-retry-scheduling`, `heartbeat-process-recovery`, `interrupted-preparation-hold-settle`, `heartbeat-issue-liveness-escalation`, `issue-recovery-actions`, `recovery-deadline-acceptance`, `legacy-continuation-authority`, `workspace-recovery-admission`, `native-workspace-export-retry`, `heartbeat-paused-assignee-new-task`, `issue-tree-pause-resume-run`, `heartbeat-stale-queue-invalidation`. 1,619 tests passed, with one failure that a change in this branch fixed; rerun of the affected suites: 560/560. - The one failure came from the first version of the settling check, which also waited for this process's own lease after a shutdown drain. The check now uses this process's live executions and other processes' leases only. - `tsc --noEmit` for `server`. - Required CI covers the full suite. **After deploy, the next audit should show:** - Tasks stopped by a shutdown while their Retry was preparing get one `scheduled_retry` successor on the first sweep. This includes tasks whose Retry selected a fresh workspace and tasks whose Retry replaced a native session. - They show no further "could not revalidate" events. - A Retry-started run that fails transiently before provider work gets its bounded retry, and the task is not escalated in between. - A task whose continuation is truly refused is `blocked` once with "Retry needed". ## Risks - **Behavior change:** the sweep now offers shutdown-stopped Retry or message preparations to the scheduler before its generic skips. All authorization checks still run at admission, under the task and run row locks. The scheduler reuses an existing successor, so two servers create one continuation. The tests run two sweeps at once. - **Settling wait:** a task whose latest terminal run still has another server's live controller lease waits until that lease expires (at most 90 s) before the sweep acts. Normal turn starts are unchanged, and nothing is added before a turn starts. - **Escalation on refusal:** a refused continuation now moves the task to `blocked` with a Retry notice, where before it stayed `in_progress` with no path. Pressing Retry is a new, separately admitted request. - **Rows from older releases:** no schema change and no backfill. A shutdown-stopped fresh-workspace Retry is continued once. An older successor receipt without `automaticRetry.readiness` behaves as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, read-only API reads, Vitest on embedded PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>