mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
c4927e1ec2
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade stops the old server gracefully. A legacy run that is still preparing ends `interrupted` with `server_shutdown_interrupted`. The new server must continue it automatically (#21019, #21046). > - A run that a person started with Retry (or a message) carries `explicitUserContinuation`. Its automatic retries need their own receipt from `admitExplicitContinuationRetry` (#14987, #19384). > - That admission refused every shutdown-interrupted Retry run. The task then stayed in progress with no run, and the stranded-task sweep refused it again about every 2.5 minutes, forever. A person had to press Retry. > - An upgrade is routine. It must never leave a task that needs a manual Retry. > - This pull request lets the person's original authorization cover a continuation after a shutdown that stopped the run before provider work started. All other checks of the gate stay. > - The benefit is that Retry-started tasks continue through an upgrade, and tasks that earlier releases left waiting continue on their own after deploy. ## Linked Issues or Issue Description Refs #21019, #21046, #14987, #19384 **What happened?** On a hosted staging server, each automatic upgrade stopped legacy runs that were still preparing (`executionRecovery {kind: bootstrap, providerWorkStarted: false}`). When the run came from a person's Retry (wake reason `task_retry`), the continuation was refused with "The automatic retry could not revalidate the original user continuation." (`continuation_user_authorization_missing`). The task stayed `in_progress` with no run and no next retry. Runs from older upgrades were refused again by each sweep. A run from the same upgrade that did not come from a Retry got its continuation in the `interrupted_preparation` lane. **Expected behavior** A Retry-started run that a graceful shutdown stopped before provider work continues automatically, once, under the person's original Retry. Tasks that an earlier release already refused continue after deploy with no manual step. **Root cause** Three conditions refused it (`server/src/services/heartbeat/retries.ts`, `server/src/services/explicit-native-continuation.ts`): 1. The scheduler admits an explicit continuation only for the `transient_failure`, `server_restart`, preparation-wait and agent-directory lanes. A shutdown-interrupted preparation with unspent failure budget uses `interrupted_preparation`, so it was refused before admission ran. 2. `admitExplicitContinuationRetry` accepted only a `failed` or `timed_out` parent. These parents are `interrupted`. 3. It required `task.executionRunId` to be the parent. The refusal released the claim (`releaseIssueExecutionAndPromote`), so every later sweep was refused by this check too. **Paperclip version or commit** master `f30005440b`. ## What Changed - `retries.ts`: an explicit continuation may also use the `interrupted_preparation` lane. Lane choice is unchanged (see below). - `explicit-native-continuation.ts`: `admitExplicitContinuationRetry` also accepts a parent that is `interrupted` by a graceful shutdown (`isServerReplacementInterruption`, so `server_shutdown_interrupted`) with bootstrap evidence `providerWorkStarted: false` (`isShutdownInterruptedPreparation`). - For such a parent only, a released claim (`executionRunId` null) is accepted. Admission restores the claim to the parent under the task lock, as the shutdown path leaves it, so the scheduler passes the claim to the successor and dispatch accepts it. If the parent validation then refuses, admission releases the claim again in the same transaction. A claim that another run holds still refuses. - `doc/execution-semantics.md`: describe the rule and the conditions that still refuse. Also apply the sentence that #19384 left pending (a plain task Retry is renewable). - No schema change, no backfill. Nothing changes on the path that starts a turn. **Which lane a graceful-shutdown legacy preparation uses (after #21046).** `scheduleBoundedRetryForRun` sets `interruptedPreparation` for an `interrupted` run with the bootstrap evidence. `usesServerRestartLane(run, true)` returns false while the failure budget is unspent, so the run uses `interrupted_preparation`: it does not spend the failure budget, the first continuation is due at once, and it is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`. With a spent failure budget it uses `server_restart`. This is the intended design (#21046 comment on `usesServerRestartLane`), so no lane change. The `server_restart` lane is for `process_lost` replaced preparations and for runs past dispatch. **Conditions that still refuse** (each has a test or an existing test): - The task has another assignee, or is `done` or `cancelled`. - Another run holds the task's claim. - Newer work on the task (a run created after the parent, a retry child, or a live run). - The Retry request changed: other actor, not admitted, other generation, bound to another run, a chat Retry, a fresh-workspace Retry. A message continuation with an edited or deleted message. - A missing or revoked receipt (dispatch validation of the parent fails). - An execution blocker on the task, or a live adapter controller for the parent. - An `interrupted` parent with another cause (for example an operator interrupt), or with no bootstrap evidence, or with `providerWorkStarted: true`. A parent that failed after provider work keeps the existing rules (`failed`/`timed_out` only). - Native runtime parents (unchanged, see below). **Native runtime.** A native `paperclip_runner` run is not interrupted by a graceful shutdown: `drainRunningRunsForShutdown` suspends it and the next server reattaches it (`restart_suspended`). It does not reach this gate. Native replacement paths (`native-safe-replacement`, `native-provider-capacity-retry`) remove `explicitUserContinuation` from the successor and use their own contract. So the native path does not have this refusal, and this pull request does not change it. ## Verification - `server/src/services/task-retry.test.ts`, new tests. Red on `origin/master` (no successor), green on this branch: - "continues a Retry-admitted run that a graceful shutdown stopped while it was preparing": production shape. A legacy run admitted by a task Retry, `preparing`, owned by this server, with bootstrap evidence. `drainRunningRunsForShutdown("SIGTERM")` interrupts it with `server_shutdown_interrupted` and schedules one `interrupted_preparation` successor (`failureRetries: 0`, `interruptedPreparations: 1`) with its own receipt. The claim passes to the successor and dispatch accepts it. A concurrent stranded sweep and a second schedule call reuse the same successor. - "continues a task an earlier release left waiting after refusing its shutdown continuation" (a direct Retry run, and an automatic retry of a Retry run): the row and task as an older release left them (claim released). Two concurrent stranded sweeps schedule exactly one successor; the claim passes to it; dispatch accepts it. - "does not continue a shutdown-interrupted Retry run without its original authority" for: provider work started, no bootstrap proof, another interruption cause, claim held by another run, reassigned, superseded, other Retry actor, cancelled task. No successor, no receipt, and the claim is left as it was. - I confirmed that each source change is needed: without the lane change both main tests fail; without the released-claim rule the older-release test fails. - Suites: `task-retry`, `explicit-native-continuation`, `legacy-explicit-continuation-recovery`, `quarantined-task-retry` (587/587); `heartbeat-process-recovery`, `heartbeat-retry-scheduling` (520/520). - `tsc --noEmit` for `server`. - `pnpm verify` selects a broad run (exit 2) because `retries.ts` is central; required CI covers that. **After deploy, the next audit should show:** a Retry-started run that a shutdown stops while preparing gets a `scheduled_retry` successor with `scheduledRetryReason: interrupted_preparation` (or `server_restart` with a spent failure budget) and no `continuation_user_authorization_missing` event. The tasks that earlier upgrades left waiting get one such successor on the first stranded sweep of the new server, with the task claim on the successor. ## Risks - Behavior change: a shutdown-interrupted preparation of a Retry or message turn now continues automatically, within the existing uncharged and bounded lanes. The run did no provider work, so there is nothing to reconcile. - Rows saved by older releases: a refused row whose claim was released is continued once. If another run took the claim, or newer work exists, it stays refused. - Two servers at once: admission runs in the scheduler transaction under the task and run row locks, and the scheduler reuses an existing successor (`retryOfRunId`). The tests run two sweeps concurrently and get one successor. An old server that still refuses cannot clear the claim after the new server moved it to the successor (its release matches only the stopped run). - The claim restore writes `executionRunId` for a terminal run inside the scheduling transaction. If admission refuses after the restore, it releases the claim again before the transaction commits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, Vitest on embedded PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>