mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
1143abaf92
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The workspace recovery worker saves the files of failed runs. It also re-arms exhausted holds under the rules of #20295, #20569, #20641 and #20648 > - After #20680 and #20661 were deployed to staging, the worker still re-armed none of 29 exhausted holds in 36 minutes. One hold's id sorts in the first percent of all candidates, so the restart starvation that #20680 fixed was not the only cause > - The worker skipped every obligation while any unfinished save was selectable. Discovery is instance-wide. A save that another server owns stays unfinished on every run, so obligations never ran. The obligation pass also put due holds behind hundreds of settled holds whose saved input waits, at two per run. A completed pass never revisited a hold that became due by time alone > - This pull request gives due exhausted holds their own lane, and it shares the worker's slots fairly between unfinished saves, due holds and the obligation pass > - The benefit is that every due hold is reached within a few runs of the worker, whatever its other load. The holds save their files and free their tasks without a person ## Linked Issues or Issue Description Refs #20680 Refs #20661 Refs #20648 Refs #20641 Refs #20569 Refs #20295 **What happened?** On staging, 36 minutes after a deploy that included #20680 and #20661, the workspace recovery worker had touched none of the 29 exhausted holds of the company (22 capture-policy holds and 7 stop-proof or fallback holds). They had no `stopProofRecovery` or `capturePolicyRecovery` marker, no re-arm activity, and `updatedAt` from Nov 2 to Nov 10. The worker did run manual "Recover saved files" requests: one stop-proof hold saved its files 57 seconds after the request. So the worker was alive and the stop proof was available. The holds were not reached. **What did you expect to happen?** The worker reaches each exhausted hold whose re-arm or fallback is due, re-arms it, and saves the files, within a bounded time. **Steps to reproduce** 1. Create a scheduled save whose run's finalization lock another database session holds (another server owns the run). 2. Create an exhausted stop-proof hold and an exhausted capture-policy hold, in the evidence shape that the Nov 4 and Nov 10 releases saved. 3. Start the real workspace recovery worker. 4. The worker retries the busy save on every run and never re-arms either hold. ## What Changed - `server/src/services/workspace-recovery.ts`: `reconcileRetainedWorkspaces` draws candidates from lanes. Each lane is a page that its own cursor advances: unfinished saves; for the worker, exhausted holds whose re-arm or fallback is due; and the obligation pass. Before, the obligation pass ran only when no unfinished save was selectable (the gate at `const unfinished = rows.length > 0; if (!rows.length) { ... }`). - Each lane with work takes one free slot before any lane takes a second. The lane that goes first rotates on each run, so with one free slot (a long local export holds the other) each lane still gets it in one run of three. Actions of a run that one lane took join that run's job without another slot. - The due lane wraps at its tail. A hold that becomes due by time alone (a lease expiry or a re-arm backoff) is reached after the obligation pass completed with no committed change. - `retainedWorkspaceCandidates` takes a `dueOnly` filter (due re-arm or fallback of an exhausted hold). The obligation filter uses the same condition. One `now` is used for the whole query. - The startup sweep (no worker sweep state) also shares its slots between unfinished saves and its general pass. - `doc/execution-semantics.md`: document the lanes and the fairness rule. - New `workspace-recovery-passive-fairness.test.ts` with four real-Postgres regressions. ## Verification - Red on `origin/master` (156212cfd5), green with this change: - "re-arms both staging holds while a save that another server owns stays unfinished": the real worker, a busy unfinished save, and the two staging hold shapes. On master neither hold is re-armed in 20 s. - "reaches due holds within two runs however many settled holds with waiting input sort before them": ten settled holds with waiting user input sort before the two holds. On master the capture-policy hold is not re-armed after two runs. - "re-arms a hold that becomes due after its pass completed, with no committed change": on master the hold is never re-armed. - "re-arms the holds with one free slot while a long local job holds the other and a busy save stays unfinished" (from the Greptile review): red on this PR's first revision (05b261e5fe), green with the lane rotation. - `npx vitest run` of the twelve suites that use discovery or the worker (`heartbeat-observed-workspace-preservation`, `server-startup-feedback-export`, `execution-control-work`, `workspace-recovery-preparation-dispatch`, `-retry-copyback`, `-fallback`, `-warm-handoff`, `-preparation-gates`, `-capture-policy`, `-preparation`, `-stop-proof`, `-admission`) plus the new suite: 646 passed. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input. Required CI supplies the broad coverage. - Staging (read-only GET requests): the 29 exhausted holds, their saved shapes, and the manual Retry timeline. Staging exposes only one company to this account and no worker logs, so the exact unfinished row that starved the pass on staging is not named. The fix removes the whole class: no lane can starve another. ## Risks - Previous-release story: no persisted state changes. Lane positions stay in memory, as the existing cursors did. Holds saved by earlier releases (no marker, no capture policy version) are due at once and are reached in the first runs after deploy, with no migration. A deploy that lands mid-run only resets the in-memory cursors. - Behavior shift: obligations now share the two slots with unfinished saves. With two free slots, a new save or a manual Retry gets one in two runs of three. With one free slot, it gets it in one run of three. - A due hold that its re-arm refuses again is revisited once per due-lane pass. The worker already ran every interval while such a hold was due, so this adds no new wakeups. - One more bounded query per worker run (the due lane, 5 rows). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, code editing, browser GET reads) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>