Files
PaperClipAI/server
DottaandClaude Opus 5.5 69c69e4c20 fix(recovery): finish the passive recovery pass before restarting it (#20680)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The workspace recovery worker saves the files of failed runs. One
passive pass of the worker visits holds with obligations: saved input
that waits, cleanup, and due re-arms and fallbacks of exhausted holds
> - #17741 restarts that pass after each committed workspace-recovery
change, so new work is found quickly
> - On staging, changes commit continuously (stop receipts, new holds,
recovery progress). Every pass restarted from the first candidates,
which are settled holds whose saved input waits behind another hold, and
it never reached later candidates
> - After #20295, #20569 and #20661 were deployed, no due re-arm ran:
five stop-proof holds and about twenty capture-policy holds kept their
exhausted state, with no marker and no activity
> - This pull request restarts the pass only after it reaches its tail,
and it continues an unfinished pass on the next run
> - The benefit is that every candidate is visited on a busy instance,
so the exhausted holds re-arm, save their files and free their tasks
without an administrator

## Linked Issues or Issue Description

Refs #17741
Refs #20295
Refs #20569
Refs #20661

**What happened?**
Exhausted workspace holds were not re-armed on staging, even after the
fixes that add their re-arm were deployed. Their recovery actions stayed
unchanged for days (`updatedAt` on Nov 5 or Nov 10). No
`stopProofRecovery` or `capturePolicyRecovery` marker and no
`issue.workspace_recovery_rearmed` activity was recorded. The worker did
process new restore holds, which are selected by its unfinished pass.

**What did you expect to happen?**
The worker's passive pass reaches every hold with a due re-arm or
fallback, re-arms it, and saves the files.

**Steps to reproduce**
1. Create several settled holds with saved user input that waits, with
action ids that sort before an exhausted stop-proof hold.
2. Start the real workspace recovery worker.
3. Commit a workspace-recovery notification every 50 ms, as stop
receipts and new holds do on a busy instance.
4. The worker never re-arms the exhausted hold. Without the
notifications, it re-arms the hold.

## What Changed

- `workspace-recovery-work.ts`: a committed change marks the passive
pass for restart (`restart`). It no longer resets the cursor. While a
restart is pending, an unfinished pass that made progress continues on
the next run.
- `workspace-recovery.ts`: when the passive cursor passes the last
candidate and a restart is pending, the same run starts a fresh pass
from the first candidate, so a new change is not delayed by another run
interval. Without a pending restart the pass records `complete` and does
not wrap, so paused input does not keep the worker active.
- `doc/execution-semantics.md`: document that a pass is never restarted
midway.
- `workspace-recovery-stop-proof.test.ts`: add a real-worker regression
in the staging shape (six earlier settled holds with waiting user input,
one exhausted stop-proof hold, continuous notifications).

## Verification

- Red: on master, the new test "re-arms the hold on a busy instance
whose passive sweep keeps selecting earlier settled holds" fails
(`stopProofRecovery` stays undefined). Control: the same test without
the notifications passes on master. This shows that the restarts cause
the starvation.
- Green: with the fix, the test passes. These suites pass:
`workspace-recovery-stop-proof` (12), `execution-control-work` (27),
which includes "visits a newly committed earlier Retry during a busy
recovered-source pass, then becomes quiet", plus
`workspace-recovery-capture-policy`,
`workspace-recovery-preparation-dispatch`,
`workspace-recovery-warm-handoff`, `workspace-recovery-preparation` and
`server-startup-feedback-export` (279). The timing-sensitive
`heartbeat-observed-workspace-preservation` suite passed 6 of 6 runs (an
earlier revision that restarted only on the next run made it flaky).
- Staging evidence (read-only GET requests): the 28 exhausted holds of
the company have no re-arm marker and no fallback check. Many blocked
tasks with settled holds have deferred wakes.

## Risks

- A change committed during a pass is now picked up at the next pass,
not at once. Unfinished physical work still uses its own pass first, so
new recoveries are not delayed. Saved input in a passive obligation can
wait up to one full pass.
- Previous-release story: no persisted state changes. Holds saved by
earlier releases are selected on the first full pass after deploy, with
no migration.
- Quiet behavior stays: a completed pass with no new change keeps the
worker idle.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
code editing, browser GET reads) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 20:44:09 -06:00
..