mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 07:42:38 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent's provider account recently reported a usage limit,
Paperclip holds new work instead of starting a run that would fail the
same way. The provider limit wait cancels the run, then schedules a
retry at the expected reset, and the retry takes over the task's
execution lock
> - The cancellation and the retry are two separate writes. Cancelling
the run wakes task recovery for that task, and its stale-lock sweep
treats a cancelled run's lock as stale and clears it
> - When the sweep wins that race, the retry scheduler finds the lock
gone and refuses to schedule ("Subscription retry suppressed because the
task execution lock changed"). The wait then releases the task, and
stranded-work recovery escalates it to a person with a Retry button
> - The stranded-work reconciler already waits while a terminal run's
own executor is still settling it (`terminalRunSettlingUntil`). The
stale-lock sweep did not
> - This pull request makes the stale-lock sweep wait for the same
executor before it clears an open task's lock, and look again at the
executor's deadline
> - The benefit is that a task waiting for a provider usage limit
resumes by itself at the reset, instead of stopping on "Couldn't finish
this attempt" until someone presses Retry
## Linked Issues or Issue Description
Refs #19246, which added the provider usage limit wait.
**What happened?**
An agent's provider account hit a usage limit. A task's next wake was
held for the reset: its run was cancelled with `provider_limit_wait`.
About 200 ms later the stale-lock sweep cleared the task's execution
lock, because the run was now cancelled. The wait then tried to schedule
its retry, found the lock gone, and released the task. The continuation
that followed hit the same race. After two lost retries, stranded-work
recovery blocked the task and asked for Retry.
On one instance, 80 of about 600 provider-limit waits in two days lost
their retry this way, across 43 tasks. 15 of those tasks were blocked
behind a Retry button.
**Steps to reproduce**
1. Give an agent a recent provider rate-limit failure with no reported
reset.
2. Wake the agent on an assigned task.
3. Let task recovery sweep the task's locks between the cancellation and
the retry (the new integration test does this from the cancellation
write).
**Expected behavior**
The task keeps its lock for a retry scheduled at the reset.
**Actual behavior**
The sweep clears the lock, no retry is scheduled, and the task waits for
a manual Retry.
## What Changed
- `recovery/service.ts`: `sweepStaleIssueLocks` keeps an open task's
lock while a run that holds it is still being settled by its executor,
using the existing `terminalRunSettlingUntil`. That is a live execution
in this process, or an unexpired controller lease in another one. The
sweep reports the deadline through `onDeferredDeadline`, so it looks
again once the executor is done. A closed task's lock is still cleared
at once.
- Tests:
- The stale-lock sweep keeps an open task's lock while the executor is
live, and clears it once the executor is gone. A done task's lock is
cleared either way.
- A provider-limit wait whose cancellation is swept straight away still
schedules its retry, and the retry holds the task's lock.
## Verification
- `npx vitest run` on the new cases: pass. Both fail without the source
change; the integration test then reproduces the incident (the sweep
clears the lock and no retry is scheduled).
- `npx vitest run` on 58 related server test files (stale-lock sweep,
stranded and assigned-work recovery, provider limit wait, native resume
locks, retry admission, recovery hints): all pass, except 6 cases in
`runtime-selection.test.ts` that read a skill file from a fixed local
path. They fail the same way on master on this machine.
- `npx tsc --noEmit` passes for the server.
## Risks
- Low. The sweep waits only while the run's executor is provably still
running: a live execution in this process, or another process's
unexpired controller lease. A crashed executor has neither, so the sweep
clears the lock as before.
- An executor that keeps running after its run is terminal now delays
the sweep's backstop clear of an open task until it finishes. The
executor releases or hands off the lock itself when it finishes, and the
stranded-work reconciler already waits for the same executor.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>