Files
PaperClipAI/server
54972096ba fix(native): wait out a provider rate limit that refuses the session start (#22386)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts each agent run in a provider session on a
remote sandbox
> - A provider can refuse `session.open` because the account reached its
rate limit or quota
> - After one same-run retry, a repeat of that refusal ended the run as
`native_provider_startup_rejected`
> - That code blocks the task for a board Retry and sends the failure to
Sentry, but the limit passes by itself
> - This pull request records that refusal as a provider usage-limit
hold, which already exists for limits on a turn
> - The benefit is that Paperclip retries the task at the reset by
itself, and Sentry gets no error for an expected wait

## Linked Issues or Issue Description

Refs #20808, #19246, #18854. No public issue exists for this case.

**What happened?**
On staging, an account-wide Anthropic 429 refused `session.open` for all
runs of one agent. Each run waited the one-hour default cooldown and
retried on the same run. The retry met the same 429. The repeat check
then ended each run as `native_provider_startup_rejected`. Sentry
recorded 25 events in two bursts. Each task became blocked and waited
for a person to click Retry.

**What did you expect to happen?**
A rate limit or quota at session start must wait for the reset, the same
as a limit that the provider reports on a turn. The usage-limit hold
retries the task at the reset by itself and does not report to Sentry.

**Steps to reproduce**
1. Use a remote native runner with an ACPX provider whose account is
rate limited.
2. Start a task run. The provider refuses `session.open` with
`classification=provider_rate_limit` and a typed `providerFailure`
(`phase: "startup"`, `category: "rate_limit"`).
3. Let the same-run retry run after the cooldown while the limit
continues.
4. See that the run fails as `native_provider_startup_rejected`, the
task becomes blocked for a board Retry, and Sentry gets a `run_failure`
event.

## What Changed

- `nativeRemoteStartupRejectionCode` returns
`native_provider_usage_limit` instead of
`native_provider_startup_rejected` when the typed startup failure is
`rate_limit` or `quota_exhausted`. The first refusal with a proven
runner exit still retries on the same run after the cooldown, as before.
Other startup failures do not change.
- `readNativeUsageLimitReset` also accepts a startup refusal when the
run has no turn event. It reads the current failure (from the executor)
or the run's saved `providerFailure`, and it requires `phase:
"startup"`. A run with a turn event keeps the old rules.
- `planProviderLimitResume` accepts the current startup failure, so the
hold gets its reset before the run row records the failure.
- The hold saves the startup refusal that created it
(`providerLimitStartup`, with its observation time). When the hold
resumes, it reads that refusal first. A same-run retry keeps the run
row's first saved failure, so the run row can name an older reset.
- Tests: a classifier unit test, an integration test for the hold and
its automatic Retry, an executor-level test for two startup attempts
with different resets, and a Sentry-report test for the startup shape.

## Verification

- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"session.open the provider itself rejected"`: 9 tests pass. The new test
fails on `master` with `Received: "native_provider_startup_rejected"`.
- `npx vitest run
src/services/native-runtime/native-provider-capacity.integration.test.ts
src/services/__tests__/run-failure-report-native-usage-limit.test.ts`:
26 tests pass. The new integration test fails on `master` (the plan is
not automatic without a turn event).
- `npx vitest run` on `native-startup-rejected-retry`,
`native-provider-capacity`, `native-startup-retry` and
`native-provider-limit-cooldown` (integration), and
`run-failure-report-native-usage-limit`: 76 tests pass. The new
two-attempt test fails on the first head (`c3ee40f166`) with `reason:
"reset_unknown"`.
- The full `native-session-executor.test.ts`,
`native-provider-limit-cooldown.integration.test.ts` and
`run-failure-report*.test.ts` files: 1865 tests pass and 1 fails. The
failure (`compares export process compatibility without changing
conversation identity (changed: false)`) also fails on an unchanged
`master` in the same sandbox.
- I could not run the full server typecheck locally, because the sandbox
has an 8 GiB memory limit and `tsc` stopped with an out-of-memory error.
CI runs the typecheck.

## Risks

- Low risk. The change applies only to a remote startup refusal with a
typed `rate_limit` or `quota_exhausted` category that would otherwise
end the run.
- The task still shows as blocked during the hold, as for a limit on a
turn. The hold now retries by itself at the reset, at most three times
in a row without a successful run.
- A quota without a reported reset gives the same "restore usage
capacity" action as before, so a person still decides.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 07:22:34 -08:00
..