Files
PaperClipAI/server
1196a3402d fix(native): do not report a provider usage-limit run that a restart ended (#22385)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs report an unexpected terminal failure to Sentry through
the run-failure reporter
> - The reporter keeps an expected provider usage-limit failure
(`native_provider_usage_limit`) out of Sentry, because the task takes
its provider-limit wait and runs again after the reset
> - A server restart can end such a run after its provider turn failed
on the limit, but before its executor saves
`native_provider_usage_limit`. Restart recovery then blocks the run as
`provider_checkpoint_permanently_failed`, and the reporter sends it to
Sentry
> - On staging, one deploy restart sent seven such reports at once. Each
task then took its provider-limit wait and finished after the reset, so
no operator action was necessary
> - This pull request re-derives the usage-limit fact from the run's
last committed runner turn terminal and keeps that expected failure out
of Sentry
> - The benefit is that a deploy during a provider limit does not create
false error reports, and real checkpoint failures are still reported

## Linked Issues or Issue Description

**What happened?**
A deploy restarted the server while several native runs were settling a
provider turn that failed on a weekly allowance (`quota_exhausted`).
Restart recovery blocked each run as `native_restart_recovery_blocked` /
`provider_checkpoint_permanently_failed`. The run-failure reporter sent
one Sentry event per run.

**Expected behavior**
A run that ended on a provider usage limit is an expected failure. The
reporter does not send it to Sentry when the executor settles it
normally. A restart during settlement must not change that.

**Steps to reproduce**
1. Start a native run whose provider turn fails with a `quota_exhausted`
or `rate_limit` provider failure.
2. Restart the server before the executor saves
`native_provider_usage_limit`.
3. Restart recovery blocks the run as
`provider_checkpoint_permanently_failed`, and the reporter sends a
Sentry event.

Refs #20149

## What Changed

- `native-provider-failure.ts`: move `nativeProviderUsageLimitFromEvent`
here from `native-session-executor.ts` so that the evidence reader can
use it. The executor re-exports it, so its callers do not change.
- `native-provider-failure-evidence.ts`: add
`readPersistedNativeTerminalUsageLimit`. It reads the run's last
committed turn lifecycle event. It returns true only when that event is
a `turn.failed` from the run's own runner instance, the PRP event
validates against the stored row (run, session, source, sequence and
payload hash), and the failure is a provider usage limit.
- `run-failure-report.ts`: for an unexpected
`native_restart_recovery_blocked` /
`provider_checkpoint_permanently_failed` failure, skip the report when
that reader returns true. A read error logs a warning and keeps the
report.
- Add a real-process restart-recovery test with four cases: usage limit
and rate limit are not reported; an unknown failure and a later turn
after the limit are still reported.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "provider usage limit|pending retry|unreported"`: 5 passed.
- Red proof: with only `run-failure-report.ts` reverted, the
`usage-limit` and `rate-limit` cases fail (`expected ... to have a
length of +0 but got 1`). The two report cases still pass.
- `cd server && PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm run typecheck`:
passed.
- `npx vitest run src/services/run-failure-report
src/services/native-runtime/native-provider-failure
src/services/native-runtime/native-session-executor.test.ts`: 1619
passed, 1 failed. The failure (`compares export process compatibility
without changing conversation identity (changed: false)`,
`db.transaction is not a function`) also fails on unchanged `master`
(`ce8006de3c`), so this change does not cause it.

## Risks

- Low risk. The change only removes a Sentry report. It does not change
run status, task status, recovery, or the provider-limit wait.
- The filter needs a validated, run-bound runner terminal. A missing,
tampered, foreign or later event keeps the report.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 11:57:54 +00:00
..