PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-08 22:13:59 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	b22ef22d56	fix(runner): show provider failure details on tasks and runs (#18622 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner starts providers and records their run outcomes. > - Provider failures can contain useful details below a generic protocol error. > - Those details were lost at the sidecar, command receipt, and run projection boundaries. > - A blocked task therefore showed a generic failure even when the provider gave a clear reason. > - This pull request carries a bounded, sanitized provider failure through those boundaries. > - Operators can read the actual cause on the task and run pages. Existing recovery policy still controls retries. ## Linked Issues or Issue Description Refs #17543 for the related closed JSON-RPC classification work. Refs #17272 for provider recovery in the legacy OpenCode adapter. This change adds display evidence for native Runner providers and keeps recovery policy separate. What happened? A native Claude session returned an HTTP 429 message below an ACP `Internal error`. The task and run showed only a generic startup failure. Operators had to inspect the provider response to learn that the account was rate limited. Expected behavior The task and run show the sanitized provider message, provider name, startup or turn phase, and available HTTP status. Unknown categories keep the message. Display evidence does not grant a retry or replace a recovery decision. Steps to reproduce 1. Start a native Claude run whose account is rate limited during model selection, or return a structured provider failure from a Codex or OpenCode turn. 2. Let the run fail and open its task page. 3. Before this change, the page shows a generic failure. With this change, the page shows the provider explanation. The run page shows the same explanation after reload. Paperclip version or commit Source checkout based on `f8a888a072`. ## What Changed - Add one JSON Schema for `ProviderFailure` and generate matching Runner and shared TypeScript readers. Keep the Runner package independent of the app workspace at runtime. - Normalize known ACPX, Codex, and OpenCode error fields. Preserve unknown messages. Redact known credential values and credential patterns before applying the 4,000-character limit. Do not copy response bodies, headers, stacks, or prompts. - Carry startup failures through the sidecar, Rust Runner, durable command receipts, and private transport error details. Keep provider text out of generic transport error strings and Sentry failure messages. - Keep turn failures in authenticated run events. Recover the same evidence from persisted events during server finalization. Store the cause in the company-scoped run result. - Show one shared notice on failed tasks and run details. Keep existing retry permissions, scheduled retry state, and resolved-task behavior. - Reapply registered run-secret and current-user redaction to task display data. Batch secret lookups for only the visible provider causes. - Document the contract and update the capability maps. Add regressions for nested Claude 429 errors, OpenAI quota and rate limits, OpenCode errors, redaction, durable receipt replay, restart finalization, and UI rendering. ## Verification - `pnpm -r typecheck` passed. `pnpm build` passed. All GitHub Actions build, test, package, and validation jobs passed on `d49c79ab4a`. The security checks passed. Greptile gave this head 5/5 with no open review threads. Paperclip Review remains pending. Its review worker exhausted six workspace recovery attempts with export_interrupted. A fresh review request was accepted and is waiting for that saved workspace. The ordinary Retry action rejected reuse of the stopped request. This is an external review-worker blocker; no check was overridden. The duplicate local `pnpm test:run` was stopped after the CI test shards passed. - Focused server settlement, provider evidence, and restart tests: 93 passed. Runner admission and normalization tests: 103 passed. Focused Runner wire and terminal tests: 29 passed. The real Rust/sidecar command receipt replay test passed. - The new projection test failed before implementation because the task had no provider cause. The startup producer regression also failed before extraction moved ahead of the generic error-code replacement. Both pass with this change. - Live check: a rebuilt local sidecar received an actual Claude HTTP 429 during startup. The new cause kept the full provider message. A synthetic task in an isolated local app then reached the same provider error through the native Runner path. The dedicated run result, task API, task banner, and run detail all showed the same message after reload. The later reporting fix keeps run.error generic; a regression proves the display cause still survives restart and stays out of Sentry capture. - The first isolated app attempt failed earlier in `admission_credential`. An ordinary manual Retry reached the provider and produced the 429 above. This PR does not claim to fix that separate admission failure. An initial parallel local database test setup also failed; the serial restart suite passed. - Security review found that copying display text into run.error could forward it to Sentry. A regression failed with that behavior. The fix keeps the cause in its display field. All 38 settlement/restart tests and the server typecheck pass. - All 29 protocol contract tests pass, including bounds and extra-field rejection. An earlier full Rust core run passed. The later full run hit one 5-second receipt-limit timing assertion; its isolated rerun passed. The full Linux and Windows Rust CI jobs passed on this head. - Review regressions pass for explicit Codex driver identity, native Rust startup and resume receipt propagation, oversized OpenCode/ACPX errors, task read-time registered-secret redaction, and the standalone source closure. The focused Runner run passed 103 tests; projection/restart tests passed 16. - Read-time redaction edge cases passed: a missing decryption key omits only the unsafe optional explanation; expanded redacted messages remain visible within the Unicode character limit. All 14 task projection tests and server typecheck pass. - The package Build check found an old Pi assertion against Error.message. The probe now checks the exact closed classification in the authenticated receipt and requires the new provider cause. Both polling transport cases pass. A local credential-free Pi probe stopped at admission_installation because this checkout lacks the Pi distribution; the installed Linux package probe passed in CI on this head. - UI coverage includes startup and turn text, unknown categories, literal rendering of provider text, read-only retry behavior, and resolved-task suppression. Token gates and map checks passed. - The live check covers Claude startup on macOS. Codex, OpenCode, terminal failures, and restart paths have deterministic regression coverage. The change is not deployed. ## Risks - Provider messages are untrusted text. The boundary selects specific fields, redacts credential patterns and known environment secret values, bounds the message, and renders plain text. New provider-specific secret formats may need added redaction patterns. - Optional fields preserve compatibility with older producers and stored runs. Old runs without the field keep their current fallback text. No database migration is required. - The cause stays in company-scoped run data. This change does not send provider text to first-party Telemetry or add it to OpenTelemetry span attributes. - Normalized categories and retryability are display evidence. Existing recovery classifiers, retry limits, ownership checks, and approval gates keep their authority. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser verification. The host does not expose the exact serving variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 06:13:26 -06:00
Devin FoleyandClaude Opus 5.5	2a221e8e80	fix(sentry): keep scheduled native model capacity retries out of run-failure reports (#18295 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The run-failure reporter sends terminal run failures to Sentry when an operator sets `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` > - In `unexpected` mode, the reporter skips failures that a bounded automatic retry already owns. The Codex ACP model capacity failure (`codex_transient_upstream`) is one of these > - The native runner has the same capacity failure: Codex `serverOverloaded` ("Selected model is at capacity"). Native finalization schedules a bounded automatic retry for it (#15347) > - The reporter has no matching exclusion for the native path. Each scheduled capacity retry also sends a Sentry error, even when the retry recovers the task > - This pull request skips the report only when the native failure has typed capacity evidence, a committed retry decision, and the retry run that the decision created > - The benefit is that Sentry shows capacity failures only when automatic recovery stops (retries are exhausted, held, or not scheduled) ## Linked Issues or Issue Description Refs #15347 (native capacity retry). Refs #16407 (Codex ACP capacity exclusion). What happened? A native Codex run failed with "Selected model is at capacity. Please try a different model." (`native_provider_overloaded`). Native finalization scheduled the automatic capacity retry. The retry run succeeded and the task completed. The run-failure reporter still sent a Sentry error event for the first run. Expected behavior When the automatic capacity retry owns recovery, the run failure stays in the local run log and does not go to Sentry. This is the same as the Codex ACP capacity path. When retries are exhausted or held, or when no retry is scheduled, the failure goes to Sentry. Steps to reproduce 1. Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` with a backend Sentry DSN. 2. Run a native Codex agent turn that ends with a Codex `serverOverloaded` terminal error. 3. See that native finalization creates a `scheduled_retry` run with `scheduledRetryReason=native_provider_overloaded`. 4. See that Sentry also receives a run-failure event for the failed run. ## What Changed - `server/src/services/run-failure-report.ts`: add `isScheduledNativeCapacityRetry`. In `unexpected` mode it skips the report only when all of these are true: - the run is native, `failed`, has `errorCode=native_provider_overloaded`, and has a native task (`nativeIssueId`); - `resultJson.nativeProviderFailure` is the closed `codex`/`server_overloaded` diagnostic; - `resultJson.finalizationPhase` is `committed` and `finalizationReasonCode` is `native_provider_overloaded`; - the report is not from setup, and the run has no signal; - a successor run for the same task exists with `retryOfRunId=run.id` and `scheduledRetryReason=native_provider_overloaded`. The same status transaction creates this run. - The successor lookup filters on `company_id`, `context_snapshot ->> 'issueId'` and `created_at >= run.createdAt`, so the existing `heartbeat_runs_company_ctx_issue_created_idx` bounds it to that task's runs since the failure. No migration is needed. - A failed read of the retry evidence logs a warning and reports the failure. - `doc/observability.md`: document the new exclusion in the run failure context section. - Unit tests: the exclusion, and twelve cases that must still report (exhausted, held, no retry run, another retry lane, successor for another task, run with no native task, finalization not committed, wrong or missing diagnostic, legacy run, signal, timeout). Also setup phase, `all` mode, and an unreadable retry record. - Integration test (real native finalizer): no Sentry event for retries 1 and 2, one event when retries are exhausted, and one event when reassignment prevents a retry. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/__tests__/run-failure-report.test.ts src/services/native-runtime/native-provider-overload.integration.test.ts src/services/native-runtime/native-provider-failure.test.ts src/services/__tests__/status-arbiter.test.ts`: 359 tests pass. - Red/green: before the fix, the new unit test captured 2 events, and the integration test captured an event on the first scheduled retry. After the fix, both pass. - `EXPLAIN` on the migrated schema: `Index Scan using heartbeat_runs_company_ctx_issue_created_idx` with company, context issue and created time in `Index Cond`. - CI provides the full workspace typecheck. ## Risks - Low risk. The change applies only in `unexpected` mode and only to native runs with typed capacity evidence and an existing retry run. `all` mode still reports every failure. - The check adds one index-bounded read of `heartbeat_runs` for runs that match the in-memory capacity checks. Other runs do not do this read. - If a scheduled retry later fails, that retry run reports its own outcome. When retries are exhausted, the final failure reports. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with tool use (shell, file edits, test execution) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-30 05:42:43 -07:00
DottaandPaperclip	f2715e02bb	fix(native): surface model capacity errors and retry automatically (#15347 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs preserve provider events and results, then decide task state. > - Codex can fail a turn because the selected model is temporarily at capacity. > - Paperclip displayed a generic native-session failure and left the task in recovery. > - A known capacity failure needs a clear message and a durable delayed retry. > - This pull request uses committed terminal evidence, the status-effect ledger, and existing dispatch gates. > - The task can continue automatically without an unbounded retry loop or a model switch. ## Linked Issues or Issue Description Related: #13993 adds launcher capacity deferrals before provider startup. This change handles a committed native Codex `serverOverloaded` terminal after provider work has started. What happened? A native Codex turn ended with `codexErrorInfo: serverOverloaded` and “Selected model is at capacity. Please try a different model.” Paperclip saved the result, showed a generic failure, and required recovery instead of waiting and retrying. Expected behavior Show the capacity error directly. Schedule bounded retries after a short delay. Preserve saved work and honor current execution gates. Steps to reproduce 1. Run a native Codex task. 2. Commit a `turn.failed` event with `serverOverloaded`, then accept a failed result for the same turn. 3. Inspect the task error and its recovery state. The regression test reproduces this without a paid provider call. Paperclip version or commit Reproduced against master `16b7db35ffa0f9a95913c8cbdeea3d595435691f`. ## What Changed - Classify capacity failures from committed runner events and the pinned execution identity. Preserve accepted results and display the specific capacity error. - Atomically persist one scheduled successor with the status decision. Retry after one minute, then two minutes. Share the existing failure budget and stop after two automatic retries. - Reuse dispatch gates for ownership, task holds, dependencies, budget, and locks. Wait for predecessor execution, finalization, and cleanup before claiming a retry. - Preserve pending reviewer authority and suppress retries after reassignment or a successor claim. - Consume the failed run's resume receipt and delivered wake input. Rebuild ordinary continuation from the failed run so explicitly resumed tasks can retry without borrowing one-run authorization. - Label scheduled retries “Model at capacity.” Add a Storybook example and avoid duplicate punctuation in the existing retry card. - Add regression coverage and document the runtime contract. ## Verification - Targeted recovery and UI suites: 125 tests passed. Additional final cleanup and lock regressions passed. - Latest-head continuation and authorization regressions: 57 tests passed, including all four capacity integration tests against an isolated PostgreSQL database. - Rechecked all four capacity integration tests with the full runner's isolated `PAPERCLIP_HOME`, config, temporary directory, and serial fork settings: passed. - `pnpm -r typecheck`: passed. Final server typecheck passed. - `pnpm build`: passed. - `pnpm check:token-gates`: passed. - Browser: checked the real Storybook card. It shows the capacity message, automatic retry time, and existing Retry now action. - `pnpm test:run`: attempted and restarted after an interruption. The resumed run started before the final review correction and was stopped after recorded workspace/native test failures and the five-minute Git streaming timeout already documented on master. Exit 130; no complete local full-suite pass is claimed. Current-head isolated capacity tests and the complete CI suite pass. - A combined local recovery-suite attempt also hit PostgreSQL initialization failures at this macOS host's global shared-memory limit (32 slots). The focused recovery suites passed separately; final-head capacity tests also pass with the full runner's isolated environment settings. - Latest-head GitHub checks: all 56 checks green, including general and serialized test shards, browser E2E, typecheck, build, Runner verification, and canary dry run. No merge conflicts. - Greptile: 5/5 on `813a470b8c18c05aeb7e31e63e573c3a3a2f5cac`, with zero unresolved findings after fixing the resumed-task continuation issue. ## Risks - Capacity retries can repeat a task turn after partial work. They start a fresh provider session with task history and wait for predecessor cleanup. They do not resume the failed turn. - Retries retain the configured model unless an operator changes configuration. Persistent overload consumes the existing failure budget and then needs an explicit retry or model change. - Only run-bound native Codex `serverOverloaded` failures qualify. Usage-limit exhaustion, model/account incompatibility, unbound text, and unknown provider failures retain their current recovery behavior. - No schema migration is required. ## Model Used - OpenAI GPT-6 through Codex. The exact model ID and context-window size are not exposed to this session. Used reasoning, repository tools, code execution, and browser verification. No subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-06 10:59:18 -05:00

Author

SHA1

Message

Date

DottaandPaperclip

b22ef22d56

fix(runner): show provider failure details on tasks and runs (#18622 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner starts providers and records their run outcomes.
> - Provider failures can contain useful details below a generic
protocol error.
> - Those details were lost at the sidecar, command receipt, and run
projection boundaries.
> - A blocked task therefore showed a generic failure even when the
provider gave a clear reason.
> - This pull request carries a bounded, sanitized provider failure
through those boundaries.
> - Operators can read the actual cause on the task and run pages.
Existing recovery policy still controls retries.

## Linked Issues or Issue Description

Refs #17543 for the related closed JSON-RPC classification work. Refs
#17272 for provider recovery in the legacy OpenCode adapter. This change
adds display evidence for native Runner providers and keeps recovery
policy separate.

**What happened?**

A native Claude session returned an HTTP 429 message below an ACP
`Internal error`. The task and run showed only a generic startup
failure. Operators had to inspect the provider response to learn that
the account was rate limited.

**Expected behavior**

The task and run show the sanitized provider message, provider name,
startup or turn phase, and available HTTP status. Unknown categories
keep the message. Display evidence does not grant a retry or replace a
recovery decision.

**Steps to reproduce**

1. Start a native Claude run whose account is rate limited during model
selection, or return a structured provider failure from a Codex or
OpenCode turn.
2. Let the run fail and open its task page.
3. Before this change, the page shows a generic failure. With this
change, the page shows the provider explanation. The run page shows the
same explanation after reload.

**Paperclip version or commit**

Source checkout based on `f8a888a072`.

## What Changed

- Add one JSON Schema for `ProviderFailure` and generate matching Runner
and shared TypeScript readers. Keep the Runner package independent of
the app workspace at runtime.
- Normalize known ACPX, Codex, and OpenCode error fields. Preserve
unknown messages. Redact known credential values and credential patterns
before applying the 4,000-character limit. Do not copy response bodies,
headers, stacks, or prompts.
- Carry startup failures through the sidecar, Rust Runner, durable
command receipts, and private transport error details. Keep provider
text out of generic transport error strings and Sentry failure messages.
- Keep turn failures in authenticated run events. Recover the same
evidence from persisted events during server finalization. Store the
cause in the company-scoped run result.
- Show one shared notice on failed tasks and run details. Keep existing
retry permissions, scheduled retry state, and resolved-task behavior.
- Reapply registered run-secret and current-user redaction to task
display data. Batch secret lookups for only the visible provider causes.
- Document the contract and update the capability maps. Add regressions
for nested Claude 429 errors, OpenAI quota and rate limits, OpenCode
errors, redaction, durable receipt replay, restart finalization, and UI
rendering.

## Verification

- `pnpm -r typecheck` passed. `pnpm build` passed. All GitHub Actions
build, test, package, and validation jobs passed on `d49c79ab4a`. The
security checks passed. Greptile gave this head 5/5 with no open review
threads. Paperclip Review remains pending. Its review worker exhausted
six workspace recovery attempts with export_interrupted. A fresh review
request was accepted and is waiting for that saved workspace. The
ordinary Retry action rejected reuse of the stopped request. This is an
external review-worker blocker; no check was overridden. The duplicate
local `pnpm test:run` was stopped after the CI test shards passed.
- Focused server settlement, provider evidence, and restart tests: 93
passed. Runner admission and normalization tests: 103 passed. Focused
Runner wire and terminal tests: 29 passed. The real Rust/sidecar command
receipt replay test passed.
- The new projection test failed before implementation because the task
had no provider cause. The startup producer regression also failed
before extraction moved ahead of the generic error-code replacement.
Both pass with this change.
- Live check: a rebuilt local sidecar received an actual Claude HTTP 429
during startup. The new cause kept the full provider message. A
synthetic task in an isolated local app then reached the same provider
error through the native Runner path. The dedicated run result, task
API, task banner, and run detail all showed the same message after
reload. The later reporting fix keeps run.error generic; a regression
proves the display cause still survives restart and stays out of Sentry
capture.
- The first isolated app attempt failed earlier in
`admission_credential`. An ordinary manual Retry reached the provider
and produced the 429 above. This PR does not claim to fix that separate
admission failure. An initial parallel local database test setup also
failed; the serial restart suite passed.
- Security review found that copying display text into run.error could
forward it to Sentry. A regression failed with that behavior. The fix
keeps the cause in its display field. All 38 settlement/restart tests
and the server typecheck pass.
- All 29 protocol contract tests pass, including bounds and extra-field
rejection. An earlier full Rust core run passed. The later full run hit
one 5-second receipt-limit timing assertion; its isolated rerun passed.
The full Linux and Windows Rust CI jobs passed on this head.
- Review regressions pass for explicit Codex driver identity, native
Rust startup and resume receipt propagation, oversized OpenCode/ACPX
errors, task read-time registered-secret redaction, and the standalone
source closure. The focused Runner run passed 103 tests;
projection/restart tests passed 16.
- Read-time redaction edge cases passed: a missing decryption key omits
only the unsafe optional explanation; expanded redacted messages remain
visible within the Unicode character limit. All 14 task projection tests
and server typecheck pass.
- The package Build check found an old Pi assertion against
Error.message. The probe now checks the exact closed classification in
the authenticated receipt and requires the new provider cause. Both
polling transport cases pass. A local credential-free Pi probe stopped
at admission_installation because this checkout lacks the Pi
distribution; the installed Linux package probe passed in CI on this
head.
- UI coverage includes startup and turn text, unknown categories,
literal rendering of provider text, read-only retry behavior, and
resolved-task suppression. Token gates and map checks passed.
- The live check covers Claude startup on macOS. Codex, OpenCode,
terminal failures, and restart paths have deterministic regression
coverage. The change is not deployed.

## Risks

- Provider messages are untrusted text. The boundary selects specific
fields, redacts credential patterns and known environment secret values,
bounds the message, and renders plain text. New provider-specific secret
formats may need added redaction patterns.
- Optional fields preserve compatibility with older producers and stored
runs. Old runs without the field keep their current fallback text. No
database migration is required.
- The cause stays in company-scoped run data. This change does not send
provider text to first-party Telemetry or add it to OpenTelemetry span
attributes.
- Normalized categories and retryability are display evidence. Existing
recovery classifiers, retry limits, ownership checks, and approval gates
keep their authority.

## Model Used

- OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and browser verification. The host does not expose the exact
serving variant or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-11-03 06:13:26 -06:00

Devin FoleyandClaude Opus 5.5

2a221e8e80

fix(sentry): keep scheduled native model capacity retries out of run-failure reports (#18295 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The run-failure reporter sends terminal run failures to Sentry when
an operator sets `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected`
> - In `unexpected` mode, the reporter skips failures that a bounded
automatic retry already owns. The Codex ACP model capacity failure
(`codex_transient_upstream`) is one of these
> - The native runner has the same capacity failure: Codex
`serverOverloaded` ("Selected model is at capacity"). Native
finalization schedules a bounded automatic retry for it (#15347)
> - The reporter has no matching exclusion for the native path. Each
scheduled capacity retry also sends a Sentry error, even when the retry
recovers the task
> - This pull request skips the report only when the native failure has
typed capacity evidence, a committed retry decision, and the retry run
that the decision created
> - The benefit is that Sentry shows capacity failures only when
automatic recovery stops (retries are exhausted, held, or not scheduled)

## Linked Issues or Issue Description

Refs #15347 (native capacity retry). Refs #16407 (Codex ACP capacity
exclusion).

**What happened?**
A native Codex run failed with "Selected model is at capacity. Please
try a different model." (`native_provider_overloaded`). Native
finalization scheduled the automatic capacity retry. The retry run
succeeded and the task completed. The run-failure reporter still sent a
Sentry error event for the first run.

**Expected behavior**
When the automatic capacity retry owns recovery, the run failure stays
in the local run log and does not go to Sentry. This is the same as the
Codex ACP capacity path. When retries are exhausted or held, or when no
retry is scheduled, the failure goes to Sentry.

**Steps to reproduce**
1. Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` with a backend Sentry
DSN.
2. Run a native Codex agent turn that ends with a Codex
`serverOverloaded` terminal error.
3. See that native finalization creates a `scheduled_retry` run with
`scheduledRetryReason=native_provider_overloaded`.
4. See that Sentry also receives a run-failure event for the failed run.

## What Changed

- `server/src/services/run-failure-report.ts`: add
`isScheduledNativeCapacityRetry`. In `unexpected` mode it skips the
report only when all of these are true:
- the run is native, `failed`, has
`errorCode=native_provider_overloaded`, and has a native task
(`nativeIssueId`);
- `resultJson.nativeProviderFailure` is the closed
`codex`/`server_overloaded` diagnostic;
- `resultJson.finalizationPhase` is `committed` and
`finalizationReasonCode` is `native_provider_overloaded`;
  - the report is not from setup, and the run has no signal;
- a successor run for the same task exists with `retryOfRunId=run.id`
and `scheduledRetryReason=native_provider_overloaded`. The same status
transaction creates this run.
- The successor lookup filters on `company_id`, `context_snapshot ->>
'issueId'` and `created_at >= run.createdAt`, so the existing
`heartbeat_runs_company_ctx_issue_created_idx` bounds it to that task's
runs since the failure. No migration is needed.
- A failed read of the retry evidence logs a warning and reports the
failure.
- `doc/observability.md`: document the new exclusion in the run failure
context section.
- Unit tests: the exclusion, and twelve cases that must still report
(exhausted, held, no retry run, another retry lane, successor for
another task, run with no native task, finalization not committed, wrong
or missing diagnostic, legacy run, signal, timeout). Also setup phase,
`all` mode, and an unreadable retry record.
- Integration test (real native finalizer): no Sentry event for retries
1 and 2, one event when retries are exhausted, and one event when
reassignment prevents a retry.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/__tests__/run-failure-report.test.ts
src/services/native-runtime/native-provider-overload.integration.test.ts
src/services/native-runtime/native-provider-failure.test.ts
src/services/__tests__/status-arbiter.test.ts`: 359 tests pass.
- Red/green: before the fix, the new unit test captured 2 events, and
the integration test captured an event on the first scheduled retry.
After the fix, both pass.
- `EXPLAIN` on the migrated schema: `Index Scan using
heartbeat_runs_company_ctx_issue_created_idx` with company, context
issue and created time in `Index Cond`.
- CI provides the full workspace typecheck.

## Risks

- Low risk. The change applies only in `unexpected` mode and only to
native runs with typed capacity evidence and an existing retry run.
`all` mode still reports every failure.
- The check adds one index-bounded read of `heartbeat_runs` for runs
that match the in-memory capacity checks. Other runs do not do this
read.
- If a scheduled retry later fails, that retry run reports its own
outcome. When retries are exhausted, the final failure reports.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with
tool use (shell, file edits, test execution) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-30 05:42:43 -07:00

DottaandPaperclip

f2715e02bb

fix(native): surface model capacity errors and retry automatically (#15347 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs preserve provider events and results, then decide task
state.
> - Codex can fail a turn because the selected model is temporarily at
capacity.
> - Paperclip displayed a generic native-session failure and left the
task in recovery.
> - A known capacity failure needs a clear message and a durable delayed
retry.
> - This pull request uses committed terminal evidence, the
status-effect ledger, and existing dispatch gates.
> - The task can continue automatically without an unbounded retry loop
or a model switch.

## Linked Issues or Issue Description

Related: #13993 adds launcher capacity deferrals before provider
startup. This change handles a committed native Codex `serverOverloaded`
terminal after provider work has started.

**What happened?**
A native Codex turn ended with `codexErrorInfo: serverOverloaded` and
“Selected model is at capacity. Please try a different model.” Paperclip
saved the result, showed a generic failure, and required recovery
instead of waiting and retrying.

**Expected behavior**
Show the capacity error directly. Schedule bounded retries after a short
delay. Preserve saved work and honor current execution gates.

**Steps to reproduce**
1. Run a native Codex task.
2. Commit a `turn.failed` event with `serverOverloaded`, then accept a
failed result for the same turn.
3. Inspect the task error and its recovery state. The regression test
reproduces this without a paid provider call.

**Paperclip version or commit**
Reproduced against master `16b7db35ffa0f9a95913c8cbdeea3d595435691f`.

## What Changed

- Classify capacity failures from committed runner events and the pinned
execution identity. Preserve accepted results and display the specific
capacity error.
- Atomically persist one scheduled successor with the status decision.
Retry after one minute, then two minutes. Share the existing failure
budget and stop after two automatic retries.
- Reuse dispatch gates for ownership, task holds, dependencies, budget,
and locks. Wait for predecessor execution, finalization, and cleanup
before claiming a retry.
- Preserve pending reviewer authority and suppress retries after
reassignment or a successor claim.
- Consume the failed run's resume receipt and delivered wake input.
Rebuild ordinary continuation from the failed run so explicitly resumed
tasks can retry without borrowing one-run authorization.
- Label scheduled retries “Model at capacity.” Add a Storybook example
and avoid duplicate punctuation in the existing retry card.
- Add regression coverage and document the runtime contract.

## Verification

- Targeted recovery and UI suites: 125 tests passed. Additional final
cleanup and lock regressions passed.
- Latest-head continuation and authorization regressions: 57 tests
passed, including all four capacity integration tests against an
isolated PostgreSQL database.
- Rechecked all four capacity integration tests with the full runner's
isolated `PAPERCLIP_HOME`, config, temporary directory, and serial fork
settings: passed.
- `pnpm -r typecheck`: passed. Final server typecheck passed.
- `pnpm build`: passed.
- `pnpm check:token-gates`: passed.
- Browser: checked the real Storybook card. It shows the capacity
message, automatic retry time, and existing Retry now action.
- `pnpm test:run`: attempted and restarted after an interruption. The
resumed run started before the final review correction and was stopped
after recorded workspace/native test failures and the five-minute Git
streaming timeout already documented on master. Exit 130; no complete
local full-suite pass is claimed. Current-head isolated capacity tests
and the complete CI suite pass.
- A combined local recovery-suite attempt also hit PostgreSQL
initialization failures at this macOS host's global shared-memory limit
(32 slots). The focused recovery suites passed separately; final-head
capacity tests also pass with the full runner's isolated environment
settings.
- Latest-head GitHub checks: all 56 checks green, including general and
serialized test shards, browser E2E, typecheck, build, Runner
verification, and canary dry run. No merge conflicts.
- Greptile: 5/5 on `813a470b8c18c05aeb7e31e63e573c3a3a2f5cac`, with zero
unresolved findings after fixing the resumed-task continuation issue.

## Risks

- Capacity retries can repeat a task turn after partial work. They start
a fresh provider session with task history and wait for predecessor
cleanup. They do not resume the failed turn.
- Retries retain the configured model unless an operator changes
configuration. Persistent overload consumes the existing failure budget
and then needs an explicit retry or model change.
- Only run-bound native Codex `serverOverloaded` failures qualify.
Usage-limit exhaustion, model/account incompatibility, unbound text, and
unknown provider failures retain their current recovery behavior.
- No schema migration is required.

## Model Used

- OpenAI GPT-6 through Codex. The exact model ID and context-window size
are not exposed to this session. Used reasoning, repository tools, code
execution, and browser verification. No subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-06 10:59:18 -05:00