Files
PaperClipAI/server
Devin FoleyandPaperclip 4a999089ef Retry transient failures in dashboard reads (#14873)
## Thinking Path

> - Paperclip shows company activity in the dashboard.
> - The dashboard reads company, task, approval, and cost data.
> - A pooled database connection can close during one of these reads.
> - The driver must reject ambiguous statements because writes may have
committed.
> - These four dashboard queries are known to be read-only.
> - This change retries only the failed read and preserves completed
work.

## Linked Issues or Issue Description

Refs #14773, which correctly removed automatic replay of ambiguous
database statements. Searched existing database and dashboard PRs.
Related #8780 changes pool recycling and logging; #13925 adds dashboard
consistency coverage. Neither provides these per-query retries.

**What happened?**

A dashboard request returns a server error when its company lookup, task
count, approval count, or monthly spend query loses its database
connection. Drizzle wraps the driver's connection error in `cause`.

**Expected behavior**

A transient connection failure gets a bounded retry of the specific
read. Completed reads and budget processing are not replayed. Persistent
outages and non-connection errors still fail the request.

**Steps to reproduce**

Inject a typed `CONNECTION_CLOSED` error into one of these reads. Then
allow the next query to succeed. Before this change, the dashboard
request fails immediately.

## What Changed

- Apply independent retries to the company lookup, task counts, pending
approval count, and monthly spend query. Each callback rebuilds its own
query with the same company scope.
- Extract the existing authentication retry helper as
`retryIdempotentDatabaseOperation`. Preserve authentication behavior and
its existing exports.
- Retain the existing limit of three total attempts with 50 ms and 100
ms pauses. Match typed connection codes through the error cause chain.
- Test later-read failures, unchanged query parameters, retry
exhaustion, missing companies, and errors that must not retry. Document
the boundary.

## Verification

- Final focused dashboard, authentication, and real database wire
suites: 38 tests passed. Six initial recovery regressions failed before
the implementation.
- `pnpm -r typecheck`: passed on the final source.
- Independent review: no actionable findings. The reviewer separately
passed all 38 focused tests and checked the code allowlist, attempt
bounds, pauses, and final error identity.
- `pnpm test:run`: the general-server group completed with 14,738 tests
passed, 13 failed, and 87 skipped. All 13 failures match the previously
reproduced clean-base macOS skill-cache failures. The two test files and
their implementations are unchanged from that baseline. The runner
exited after this group, so the remaining local workspace and serialized
groups did not run. All corresponding Linux CI groups passed on this
commit.
- `pnpm build`: passed on the final source.
- Full CI: 53 successful checks and 2 skips on `6a113529c0`. Greptile:
5/5 on that commit, with no review threads or remaining findings.
- Merge compatibility with master `f2e0f19630`, including #14866: no
conflicts. The five reviewed files are unchanged in the resulting merge
tree.

## Risks

A persistent outage adds at most two retries per covered query. Each
connection attempt retains the configured driver timeout. The change
does not repair the underlying network or database failure. Agent
counts, run-activity queries, and the budget workflow stay outside these
retry boundaries. General database statements and disconnected
transactions are not replayed. There is no schema or authorization
change.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
editing, and test execution. The runtime does not expose a more specific
serving model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (38 focused tests; the full
local run has the baseline limitation documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-01 19:23:47 -07:00
..