Files
PaperClipAI/server
DottaandPaperclip 057898530c fix(server): handle wrapped recovery action conflicts (#18885)
## Thinking Path

> - Paperclip coordinates durable agent work and recovery.
> - Independent workers can find the same task needs recovery.
> - The database permits one active recovery action for that task.
> - A losing insert must read the action that already committed.
> - Drizzle wraps the PostgreSQL unique violation in its cause.
> - The current classifier reads only the outer error and misses that
conflict.
> - This PR recognizes the same allowed conflict through a bounded cause
chain.
> - Both workers can then return the one durable action through the
existing retry path.

## Linked Issues or Issue Description

Refs: #18783.

**What happened?**

The saved-disposition-debt recovery regression failed in CI
`41733864320`. Two recovery workers collided on
`issue_recovery_actions_active_source_uq`. The fixture captured an
unhandled PostgreSQL `23505` error. A separate canceled shard in that
run lost its AWS Spot instance; that interruption is a different
failure.

**Expected behavior**

Two service instances that race to create the same active action must
return that one action. Unrelated database failures must still
propagate. Existing transaction and ownership rules must stay in force.

**Steps to reproduce**

Hold a PostgreSQL SHARE lock on the recovery-action table. Start the
same source-scoped upsert on two independent service instances. Confirm
both real INSERTs are waiting in `pg_stat_activity`, then release the
lock. The original code persists one action but rejects one caller with
a `DrizzleQueryError`. Its outer code is absent; its `PostgresError`
cause has SQLSTATE `23505` and the expected active-source constraint.

**Paperclip version or commit**

Supporting branch starts at master
`54782706e1d83d92295246e499627930df03662d`. It now contains a normal
merge of master `8771606fd87f69ed4057779e3dce3a9bca8088e6`. The
classifier correction stays unchanged. The PR diff against that master
still contains only the recovery service and its tests.

## What Changed

- Inspect up to eight error causes with cycle detection.
- Require SQLSTATE `23505` and one of the same two recovery-action
constraint names on the same error. Support the PostgreSQL driver's
`constraint_name` field.
- Reuse the existing source-scoped retry path and its unchanged
three-retry limit.
- Add a deterministic real-database collision regression and no-retry
controls for unrelated uniqueness, other SQLSTATEs and cyclic errors.
- Keep the original recovery regression, assertions and timeouts
unchanged. Keep the saved-skill journey separate in #18783.

## Verification

- Review follow-up: the real collision test now checks the intended
retry merge. The surviving action has attempt count 2, active status,
the default unlimited budget, the expected owner, supplied evidence and
supplied attempt timestamp. Its wake and monitor fields remain null.
Existing identity and convergence assertions run first.
- A disposable counterfactual returned the winner unchanged after the
lost insert. It passed the original barrier, one-row, no-rejection and
same-ID assertions, then failed the new attempt-count check with 1
instead of 2. The counterfactual was not committed. The production
service was restored byte-for-byte. The actual service passed all seven
focused race, no-retry and sequential controls. The affected canonical
test-tree-2 typecheck passed against verified unchanged local
declarations.
- Parent `4f8abfe4f2244649219ef6ced3534634193f0b53` full CI
`41746314291` passed all 48 selected jobs with one intended skip. Its
5/5 summary also had two duplicate unresolved reliability comments about
the missing state assertions. This test-only follow-up addresses both.
Fresh full CI and current-head review are required again.
- The deterministic real-database regression failed before the
correction. Both inserts reached the lock and exactly one active action
persisted. The losing error was a `DrizzleQueryError` with a
`PostgresError` cause, SQLSTATE `23505` and constraint
`issue_recovery_actions_active_source_uq`.
- **11 focused checks passed across two files on the integrated head**
(181.72 seconds). This includes the real-database collision, no-retry
controls and all four unchanged recovery producer/dependency
regressions, including the original failed real-clock saved-debt case.
Both callers return the one persisted action.
- Current-head canonical server typecheck passed on Node24.22.2 with all
seven layers: below, cycle, above, tests and test-tree-0 through
test-tree-2. Canonical shared, SDK, DB and Runner TypeScript
prerequisites passed. The first filtered installs lacked CLI and UI
imports required by the new test layers. Those module-resolution
failures remain recorded as setup failures. A full frozen workspace
install with lifecycle scripts disabled supplied the same locked
dependencies. No source or type-configuration workaround was used. No
Rust or broad workspace build ran locally; full repository typecheck,
tests and build remain hosted-CI gates.
- The actual supporting master baseline reproduced the same wrapped23505
failure before the correction. Earlier configuration-resolution, local
IPC and native-library prerequisites remain separate setup failure
evidence. No test failure was reclassified as a pass.
- First supporting CI `41739489620` was canceled before its build/test
jobs completed. Its later automatic full-CI run `41739747947` failed.
The recovery server shard passed. The Runner failure was the Cursor
fixture case repaired upstream in #18782, now included by the normal
master merge. That CI also observed an attachment-browser timeout, a
repository-rehome timeout and a lost-runner connection. Their underlying
causes remain unknown. A later pass will establish recovery, not explain
those earlier failures. These original results remain retained.
- Current `fbd392b9b65dbcf4ac078825096037ed0d490a0b` full CI
[41751706634 attempt
2](https://github.com/paperclipai/paperclip/actions/runs/41751706634/attempts/2)
passed: 48 successful jobs and one intended skip. Attempt 1 remains
retained: ten primary jobs lost their bound Spot capacity and the verify
job failed as a dependent. A single ordinary failed-job recheck produced
eleven fresh successful executions (ten primaries and verify);
timestamps prove 37 successful executions were retained. The skip is not
counted as execution. Current Greptile is 5/5 with both addressed
reliability threads resolved and no unresolved findings. All 57 current
check runs are successful, neutral or skipped, and Snyk is successful.
This is CI recovery; it does not explain the separately retained older
attachment and repository timeouts.
- The original CI failure and both failed saved-skill QA attempts remain
unchanged. This PR does not establish a live QA pass or deploy a
production change.

## Risks

Conflict classification must not turn unrelated errors into retries. The
new controls require those errors to propagate after one insert attempt.
The traversal is bounded and stops at cycles. Transaction, company
scope, ownership, notifications, retry count and database constraints
are unchanged. No schema or public API change is required.

## Model Used

OpenAI GPT-6 through Codex with reasoning and code execution. The
runtime does not expose a more specific model ID or context-window size.
These checks make no model or provider calls.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 19:51:37 -06:00
..