Files
PaperClipAI/server
9874608d0a fix(native): retry a workspace failure write that loses a deadlock (#21507)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime finalizes each agent run after the provider turn
ends. It copies the workspace back and records the result.
> - When the workspace copy-back fails, `recordRetryableFailure` records
the failure and a bounded export retry in one transaction.
> - That transaction locks the coordinator, issue and run rows. Then the
recovery-action insert takes `FOR KEY SHARE` on the agent row for its
foreign keys.
> - A different transaction that holds the agent row and waits on the
run row deadlocks with it. Postgres aborts the failure write with 40P01.
> - The run then fails with `Database query failed (40P01)`, and the
export retry is never recorded.
> - This pull request runs the whole failure write through the existing
`retryTransientLockContention` helper.
> - The benefit is that a lost deadlock no longer fails the run or drops
its workspace export retry.

## Linked Issues or Issue Description

Refs #21494 (same mechanism, but for the provider-turn failure write in
`native-session-executor.ts`. This PR covers `native-run-finalizer.ts`.)
Refs #21506 (a different agent-row deadlock in run setup. It does not
change this write.)

**What happened?**
On a hosted staging server, two runs finished their provider turn, and
then the workspace copy-back failed. The server tried to record that
failure and lost a Postgres deadlock on `insert into
issue_recovery_actions`. The stack was `recordNativeFinalizationFailure`
-> `recordRetryableFailure` -> `upsertSourceScoped`. Each run failed
with `Database query failed (40P01)`.

**Expected behavior**
A lost deadlock in the failure write is retried. The run records
`retryable_failure` and an active `active_run_watchdog` recovery action
for the bounded workspace export retry.

**Steps to reproduce**
1. Hold `FOR UPDATE` on the agent row in one transaction.
2. Start `recordRetryableFailure` for a run of that agent. It locks the
run and issue rows, then waits on the agent row.
3. In the first transaction, request a lock on the run row. Postgres
detects the deadlock and aborts one side with 40P01.

The new integration test does these steps with embedded Postgres.

**Paperclip version or commit**
`master` at 0ebe6d846d.

## What Changed

- `recordRetryableFailure` now wraps its transaction in
`retryTransientLockContention`. That helper retries only 40P01, 40001
and 55P03, with at most 5 attempts.
- Each attempt resets its local state (`input`, `terminalRunToEmit`).
Postgres rolls back the aborted attempt completely, and the next attempt
re-reads all state under the same lock order.
- Post-commit reporting (`emitAgentTaskRun`, `reportRunFailure`) runs
one time, after the attempt that commits.
- Added `native-workspace-failure-write-deadlock.integration.test.ts`.
It makes a real deadlock against the failure write and checks the
committed coordinator, recovery action and run rows.

## Verification

- Red before the fix: `pnpm exec vitest run
src/services/native-runtime/native-workspace-failure-write-deadlock.integration.test.ts`
failed with 40P01 while locking a tuple in `agents`. This is the same
error and stack as the staging event.
- Green after the fix: the same test passed 4 times in a row.
- 7 related finalizer and export test files (288 tests) pass.
- `pnpm typecheck` (server) passes.

## Risks

- Low risk. The change only adds a retry around an existing transaction.
The retry is bounded and only applies to lock-contention error codes.
- A retried attempt re-reads its state. If a different owner committed a
result in the meantime, the existing admission checks reject the stale
write as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
2026-11-20 04:42:41 -08:00
..