Files
PaperClipAI/doc
DottaandPaperclip d180ad8c77 fix(heartbeat): retry chat execution authorization contention (#19017)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task Retry must use the current task and chat authority.
> - The request and queue paths already retain a Retry through brief
lock contention.
> - Execution checks that authority twice, including after native input
preparation.
> - A busy task row or chat identity could fail the admitted run at
either check.
> - This pull request retries only the rolled-back authorization
transaction.
> - The same run can proceed after the lock clears. The final dispatch
gate still rejects changed ownership.

## Linked Issues or Issue Description

**What happened?**

A failed chat task can accept a normal Retry and then fail during
execution setup when another PostgreSQL transaction holds the task row
or chat identity lock. The late check can fail after native input and
workspace preparation. The exception can then enter the workspace
preservation path.

**Expected behavior**

Keep the same admitted run and its frozen input. Retry a recognized,
rolled-back authorization conflict within the existing bound. Recheck
current authority before each attempt. Reject Stop, paused agents,
revoked chat access, or a changed task owner before the execution
callback.

**Steps to reproduce**

1. Admit a normal failed-chat task Retry with the existing PostgreSQL
fixture.
2. On a second connection, hold the task row or the chat identity
advisory lock when execution authorization starts.
3. Release the lock after the real authorizer reports contention.
4. Observe that the original code fails the run before the authorized
callback. Repeat at the late native check after input preparation.

**Paperclip version or commit**

Reproduced on master `039e0dad2c51c80251e1302a19c5566e039c357a`. The
branch is rebased on `8f06bd4e1`.

**Deployment mode**

Source checkout with real PostgreSQL and the local HTTP Retry route.
Provider and chat transport responses use controlled test fixtures.

Refs: #18895 and #18896 cover request and queue admission. This change
covers execution authorization. #18898 and #18989 cover predecessor
cleanup, a separate boundary. The roadmap and public PR search show no
duplicate execution-authorization repair.

## What Changed

- Recognize the existing typed chat-presentation contention error in the
bounded admission retry helper. Keep its existing PostgreSQL
lock-unavailable handling.
- Use that helper at both execution authorization checks. Each attempt
checks cancellation and the applicable legacy controller lease, then
opens a fresh transaction. The loop contains no provider or workspace
callback.
- Include failed-chat Retry in the existing final dispatch ownership
gate. This preserves current task ownership and native recovery owner
checks after a delay.
- Add real PostgreSQL contention, late prepared-input, Stop, pause,
revocation, owner replacement, native completion, and ordinary legacy
completion controls. Document the retry and dispatch boundaries.

## Verification

- The unchanged base failed real task-row and chat-identity contention
cases. A separate late native case failed after frozen input
preparation. A safety control then proved that a replaced task owner
could still reach the backend; the final dispatch gate fixes that path.
- All 11 PostgreSQL cases and all four helper controls pass. The tests
keep exact frozen model and trust input, reject changed authority, and
do not replay a provider callback.
- A real local HTTP Retry reaches the native executor, one controlled
provider turn, one accepted result, a committed finalizer, and one
published chat reply. Reprocessing the canonical request does not start
another turn.
- A separate real HTTP ordinary legacy Retry runs one controlled adapter
callback, completes its task through the normal service, and does not
replay the canonical request.
- Independent review found no material issue in the two production
files. It checked transaction scope, lock order, current ownership,
native lease expiry, and import cycles.
- The full workspace typecheck and full `pnpm build` pass on this head.
Full `pnpm test:run` passed all 1,916 chat integration cases. Its
broader server group reported the existing workspace-runtime case
`returns a bounded structured conflict and exposes only same-company
workspace references` as failed after 46 ms. The unchanged focused case
and the complete 179-case file both pass on rerun. The active broad run
was then stopped for this finite handoff; its child exited 130 and the
command exited 1. The detailed assertion was buffered and was not
retained, so its exact cause remains unknown. This is a partial local
run, not a full local pass; later local groups did not run. All three CI
chat shards pass. [Greptile reviewed this exact head at 5/5 with no
findings](https://github.com/paperclipai/paperclip/pull/19017#issuecomment-6638521119).
- Retained setup and fixture failures are not counted as product
regressions or passes. An incomplete copied Copilot runtime failed the
first typecheck; the supported installer repaired that isolated
dependency, then typecheck passed. Early acceptance fixtures used a
wrong publication field or did not complete their task; the final
controls use the real field and completion service.
- The first CI run failed [workspace shard
1](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126769239992)
on child-server startup and a five-second dense ZIP test deadline.
[Server shard
2](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126769239908)
failed an existing local runtime readiness probe. All 18 cases in the
two CLI files and the runtime case then passed unchanged locally on this
head. The CLI files also match an earlier independently verified head
byte for byte. One failed-job rerun was requested. Its [server
job](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126777070120)
was cancelled after the runner received a shutdown signal. It produced
no test verdict. After that workflow ended, only the interrupted server
job and its required dependents were rescheduled once. The [complete
workspace rerun
passed](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126777070125).
The [rescheduled server
job](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126781960353)
and its aggregate passed. The [complete current-head workflow is
green](https://github.com/paperclipai/paperclip/actions/runs/41885475363).
All required current-head CI and security gates pass, including the
Ready-triggered scans; three conditional checks are skipped. The
complete unchanged idle-history file also passes all 75 cases locally.
No timeout or assertion changed; all attempts remain part of this
result.
- These local controls do not claim hosted-provider or Linux sandbox
acceptance. The original task-row failure did not retain a SQLSTATE; the
PostgreSQL reproduction proves the local gap without assigning that
original cause.

## Risks

- The existing bound allows 50 backoffs of 100 ms. It limits backoff
time, not total database or lease wall time.
- Native ownership is checked at the final dispatch gate. The
per-attempt controller lease assertion applies to legacy execution.
- A persistent lock or changed authority still fails closed. Unknown
prior provider effects and accepted work retain their existing guards.
- No migration or model, account, trust, credential, or permission
policy change is included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, code editing, and
PostgreSQL verification. The host does not expose the exact serving
model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass within the stated scope:
relevant regression and acceptance controls, and the complete chat,
idle-history, and workspace-runtime files. The broad local run remains
partial as recorded above.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 13:07:50 -06:00
..
2026-02-16 14:25:00 -06:00