mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude
c37eacbee3
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save an accepted result. Then a finalizer applies that
result to the task.
> - Finalization has its own budget of 3 failed attempts (#20555). When
the budget runs out, the coordinator goes to `terminal_failure` with
`native_finalization_retry_exhausted`.
> - For a provider-succeeded run, the task then rests `in_review` with a
board recovery card. The task shows "resolved" and offers no Retry. The
user is stuck.
> - The ordinary Retry is also wrong here: it starts a new provider run,
which repeats work the agent already finished.
> - This pull request adds two things. First, contention failures on
such a result get two longer automatic delays before the board gets the
task. Second, when the state is still exhausted, the task offers Retry,
and that Retry applies the same saved result again.
> - The benefit: fewer tasks need a Retry, and when one does, Retry
works and does no duplicate provider work.
## Linked Issues or Issue Description
Refs #20555 (it names this gap under Risks).
**What happened?**
In restart-chaos wave `w-20261110T044338-s20385`, task t03 ended with a
succeeded run, an accepted result, and a coordinator in
`terminal_failure` / `native_finalization_retry_exhausted`. The issue
stayed `in_review` with a board-owned `active_run_watchdog` card and
`canRetry=false`. #20555 stopped the early exhaustion. A real exhaustion
still leaves the same stuck state. Coordinators that older builds
already left in this state are stuck the same way.
**Expected behavior**
Transient finalization failures recover by themselves when that is safe.
When they do not, the task offers Retry. That Retry re-finalizes the
saved result. It does not start a provider run and does not charge a
provider attempt.
**Steps to reproduce**
1. Let a native run save an accepted result and end `succeeded`.
2. Make each finalization of that result fail until the budget of 3 runs
out.
3. Before this fix, the task presentation is `resolved` with
`canRetry=false`, and the board card has no action that applies the
result. The new test
`server/src/__tests__/native-exhausted-finalization-retry.test.ts`
reproduces this.
**Paperclip version or commit**
master at c58356003f (after #20555).
## What Changed
- `native-run-finalizer.ts` (`recordRetryableFailure`): when the budget
runs out on an accepted result of a provider-succeeded run, and the
cause is contention (`status_cas_exhausted` or
`side_effect_planning_failed`), the coordinator stays
`retryable_failure`. It tries again after 2 minutes, then after 10
minutes, each with a fresh budget. Each delay is recorded once in
`recoveryHistory` (`automatic_finalization_backoff`). After both delays,
the old terminal path runs. Deterministic causes (for example
`native_finalization_invalid`) and workspace-scope failures are
unchanged. The existing deadline worker (`native-result-commit-work`)
runs the retry, so no new timer or poll is added.
- New `native-runtime/exhausted-finalization-retry.ts`:
- `exhaustedNativeFinalizationCondition()`: one SQL predicate for
"succeeded native run, accepted result, coordinator `terminal_failure`
with `native_finalization_retry_exhausted`, no lease, no successor".
- `rearmExhaustedNativeFinalization()`: under the task lock, reopens the
same coordinator (`arbitrating` or `workspace_finalizing`), records
`exhausted_finalization_retry` in `recoveryHistory`, resolves the board
card (`handed_back`), and resets `finalizationFailureCount` to 0. It
sets a 30 s fallback `nextAttemptAt`, so the deadline worker finishes
the job if the process stops first.
- `refinalizeSavedNativeResult()`: runs `finalizeNativeRun` for the same
run with `preserveProviderAttempt: true`. A failure goes through the
finalizer's normal bounded budget.
- `task-interruption-presentation.ts`: the run read adds
`nativeFinalizationRetry` from that predicate. A succeeded run with this
flag shows `state: "failed"`, `canRetry: true`. It also adds
`nativeFinalizationRetryPending` (`savedResultRetryPendingCondition()`:
coordinator not committed or terminal, with an
`exhausted_finalization_retry` entry). While it is true the task shows
`state: "working"` with `recoveryStep: "finishing_recovery"`, so the
user sees the Retry took effect. Done or cancelled tasks, user gates,
and ordinary runs are unchanged.
- `task-retry.ts` (`admitTaskRetry`): for a succeeded native run in the
exhausted state, Retry re-arms the coordinator instead of creating a
provider retry. It writes a `coalesced` `task_retry` receipt bound to
the same run with strategy `finalize_result` (an existing strategy
value), joins any unadmitted Retry intent to it, and logs
`issue.retry_requested` with `strategy: "finalize_result"`. The
idempotency key still returns the same receipt on replay. A later click
with a new key, after an earlier saved-result Retry of the same run
(still finalizing, or applied), joins that Retry with another coalesced
`finalize_result` receipt and starts no provider work.
- `routes/issues.ts` (`POST /issues/:id/retry`): after the receipt
commits, it starts `refinalizeSavedNativeResult` for that run instead of
the provider Retry dispatch.
- `doc/execution-semantics.md`: documents the automatic delays and the
re-finalizing Retry.
## Verification
- `cd server && npx vitest run
src/__tests__/native-exhausted-finalization-retry.test.ts` (4 tests):
- Exhausted state reports `state: "failed"`, `canRetry: true`. Retry
re-arms the coordinator, writes a `finalize_result` receipt and an
`issue.retry_requested` activity row, and replays idempotently. A second
click with a new key while pending, and a third after commit, join the
same Retry; no provider Retry wake exists. While pending, the task shows
`working` / `finishing_recovery`. Re-finalization commits, and the task
shows resolved. The run count for the task stays 1, the coordinator
`attempt` is unchanged, the run stays `succeeded`, and no board card
stays active.
- Old-shape row (no `finalizationFailureCount`, `attempt: 3`, already
terminal): the same Retry works with no migration.
- Re-finalization that fails again: the coordinator goes to
`retryable_failure` with a fresh count of 1, then to `terminal_failure`
after the budget, and the task again shows `canRetry: true`. Nothing is
lost silently.
- Contention failures (failpoint `status_projection`): two automatic
delays (2 min, 10 min) with the task status unchanged, then the board
Retry, which commits with no new run.
- Red/green: with only the new module and test applied (finalizer,
presentation, task Retry, and route reverted), all 4 tests fail
(`canRetry` false; no automatic delay).
- `npx vitest run src/services/task-interruption-presentation.test.ts`
adds unit cases: the flag offers Retry; ordinary succeeded runs stay
resolved; ordinary failed runs keep `canRetry: true`; done tasks stay
resolved.
- Also passing: `native-finalization-retry-budget`,
`native-finalization-recovery`, `native-run-finalizer-telemetry`,
`task-interruption-query`, `interrupted-preparation-restore-retry`,
`native-result-commit-work`, `workspace-recovery-admission`,
`workspace-recovery-preparation-gates`,
`deferred-execution-receipt-work`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Not run: the full local suite (`pnpm verify` selects about 2,450 files
for these shared modules; CI covers it), and no live restart-chaos wave.
## Risks
- Previous-release story: coordinators saved by older builds in
`terminal_failure` / `native_finalization_retry_exhausted` qualify by
phase and code. The count is optional. Nothing is migrated. Older rows
do not get the automatic delays; they get the working Retry.
- Automatic delays add at most two extra finalization rounds of the same
saved result, only for contention causes, only for provider-succeeded
accepted results. During the delays the task keeps its prior status, as
it does during the normal 30 s retries. Supersession by a newer run is
still checked first on every attempt.
- The re-finalizing Retry applies the agent's saved result after a
person may have looked at the task in review. It uses the normal status
arbiter and CAS. Done and cancelled tasks, other assignees, and newer
runs refuse it.
- No change before `turn.start`: all new work is on the finalization,
presentation, and Retry paths.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (shell, code editing, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>