mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude
7368a0b15c
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native (Paperclip Runner) runs prepare a task, pass a final dispatch
gate, and only then start a provider.
> - A native run can fail before any provider work starts. Examples are
a Postgres serialization or deadlock error (`40001`, `40P01`) at budget
admission, or a transient database error during setup.
> - Today such a run ends `adapter_failed` or `setup_failed`. `failRun`
writes the "no provider work" record only for legacy runs, and the
bounded retry paths do not retry these codes for native runs. The
release then opens the native hold
(`native_continuation_requires_reconciliation`), and a person must click
Retry.
> - The goal is that 0% of tasks need a Retry click after a restart or a
storage/DB fault. Duplicate provider work is worse than a Retry click,
so the fix must only act when the server can prove no provider work
started.
> - This pull request records that proof for native runs and sends a
transient database failure to the existing uncharged, bounded
`interrupted_preparation` retry lane.
> - The benefit is that these runs re-queue by themselves, and runs with
any sign of provider work keep today's hold.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos waves with database pressure, a native run failed at
budget admission with exhausted serialization retries (`40001`/`40P01`).
The run event history had no `native.process_start_requested` event.
About 15 s later the server recorded "Automatic recovery stopped" with
cause `native_continuation_requires_reconciliation`. The task stayed
blocked until a person clicked Retry.
**Expected behavior**
A native run that failed before any provider work, on a fault that
clears by itself, re-queues automatically. The attempt is bounded and
does not spend the failure budget. A persistent failure still ends
failed, with a working Retry.
**Steps to reproduce**
1. Start a native (Paperclip Runner) task.
2. Make budget admission or a setup database write fail with SQLSTATE
`40001` or `40P01` until its bounded retries are exhausted.
3. See the run end `adapter_failed`/`setup_failed`, the task go to
`blocked`, and a `native_continuation_requires_reconciliation` recovery
action that waits for Retry.
**Paperclip version or commit**
`master` at 902f376796.
## What Changed
- `run-failure-cause.ts`: new durable record
`resultJson.nativePreDispatchFailure = { version: 1,
providerWorkStarted: false }`, and
`isNativePreDispatchTransientFailure`. The predicate needs a native
failed run, a generic code (`adapter_failed`/`setup_failed`), the
record, and a transient database `failureCause` (`40001`, `40P01`,
`55P03`, `57P01`-`57P03`, `53300`, `08xxx`, postgres.js connection
codes).
- `run-completion.ts`: `failRun` and `failRunSetup` write the record for
a native run only when all of these are true:
- the caller reports `nativeDispatchMayHaveStarted: false` (an omitted
value means "may have started");
- the run was not stopped during the failure, and has no terminal native
coordinator failure;
- the run's `native_run_finalizations` coordinator, if any, is still the
unclaimed row that runtime selection creates (`observed`, attempt 0, no
lease owner, no controller, no result). Any executor claim, in any
generation, disqualifies;
- the run has no recorded process (`processPid`, `processGroupId`,
`processStartedAt`), no `sessionIdAfter`, and no `native_run_results`
row. Process fields copied from a warm predecessor (marked by
`runnerProfileJson.inheritedProcessIdentityRunId`) are not this run's
own process and do not disqualify; the run's own start events are still
checked;
- the run's whole event history has no execution-start evidence. This
uses the canonical `hasNativeExecutionStartEvidence` helper from restart
recovery (`native.process_start_requested`,
`native.process_identity_recorded`, `adapter.invoke`,
`tool.execution.started`, or any runner event with a source event ID).
The helper is unchanged; the earlier inline copy matched it exactly and
was removed.
- `run-completion.ts`: a recorded transient failure closes the unclaimed
coordinator as `terminal_failure` (`native_pre_dispatch_failure`),
because `hasPendingRunCleanup` otherwise holds the successor on it. It
then calls the existing `scheduleBoundedRetryForRun` with `retryReason:
"interrupted_preparation"`. An answered-interaction continuation keeps
its own retry lane. The storage-outage lane is unchanged.
- `heartbeat.ts`: passes `nativeDispatchMayHaveStarted`. It is true once
native dispatch started, once the dispatch gate entered the executor
(also when the gate then fails), while native ownership is held, or for
a restart-recovery or native-lease resume.
- No change to the native hold in
`modules/wake-queue/adapters/postgres.ts`. The release already returns
early when a queued/scheduled successor exists, so a scheduled retry
never opens the hold. See Risks for why the hold stays for the exhausted
and no-successor cases.
## Verification
- `heartbeat-native-predispatch-retry.test.ts` (assembled heartbeat,
mock native backend): budget admission fails once with `40001`. The run
records the proof, one uncharged `interrupted_preparation` successor is
scheduled, no recovery action opens, and the promoted retry succeeds
once. A failure after the dispatch gate entered the executor gets no
record and no such successor.
- `run-completion.test.ts` (embedded Postgres):
- native pre-dispatch `40P01` in `failRun` writes the record and
schedules `interrupted_preparation`; the release suppresses immediate
recovery;
- native `setup_failed` on `40001` does the same;
- a run with `native.process_start_requested` in its history gets no
record and no retry;
- `nativeDispatchMayHaveStarted: true` and an omitted value (older
caller) get no record and no retry;
- a claimed native coordinator gets no record and no retry; an unclaimed
one is closed and the run re-queues;
- a run with a recorded process or provider session gets no record and
no retry;
- a non-transient cause gets the record but no retry.
- `heartbeat-retry-scheduling.test.ts`: the real scheduler chains two
`interrupted_preparation` retries for a recorded native failure. The
first is due at once. `failureRetries` stays 0. A spent lane returns
`retry_exhausted`, queues nothing, and the run stays `failed`.
- `wake-queue/adapters/postgres.test.ts`: with a scheduled successor,
the release opens no recovery action and the task stays `in_progress`.
Without one, today's hold opens (`blocked`,
`native_continuation_requires_reconciliation`, Retry available).
- `run-failure-cause.test.ts`: the predicate is false for a row without
the record (older server), for `providerWorkStarted: true`, for a
non-transient cause, a non-generic code, a legacy run, and a cancelled
run.
- Command: `cd server && npx vitest run
src/__tests__/heartbeat-native-predispatch-retry.test.ts
src/services/heartbeat/run-failure-cause.test.ts
src/services/heartbeat/run-completion.test.ts
src/modules/wake-queue/adapters/postgres.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts` (all new cases pass).
`tsc --noEmit` for `server` passes. `pnpm verify` selected 2451 files
because `heartbeat.ts` is a hub; I stopped it and rely on CI for the
broad suite.
- Live path unproven: I did not run a restart-chaos wave for this
change.
## Risks
- A wrong "no provider work" record could cause duplicate work. The
record needs several independent facts (no gate entry, no resume claim,
no claimed coordinator, no process, session or result, no
execution-start evidence in the whole history). Any read error keeps the
hold.
- Rows saved by an older server have no record. They keep today's hold
and manual Retry. Nothing is migrated. A run resumed after a deploy is a
restart-recovery or lease resume, so it gets no record.
- The normal path gains no I/O. The evidence reads run only on the
failure path, and only when dispatch was never entered.
- The native hold is not skipped when no successor exists. Skipping it
would send the run to the generic immediate-recovery tail, whose new run
does not carry this lane's counter. That could loop without a bound. So
a spent lane, and a crash between the failure write and the retry
scheduling, still end in today's hold with a working Retry.
- The lane is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`
(12) with backoff after the first attempt, and it never revives a chain
whose failure budget is already spent.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (file edits, shell, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #20484, #20385.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>