mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Automatic task recovery continues a task when its run ends for a
reason that is not a person's to fix
> - A continuation or recovery run can itself fail before it does any
work: a sandbox create timeout, a full sandbox disk, a lease mismatch, a
provider pack check, or a database deadlock
> - Today the stranded reconciler treats such a failure as non-retryable
and blocks the task for the board. The assigned-work sweep then leaves
the board hold alone, because its list of temporary causes is a short
allow-list. The held backlog pass uses a similar allow-list. Nothing
continues the task, and it shows "failed, Retry"
> - This pull request makes the cause test "not owned by a person" on
these paths, and routes a failed continuation run back through the
fresh-continuation fallback
> - The benefit is that such a task continues from its saved work, under
the existing hourly cap, and nobody needs to press Retry
## Linked Issues or Issue Description
**What happened?**
A task was continued automatically after an interruption. The
continuation run failed during setup, before any provider turn (for
example `setup_failed` with a sandbox HTTP timeout). The stranded
reconciler then set the task to `blocked` with a board-owned recovery
hold. The assigned-work sweep left the task to that hold, and the
fresh-continuation fallback never owns that hold kind. The task stayed
blocked and showed a Retry button. Settled holds whose run was a
recovery run that failed with `adapter_failed`, `setup_failed` or a
provider pack error stayed in the same state, because the held backlog
pass accepted only restart-class codes.
**Expected behavior**
A run that ends for a reason a person does not own is continued again
from saved work. The continuation goes through the fresh-continuation
fallback, so the old workspace epoch is fenced or a new isolated
workspace is used. The per-task limit of 6 automatic continuations per
hour still applies. At the limit the task waits for the next window and
is not blocked.
**Steps to reproduce**
1. Record a fresh continuation for an interrupted run of an
agent-assigned task.
2. Let the continuation run fail with `setup_failed` before provider
work.
3. Run the stranded reconciler. The task becomes `blocked` with a board
hold.
4. Run the assigned-work sweep. It returns `left_to_hold_owner`. No
continuation follows.
**Paperclip version or commit**
master at 6917d0568d
**Deployment mode**
Hosted (two servers can run at once during a deploy)
## What Changed
- New `continuation-cause.ts` holds the cause classification:
`PERSON_OWNED_FAILURE_CODES` (moved, unchanged), a new short list
`PERMANENT_FAILURE_CODES` (low-trust boundary codes,
`adapter_engine_unavailable`, `agent_not_invokable`, `agent_not_found`,
`agent_interaction_delivery_denied`), `isSystemOwnedFailureCode`, and
`isAutomaticContinuationRun` (a retry, scheduled retry, fresh
continuation, sweep or recovery run; never a person's Retry).
`fresh-continuation-fallback.ts` re-exports what callers use.
- `stranded-hold-continuation.ts`: the sweep continues a recent (24
hours) board hold when the failure cause is system-owned, not only for a
short allow-list. A reusable sandbox resume failure still needs a
transient cause.
- `recovery/service.ts`: the stranded reconciler does not escalate an
agent-assigned `todo` or `in_progress` task to the board when its latest
run was an automatic continuation or recovery run that ended within 24
hours with a system-owned cause. The assigned-work sweep owns that task.
A person's Retry, a person-owned cause, a permanent cause, or an older
failure escalates as before.
- `assigned-work-sweep.ts`: for a failed automatic continuation run, the
sweep asks the fresh-continuation fallback first. A recorded
continuation is the repair (fenced epoch or new isolated workspace). On
the fallback's hourly backoff the sweep records "not started" with the
retry time and keeps the task unblocked. While exact recovery still
waits, it records the retry time. A person-owned reason from the
fallback is written on a refused attempt.
- `task-automatic-attempts.ts`: the shared 6-per-hour ledger also counts
`server_restart` and `interrupted_preparation` scheduled retries, so the
sweep cannot add attempts on top of those lanes in the same hour.
- `fresh-continuation-fallback.ts`: the held backlog pass also accepts a
source run that was an automatic continuation or recovery run with a
system-owned code (24 hours, 5 per pass, 10 per minute unchanged). With
no hold to carry a pending check, a wait is measured from the source
run's end, so it is bounded.
- `fresh-continuation-fallback.ts`: rechecks the cause with the shared
classifier and applies the shared hourly cap inside the task lock. A
continuation keeps the old writer's workspace unless a wait was actually
cut short (bound measured from a recorded check, or a wait skipped).
- New test file `failed-continuation-continues-again.test.ts` (15
tests).
### Changed test expectations
- `interrupted-preparation-hold-settle.test.ts`, "bounds repeated lost
leases before provider work" (both cases): the lane cap still holds (2
and 12 successors). The end state is no longer `blocked`: the task keeps
its status, no further run exists, and the sweep returns a deferral with
a future retry time (`deferred_cap` at 12 ledger attempts,
`deferred_rate_limit` spacing at 2). The fixture now settles the
scheduled wakes and retry times the loop leaves behind, so the sweep
reads the real state. A person's Retry is still accepted.
- Same file, "blocks the task when a replaced preparation's lane is
spent" is renamed "leaves the task to the assigned-work sweep, not
blocked, when a replaced preparation's lane is spent". It still expects
no successor run, and now expects `todo` with no active hold.
- `heartbeat-process-recovery.test.ts` and `task-retry.test.ts`: no
change. Their runs either ended months before (outside the 24-hour
window) or were a person's Retry, so they escalate as before. The bound,
no-duplicate, and redaction tests keep their assertions.
## The one classifier
`continuation-cause.ts` `continuesAutomatically(run)` (and its SQL twin
`continuesAutomaticallySql`) decides whether a failed run continues on
its own:
- the cause is system-owned: not in `PERSON_OWNED_FAILURE_CODES`
(unchanged), not in the new `PERMANENT_FAILURE_CODES`, not an external
execution whose stop cannot be confirmed (Dot and Muse, the same rule
the retry scheduler uses to decline them permanently), and a reusable
sandbox resume failure only with a transient cause; and
- the run was an automatic continuation or recovery run (a scheduled
retry, a recovery wake, a fresh continuation; never a person's Retry),
or a first run that ended with a restart-class code or a transient setup
failure before provider work (the pre-existing allow-list for the
backlog and the board-hold takeover).
A person's failed Retry keeps master's outcome at every entry point: the
classifier never treats it as an automatic run. Whether it should
continue on its own is a separate product decision.
## Every entry point and the cap it uses
Shared cap = `sharedContinuationAttempts` in
`recovery/continuation-ledger.ts`: 6 per task per rolling hour, counting
fresh continuation records, settled sweep wakes, and transient, restart
and preparation retries. Only attempts that started or created a run
count.
- Disposition pass (`settleUnrecoverableExecutions` to
`continueFreshInTransaction`): cause test of the classifier and the
shared cap, inside the fallback under the task lock. Run kind is not
required on this path: it handles fresh failures of any run.
- Held backlog (`continueHeldExecutions`): `continuesAutomaticallySql`
in the listing; then the fallback's cause test and shared cap. Pacing 5
per pass, 10 per minute, 24 hours.
- System continuation wake in the queue (`heartbeat/queue.ts`,
unchanged) to the fallback: cause test and shared cap.
- Dispatch refusal (`run-dispatch` Postgres adapter, unchanged) to the
fallback: cause test and shared cap.
- Sweep board-hold takeover (`strandedHoldContinuesAutomatically`):
`continuesAutomatically`, 24 hours.
- Sweep route to the fallback (trigger `assigned_work`): recovery run
with a system-owned cause, then the fallback's cause test and shared
cap; the sweep claim also applies its ledger.
- Old reconciler deferral (`assignedWorkSweepOwnsFailedRun`): automatic
run with a system-owned cause (`canContinueRecoveryRunAutomatically`),
24 hours. First runs keep their existing escalation and budgets.
- Sweep plain wake (unchanged path, for first runs): the sweep claim
ledger (`taskAutomaticAttempts`: started sweep attempts, scheduler
retries, and fresh records from other paths), spacing from every
attempt.
## Self-review: places where the rule still differs, on purpose
- The fallback's fresh path continues a first run with any system-owned
cause, while the backlog limits first runs to restart-class and
transient setup codes. The backlog limit is the existing herd rule for
records older releases left behind.
- The fallback treats a pending question as a person's wait; the sweep
treats a question asked before a person's Retry as superseded. In that
one shape the sweep sends its ordinary wake, as before.
- The scheduler lanes (transient, restart, preparation) keep their own
bounds and do not read the shared cap before scheduling; their runs
count in it, so every other path stops once they fill the hour. Changing
the scheduler is outside these files.
- The sweep's plain wake path does not test the cause; it relies on the
invariant's waiting reasons, as before.
## Waits and their bounds
- Every exact-recovery wait in the fallback (settling execution, native
finalizer, live controller lease, preparation lane, saved-work recovery,
workspace restore owner, retained-source check, previous local process):
it starts at the first recorded pending check on the run's hold. When no
check is recorded (no hold, a hold nobody checks, or a replaced record),
it starts at the source run's end (`finished_at`, a database row that
every server reads the same). Bound: 10 minutes idle, 15 minutes while
the owner shows progress, 30 minutes while saved-work recovery advances.
Then the fallback continues from saved work.
- A source run that ended more than 24 hours ago with no recorded check
does not wait: the fallback answers `retry_needed` (the existing named
state that leaves an old hold for a person's Retry), so nothing waits
silently and the 24-hour rule holds.
- The sweep on a fallback backoff: waits until the oldest continuation
leaves the hour (6 per task per hour), task stays unblocked.
- The sweep while exact recovery waits: rechecks at the fallback's retry
time (at most 4 minutes), until the bound above.
## Decisions on permanent causes
- `PERSON_OWNED_FAILURE_CODES` is not changed. A separate
`PERMANENT_FAILURE_CODES` list keeps causes that cannot heal by
themselves out of the new automatic paths, so they still block and tell
a person. `repository_catalog_unavailable` was transient in the old
allow-list too; in the failing Retry test it was a person's Retry, which
the reconciler still escalates.
## Verification
- `failed-continuation-continues-again.test.ts`: 28 tests, including a
12-case table (person's Retry, automatic continuation, first assignment;
a person's Retry gets master's outcome everywhere) x (setup failure,
restart code, permanent code, person-owned code) through the backlog,
the sweep takeover and route, and the reconciler, and a test that
waiting checks spend no attempt (red before the fix for the first-run
case and for the cap); earlier: 15 tests (the latest three added
red-first: permanent codes and a person's Retry stay blocked on the
backlog; no entry point continues a person-owned or permanent code; six
mixed attempts stop the backlog until the next window; plus a pin that
the ledger uses the restart lane's window, cap and reason). Earlier:
Before the fix, the 10 behavior tests failed for the expected reasons
(board escalation, `left_to_hold_owner`, no backlog continuation,
endless wait with no hold, endless wait on an unchecked hold at 20
minutes and 30 hours, checked alternately by two sweep instances). The
person-owned guard passed before and after. After: 11 of 11 pass. After
the last change, alone: `fresh-continuation-fallback.test.ts` 72/72,
`fresh-continuation-move-forward-offer.test.ts` 14/14,
`assigned-work-invariant.test.ts` 31/31,
`interrupted-preparation-hold-settle.test.ts` 29/29.
- Locally after the change, one file at a time:
`assigned-work-invariant.test.ts` 31/31,
`recovery/assigned-work-invariant.test.ts` 50/50,
`interrupted-preparation-hold-settle.test.ts` 29/29,
`task-retry.test.ts` 117/117, the 10 previously failing
`heartbeat-process-recovery.test.ts` tests 12/12 (filtered run),
`fresh-continuation-fallback.test.ts` 66/72 in a loaded run with 6
timeouts at 30 to 90 s that pass alone (8/8 filtered).
- `tsc --noEmit` for `server`: clean.
- CI supplies the full suite.
## Risks
- Behavior change: a recent failure with a cause outside the
person-owned list is no longer escalated to the board by the stranded
reconciler. The sweep continues it instead, at most 6 times per hour per
task, then once per window. Some codes outside that list may be
configuration a person must change (for example `low_trust_*`,
`adapter_engine_unavailable`, `agent_not_found`). Such tasks now retry
at the capped pace instead of blocking. The person-owned list is not
changed in this pull request.
- Upgrade: tasks blocked by an older release with a board hold for such
a cause, within 24 hours, are continued by the sweep on its first
passes. The sweep's 10 repairs per minute per process, the delivery
pacing, and the hourly cap bound the burst.
- Two servers: the sweep's claim and the fallback each run under the
task row lock, and the fallback is idempotent per source run.
## Model Used
- Claude Fable 5.2 (Anthropic), via Claude Code with tool use and code
execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>