fix(presentation): a provider failure the system is still recovering shows as working, not as needing Retry (#21369)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a provider ends a run's session with a terminal failure
(`native_provider_terminal_failed`), Paperclip recovers by itself: it
checks whether a safe replacement is possible, saves the remote files,
and then continues the task in a fresh run from saved work
> - These steps take 15 to 60 seconds. Until the continuation is
recorded, the task view did not know the system was still deciding
> - In a restart-chaos wave, two tasks showed a failed attempt with a
Retry button for 13 to 27 seconds, and one of them changed state in a
loop: failed, then working, then failed with Retry, then working
> - Product rule: a task that the system will continue by itself never
shows as failed with a Retry button, and the state does not change back
and forth during one recovery
> - This pull request reads the record that says the system is still
deciding (the native coordinator's terminal-failure row) and shows the
task as working during that time, within the existing 2-minute bound
> - The benefit is one steady "working" state from the failure to the
new run, and Retry comes back at once when the system hands the task to
a person

## Linked Issues or Issue Description

Refs #21264, #21353, #21200, #21320

**What happened?**

Restart-chaos wave `w-20261113T092657-s7302` on master d1d71a7755, tasks
t030 and t039 (Claude through ACPX). The provider returned a temporary
server error ("ACP agent reported a terminal service failure. API Error:
503 ..."). The run failed with `native_provider_terminal_failed`. The
system then continued the task by itself
(`issue_recovery_action_restored`), and a new run completed the task. In
between, the task view showed:

- t030: 10:20:18 working → 10:20:32 failed, Retry offered → 10:20:45
working (13.4 s with Retry).
- t039: 10:21:15 failed, no Retry → 10:21:29 working → 10:21:46 failed,
Retry offered → 10:22:12 working (26.8 s from the failure; it changed
state in a loop).

The wave database is not kept, so the record at each step comes from the
samples (`recoveryOwner`, `executionBlocker`, issue status), the server
log, and the code paths that write those records:

1. The run ends. The native executor saves the coordinator row (`phase:
terminal_failure`, `failureCode: native_provider_terminal_failed`, no
`replacementDenied`) and a board-owned hold with cause
`native_provider_terminal_failed`, and blocks the task. The projection
fell through to "failed". At t039 10:21:15, the execution-blocker check
removed Retry because the remote files were not saved yet.
2. Workspace recovery adopts the hold (`ownerType: system`,
`systemRecovery` running). While it runs, the task shows "working" (t039
10:21:29).
3. The files are saved (`systemRecovery` complete). No record now says
"running", the execution-blocker check no longer removes Retry, and the
task shows "failed" with Retry (t039 10:21:46, t030 10:20:32).
4. Safe replacement is denied (`replacementDenied` on the coordinator
row, and the hold is upserted as board-owned). The settle pass then
records a fresh continuation (log: "Unfinished run continues in a fresh
run from saved work", `trigger: recovery_refused`), sets the task back
to `todo`, and the task shows "working" (t039 10:22:10–12).

**Expected behavior**

From the failure until the continuation is recorded, the task shows
"working" with no Retry, without changing back and forth. When recovery
is handed to a person, or nothing is recorded within the bound, Retry is
offered.

**Steps to reproduce**

1. Let a native run fail with `native_provider_terminal_failed` on a
remote workspace.
2. Read the task before the fresh continuation is recorded.
3. The task shows "failed" with Retry.

## What Changed

- `server/src/services/task-interruption-presentation.ts`:
- The run read selects `providerRecoveryAt`: the coordinator row's
update time while the row says the system is still deciding (a failed
native run, `phase: terminal_failure`, `failureCode:
native_provider_terminal_failed`, no result, no lease owner, no
successor run). This matches the rule that `execution-projection.ts`
already uses to show "Checking recovery" for this failure.
- A denied replacement (`replacementDenied`) does not end this time.
After a denial, the settle pass continues the task in a fresh run unless
the failure code is a person's. `native_provider_terminal_failed` is not
a person's code.
- While the time is inside the existing 2-minute bound
(`AUTOMATIC_DISPOSITION_GRACE_MS`), the task shows `working`, `canRetry:
false`, responsibility `system`, with the saved-files step text when
there is one. The bound counts from the latest of: the run end, the
coordinator row update, and the recovery record update. So each recovery
step extends it, and the steps do not make the state change back and
forth.
- Retry is offered at once when recovery was handed to a person: a
fresh-continuation check that decided it cannot continue (any kind other
than `exact_recovery_pending` or `backoff`), `ownerGoneAttestation`,
`explicitUserContinuation`, or an exhausted saved-work recovery.
- No change to recovery decision code, the fallback, or the Retry path.
No new record or marker. The coordinator row already records "still
deciding".

## Verification

- Red (before the fix), in `server/`: `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts`: the new t039
sequence test failed at its first step (`state: "failed"`, `canRetry:
true` right after the failure).
- Green (after the fix), in `server/`: `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/task-interruption-query.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 4 files, 148 tests pass.
- New tests in `task-interruption-automatic-path.test.ts`:
- The t039 sequence (failure, files being saved, files saved,
replacement denied with the hold upserted to board) shows `working` with
`canRetry: false` at every step.
  - The bound passed with nothing recorded: Retry.
  - The continuation worker decided it cannot continue: Retry at once.
  - A person took over the hold: Retry at once.
- The #21200, #21264, #21320 and #21353 guard tests pass unchanged. This
includes "still offers Retry when recovery gave up and handed the hold
to the board" and the person-owned hold tests.
- Related suites: `workspace-recovery-preparation-dispatch`,
`explicit-native-continuation`, `workspace-recovery-retry-copyback`,
`workspace-recovery-admission`, `native-safe-replacement` and
`run-failure-report`: 1046 of 1046 tests pass. One run on a loaded
machine had a 30-second `afterAll` cleanup timeout in
`native-safe-replacement` after all its tests passed. An earlier
parallel run had one timing failure in
`workspace-recovery-preparation-dispatch` that did not repeat.
- `node scripts/typecheck-layers.mjs` in `server/`: passes.

## Risks

- A provider failure that the system will not continue (for example, the
task was reassigned) can show "working" for up to 2 minutes before Retry
appears. This is the same bound #21264 uses for a system-owned hold on
an ended run.
- The rule applies only to `native_provider_terminal_failed`, which is
the failure that the safe-replacement evaluation owns. Other terminal
failures are unchanged.
- One more small read per page, only when a listed task has a run in
this state.
- Note for the recovery owner, not changed here: the provider's 503
("credential validation failed. This is a server-side issue") is
classified as a permanent provider failure
(`native_provider_terminal_failed`). Whether a 5xx from the provider
should instead be a temporary retry on the same run is the recovery
owner's decision.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs, reading
restart-chaos wave evidence).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 06:43:34 -06:00
1 parent 8e41247b65
commit 7b27fc453f
2 files changed
+86 -1

No files matched your search

@@ -1,5 +1,5 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { and, eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, agentWakeupRequests, applyPendingMigrations, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db";
@@ -158,6 +158,70 @@ describe.skipIf(!support.supported)("Retry is offered only when no automatic pat
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
});
async function seedProviderTerminalFailure(finishedAt: Date) {
const task = await seedTask("blocked");
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: task.companyId, agentId: task.agentId,
nativeIssueId: task.issue.id, scopeKind: "issue", status: "failed", runtimeMode: "native",
nativePhase: "terminal_failure", errorCode: "native_provider_terminal_failed",
error: "ACP agent reported a terminal service failure.", startedAt: new Date(finishedAt.getTime() - 60_000), finishedAt });
await db.insert(nativeRunFinalizations).values({ companyId: task.companyId, issueId: task.issue.id, runId,
phase: "terminal_failure", attempt: 1, failureCode: "native_provider_terminal_failed", updatedAt: finishedAt });
const [hold] = await db.insert(issueRecoveryActions).values({ companyId: task.companyId, sourceIssueId: task.issue.id,
kind: "active_run_watchdog", status: "active", ownerType: "board", returnOwnerAgentId: task.agentId,
cause: "native_provider_terminal_failed", fingerprint: `provider-terminal:${runId}`,
evidence: { runId, nativeFailureBlock: { runId, statusVersion: 1 }, coordinatorAttempt: 1,
sourceFailureCode: "native_provider_terminal_failed", recoveryDisposition: "native_provider_terminal_failed",
recoveryMode: "replacement", providerSessionEstablished: true },
nextAction: "Verify that the failed provider stopped and reconcile its action outcomes.",
maxAttempts: 3, createdAt: finishedAt, updatedAt: finishedAt }).returning();
return { ...task, issue: { ...task.issue, status: "blocked" }, runId, hold: hold! };
}
it("shows a provider failure the system is still recovering as working through every recovery step, never offering Retry in between", async () => {
const { issue, runId, hold, companyId } = await seedProviderTerminalFailure(new Date(Date.now() - 2_000));
const working = { responsibility: "system", state: "working", canRetry: false, observedRunId: runId };
expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working);
const running = { ...createSystemRecovery(runId), state: "running" as const, phase: "preserve_files" as const };
await db.update(issueRecoveryActions).set({ ownerType: "system", updatedAt: new Date(),
evidence: { ...hold.evidence, systemRecovery: running, systemRecoveryActiveAt: Date.now() } })
.where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working);
await db.update(issueRecoveryActions).set({ updatedAt: new Date(),
evidence: { ...hold.evidence, systemRecovery: { ...running, state: "complete" }, systemRecoveryActiveAt: Date.now() - 60_000 } })
.where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working);
await db.update(nativeRunFinalizations).set({ failureDetail: { replacementDenied: "provider_failure_meaning_unverified" },
updatedAt: new Date() }).where(eq(nativeRunFinalizations.runId, runId));
await db.update(issueRecoveryActions).set({ ownerType: "board", cause: "provider_failure_meaning_unverified",
fingerprint: `native-replacement:${runId}`, evidence: { runId, attempts: 1 }, nextAction: "Review the provider failure.",
updatedAt: new Date() }).where(and(eq(issueRecoveryActions.id, hold.id), eq(issueRecoveryActions.companyId, companyId)));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working);
});
it("offers Retry for a provider failure once the recovery bound passes with nothing recorded", async () => {
const { issue } = await seedProviderTerminalFailure(new Date(Date.now() - 3 * 60_000));
expect(await taskInterruptionPresentation(db, { ...issue, status: "todo" })).toMatchObject({ state: "failed", canRetry: true });
});
it("offers Retry for a provider failure at once when the continuation worker decided it cannot continue the task", async () => {
const { issue, hold } = await seedProviderTerminalFailure(new Date(Date.now() - 2_000));
await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence,
freshContinuationCheck: { kind: "not_applicable", reason: "task_changed", checks: 1, checkedAt: Date.now(), nextCheckAt: null } } })
.where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, { ...issue, status: "todo" })).toMatchObject({ state: "failed", canRetry: true });
});
it("offers Retry for a provider failure at once when a person took over its hold", async () => {
const { issue, hold, companyId } = await seedProviderTerminalFailure(new Date(Date.now() - 2_000));
await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence, ownerGoneAttestation: { companyId } } })
.where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, { ...issue, status: "todo" })).toMatchObject({ state: "failed", canRetry: true });
});
it("shows a healthy task as working while the message queued during its succeeded run is delivered", async () => {
const { companyId, agentId, issue } = await seedTask();
const runId = randomUUID(), finishedAt = new Date(Date.now() - 3_000);
@@ -52,6 +52,8 @@ type ProjectionRun = Pick<typeof heartbeatRuns.$inferSelect,
nativeResumeAt?: Date | null;
/** When the native coordinator saved this run's terminal failure, if it did. */
nativeFailedAt?: Date | null;
/** When native recovery of this run's provider failure last moved, while that recovery is still undecided. */
providerRecoveryAt?: Date | null;
/** A succeeded native run whose accepted result ran out of finalization retries; Retry applies it again. */
nativeFinalizationRetry?: boolean;
/** A saved-result Retry of this succeeded run is still being applied. */
@@ -98,6 +100,7 @@ type ProjectionState = {
* not yet started) or will check it again at `nextCheckAt`. */
freshContinuation?: { state: "continuing" } | { state: "scheduled"; nextCheckAt: number };
automaticDispositionAt?: number;
recoveryHandedToPerson?: boolean;
};
// A task read reports the change that made a task fail, not every later read of
@@ -334,6 +337,10 @@ function projectTaskInterruptionState(issue: ProjectionIssue, input: ProjectionS
if (nativeResumeAt && !system && now - nativeResumeAt < NATIVE_RESUME_DISPATCH_GRACE_MS)
return { ...base, state: "working" };
if (recoveryRunning) return { ...base, state: "working", ...(recoveryStep ? { recoveryStep } : {}) };
const providerRecoveryAt = run.providerRecoveryAt?.getTime();
if (providerRecoveryAt !== undefined && run.status === "failed" && !input.recoveryHandedToPerson && system?.state !== "exhausted"
&& now - Math.max(providerRecoveryAt, run.finishedAt?.getTime() ?? 0, action?.updatedAt.getTime() ?? 0) < AUTOMATIC_DISPOSITION_GRACE_MS)
return { ...base, state: "working", ...(recoveryStep ? { recoveryStep } : {}) };
const dependencies = input.retryDependencies;
if (retryWait?.sourceRunId === run.id && !retryWait.executionWaitReason && !retryWait.preparationStatus && dependencies?.count)
return { ...base, intentId: retryWait.id, state: "working",
@@ -453,6 +460,12 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
nativeFailedAt: sql<Date | null>`(select ${nativeRunFinalizations.updatedAt} from ${nativeRunFinalizations}
where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
and ${nativeRunFinalizations.phase} = 'terminal_failure')`.mapWith(value => value == null ? null : new Date(value)),
providerRecoveryAt: sql<Date | null>`(select ${nativeRunFinalizations.updatedAt} from ${nativeRunFinalizations}
where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
and ${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} = 'failed'
and ${nativeRunFinalizations.phase} = 'terminal_failure' and ${nativeRunFinalizations.failureCode} = 'native_provider_terminal_failed'
and ${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.leaseOwner} is null
and coalesce(${nativeRunFinalizations.failureDetail}->>'successorRunId', '') = '')`.mapWith(value => value == null ? null : new Date(value)),
nativeFinalizationRetry: exhaustedNativeFinalizationCondition(),
nativeFinalizationRetryPending: savedResultRetryPendingCondition(),
// The run whose resultJson supplied providerFailure. A scheduled retry
@@ -613,6 +626,13 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
inArray(issueRecoveryActions.cause, [...EXECUTION_RECONCILIATION_CAUSES]), isNull(issueRecoveryActions.ownerUserId),
sql`not (${issueRecoveryActions.evidence} ?| array['systemRecovery', 'executionReconciliation', 'freshContinuation',
'freshContinuationCheck', 'ownerGoneAttestation', 'explicitUserContinuation'])`)) : [];
const recoveringRunIds = runs.flatMap(run => run.providerRecoveryAt ? [run.id] : []);
const handedToPerson = new Set((recoveringRunIds.length ? await db.select({ issueId: issueRecoveryActions.sourceIssueId })
.from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids),
inArray(sql`${issueRecoveryActions.evidence}->>'runId'`, recoveringRunIds),
sql`(${issueRecoveryActions.evidence} ?| array['ownerGoneAttestation', 'explicitUserContinuation']
or coalesce(${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,kind}', 'exact_recovery_pending') not in ('exact_recovery_pending', 'backoff'))`)) : [])
.map(row => row.issueId));
const automaticDispositionAt = new Map<string, number>();
if (dispositionRows.length) {
const { isPersonOwnedFailureCode } = await import("./fresh-continuation-fallback.js");
@@ -735,6 +755,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
const presentation = projectTaskInterruption(issue, { ...facts,
freshContinuation: freshContinuationFor(issue.id, facts.run?.id),
automaticDispositionAt: automaticDispositionAt.get(issue.id),
recoveryHandedToPerson: handedToPerson.has(issue.id),
interruptWait: interruptWaitFor(issue.id),
projectAccessAllowed: denied?.length === 0,
projectAccessProjectId: denied?.includes(issue.projectId ?? "") ? issue.projectId : null,