From 7b27fc453fbac0d7474a4ddd3e4e42557a4450af Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 06:43:34 -0600 Subject: [PATCH] fix(presentation): a provider failure the system is still recovering shows as working, not as needing Retry (#21369) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a provider ends a run's session with a terminal failure (`native_provider_terminal_failed`), Paperclip recovers by itself: it checks whether a safe replacement is possible, saves the remote files, and then continues the task in a fresh run from saved work > - These steps take 15 to 60 seconds. Until the continuation is recorded, the task view did not know the system was still deciding > - In a restart-chaos wave, two tasks showed a failed attempt with a Retry button for 13 to 27 seconds, and one of them changed state in a loop: failed, then working, then failed with Retry, then working > - Product rule: a task that the system will continue by itself never shows as failed with a Retry button, and the state does not change back and forth during one recovery > - This pull request reads the record that says the system is still deciding (the native coordinator's terminal-failure row) and shows the task as working during that time, within the existing 2-minute bound > - The benefit is one steady "working" state from the failure to the new run, and Retry comes back at once when the system hands the task to a person ## Linked Issues or Issue Description Refs #21264, #21353, #21200, #21320 **What happened?** Restart-chaos wave `w-20261113T092657-s7302` on master d1d71a7755, tasks t030 and t039 (Claude through ACPX). The provider returned a temporary server error ("ACP agent reported a terminal service failure. API Error: 503 ..."). The run failed with `native_provider_terminal_failed`. The system then continued the task by itself (`issue_recovery_action_restored`), and a new run completed the task. In between, the task view showed: - t030: 10:20:18 working → 10:20:32 failed, Retry offered → 10:20:45 working (13.4 s with Retry). - t039: 10:21:15 failed, no Retry → 10:21:29 working → 10:21:46 failed, Retry offered → 10:22:12 working (26.8 s from the failure; it changed state in a loop). The wave database is not kept, so the record at each step comes from the samples (`recoveryOwner`, `executionBlocker`, issue status), the server log, and the code paths that write those records: 1. The run ends. The native executor saves the coordinator row (`phase: terminal_failure`, `failureCode: native_provider_terminal_failed`, no `replacementDenied`) and a board-owned hold with cause `native_provider_terminal_failed`, and blocks the task. The projection fell through to "failed". At t039 10:21:15, the execution-blocker check removed Retry because the remote files were not saved yet. 2. Workspace recovery adopts the hold (`ownerType: system`, `systemRecovery` running). While it runs, the task shows "working" (t039 10:21:29). 3. The files are saved (`systemRecovery` complete). No record now says "running", the execution-blocker check no longer removes Retry, and the task shows "failed" with Retry (t039 10:21:46, t030 10:20:32). 4. Safe replacement is denied (`replacementDenied` on the coordinator row, and the hold is upserted as board-owned). The settle pass then records a fresh continuation (log: "Unfinished run continues in a fresh run from saved work", `trigger: recovery_refused`), sets the task back to `todo`, and the task shows "working" (t039 10:22:10–12). **Expected behavior** From the failure until the continuation is recorded, the task shows "working" with no Retry, without changing back and forth. When recovery is handed to a person, or nothing is recorded within the bound, Retry is offered. **Steps to reproduce** 1. Let a native run fail with `native_provider_terminal_failed` on a remote workspace. 2. Read the task before the fresh continuation is recorded. 3. The task shows "failed" with Retry. ## What Changed - `server/src/services/task-interruption-presentation.ts`: - The run read selects `providerRecoveryAt`: the coordinator row's update time while the row says the system is still deciding (a failed native run, `phase: terminal_failure`, `failureCode: native_provider_terminal_failed`, no result, no lease owner, no successor run). This matches the rule that `execution-projection.ts` already uses to show "Checking recovery" for this failure. - A denied replacement (`replacementDenied`) does not end this time. After a denial, the settle pass continues the task in a fresh run unless the failure code is a person's. `native_provider_terminal_failed` is not a person's code. - While the time is inside the existing 2-minute bound (`AUTOMATIC_DISPOSITION_GRACE_MS`), the task shows `working`, `canRetry: false`, responsibility `system`, with the saved-files step text when there is one. The bound counts from the latest of: the run end, the coordinator row update, and the recovery record update. So each recovery step extends it, and the steps do not make the state change back and forth. - Retry is offered at once when recovery was handed to a person: a fresh-continuation check that decided it cannot continue (any kind other than `exact_recovery_pending` or `backoff`), `ownerGoneAttestation`, `explicitUserContinuation`, or an exhausted saved-work recovery. - No change to recovery decision code, the fallback, or the Retry path. No new record or marker. The coordinator row already records "still deciding". ## Verification - Red (before the fix), in `server/`: `pnpm exec vitest run src/__tests__/task-interruption-automatic-path.test.ts`: the new t039 sequence test failed at its first step (`state: "failed"`, `canRetry: true` right after the failure). - Green (after the fix), in `server/`: `pnpm exec vitest run src/__tests__/task-interruption-automatic-path.test.ts src/__tests__/task-interruption-query.test.ts src/services/task-interruption-presentation.test.ts src/__tests__/recovery-wake-herd.test.ts`: 4 files, 148 tests pass. - New tests in `task-interruption-automatic-path.test.ts`: - The t039 sequence (failure, files being saved, files saved, replacement denied with the hold upserted to board) shows `working` with `canRetry: false` at every step. - The bound passed with nothing recorded: Retry. - The continuation worker decided it cannot continue: Retry at once. - A person took over the hold: Retry at once. - The #21200, #21264, #21320 and #21353 guard tests pass unchanged. This includes "still offers Retry when recovery gave up and handed the hold to the board" and the person-owned hold tests. - Related suites: `workspace-recovery-preparation-dispatch`, `explicit-native-continuation`, `workspace-recovery-retry-copyback`, `workspace-recovery-admission`, `native-safe-replacement` and `run-failure-report`: 1046 of 1046 tests pass. One run on a loaded machine had a 30-second `afterAll` cleanup timeout in `native-safe-replacement` after all its tests passed. An earlier parallel run had one timing failure in `workspace-recovery-preparation-dispatch` that did not repeat. - `node scripts/typecheck-layers.mjs` in `server/`: passes. ## Risks - A provider failure that the system will not continue (for example, the task was reassigned) can show "working" for up to 2 minutes before Retry appears. This is the same bound #21264 uses for a system-owned hold on an ended run. - The rule applies only to `native_provider_terminal_failed`, which is the failure that the safe-replacement evaluation owns. Other terminal failures are unchanged. - One more small read per page, only when a listed task has a run in this state. - Note for the recovery owner, not changed here: the provider's 503 ("credential validation failed. This is a server-side issue") is classified as a permanent provider failure (`native_provider_terminal_failed`). Whether a 5xx from the provider should instead be a temporary retry on the same run is the recovery owner's decision. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs, reading restart-chaos wave evidence). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../task-interruption-automatic-path.test.ts | 66 ++++++++++++++++++- .../task-interruption-presentation.ts | 21 ++++++ 2 files changed, 86 insertions(+), 1 deletion(-) diff --git a/server/src/__tests__/task-interruption-automatic-path.test.ts b/server/src/__tests__/task-interruption-automatic-path.test.ts index 3c9c179760d..1fea7c136d6 100644 --- a/server/src/__tests__/task-interruption-automatic-path.test.ts +++ b/server/src/__tests__/task-interruption-automatic-path.test.ts @@ -1,5 +1,5 @@ import { randomUUID } from "node:crypto"; -import { eq } from "drizzle-orm"; +import { and, eq } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, agentWakeupRequests, applyPendingMigrations, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db"; @@ -158,6 +158,70 @@ describe.skipIf(!support.supported)("Retry is offered only when no automatic pat expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true }); }); + async function seedProviderTerminalFailure(finishedAt: Date) { + const task = await seedTask("blocked"); + const runId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: runId, companyId: task.companyId, agentId: task.agentId, + nativeIssueId: task.issue.id, scopeKind: "issue", status: "failed", runtimeMode: "native", + nativePhase: "terminal_failure", errorCode: "native_provider_terminal_failed", + error: "ACP agent reported a terminal service failure.", startedAt: new Date(finishedAt.getTime() - 60_000), finishedAt }); + await db.insert(nativeRunFinalizations).values({ companyId: task.companyId, issueId: task.issue.id, runId, + phase: "terminal_failure", attempt: 1, failureCode: "native_provider_terminal_failed", updatedAt: finishedAt }); + const [hold] = await db.insert(issueRecoveryActions).values({ companyId: task.companyId, sourceIssueId: task.issue.id, + kind: "active_run_watchdog", status: "active", ownerType: "board", returnOwnerAgentId: task.agentId, + cause: "native_provider_terminal_failed", fingerprint: `provider-terminal:${runId}`, + evidence: { runId, nativeFailureBlock: { runId, statusVersion: 1 }, coordinatorAttempt: 1, + sourceFailureCode: "native_provider_terminal_failed", recoveryDisposition: "native_provider_terminal_failed", + recoveryMode: "replacement", providerSessionEstablished: true }, + nextAction: "Verify that the failed provider stopped and reconcile its action outcomes.", + maxAttempts: 3, createdAt: finishedAt, updatedAt: finishedAt }).returning(); + return { ...task, issue: { ...task.issue, status: "blocked" }, runId, hold: hold! }; + } + + it("shows a provider failure the system is still recovering as working through every recovery step, never offering Retry in between", async () => { + const { issue, runId, hold, companyId } = await seedProviderTerminalFailure(new Date(Date.now() - 2_000)); + const working = { responsibility: "system", state: "working", canRetry: false, observedRunId: runId }; + expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working); + + const running = { ...createSystemRecovery(runId), state: "running" as const, phase: "preserve_files" as const }; + await db.update(issueRecoveryActions).set({ ownerType: "system", updatedAt: new Date(), + evidence: { ...hold.evidence, systemRecovery: running, systemRecoveryActiveAt: Date.now() } }) + .where(eq(issueRecoveryActions.id, hold.id)); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working); + + await db.update(issueRecoveryActions).set({ updatedAt: new Date(), + evidence: { ...hold.evidence, systemRecovery: { ...running, state: "complete" }, systemRecoveryActiveAt: Date.now() - 60_000 } }) + .where(eq(issueRecoveryActions.id, hold.id)); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working); + + await db.update(nativeRunFinalizations).set({ failureDetail: { replacementDenied: "provider_failure_meaning_unverified" }, + updatedAt: new Date() }).where(eq(nativeRunFinalizations.runId, runId)); + await db.update(issueRecoveryActions).set({ ownerType: "board", cause: "provider_failure_meaning_unverified", + fingerprint: `native-replacement:${runId}`, evidence: { runId, attempts: 1 }, nextAction: "Review the provider failure.", + updatedAt: new Date() }).where(and(eq(issueRecoveryActions.id, hold.id), eq(issueRecoveryActions.companyId, companyId))); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working); + }); + + it("offers Retry for a provider failure once the recovery bound passes with nothing recorded", async () => { + const { issue } = await seedProviderTerminalFailure(new Date(Date.now() - 3 * 60_000)); + expect(await taskInterruptionPresentation(db, { ...issue, status: "todo" })).toMatchObject({ state: "failed", canRetry: true }); + }); + + it("offers Retry for a provider failure at once when the continuation worker decided it cannot continue the task", async () => { + const { issue, hold } = await seedProviderTerminalFailure(new Date(Date.now() - 2_000)); + await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence, + freshContinuationCheck: { kind: "not_applicable", reason: "task_changed", checks: 1, checkedAt: Date.now(), nextCheckAt: null } } }) + .where(eq(issueRecoveryActions.id, hold.id)); + expect(await taskInterruptionPresentation(db, { ...issue, status: "todo" })).toMatchObject({ state: "failed", canRetry: true }); + }); + + it("offers Retry for a provider failure at once when a person took over its hold", async () => { + const { issue, hold, companyId } = await seedProviderTerminalFailure(new Date(Date.now() - 2_000)); + await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence, ownerGoneAttestation: { companyId } } }) + .where(eq(issueRecoveryActions.id, hold.id)); + expect(await taskInterruptionPresentation(db, { ...issue, status: "todo" })).toMatchObject({ state: "failed", canRetry: true }); + }); + it("shows a healthy task as working while the message queued during its succeeded run is delivered", async () => { const { companyId, agentId, issue } = await seedTask(); const runId = randomUUID(), finishedAt = new Date(Date.now() - 3_000); diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index 2f3d1dc02bd..53020da5e06 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -52,6 +52,8 @@ type ProjectionRun = Pick`(select ${nativeRunFinalizations.updatedAt} from ${nativeRunFinalizations} where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id} and ${nativeRunFinalizations.phase} = 'terminal_failure')`.mapWith(value => value == null ? null : new Date(value)), + providerRecoveryAt: sql`(select ${nativeRunFinalizations.updatedAt} from ${nativeRunFinalizations} + where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id} + and ${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} = 'failed' + and ${nativeRunFinalizations.phase} = 'terminal_failure' and ${nativeRunFinalizations.failureCode} = 'native_provider_terminal_failed' + and ${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.leaseOwner} is null + and coalesce(${nativeRunFinalizations.failureDetail}->>'successorRunId', '') = '')`.mapWith(value => value == null ? null : new Date(value)), nativeFinalizationRetry: exhaustedNativeFinalizationCondition(), nativeFinalizationRetryPending: savedResultRetryPendingCondition(), // The run whose resultJson supplied providerFailure. A scheduled retry @@ -613,6 +626,13 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i inArray(issueRecoveryActions.cause, [...EXECUTION_RECONCILIATION_CAUSES]), isNull(issueRecoveryActions.ownerUserId), sql`not (${issueRecoveryActions.evidence} ?| array['systemRecovery', 'executionReconciliation', 'freshContinuation', 'freshContinuationCheck', 'ownerGoneAttestation', 'explicitUserContinuation'])`)) : []; + const recoveringRunIds = runs.flatMap(run => run.providerRecoveryAt ? [run.id] : []); + const handedToPerson = new Set((recoveringRunIds.length ? await db.select({ issueId: issueRecoveryActions.sourceIssueId }) + .from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids), + inArray(sql`${issueRecoveryActions.evidence}->>'runId'`, recoveringRunIds), + sql`(${issueRecoveryActions.evidence} ?| array['ownerGoneAttestation', 'explicitUserContinuation'] + or coalesce(${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,kind}', 'exact_recovery_pending') not in ('exact_recovery_pending', 'backoff'))`)) : []) + .map(row => row.issueId)); const automaticDispositionAt = new Map(); if (dispositionRows.length) { const { isPersonOwnedFailureCode } = await import("./fresh-continuation-fallback.js"); @@ -735,6 +755,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i const presentation = projectTaskInterruption(issue, { ...facts, freshContinuation: freshContinuationFor(issue.id, facts.run?.id), automaticDispositionAt: automaticDispositionAt.get(issue.id), + recoveryHandedToPerson: handedToPerson.has(issue.id), interruptWait: interruptWaitFor(issue.id), projectAccessAllowed: denied?.length === 0, projectAccessProjectId: denied?.includes(issue.projectId ?? "") ? issue.projectId : null,