From e23549cd9c631e4595a3213e03d0c960360452c6 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 05:24:53 -0600 Subject: [PATCH] fix(presentation): a task with a continuation waiting to start shows as continuing, not as needing Retry (#21353) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run dies (for example during a server swap), the fresh-continuation fallback records that Paperclip will continue the task in a fresh run from saved work > - The record says `continuationDelivery: "pending"` until a delivery pass starts the new run. Delivery is paced, so a record can wait a long time behind an older backlog > - The task view showed such a task as "working" only for 15 minutes after the record was made. After that it showed a failed attempt with a Retry button, although the record still said Paperclip would continue the task > - Product rule: a task that the system has recorded it will continue by itself never shows as failed with a Retry button > - This pull request ties the "continuing" state to the record's own delivery state instead of a timer > - The benefit is that the task view stays truthful however long delivery takes, and Retry comes back as soon as the system gives up ## Linked Issues or Issue Description Refs #21264, #21324, #21297, #21320, #21200 **What happened?** On hosted staging (build d1d71a7755, which contains #21264 and #21324), five tasks had a run that died after a server swap. Each task had a recorded fresh continuation that was not delivered yet. The primary recovery record had cause `native_continuation_requires_reconciliation`, status `resolved`, `continuationDelivery: "pending"`, isolation `remote_epoch`, and no `continuationDeliveryAttemptedAt`. The other holds of the task were coalesced into it. The records were 30 to 110 minutes old. The task view showed responsibility `system`, state `failed`, and `canRetry: true`. **Expected behavior** While a continuation is recorded and its delivery is pending, the task shows as continuing (`working`, `canRetry: false`, responsibility `system`). This lasts until delivery starts the new run (then the new run decides the state) or the system gives up (`invalidated`; then Retry is offered). **Steps to reproduce** 1. Record a fresh continuation for a failed run (`continuationDelivery: "pending"`) and coalesce the task's other holds into it. 2. Let more than 15 minutes pass without delivery. 3. Read the task. It shows `failed` with Retry. ## What Changed - `taskInterruptionPresentations` in `server/src/services/task-interruption-presentation.ts`: a primary fresh-continuation record with `continuationDelivery: "pending"` reads as "continuing" with no time limit. Before, this applied only for 15 minutes (`RECOVERY_CONTINUATION_DELIVERY_BOUND_MS`). The state is the existing `working` state that the code already uses for a recorded continuation; no new state is added. The UI shows the agent's turn as queued and shows no Retry button. - A refused delivery attempt (`continuationDeliveryAttemptedAt`) does not end the state. The record stays `pending` and the delivery pass tries again on the next pass, so the system still owns the next step. - The select no longer reads the hold time columns, because the projection does not use them now. - `recovery-wake-herd.test.ts`: the #21324 test that expected "not working" past the bound now expects `working` with `canRetry: false`, and then proves that the delivery pass still delivers that overdue continuation. - No change to the delivery pass, the fallback, or the Retry path. ## What a Retry click does in this state today `requestTaskRetry` admits a person's Retry as a user wake. The queue's "a pending fresh continuation absorbs this wake" check (`hasPendingFreshContinuation` in `heartbeat/queue.ts`) applies only to system wakes, so the Retry starts its own run from saved work. The delivery pass then sees a run created after the record and marks the continuation `delivered` (superseded). In the normal order this does not start a second run. But the click only asks for what the system already recorded it would do, and it races the delivery pass: if the pass has already checked for a newer run and is waking the agent when the click is admitted, the task can get two continuation runs, one after the other. That is why the button must not show while the continuation is pending. ## Verification - Red (before the fix), `pnpm exec vitest run src/__tests__/task-interruption-automatic-path.test.ts` in `server/`: 2 failed. The exact staging record shape at 30 minutes read `state: "failed"`, `canRetry: true`, and a pending record after a refused attempt read the same. - Green (after the fix), in `server/`: `pnpm exec vitest run src/__tests__/task-interruption-automatic-path.test.ts src/__tests__/recovery-wake-herd.test.ts src/__tests__/task-interruption-query.test.ts src/services/task-interruption-presentation.test.ts src/services/fresh-continuation-fallback.test.ts`: 5 files, 202 tests pass. - New tests in `task-interruption-automatic-path.test.ts`: - The staging record shape at 1, 30 and 110 minutes shows `working`, `canRetry: false`, responsibility `system`. - A refused delivery attempt keeps `working`. - Delivery given up (`invalidated`, with the coalesced hold) offers Retry. - Delivered: the task follows the new run (queued: `working` on the new run; the new run fails: Retry on the new run). - A resolved recovery record with no recorded continuation offers Retry as before. - A board-owned hold with no recorded continuation offers Retry as before. - The #21200, #21264 and #21320 guard tests in the same file pass unchanged. - `node scripts/typecheck-layers.mjs` in `server/` passes. ## Risks - #21324 limited "working" to 15 minutes so that a continuation that never gets delivered does not hide behind "working". With this change, a continuation that stays pending forever shows "working" forever. #21324 also made delivery bounded (an overdue record is read on every pass and gets one wake per agent per pass), and a separate change is making delivery prompt. A record that cannot be delivered (task done, cancelled, or reassigned) is marked `invalidated`, and then Retry shows. - Person-owned waits still win: approval, reconnect, budget, pause and project access are checked before this state. - Presentation only. No schema, migration, or execution change. - This change supersedes the expectation of the #21324 test "does not show a task as working when its continuation is pending past the delivery bound". That test hid the continuing state after 15 minutes. The state must stay truthful: the system has recorded that it will continue the task, so the task must not show a failed attempt or a Retry that can start a second run. - Follow-up: while a continuation waits, a person has no control. On staging a continuation can wait a long time behind a deep queue. After #21348 is on master (a person's Retry folds into the pending continuation in the admitting transaction, and runs that a person starts rank ahead of system recovery), the planned follow-up keeps the continuing state and, after 15 minutes, also offers a Retry worded as "move it forward now" that folds into the pending continuation. This pull request does not add that Retry. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../src/__tests__/recovery-wake-herd.test.ts | 10 ++- .../task-interruption-automatic-path.test.ts | 76 +++++++++++++++++++ .../task-interruption-presentation.ts | 6 +- 3 files changed, 84 insertions(+), 8 deletions(-) diff --git a/server/src/__tests__/recovery-wake-herd.test.ts b/server/src/__tests__/recovery-wake-herd.test.ts index 5edf8550679..5025b543e7f 100644 --- a/server/src/__tests__/recovery-wake-herd.test.ts +++ b/server/src/__tests__/recovery-wake-herd.test.ts @@ -535,13 +535,17 @@ suite("recovery wake herd", () => { expect(await deliveryOf(healthy.action.id)).toBe("delivered"); }, 120_000); - it("does not show a task as working when its continuation is pending past the delivery bound and nothing runs", async () => { + it("shows a task as continuing, never as needing Retry, while its recorded continuation is pending past the delivery bound", async () => { const agentId = await seedCompany(1); const record = await pendingContinuation(agentId, 240); await holdPastBound(record, RECOVERY_CONTINUATION_DELIVERY_BOUND_MS); const [issueRow] = await db.select().from(issues).where(eq(issues.id, record.issue.id)); - const presentation = await taskInterruptionPresentation(db, issueRow!); - expect(presentation?.state).not.toBe("working"); + expect(await taskInterruptionPresentation(db, issueRow!)).toMatchObject({ state: "working", canRetry: false, responsibility: "system" }); + + const wake = fakeWake(); + await deliverReconciledExecutions(db, wake as never); + expect(await deliveryOf(record.action.id)).toBe("delivered"); + expect(wake).toHaveBeenCalledTimes(1); const fresh = await pendingContinuation(agentId, 30); const [freshRow] = await db.select().from(issues).where(eq(issues.id, fresh.issue.id)); diff --git a/server/src/__tests__/task-interruption-automatic-path.test.ts b/server/src/__tests__/task-interruption-automatic-path.test.ts index 74f0da150a3..3c9c179760d 100644 --- a/server/src/__tests__/task-interruption-automatic-path.test.ts +++ b/server/src/__tests__/task-interruption-automatic-path.test.ts @@ -82,6 +82,82 @@ describe.skipIf(!support.supported)("Retry is offered only when no automatic pat expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true }); }); + async function seedRecordedContinuation(minutesAgo: number) { + const recordedAt = new Date(Date.now() - minutesAgo * 60_000); + const task = await seedSwapEndedRun(new Date(recordedAt.getTime() - 30_000)); + const freshContinuation = { schema: "paperclip.fresh-continuation.v1", sourceRunId: task.runId, trigger: "held_recovery", + failureCode: "native_restart_recovery_blocked", delivery: "recorded", recordedAt: recordedAt.toISOString(), + isolation: "remote_epoch", stopProof: "unproven_epoch_abandoned", unverifiedActions: [], createdObjects: [] }; + const settlement = { schema: "paperclip.fresh-continuation.v1", actorType: "system", actorId: "fresh-continuation", + decision: "continue_in_fresh_run", replay: "none", sourceRunId: task.runId, reason: "exact_resumption_unavailable", + isolation: "remote_epoch", recordedAt: recordedAt.toISOString() }; + const reconciliation = { runId: task.runId, providerStopped: true, providerStopProof: "isolated", actionOutcome: "mixed", + actorId: "fresh-continuation", recordedAt: recordedAt.toISOString() }; + const [primary] = await db.insert(issueRecoveryActions).values({ companyId: task.companyId, sourceIssueId: task.issue.id, + kind: "active_run_watchdog", status: "resolved", outcome: "restored", ownerType: "system", returnOwnerAgentId: task.agentId, + cause: "native_continuation_requires_reconciliation", fingerprint: `native-continuation:${task.runId}`, + nextAction: "Paperclip continues the task in a fresh run from its saved work.", resolutionNote: "continued_fresh_from_last_saved", + resolvedAt: recordedAt, createdAt: recordedAt, updatedAt: recordedAt, + evidence: { runId: task.runId, executionReconciliation: reconciliation, freshContinuationSettlement: settlement, + freshContinuation, continuationDelivery: "pending" } }).returning(); + await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: recordedAt, updatedAt: recordedAt, + evidence: { ...task.hold.evidence, executionReconciliation: reconciliation, freshContinuationSettlement: settlement, + continuationDelivery: "coalesced", coalescedIntoRecoveryActionId: primary!.id } }) + .where(eq(issueRecoveryActions.id, task.hold.id)); + return { ...task, primary: primary! }; + } + + it("shows a task whose recorded continuation is still waiting to start as continuing, however long delivery takes", async () => { + for (const minutesAgo of [1, 30, 110]) { + const { issue, runId } = await seedRecordedContinuation(minutesAgo); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "working", + canRetry: false, observedRunId: runId }); + } + }); + + it("keeps a recorded continuation as continuing after a refused delivery attempt, since delivery tries again", async () => { + const { issue, primary } = await seedRecordedContinuation(45); + await db.update(issueRecoveryActions).set({ evidence: { ...primary.evidence, + continuationDeliveryAttemptedAt: new Date(Date.now() - 60_000).toISOString() } }).where(eq(issueRecoveryActions.id, primary.id)); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: false }); + }); + + it("offers Retry once the system gives up on a recorded continuation", async () => { + const { issue, primary, hold } = await seedRecordedContinuation(45); + await db.update(issueRecoveryActions).set({ evidence: { ...primary.evidence, continuationDelivery: "invalidated" } }) + .where(eq(issueRecoveryActions.id, primary.id)); + await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence, continuationDelivery: "invalidated", + coalescedIntoRecoveryActionId: primary.id } }).where(eq(issueRecoveryActions.id, hold.id)); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "failed", canRetry: true }); + }); + + it("follows the new run once a recorded continuation is delivered", async () => { + const { companyId, agentId, issue, primary } = await seedRecordedContinuation(45); + const nextRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue", + status: "queued", runtimeMode: "native", retryOfRunId: primary.evidence.runId as string }); + await db.update(issueRecoveryActions).set({ evidence: { ...primary.evidence, continuationDelivery: "delivered", + continuationRunId: nextRunId } }).where(eq(issueRecoveryActions.id, primary.id)); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: false, observedRunId: nextRunId }); + await db.update(heartbeatRuns).set({ status: "failed", errorCode: "adapter_failed", finishedAt: new Date(Date.now() - 10 * 60_000) }) + .where(eq(heartbeatRuns.id, nextRunId)); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true, observedRunId: nextRunId }); + }); + + it("still offers Retry for a resolved recovery record that recorded no continuation", async () => { + const { issue, hold } = await seedSwapEndedRun(new Date(Date.now() - 45 * 60_000)); + await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: new Date(Date.now() - 40 * 60_000), + evidence: { ...hold.evidence, freshContinuationSettlement: { schema: "paperclip.fresh-continuation.v1", decision: "continue_in_fresh_run" } } }) + .where(eq(issueRecoveryActions.id, hold.id)); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "failed", canRetry: true }); + }); + + it("still offers Retry when the board owns the hold and no continuation is recorded", async () => { + const { issue, hold } = await seedSwapEndedRun(new Date(Date.now() - 45 * 60_000)); + await db.update(issueRecoveryActions).set({ ownerType: "board" }).where(eq(issueRecoveryActions.id, hold.id)); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true }); + }); + it("shows a healthy task as working while the message queued during its succeeded run is delivered", async () => { const { companyId, agentId, issue } = await seedTask(); const runId = randomUUID(), finishedAt = new Date(Date.now() - 3_000); diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index 17e8cd7664a..2004d4387dc 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -11,7 +11,6 @@ import { EXECUTION_RECONCILIATION_CAUSES, HEARTBEAT_RUN_STATUSES, readProviderFa import { captureTaskInterruptionFailure } from "../sentry.js"; import { isAiAuthenticationBlocked } from "./ai-auth-failure.js"; import { readSystemRecovery } from "./system-recovery-state.js"; -import { RECOVERY_CONTINUATION_DELIVERY_BOUND_MS, recoveryContinuationHeldSince } from "./recovery-continuation-pacing.js"; import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js"; import { authorizationService } from "./authorization.js"; import { taskWorkspaceProjectRequirements } from "./task-workspace-access.js"; @@ -592,8 +591,6 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i runId: sql`${issueRecoveryActions.evidence}->>'runId'`, delivery: sql`${issueRecoveryActions.evidence}->>'continuationDelivery'`, recorded: sql`${issueRecoveryActions.evidence} ? 'freshContinuation'`, - evidence: sql>`jsonb_build_object('freshContinuation', ${issueRecoveryActions.evidence}->'freshContinuation')`, - resolvedAt: issueRecoveryActions.resolvedAt, updatedAt: issueRecoveryActions.updatedAt, checkKind: sql`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,kind}'`, nextCheckAt: sql`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}'` }) .from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids), @@ -603,8 +600,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i const rows = freshRows.filter(row => row.issueId === issueId && row.runId === runId); // Only the primary record (it carries `freshContinuation`) says whether a // continuation is still on its way; holds coalesced into it never do. - if (rows.some(row => row.recorded && (row.delivery === "not_requested" || (row.delivery === "pending" - && Date.now() - recoveryContinuationHeldSince(row) < RECOVERY_CONTINUATION_DELIVERY_BOUND_MS)))) return { state: "continuing" }; + if (rows.some(row => row.recorded && ["not_requested", "pending"].includes(row.delivery ?? ""))) return { state: "continuing" }; const scheduled = rows.filter(row => ["exact_recovery_pending", "backoff"].includes(row.checkKind ?? "") && row.nextCheckAt) .map(row => Number(row.nextCheckAt)).sort((a, b) => a - b)[0]; return scheduled ? { state: "scheduled", nextCheckAt: scheduled } : undefined;