From 5561a4068cd0223dfe7eb521f4e7d8b5660c5d15 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 16 Nov 2026 09:43:36 -0600 Subject: [PATCH] fix(recovery): keep a running run visible when a dropped continuation is newer (#21907) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task shows a Retry banner when its latest run failed and nothing else continues the task > - The banner reads "the latest run" by creation time only > - The queue cancels a queued system continuation before it starts when the task no longer needs it. That run is newer than the run that is really working > - So a task showed "failed" with Retry for about 79 seconds while its run was working > - The stranded-work sweep also queued one of those extra continuations, because it did not see a native status wake that was still waiting for dispatch > - This pull request ignores never-started, superseded continuations in the banner, and makes the sweep see a pending native status wake > - The benefit is no false Retry offer while an agent works, and no duplicate continuation after a slow finalization ## Linked Issues or Issue Description **What happened?** In a restart-chaos wave (6 rolling deploys, 207 tasks), one task showed state `failed` with Retry for 75–79 seconds while its run was running. Two queued system continuations for that task were cancelled before they started with `recovery_no_longer_needed`. That cancel is correct. But the newest cancelled run became the "latest run" in the interruption projection. The projection then fell through to `{ state: "failed", canRetry: true }`. The first of the two extra continuations came from a double queue. The finishing run asked for its own continuation through a native status wake (`agent_tool_call_loop`). Its finalization did not apply at once ("native result persisted but finalization did not apply; the reconciliation loop will retry"), so the dispatch of that wake waited about 17 seconds. In that window, the stranded-work sweep saw an `in_progress` task with a succeeded run and no live path, and queued its own continuation (`issue.productive_terminal_continuation_recovery`). The sweep's live-path check counted only runs and `deferred_issue_execution` wakes. It did not count a native status wake that was still `queued` or `claimed` without a run. When the native status wake was dispatched later, the queue correctly dropped it. The dropped wake also carried the better instruction (the tool-loop guidance). **Expected behavior** A task with a running run shows "working" with no Retry. One finished run produces one continuation. **Steps to reproduce** 1. Give a task a running run R1. 2. Add a newer run R2 for the task with status `cancelled`, no `startedAt`, and error code `recovery_no_longer_needed`. 3. Read the task interruption. Before this change: `failed`, `canRetry: true`, observed run R2. For the sweep: give an `in_progress` task a succeeded native run and a `native-status-committer` wake in `queued` or `claimed` with no run. Run `reconcileStrandedAssignedIssues`. Before this change it queues a second continuation. **Paperclip version or commit** master `c5218f5ec4` (wave), reproduced on `b4fa1c7fb1`. ## What Changed - `task-interruption-presentation.ts`: the latest-run candidates skip a run that is `cancelled`, never started (`startedAt` is null), and has error code `recovery_no_longer_needed` or `issue_closed_before_start`. The filter is in each per-arm `limit 1` index scan, so the query shape and cost stay the same. - `recovery/service.ts`: `hasActiveExecutionPath` also counts a `native-status-committer` wake in `queued` or `claimed` with no run for the issue. The native status wake dispatcher owns that wake, so the sweep does not queue a second continuation and does not escalate the task while it waits. - Tests (red on master, green here): - `task-interruption-query.test.ts`: a running R1 plus a newer never-started R2 cancelled with each code shows `working`, `canRetry: false`, observed R1. Negatives keep their behavior: a cancelled run that had started, and a run a person cancelled, still show `failed` with Retry. - `heartbeat-process-recovery.test.ts`: a succeeded native run with its own status wake still `queued` or `claimed` gets no second continuation and no escalation. ## Verification - `pnpm verify` exited 2 (opaque setup dependency), so I ran focused suites: - `server/src/__tests__/task-interruption-query.test.ts`, `task-interruption-automatic-path.test.ts`, `issue-confirm-run-stopped-routes.test.ts`: 83/83 pass. This includes the query-plan test that bounds `heartbeat_runs` rows read. - `server/src/__tests__/heartbeat-process-recovery.test.ts`, `server/src/services/recovery/*.test.ts`, the interruption suites and `task-interruption-presentation.test.ts` together: 18 files, 871/871 pass. - Red/green: the new interruption tests fail on master with `state: "failed", canRetry: true` and the dropped run as observed run. The new sweep tests fail on master with `continuationRequeued: 1`. - `tsc --noEmit` for `server` passes. ## Risks - Low. The banner change only hides runs that never started and were dropped as superseded. Person-cancelled runs, started runs, and other cancel codes behave as before. - The live-path change makes the sweep wait while a native status wake is pending. The dispatcher always settles such a wake: it dispatches it, or marks it skipped. A throttled wake stays queued only until its throttle time, so the sweep waits at most that long. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../heartbeat-process-recovery.test.ts | 34 +++++++++++++++++ .../__tests__/task-interruption-query.test.ts | 37 +++++++++++++++++++ server/src/services/recovery/service.ts | 11 +++++- .../task-interruption-presentation.ts | 4 +- 4 files changed, 84 insertions(+), 2 deletions(-) diff --git a/server/src/__tests__/heartbeat-process-recovery.test.ts b/server/src/__tests__/heartbeat-process-recovery.test.ts index 73cc77c5985..0a662626ef2 100644 --- a/server/src/__tests__/heartbeat-process-recovery.test.ts +++ b/server/src/__tests__/heartbeat-process-recovery.test.ts @@ -17312,6 +17312,40 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { } }); + it.each(["queued", "claimed"] as const)( + "does not queue a second continuation while the finished run's own status wake (%s) awaits dispatch (Gate C C1 t169)", + async (intentStatus) => { + const { companyId, agentId, issueId, runId } = + await seedStrandedIssueFixture({ + status: "in_progress", + runStatus: "succeeded", + livenessState: "advanced", + }); + await db.insert(agentWakeupRequests).values({ + companyId, + agentId, + source: "automation", + triggerDetail: "system", + reason: "issue_continuation_needed", + status: intentStatus, + claimedAt: intentStatus === "claimed" ? new Date() : null, + requestedByActorType: "system", + requestedByActorId: "native-status-committer", + idempotencyKey: `agent_tool_call_loop:${runId}`, + payload: { issueId, taskId: issueId, sourceRunId: runId, continuationCause: "agent_tool_call_loop" }, + }); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId }).where(eq(heartbeatRuns.id, runId)); + const heartbeat = heartbeatService(db); + + const result = await heartbeat.reconcileStrandedAssignedIssues(); + expect(result.continuationRequeued).toBe(0); + expect(result.escalated).toBe(0); + expect(result.skipped).toBe(1); + expect(await selectRecoveryRunsByAgent(agentId)).toHaveLength(1); + expect((await selectRecoveryIssueById(issueId))?.status).toBe("in_progress"); + }, + ); + it("does not replenish exhausted legacy repair from a recent comment", async () => { const { companyId, agentId, issueId, runId } = await seedStrandedIssueFixture({ diff --git a/server/src/__tests__/task-interruption-query.test.ts b/server/src/__tests__/task-interruption-query.test.ts index 0b81445c7a2..cf28f663f43 100644 --- a/server/src/__tests__/task-interruption-query.test.ts +++ b/server/src/__tests__/task-interruption-query.test.ts @@ -747,6 +747,43 @@ describe.skipIf(!support.supported)("task interruption latest run query", () => taskId: reassigned!.id, agentAdapter: "process", })); }); + describe("a newer continuation dropped before it started (Gate C C1 t169)", () => { + const seed = async (newer: Partial) => { + const companyId = randomUUID(), agentId = randomUUID(), runningId = randomUUID(), droppedId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Continuing task", issuePrefix: `C${companyId.slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Continuing agent", adapterType: "paperclip_runner" }); + const [issue] = await db.insert(issues).values({ companyId, title: "Keep working", + status: "in_progress", assigneeAgentId: agentId }).returning(); + const now = Date.now(); + await db.insert(heartbeatRuns).values({ id: runningId, companyId, agentId, nativeIssueId: issue!.id, scopeKind: "issue", + status: "running", runtimeMode: "native", createdAt: new Date(now - 60_000), startedAt: new Date(now - 59_000) }); + await db.insert(heartbeatRuns).values({ id: droppedId, companyId, agentId, nativeIssueId: issue!.id, scopeKind: "issue", + status: "cancelled", runtimeMode: "native", createdAt: new Date(now - 30_000), finishedAt: new Date(now - 30_000), + contextSnapshot: { issueId: issue!.id, wakeReason: "issue_continuation_needed" }, ...newer }); + return { companyId, issue: issue!, runningId, droppedId }; + }; + + it.each(["recovery_no_longer_needed", "issue_closed_before_start"])( + "shows the running run, not a failed attempt (%s)", async errorCode => { + const f = await seed({ errorCode, error: "The task no longer needed this recovery run when its turn came." }); + for (const presentation of [(await taskInterruptionPresentations(db, f.companyId, [f.issue])).get(f.issue.id), + await taskInterruptionPresentation(db, f.issue)]) { + expect(presentation).toMatchObject({ state: "working", canRetry: false, observedRunId: f.runningId }); + } + }); + + it("still reports a cancelled run that had started", async () => { + const f = await seed({ errorCode: "recovery_no_longer_needed", startedAt: new Date(Date.now() - 29_000) }); + expect((await taskInterruptionPresentations(db, f.companyId, [f.issue])).get(f.issue.id)) + .toMatchObject({ state: "failed", canRetry: true, observedRunId: f.droppedId }); + }); + + it("still reports a run a person cancelled before it started", async () => { + const f = await seed({ errorCode: "cancelled", resultJson: { cancelledByActorType: "user", cancelledByUserId: "board" } }); + expect((await taskInterruptionPresentations(db, f.companyId, [f.issue])).get(f.issue.id)) + .toMatchObject({ state: "failed", canRetry: true, observedRunId: f.droppedId }); + }); + }); // N-03: ONB-1 on staging waited 16 minutes on an overdue workspace_busy // retry. Nothing failed, yet the banner offered a Retry that cannot help. it.each(["workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", "runner_revision_wait", "storage_unavailable_wait"])( diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 0cb4576f41e..75efec067d3 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -42,6 +42,7 @@ import { import { CONTROLLER_LEASE_LOST_FAILURE_CODE, hasLiveLegacyController, legacyControllerBootId, liveLegacyControllerExpiresAt } from "../legacy-controller-lease.js"; import { isCompletedOnboardingHandoffWake } from "../chat-completion-delivery.js"; import { instanceSettingsService } from "../instance-settings.js"; +import { NATIVE_STATUS_COMMITTER_ACTOR_ID } from "../issue-rewake-throttle.js"; import { isWaitingConversation, settleConversationTurnForRun, deliverConversationComments } from "../agent-conversations.js"; import { and, @@ -1259,7 +1260,15 @@ export function recoveryService( .where( and( eq(agentWakeupRequests.companyId, companyId), - eq(agentWakeupRequests.status, "deferred_issue_execution"), + or( + eq(agentWakeupRequests.status, "deferred_issue_execution"), + and( + eq(agentWakeupRequests.requestedByActorType, "system"), + eq(agentWakeupRequests.requestedByActorId, NATIVE_STATUS_COMMITTER_ACTOR_ID), + inArray(agentWakeupRequests.status, ["queued", "claimed"]), + isNull(agentWakeupRequests.runId), + ), + ), sql`${agentWakeupRequests.payload} ->> 'issueId' = ${issueId}`, agentId ? eq(agentWakeupRequests.agentId, agentId) : sql`true`, ), diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index ca13eebd909..87b18917f83 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -440,8 +440,10 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i const requestedUuid = sql`case when ${requestedIssueId} ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' then ${requestedIssueId}::uuid else null end`; const legacyIssue = sql`${heartbeatRuns.contextSnapshot}->>'issueId'`; + const attempt = sql`not (${heartbeatRuns.status} = 'cancelled' and ${heartbeatRuns.startedAt} is null + and coalesce(${heartbeatRuns.errorCode}, '') in ('recovery_no_longer_needed', 'issue_closed_before_start'))`; const candidate = (condition: ReturnType) => db.select({ id: heartbeatRuns.id, createdAt: heartbeatRuns.createdAt }) - .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), condition)) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), condition, attempt)) .orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1); const candidates = unionAll( candidate(eq(heartbeatRuns.nativeIssueId, requestedUuid)),