From ed16a5b1a89b00f0bb43a7b5fb2d1d60ee8f13e1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 07:28:44 -0600 Subject: [PATCH] fix(recovery): do not continue from a latest run read while it was still finishing (#21365) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent's run ends without a live next step, a background sweep finds the stranded task and queues a continuation > - The sweep reads the task's latest run first, and checks for a live execution path several queries later > - A run that finishes between those two reads looks neither live nor failed. The sweep fell through to its generic continuation (`issue.continuation_recovery`) with the stale "running" snapshot > - In a chat, that woke the agent about a second after its turn had answered the person, and the agent posted a redundant "nothing new is pending" reply > - This pull request makes the sweep act only on a latest run it observed as finished. A run it read while still active is left to its own completion and to the next sweep > - The benefit is that a chat turn that already answered is never woken again by recovery, with no change to real recovery ## Linked Issues or Issue Description **What happened?** In a chat, the person asked a follow-up and the agent answered. The answering run's success was written at 10:09:28.840. A continuation run was queued at 10:09:28.841 with `wakeReason: issue_continuation_needed`, `source: issue.continuation_recovery`, and `retryOfRunId` set to the run that had just succeeded. The agent ran again and posted an extra reply. The generic continuation source is only reachable when the sweep's latest-run snapshot is neither succeeded nor live, so the snapshot was read while the run was still `running`, and the run had finished by the time of the live-path check. **Steps to reproduce** 1. Have an assigned `in_progress` task whose latest run is `running`. 2. Start the stranded-work sweep. It reads the latest run as `running`. 3. Before the sweep's live-path check, let the run finish with `succeeded` and release the task lock. 4. The sweep finds no live path and queues `issue.continuation_recovery` for the finished run. **Expected behavior** The sweep queues nothing for a run it read while still active. Its own completion settles it, and the next sweep decides from the committed outcome. ## What Changed - `server/src/services/recovery/service.ts`: after the live-path check, the stranded sweep skips a non-review task whose latest-run snapshot was not terminal. Review tasks keep their participant-run check, which already does this. - `server/src/services/recovery/service.ts`: an optional test hook, `afterStrandedLatestRunRead`, runs after the sweep reads the latest run, alongside the existing `beforeStaleLockClear` and `beforeOrphanedRunTerminalWrite` hooks. - `server/src/__tests__/heartbeat-process-recovery.test.ts`: a regression test finishes the run between the two reads and expects no continuation. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "still finishing"` fails without the fix (one `issue.continuation_recovery` wake with `retryOfRunId` set to the finished run) and passes with it. - `cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts src/__tests__/issue-recovery-actions.test.ts src/__tests__/heartbeat-issue-liveness-escalation.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/durable-chat-wakeup.test.ts src/__tests__/live-recovery-work.test.ts`: 790 passed. One unrelated test timed out under parallel load and passes on its own (its latest run is already terminal, so the new check does not apply). - `cd server && npx tsc --noEmit` passes. ## Risks - Behavior: for any task whose latest run is `queued`, `running` or `scheduled_retry`, the live-path check already skipped it, so the only changed outcome is the race above. A run stuck in `running` with no owner is still settled by the existing orphaned-run reaper; the sweep then sees its terminal status. - Latency: the check is in memory on an already-read row. It adds no query and nothing before a turn starts; recovery after a real stall is at most one sweep later only inside the race window. - Saved state: no persisted shape changes, so old saved runs and mid-run deploys behave as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../heartbeat-process-recovery.test.ts | 24 +++++++++++++++++++ server/src/services/recovery/service.ts | 6 +++++ 2 files changed, 30 insertions(+) diff --git a/server/src/__tests__/heartbeat-process-recovery.test.ts b/server/src/__tests__/heartbeat-process-recovery.test.ts index 19eec94b7dc..95ed9b49c06 100644 --- a/server/src/__tests__/heartbeat-process-recovery.test.ts +++ b/server/src/__tests__/heartbeat-process-recovery.test.ts @@ -15455,6 +15455,30 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { expect(mockAdapterExecute).not.toHaveBeenCalled(); }); + it("does not queue a continuation from a latest run it read while that run was still finishing", async () => { + const { companyId, issueId, runId } = await seedStrandedIssueFixture({ + status: "in_progress", + runStatus: "succeeded", + livenessState: "advanced", + }); + await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, runId)); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const enqueueWakeup = vi.fn(async () => null); + const recovery = recoveryService(db, { + enqueueWakeup, + afterStrandedLatestRunRead: async (read) => { + if (read.runId !== runId) return; + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, runId)); + await db.update(issues).set({ executionRunId: null, checkoutRunId: null }).where(eq(issues.id, issueId)); + }, + }); + + const result = await recovery.reconcileStrandedAssignedIssues({ companyId, issueId }); + + expect(result.continuationRequeued).toBe(0); + expect(enqueueWakeup).not.toHaveBeenCalled(); + }); + it("leaves a chat turn that yielded to wait for a person without starting another run", async () => { const fixture = await seedNativePassiveBoardResponse("response_wake", { blockingWork: true, objectiveUnsatisfied: true }); const previousSettings = await instanceSettingsService(db).getExperimental(); diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index e9c5e148fd1..43c5e890322 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -998,6 +998,7 @@ export function recoveryService( settleLostRunProcess?: (scope: { companyId: string; runId: string; onDeferredDeadline?: (at: number) => void }) => Promise; beforeOrphanedRunTerminalWrite?: (runId: string) => Promise; beforeStaleLockClear?: (issueId: string) => Promise; + afterStrandedLatestRunRead?: (input: { issueId: string; runId: string | null }) => Promise; }, ) { const issuesSvc = issueService(db); @@ -4835,6 +4836,7 @@ export function recoveryService( } let latestRun = await getLatestIssueRun(issue.companyId, issue.id); + await deps.afterStrandedLatestRunRead?.({ issueId: issue.id, runId: latestRun?.id ?? null }); // The run's own executor still settles it: it schedules the bounded retry // and releases the claim itself. Escalating in that window clears the // claim the retry needs. Re-check once its controller lease lapses. @@ -4977,6 +4979,10 @@ export function recoveryService( result.skipped += 1; continue; } + if (issue.status !== "in_review" && latestRun && !isTerminalIssueRun(latestRun)) { + result.skipped += 1; + continue; + } if (await hasPendingWakeInteraction(issue.companyId, issue.id)) { result.skipped += 1;