From d702f567bcf3721f349cc249be613747bb55574c Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Wed, 18 Nov 2026 15:39:14 -0800 Subject: [PATCH] fix(recovery): keep a task's lock while its run's executor still settles it (#22248) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent's provider account recently reported a usage limit, Paperclip holds new work instead of starting a run that would fail the same way. The provider limit wait cancels the run, then schedules a retry at the expected reset, and the retry takes over the task's execution lock > - The cancellation and the retry are two separate writes. Cancelling the run wakes task recovery for that task, and its stale-lock sweep treats a cancelled run's lock as stale and clears it > - When the sweep wins that race, the retry scheduler finds the lock gone and refuses to schedule ("Subscription retry suppressed because the task execution lock changed"). The wait then releases the task, and stranded-work recovery escalates it to a person with a Retry button > - The stranded-work reconciler already waits while a terminal run's own executor is still settling it (`terminalRunSettlingUntil`). The stale-lock sweep did not > - This pull request makes the stale-lock sweep wait for the same executor before it clears an open task's lock, and look again at the executor's deadline > - The benefit is that a task waiting for a provider usage limit resumes by itself at the reset, instead of stopping on "Couldn't finish this attempt" until someone presses Retry ## Linked Issues or Issue Description Refs #19246, which added the provider usage limit wait. **What happened?** An agent's provider account hit a usage limit. A task's next wake was held for the reset: its run was cancelled with `provider_limit_wait`. About 200 ms later the stale-lock sweep cleared the task's execution lock, because the run was now cancelled. The wait then tried to schedule its retry, found the lock gone, and released the task. The continuation that followed hit the same race. After two lost retries, stranded-work recovery blocked the task and asked for Retry. On one instance, 80 of about 600 provider-limit waits in two days lost their retry this way, across 43 tasks. 15 of those tasks were blocked behind a Retry button. **Steps to reproduce** 1. Give an agent a recent provider rate-limit failure with no reported reset. 2. Wake the agent on an assigned task. 3. Let task recovery sweep the task's locks between the cancellation and the retry (the new integration test does this from the cancellation write). **Expected behavior** The task keeps its lock for a retry scheduled at the reset. **Actual behavior** The sweep clears the lock, no retry is scheduled, and the task waits for a manual Retry. ## What Changed - `recovery/service.ts`: `sweepStaleIssueLocks` keeps an open task's lock while a run that holds it is still being settled by its executor, using the existing `terminalRunSettlingUntil`. That is a live execution in this process, or an unexpired controller lease in another one. The sweep reports the deadline through `onDeferredDeadline`, so it looks again once the executor is done. A closed task's lock is still cleared at once. - Tests: - The stale-lock sweep keeps an open task's lock while the executor is live, and clears it once the executor is gone. A done task's lock is cleared either way. - A provider-limit wait whose cancellation is swept straight away still schedules its retry, and the retry holds the task's lock. ## Verification - `npx vitest run` on the new cases: pass. Both fail without the source change; the integration test then reproduces the incident (the sweep clears the lock and no retry is scheduled). - `npx vitest run` on 58 related server test files (stale-lock sweep, stranded and assigned-work recovery, provider limit wait, native resume locks, retry admission, recovery hints): all pass, except 6 cases in `runtime-selection.test.ts` that read a skill file from a fixed local path. They fail the same way on master on this machine. - `npx tsc --noEmit` passes for the server. ## Risks - Low. The sweep waits only while the run's executor is provably still running: a live execution in this process, or another process's unexpired controller lease. A crashed executor has neither, so the sweep clears the lock as before. - An executor that keeps running after its run is terminal now delays the sweep's backstop clear of an open task until it finishes. The executor releases or hands off the lock itself when it finishes, and the stranded-work reconciler already waits for the same executor. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .../heartbeat-provider-limit-wait.test.ts | 36 +++++++++++++++++++ .../recovery-stale-issue-lock-sweep.test.ts | 17 +++++++++ server/src/services/recovery/service.ts | 11 ++++++ 3 files changed, 64 insertions(+) diff --git a/server/src/__tests__/heartbeat-provider-limit-wait.test.ts b/server/src/__tests__/heartbeat-provider-limit-wait.test.ts index 66067a46441..551bee138ef 100644 --- a/server/src/__tests__/heartbeat-provider-limit-wait.test.ts +++ b/server/src/__tests__/heartbeat-provider-limit-wait.test.ts @@ -11,6 +11,23 @@ import { createProviderLimitWait } from "../services/heartbeat/provider-limit-wa import { executionProjectionsForRuns } from "../services/execution-projection.js"; import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js"; +const providerLimitCancellation = vi.hoisted(() => ({ after: null as null | (() => Promise) })); +vi.mock("../services/heartbeat/provider-limit-wait.js", async importOriginal => { + const actual = await importOriginal(); + return { + ...actual, + createProviderLimitWait: (...[database, deps]: Parameters) => + actual.createProviderLimitWait(database, { + ...deps, + setRunStatusIfRunning: async (...args) => { + const result = await deps.setRunStatusIfRunning(...args); + await providerLimitCancellation.after?.(); + return result; + }, + }), + }; +}); + const describePostgres = recoveryPostgresSupport.supported ? describe : describe.skip; if (!recoveryPostgresSupport.supported) console.warn(`Skipping provider limit wait tests: ${recoveryPostgresSupport.reason}`); @@ -132,6 +149,25 @@ describePostgres("heartbeat provider usage limit wait", () => { expect(resumed.status).toBe("succeeded"); }, 60_000); + it("keeps the task lock for its retry when recovery sweeps the task right after the cancellation", async () => { + const f = await seed([rateLimited(16)]); + const sweeps: Array>> = []; + providerLimitCancellation.after = async () => { + sweeps.push(await heartbeat.sweepStaleIssueLocks({ companyId: f.companyId, issueId: f.issueId })); + }; + try { + const finished = await wake(f); + expect(finished).toMatchObject({ status: "cancelled", errorCode: "provider_limit_wait" }); + expect(sweeps).toEqual([expect.objectContaining({ cleared: 0 })]); + const [retry] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, finished.id)); + expect(retry).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "ai_connection_pool_wait" }); + const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + expect(issue.executionRunId).toBe(retry.id); + } finally { + providerLimitCancellation.after = null; + } + }, 60_000); + it("releases the task when its retry cannot be scheduled, even if diagnostics fail", async () => { const f = await seed([rateLimited(16)]); const [agent] = await db.select().from(agents).where(eq(agents.id, f.agentId)); diff --git a/server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts b/server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts index 535c1c9de6a..ec74dd0e310 100644 --- a/server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts +++ b/server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts @@ -229,6 +229,23 @@ describeEmbeddedPostgres("recovery sweepStaleIssueLocks", () => { ); }); + it.each(["in_progress", "done"] as const)("waits for a live executor to settle its run before clearing the lock of a %s task", async status => { + const { companyId, agentId, failedRunId } = await seed(); + const issueId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Executor still settling", status, + priority: "high", assigneeAgentId: agentId, executionRunId: failedRunId, executionLockedAt: new Date() }); + const deadlines: number[] = []; + const whileLive = await recoveryService(db, { enqueueWakeup: vi.fn(), liveRunExecutions: new Set([failedRunId]) }) + .sweepStaleIssueLocks({ companyId, issueId, onDeferredDeadline: at => deadlines.push(at) }); + expect(whileLive.issueIds).toEqual(status === "done" ? [issueId] : []); + expect(deadlines).toHaveLength(status === "done" ? 0 : 1); + const afterExecutor = await recoveryService(db, { enqueueWakeup: vi.fn(), liveRunExecutions: new Set() }) + .sweepStaleIssueLocks({ companyId, issueId }); + expect(afterExecutor.issueIds).toEqual(status === "done" ? [] : [issueId]); + const [row] = await db.select({ executionRunId: issues.executionRunId }).from(issues).where(eq(issues.id, issueId)); + expect(row.executionRunId).toBeNull(); + }); + it("does not clear locks while the referenced run is still running", async () => { const { companyId, agentId, runningRunId } = await seed(); const issueId = randomUUID(); diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index cd7bdb1c7b1..513d1b1ed10 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -6883,6 +6883,17 @@ export function recoveryService( .some(runId => runId !== null && pendingResumeRunIds.has(runId))) { continue; } + if (!taskClosed) { + let settlingUntil: Date | null = null; + for (const runId of new Set([issue.checkoutRunId, issue.executionRunId])) { + const lockRun = runId ? runById.get(runId) : undefined; + if (lockRun && !settlingUntil) settlingUntil = await terminalRunSettlingUntil(issue.companyId, lockRun.id, lockRun); + } + if (settlingUntil) { + opts?.onDeferredDeadline?.(settlingUntil.getTime()); + continue; + } + } await deps.beforeStaleLockClear?.(issue.id); const updated = await db.transaction(async tx => {