From 0e38c80323b4a5c9c50da31c86f64fbd9eb2bcef Mon Sep 17 00:00:00 2001 From: paperclipbugbot Date: Fri, 20 Nov 2026 07:53:08 -0800 Subject: [PATCH] fix(recovery): do not auto-replace a chat Retry under its consumed exact-request authority (#22414) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs that fail before they do provider work can continue automatically. The native safe-replacement sweep schedules a system successor run for them. > - The successor copies the failed run's context. The sweep removes consumed wake fields, but it keeps the `chatFailedRunRetry` selector and `taskRetryRequestId`. > - That selector is the authority of an exact chat Retry. It is bound to the failed run's own chat action and to the user's Retry intent. The successor gets a new system wakeup request, so admission can never authorize it. > - Thus every automatic successor of a chat Retry fails at setup with `FailedChatRunRetryAuthorizationError`. The sweep has already retired the task's recovery hold at that time, so the task can be left with no owner of the next step. > - This pull request stops the sweep from replacing a run that carries `chatFailedRunRetry`. > - The benefit is that the task keeps its hold. The user's next Retry or message continues it through the normal authorized path, and no run fails for a known reason. ## Linked Issues or Issue Description **What happened?** On staging, a user pressed Retry on a failed GitHub chat run. The Retry run failed before its first claim, after a workspace restore failure. Its own exact-request authority refused it (`chat_failed_run_retry_not_authorized`). The pre-claim continuation path then scheduled a `native_safe_replacement` successor. The successor kept the copied `chatFailedRunRetry` selector under a new wakeup request ID, and `authorizeFailedChatRunRetryWake` refused it at setup. **Expected behavior** An automatic system successor must not be scheduled under chat Retry authority that it cannot hold. The task must keep its recovery hold for the user's next Retry or message. **Steps to reproduce** 1. Make a native run whose context has `chatFailedRunRetry` (an exact chat Retry). 2. Fail it before its first coordinator claim, or with a provider terminal failure, so it qualifies for safe replacement. 3. Run `reconcileSafeNativeReplacements`. 4. See that a successor run is scheduled with the copied selector. At admission it always fails with `FailedChatRunRetryAuthorizationError`. Related: #22413 changes the same file for a different defect (the reviewed chat message batch on fresh-session retries). It does not touch the eligibility predicate or `chatFailedRunRetry`. The two changes are independent. ## What Changed - `nativeReplacementPending()` now excludes a failed run whose context carries `chatFailedRunRetry`. Because the shared eligibility predicate has the guard, the pre-claim, historical listener, and provider terminal-failure replacement paths all get it. - New regression in `native-preclaim-continuation.test.ts`: this is the staging shape (a pre-claim failure of an exact chat Retry after its saved work is recovered). It schedules no successor, the hold stays effective, and the task stays blocked. - New paired regression in `native-safe-replacement.test.ts` for the provider terminal-failure path. An ordinary run still gets exactly one successor. An exact chat Retry gets none. ## Verification - Red before the fix: the pre-claim chat Retry case failed with `expected length 0, got 1`. The terminal-failure chat Retry case also failed. Its ordinary-run control passed. - Green after the fix, rebased on `fe0c4aa95d`: - `npx vitest run src/services/native-preclaim-continuation.test.ts src/services/native-runtime/native-safe-replacement.test.ts`: 89 passed. - `npx vitest run src/services/native-runtime/native-safe-replacement.test.ts src/services/quarantined-task-retry.test.ts src/__tests__/execution-control-work.test.ts src/__tests__/recovery-settled-records-census.test.ts src/__tests__/durable-chat-wakeup.test.ts`: 206 passed. - `pnpm --filter @paperclipai/server typecheck`: passed. - Second-order check: `nativeInterruptionHold()` uses `not(nativeReplacementPending())`. Thus an excluded run can now reach the existing fresh-continuation fallback, the same as other pre-claim runs that do not qualify. That fallback records a continuation decision and lets normal admission build a fresh context. It does not copy the predecessor's selector. - `pnpm verify` selects 2,950 of 3,817 files for this widely imported module. I ran the focused suites above and rely on required CI for the broad coverage. ## Risks - Low risk. The change only removes candidates from automatic replacement. Before this change, a removed candidate always produced a successor that failed at setup. - Behavior shift: after a chat Retry run fails, the user (or a new chat message) continues the task. A system successor that cannot be authorized does not start. Ordinary chat-message runs and non-chat runs are not changed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in an agent harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- .../native-preclaim-continuation.test.ts | 30 +++++++++++++++++++ .../native-safe-replacement.test.ts | 18 +++++++++++ .../native-runtime/native-safe-replacement.ts | 4 +++ 3 files changed, 52 insertions(+) diff --git a/server/src/services/native-preclaim-continuation.test.ts b/server/src/services/native-preclaim-continuation.test.ts index 95929b02e1f..6e1b2af2610 100644 --- a/server/src/services/native-preclaim-continuation.test.ts +++ b/server/src/services/native-preclaim-continuation.test.ts @@ -136,6 +136,36 @@ describe("native run that failed before its first coordinator claim", () => { expect(await successorsOf(f.runId)).toHaveLength(0); }, 60_000); + // Staging 2026-11-20: a user's exact chat Retry failed before its first + // claim. Its authority is bound to its own chat action and Retry intent, so + // a system successor carrying the copied selector is always refused at + // setup. The saved hold keeps the task for the user's next Retry or message. + it("does not continue a chat Retry automatically under its consumed exact-request authority", async () => { + const f = await preClaimNativeFailure(); + const current = await run(f.runId); + // The failed run the user retried. It precedes the Retry, so it does not + // supersede it. + const [failedSource] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, + status: "failed", contextSnapshot: { issueId: f.issueId }, createdAt: new Date(current.createdAt.getTime() - 60_000) }) + .returning(); + const retryOfRunId = failedSource!.id; + await db.update(heartbeatRuns).set({ + errorCode: "chat_failed_run_retry_not_authorized", retryOfRunId, + contextSnapshot: { ...current.contextSnapshot, retryOfRunId, forceFreshSession: true, taskRetryRequestId: randomUUID(), + chatFailedRunRetry: { version: 1, actionId: current.wakeupRequestId ?? randomUUID(), failedRunId: retryOfRunId } }, + }).where(eq(heartbeatRuns.id, f.runId)); + await recoverRetainedWorkspace(f.input); + expect(readWorkspaceRecovery((await action(f.actionId)).evidence)).toMatchObject({ status: "recovered" }); + + await sweep(); + + expect(await successorsOf(f.runId)).toHaveLength(0); + const hold = await action(f.actionId); + expect(hold).toMatchObject({ outcome: "blocked", evidence: { automaticRecovery: { replay: "blocked" } } }); + const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + expect(task.status).toBe("blocked"); + }, 60_000); + it.each(["claimed", "provider_event", "tool_call", "uncertain_side_effect", "process", "superseded"] as const)( "keeps reconciliation when the run may have acted: %s", async gate => { const f = await preClaimNativeFailure(); diff --git a/server/src/services/native-runtime/native-safe-replacement.test.ts b/server/src/services/native-runtime/native-safe-replacement.test.ts index e7c4b033609..5ffa2adee33 100644 --- a/server/src/services/native-runtime/native-safe-replacement.test.ts +++ b/server/src/services/native-runtime/native-safe-replacement.test.ts @@ -527,6 +527,24 @@ const support = externalDatabaseUrl const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId)); expect(action.status).toBe("active"); }); + it.each([["an ordinary run", false, 1], ["an exact chat Retry", true, 0]] as const)( + "replaces a provider terminal failure only without consumed chat Retry authority (%s)", async (_label, chatRetry, expected) => { + const source = await seed(); + if (chatRetry) { + const [failed] = await db.insert(heartbeatRuns).values({ companyId: source.companyId, agentId: source.agentId, + status: "failed", contextSnapshot: { issueId: source.issueId }, createdAt: new Date(Date.now() - 60_000) }).returning(); + const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, source.runId)); + await db.update(heartbeatRuns).set({ retryOfRunId: failed!.id, contextSnapshot: { ...current!.contextSnapshot, + retryOfRunId: failed!.id, taskRetryRequestId: randomUUID(), + chatFailedRunRetry: { version: 1, actionId: randomUUID(), failedRunId: failed!.id } } }) + .where(eq(heartbeatRuns.id, source.runId)); + } + invalidateNativeReplacementDiscovery(db); + await reconcileSafeNativeReplacements(db); + const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId)); + expect(successors).toHaveLength(expected); + for (const successor of successors) expect(successor.contextSnapshot).not.toHaveProperty("chatFailedRunRetry"); + }); async function seedUnprovenOwner(finishedAgoMs: number) { const source = await seed(); await db.update(heartbeatRuns).set({ finishedAt: sql`now() - ${finishedAgoMs}::integer * interval '1 millisecond'`, diff --git a/server/src/services/native-runtime/native-safe-replacement.ts b/server/src/services/native-runtime/native-safe-replacement.ts index bcb960876bc..7e619c53537 100644 --- a/server/src/services/native-runtime/native-safe-replacement.ts +++ b/server/src/services/native-runtime/native-safe-replacement.ts @@ -240,6 +240,10 @@ export async function remoteWorkspaceReplacementEvidence( export function nativeReplacementPending() { return and( eq(heartbeatRuns.status, "failed"), + // An exact chat Retry is authorized only through its own chat action and + // the user's Retry intent. A system successor cannot hold that authority, + // so admission would always refuse it. Keep the task's hold for the user. + sql`not (coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) ? 'chatFailedRunRetry')`, eq(nativeRunFinalizations.phase, "terminal_failure"), or(inArray( nativeRunFinalizations.failureCode,