From 88469bd2bc57add904ec8fc93a4265be47e8e8f4 Mon Sep 17 00:00:00 2001 From: paperclipbugbot Date: Fri, 20 Nov 2026 08:47:32 -0800 Subject: [PATCH] fix(retry): wait for the failed run's sandbox to stop before repeating its chat request (#22423) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task from a connected chat (for example a GitHub pull request review) can fail. Retry then repeats the exact failed chat request in a new run > - Before that run starts the agent, it checks again that the failed run's executor is stopped. For a remote sandbox, the proof is a stop receipt on the sandbox lease > - Workspace recovery can restart the failed run's sandbox at the same time, to save its files. While it does this, the lease has no stop receipt > - The check then refused the repeated request. The run failed for good, although recovery stopped the sandbox again a few seconds later > - This pull request makes the repeated request wait while a stop of that sandbox is in progress, and then ask the retry authority again > - The benefit is that a correct Retry is not lost because of a short overlap with workspace recovery ## Linked Issues or Issue Description Refs #22414. That PR fixes the next step of the same incident: the system replacement run that copied the consumed retry selector. This PR fixes the first refusal. The two PRs change different files and are independent. **What happened?** On a hosted staging deploy, a GitHub review reply task failed (`native_continuation_requires_reconciliation`). Its Daytona sandbox went into workspace recovery. A Retry started a new run that repeats the exact chat request, in a fresh isolated workspace. The run passed admission and its first execution check. After about 14 seconds of workspace preparation, the `dispatching` recheck refused it with 409 `chat_failed_run_retry_not_authorized` ("This chat request can't be repeated exactly as it was sent."). No provider work started. The recovery reported "Saved work recovered" 12 seconds later. The GitHub reply request was not answered. **Expected behavior** The repeated request waits while the failed run's sandbox stops again, and then continues. If the stop does not settle, the request is refused as before. **Steps to reproduce** 1. Make a native chat task fail on a remote sandbox with a recorded stop receipt. 2. Start workspace recovery for that run. `prepareStoppedSandboxForRecovery` moves the lease to `pending_cleanup` and removes `remoteExecutionTermination` while the sandbox restarts. 3. Press Retry during that interval. `taskChatRetryExecutionStopped(source)` is false, so `assertFailedNativeRetryState` refuses the run at the `dispatching` recheck. The new test does steps 1 and 2 with the real lease transitions on an embedded Postgres database. **Paperclip version or commit** `201abacd6ee5` (the staging release in the incident). ## What Changed - Add `server/src/services/chat-channels/failed-chat-retry-source-stop-wait.ts`: - `failedChatRetrySourceStopPending()` is true when a non-local lease of the failed run is `pending_cleanup` with a valid stop-only cleanup intent (`readStopOnlyCleanup`). Workspace recovery holds such an intent while it restarts and stops the sandbox. - `awaitFailedChatRetrySourceStop()` runs the authorization again every 5 seconds, for up to 3 minutes, while that stop is pending. It reads the pending stop before and after each refused attempt. If a stop was pending before the attempt but is gone after it, the stop settled while the refused transaction held no locks, so it reads the authorization once more at once. If no stop is pending before or after, it returns the refusal after one attempt, exactly as before. It waits only for the reason-less containment refusal. It does not wait for a reasoned refusal, a mismatched retry selector (`FailedChatRunRetryAuthorizationError`), or any other error. It stops when the run is stopped, and it returns the last refusal at the deadline. - In `heartbeat.ts`, both execution-phase checks of an exact replay (the first check and the `dispatching` recheck) use this wait. The run log records one line when the wait starts. The wait grants no authority. Each attempt still runs the unchanged `authorizeFailedChatRunRetryWake`. The controller lease is checked on each attempt, as before. ## Verification - `cd server && npx vitest run src/services/chat-channels/failed-chat-retry-source-stop-wait.test.ts`: 10 tests pass. They use real lease rows and the real `prepareStoppedSandboxForRecovery`, `restoreSandboxStopAfterFailedRestart` and `taskChatRetryExecutionStopped`. - Red check: with the helper changed to make one attempt only (the old behavior), the wait test fails with the incident's refusal: `promise rejected "Error: This chat request can't be repeated…" { status: 409, details: { code: 'chat_failed_run_retry_not_authorized' } }`. The abort test also fails. Then I restored the helper. - Red check for the review finding: with the old "no pending stop, so refuse" line, the new test that commits the stop receipt between the refusal and the pending-stop read fails with the old refusal. It passes with the final read. - A first version of that fix read the authorization again after every refusal. CI showed this was wrong: `durable-chat-wakeup.test.ts` counts authorization calls, and a budget-hold case in `chat-channels.integration.test.ts` changed outcome. I reproduced both failures locally with that version. The current version rechecks only when a stop was pending, and those tests pass: `durable-chat-wakeup.test.ts` (51), the CI-failing chat integration cases by name (73), the retry-filtered chat integration run (261), and the focused suites (300). - Safety cases in the same file: refusal at the deadline when the sandbox stays held; immediate refusal when no stop is in progress (an active lease whose executor can still run); no wait for a reasoned refusal, a selector mismatch, or another error; no wait for a run that is not an exact replay; the wait stops when the run is stopped. - Focused existing suites pass: `authorization-run-read.test.ts`, `wake-queue/application/use-cases.test.ts`, `__tests__/durable-chat-wakeup.test.ts`, `task-retry.test.ts`, `budget-resume-continuation.test.ts` (299 tests in total, including the new file). - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify` selects almost the full suite because `heartbeat.ts` is a shared module. I did not run that locally. Required CI supplies that coverage. - I did not do a live staging reproduction. The incident timing cannot be produced on demand. The test reproduces the lease state that the refusal saw. ## Risks - A repeated request can now hold its own prepared sandbox for up to 3 more minutes before it fails. This happens only while a stop of the failed run's sandbox is in progress. - The wait can not let a second executor run. It changes only when the check runs, and every attempt uses the unchanged authority. - If workspace recovery takes longer than 3 minutes, the run is refused as before. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (not needed: the code and tests describe the behavior) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- ...failed-chat-retry-source-stop-wait.test.ts | 161 ++++++++++++++++++ .../failed-chat-retry-source-stop-wait.ts | 85 +++++++++ server/src/services/heartbeat.ts | 14 +- 3 files changed, 258 insertions(+), 2 deletions(-) create mode 100644 server/src/services/chat-channels/failed-chat-retry-source-stop-wait.test.ts create mode 100644 server/src/services/chat-channels/failed-chat-retry-source-stop-wait.ts diff --git a/server/src/services/chat-channels/failed-chat-retry-source-stop-wait.test.ts b/server/src/services/chat-channels/failed-chat-retry-source-stop-wait.test.ts new file mode 100644 index 00000000000..00708eb9edf --- /dev/null +++ b/server/src/services/chat-channels/failed-chat-retry-source-stop-wait.test.ts @@ -0,0 +1,161 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, issues } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { conflict } from "../../errors.js"; +import { FailedChatRunRetryAuthorizationError } from "../durable-chat-wakeup.js"; +import { remoteTerminationReceipt } from "../remote-execution-termination.js"; +import { prepareStoppedSandboxForRecovery, restoreSandboxStopAfterFailedRestart } from "../sandbox-stop-and-retain.js"; +import { taskChatRetryExecutionStopped } from "./task-retry-authority.js"; +import { awaitFailedChatRetrySourceStop, failedChatRetrySourceStopPending } from "./failed-chat-retry-source-stop-wait.js"; + +describe("exact chat replay while workspace recovery restarts the failed run's sandbox", () => { + let postgres: Awaited>; + let db: ReturnType; + beforeAll(async () => { + postgres = await startEmbeddedPostgresTestDatabase("chat-retry-source-stop-"); + db = createDb(postgres.connectionString); + }, 30_000); + afterAll(async () => { await db?.$client.end(); await postgres?.cleanup(); }); + + /** A failed native run whose remote sandbox was stopped with a bound receipt. */ + async function stoppedSource() { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Chat retry", issuePrefix: `C${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Reviewer", adapterType: "paperclip_runner", status: "idle" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Review", status: "blocked", assigneeAgentId: agentId }); + const [source] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, status: "failed", + runtimeMode: "native", nativeIssueId: issueId, nativePhase: "terminal_failure", + errorCode: "native_continuation_requires_reconciliation", contextSnapshot: { issueId }, finishedAt: new Date() }).returning(); + const [environment] = await db.insert(environments).values({ name: runId, driver: "sandbox" }).returning(); + const identity = { id: randomUUID(), companyId, heartbeatRunId: runId, provider: "fake-sandbox", providerLeaseId: randomUUID() }; + const [lease] = await db.insert(environmentLeases).values({ ...identity, issueId, environmentId: environment.id, + status: "released", cleanupStatus: "success", releasedAt: new Date(), leasePolicy: "retain_on_failure", + metadata: { pluginId: "fake-plugin", remoteCwd: "/work", remoteExecutionTermination: + remoteTerminationReceipt(identity, { providerLeaseId: identity.providerLeaseId, state: "stopped" }) }, + }).returning(); + return { source: source!, lease: lease!, companyId }; + } + + /** Workspace recovery suspends the stop receipt and restarts the sandbox to save its files. */ + async function recoveryRestart(lease: typeof environmentLeases.$inferSelect) { + const pending = await prepareStoppedSandboxForRecovery(db, lease); + expect(pending).not.toBeNull(); + return pending!; + } + + async function restop(pending: typeof environmentLeases.$inferSelect) { + const attemptId = String(pending.metadata?.pendingCleanupAttemptId); + expect(await restoreSandboxStopAfterFailedRestart(db, pending, { attemptId })).not.toBeNull(); + } + + /** The containment part of the retry authority, as assertFailedNativeRetryState applies it. */ + const authorize = (sourceRunId: string) => async () => { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, sourceRunId)); + if (!await taskChatRetryExecutionStopped(db, run!)) + throw conflict("This chat request can't be repeated exactly as it was sent.", { code: "chat_failed_run_retry_not_authorized" }); + return true; + }; + + it("loses the source's containment proof while recovery holds the sandbox, then regains it", async () => { + const { source, lease, companyId } = await stoppedSource(); + expect(await taskChatRetryExecutionStopped(db, source)).toBe(true); + expect(await failedChatRetrySourceStopPending(db, companyId, source.id)).toBe(false); + + const pending = await recoveryRestart(lease); + // This is the state the replay's dispatch recheck saw in the incident. + expect(await taskChatRetryExecutionStopped(db, source)).toBe(false); + expect(await failedChatRetrySourceStopPending(db, companyId, source.id)).toBe(true); + + await restop(pending); + expect(await taskChatRetryExecutionStopped(db, source)).toBe(true); + expect(await failedChatRetrySourceStopPending(db, companyId, source.id)).toBe(false); + }); + + it("waits for the sandbox to stop again instead of failing the replay", async () => { + const { source, lease, companyId } = await stoppedSource(); + const pending = await recoveryRestart(lease); + await expect(authorize(source.id)()).rejects.toMatchObject({ details: { code: "chat_failed_run_retry_not_authorized" } }); + + const onWait = vi.fn(); + const attempt = vi.fn(authorize(source.id)); + const restopped = new Promise(resolve => setTimeout(() => void restop(pending).then(resolve), 150)); + await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id, + signal: new AbortController().signal, onWait, pollMs: 25 }, attempt)).resolves.toBe(true); + await restopped; + expect(onWait).toHaveBeenCalledTimes(1); + expect(attempt.mock.calls.length).toBeGreaterThan(1); + }); + + it("checks once more when the stop settles between the refusal and the pending-stop read", async () => { + const { source, lease, companyId } = await stoppedSource(); + const pending = await recoveryRestart(lease); + const check = authorize(source.id); + let first = true; + // The refused transaction has released its locks; recovery then commits the stop receipt. + const attempt = vi.fn(async () => { + if (!first) return check(); + first = false; + await check().catch(async error => { await restop(pending); throw error; }); + throw new Error("expected the first attempt to be refused"); + }); + const onWait = vi.fn(); + await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id, + signal: new AbortController().signal, onWait, pollMs: 25 }, attempt)).resolves.toBe(true); + expect(attempt).toHaveBeenCalledTimes(2); + expect(onWait).not.toHaveBeenCalled(); + }); + + it("refuses at the deadline when the sandbox stays held", async () => { + const { source, lease, companyId } = await stoppedSource(); + await recoveryRestart(lease); + await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id, + signal: new AbortController().signal, timeoutMs: 100, pollMs: 25 }, authorize(source.id))) + .rejects.toMatchObject({ details: { code: "chat_failed_run_retry_not_authorized" } }); + }); + + it("refuses without waiting when no stop of the source sandbox is in progress", async () => { + const { source, companyId } = await stoppedSource(); + // An unreleased lease without a stop intent: the old executor may still be running. + await db.update(environmentLeases).set({ status: "active", cleanupStatus: null, releasedAt: null, metadata: { pluginId: "fake-plugin" } }) + .where(eq(environmentLeases.heartbeatRunId, source.id)); + const attempt = vi.fn(authorize(source.id)); + const onWait = vi.fn(); + await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id, + signal: new AbortController().signal, onWait, pollMs: 25 }, attempt)).rejects.toMatchObject({ status: 409 }); + expect(attempt).toHaveBeenCalledTimes(1); + expect(onWait).not.toHaveBeenCalled(); + }); + + it.each([ + ["a reasoned refusal", () => conflict("saved", { code: "chat_failed_run_retry_not_authorized", reason: "prior_result_publication" })], + ["a mismatched retry selector", () => new FailedChatRunRetryAuthorizationError()], + ["another error", () => new Error("database unavailable")], + ])("does not wait on %s", async (_label, error) => { + const { source, lease, companyId } = await stoppedSource(); + await recoveryRestart(lease); + const attempt = vi.fn(async () => { throw error(); }); + await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id, + signal: new AbortController().signal, pollMs: 25 }, attempt)).rejects.toBeInstanceOf(Error); + expect(attempt).toHaveBeenCalledTimes(1); + }); + + it("does not wait when the run is not an exact replay of a failed run", async () => { + const { source, lease, companyId } = await stoppedSource(); + await recoveryRestart(lease); + const attempt = vi.fn(authorize(source.id)); + await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: null, + signal: new AbortController().signal, pollMs: 25 }, attempt)).rejects.toMatchObject({ status: 409 }); + expect(attempt).toHaveBeenCalledTimes(1); + }); + + it("stops waiting when the run is stopped", async () => { + const { source, lease, companyId } = await stoppedSource(); + await recoveryRestart(lease); + const control = new AbortController(); + const waiting = awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id, + signal: control.signal, pollMs: 10_000, onWait: () => control.abort(new Error("stopped")) }, authorize(source.id)); + await expect(waiting).rejects.toThrow("stopped"); + }); +}); diff --git a/server/src/services/chat-channels/failed-chat-retry-source-stop-wait.ts b/server/src/services/chat-channels/failed-chat-retry-source-stop-wait.ts new file mode 100644 index 00000000000..c1db42aae3d --- /dev/null +++ b/server/src/services/chat-channels/failed-chat-retry-source-stop-wait.ts @@ -0,0 +1,85 @@ +import { and, eq } from "drizzle-orm"; +import { environmentLeases, type Db } from "@paperclipai/db"; +import { HttpError } from "../../errors.js"; +import { FailedChatRunRetryAuthorizationError } from "../durable-chat-wakeup.js"; +import { readStopOnlyCleanup } from "../sandbox-stop-and-retain.js"; + +/** Long enough for workspace recovery to save files and stop the sandbox again. */ +export const FAILED_CHAT_RETRY_SOURCE_STOP_WAIT_MS = 3 * 60_000; +export const FAILED_CHAT_RETRY_SOURCE_STOP_POLL_MS = 5_000; + +/** The containment refusal carries no reason. Reasoned refusals (an accepted + * result, a prior publication or interaction, source authorization) and a + * mismatched retry selector do not change while a sandbox stops. */ +function isSourceContainmentRefusal(error: unknown) { + if (!(error instanceof HttpError) || error instanceof FailedChatRunRetryAuthorizationError || + error.status !== 409) return false; + const details = error.details as Record | undefined; + return details?.code === "chat_failed_run_retry_not_authorized" && details.reason === undefined; +} + +/** A stop of the failed run's sandbox is in progress. Workspace recovery holds + * one while it restarts the sandbox to save its files, and removes the stop + * receipt until it stops it again. This only permits a wait: containment is + * still decided by the unchanged retry authority on each attempt. */ +export async function failedChatRetrySourceStopPending(db: Db, companyId: string, sourceRunId: string) { + const leases = await db.select().from(environmentLeases).where(and( + eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, sourceRunId), + )); + return leases.some(lease => lease.provider !== "local" && lease.status === "pending_cleanup" && + readStopOnlyCleanup(lease) !== null); +} + +function delay(ms: number, signal: AbortSignal) { + return new Promise((resolve, reject) => { + if (signal.aborted) return reject(signal.reason); + const onAbort = () => { clearTimeout(timer); reject(signal.reason); }; + const timer = setTimeout(() => { signal.removeEventListener("abort", onAbort); resolve(); }, ms); + signal.addEventListener("abort", onAbort, { once: true }); + }); +} + +/** Re-run an exact chat replay's authorization while the failed run's sandbox + * stop settles, instead of failing the replay on a transient missing receipt. + * The wait is bounded and stops with the run; the last refusal is returned. + * The pending stop is read before and after each refused attempt. A stop can + * settle after the refused transaction releases its locks, so a stop seen + * before the attempt earns one more attempt even when it is gone after. A + * refusal with no pending stop on either side is returned at once. */ +export async function awaitFailedChatRetrySourceStop(db: Db, input: { + companyId: string; + sourceRunId: string | null; + signal: AbortSignal; + onWait?: () => Promise | void; + timeoutMs?: number; + pollMs?: number; +}, attempt: () => Promise): Promise { + const sourceRunId = input.sourceRunId; + if (!sourceRunId) return attempt(); + const deadline = Date.now() + (input.timeoutMs ?? FAILED_CHAT_RETRY_SOURCE_STOP_WAIT_MS); + let announced = false; + let settledRetry = false; + for (;;) { + const pendingBefore = await failedChatRetrySourceStopPending(db, input.companyId, sourceRunId); + let pendingAfter: boolean; + try { + return await attempt(); + } catch (error) { + if (!isSourceContainmentRefusal(error)) throw error; + pendingAfter = await failedChatRetrySourceStopPending(db, input.companyId, sourceRunId); + if ((!pendingBefore && !pendingAfter) || Date.now() >= deadline) throw error; + } + // The stop settled during the refused attempt: read the authority again + // now, once. A later flip waits for the normal poll interval. + if (!pendingAfter && !settledRetry) { + settledRetry = true; + continue; + } + if (!announced) { + announced = true; + await input.onWait?.(); + } + await delay(Math.min(input.pollMs ?? FAILED_CHAT_RETRY_SOURCE_STOP_POLL_MS, Math.max(0, deadline - Date.now())), + input.signal); + } +} diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 200faf00f62..be7c0cf9ea1 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -314,6 +314,7 @@ import { watchLegacyControllerLease, } from "./legacy-controller-lease.js"; import { retryChatControlAdmission } from "./chat-control-admission-retry.js"; +import { awaitFailedChatRetrySourceStop } from "./chat-channels/failed-chat-retry-source-stop-wait.js"; import { @@ -2268,8 +2269,17 @@ export function heartbeatService( } const runtime = await timePreparation("runtime_state", () => ensureRuntimeState(agent)); const context = parseObject(run.contextSnapshot); + // Workspace recovery can restart the failed run's sandbox to save its + // files while this replay prepares. Wait for that stop to settle; the + // retry authority still decides each attempt. const authorizeFailedChatRetryExecution = (stage?: "dispatching") => - retryChatControlAdmission(async () => { + awaitFailedChatRetrySourceStop(db, { + companyId: run.companyId, + sourceRunId: Object.hasOwn(context, "chatFailedRunRetry") ? readNonEmptyString(context.retryOfRunId) : null, + signal: executionControl.controller.signal, + onWait: () => appendRunEvent(run, { eventType: "lifecycle", stream: "system", level: "info", + message: "Waiting for the failed run's sandbox to stop before repeating the chat request" }), + }, () => retryChatControlAdmission(async () => { executionControl.controller.signal.throwIfAborted(); await controllerLease.assertOwned(stage); return db.transaction((tx) => @@ -2282,7 +2292,7 @@ export function heartbeatService( runId: run.id, contextSnapshot: context, })); - }); + })); const isFailedChatRunRetry = await timePreparation("chat_retry_admission", () => authorizeFailedChatRetryExecution()); // Never adopt a chat-execution attestation supplied in a wake payload. // Reviewed chat turns rebuild it from the current durable owner below.