fix(retry): wait for the failed run's sandbox to stop before repeating its chat request (#22423)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task from a connected chat (for example a GitHub pull request
review) can fail. Retry then repeats the exact failed chat request in a
new run
> - Before that run starts the agent, it checks again that the failed
run's executor is stopped. For a remote sandbox, the proof is a stop
receipt on the sandbox lease
> - Workspace recovery can restart the failed run's sandbox at the same
time, to save its files. While it does this, the lease has no stop
receipt
> - The check then refused the repeated request. The run failed for
good, although recovery stopped the sandbox again a few seconds later
> - This pull request makes the repeated request wait while a stop of
that sandbox is in progress, and then ask the retry authority again
> - The benefit is that a correct Retry is not lost because of a short
overlap with workspace recovery

## Linked Issues or Issue Description

Refs #22414. That PR fixes the next step of the same incident: the
system replacement run that copied the consumed retry selector. This PR
fixes the first refusal. The two PRs change different files and are
independent.

**What happened?**

On a hosted staging deploy, a GitHub review reply task failed
(`native_continuation_requires_reconciliation`). Its Daytona sandbox
went into workspace recovery. A Retry started a new run that repeats the
exact chat request, in a fresh isolated workspace. The run passed
admission and its first execution check. After about 14 seconds of
workspace preparation, the `dispatching` recheck refused it with 409
`chat_failed_run_retry_not_authorized` ("This chat request can't be
repeated exactly as it was sent."). No provider work started. The
recovery reported "Saved work recovered" 12 seconds later. The GitHub
reply request was not answered.

**Expected behavior**

The repeated request waits while the failed run's sandbox stops again,
and then continues. If the stop does not settle, the request is refused
as before.

**Steps to reproduce**

1. Make a native chat task fail on a remote sandbox with a recorded stop
receipt.
2. Start workspace recovery for that run.
`prepareStoppedSandboxForRecovery` moves the lease to `pending_cleanup`
and removes `remoteExecutionTermination` while the sandbox restarts.
3. Press Retry during that interval.
`taskChatRetryExecutionStopped(source)` is false, so
`assertFailedNativeRetryState` refuses the run at the `dispatching`
recheck.

The new test does steps 1 and 2 with the real lease transitions on an
embedded Postgres database.

**Paperclip version or commit**

`201abacd6ee5` (the staging release in the incident).

## What Changed

- Add
`server/src/services/chat-channels/failed-chat-retry-source-stop-wait.ts`:
- `failedChatRetrySourceStopPending()` is true when a non-local lease of
the failed run is `pending_cleanup` with a valid stop-only cleanup
intent (`readStopOnlyCleanup`). Workspace recovery holds such an intent
while it restarts and stops the sandbox.
- `awaitFailedChatRetrySourceStop()` runs the authorization again every
5 seconds, for up to 3 minutes, while that stop is pending. It reads the
pending stop before and after each refused attempt. If a stop was
pending before the attempt but is gone after it, the stop settled while
the refused transaction held no locks, so it reads the authorization
once more at once. If no stop is pending before or after, it returns the
refusal after one attempt, exactly as before. It waits only for the
reason-less containment refusal. It does not wait for a reasoned
refusal, a mismatched retry selector
(`FailedChatRunRetryAuthorizationError`), or any other error. It stops
when the run is stopped, and it returns the last refusal at the
deadline.
- In `heartbeat.ts`, both execution-phase checks of an exact replay (the
first check and the `dispatching` recheck) use this wait. The run log
records one line when the wait starts.

The wait grants no authority. Each attempt still runs the unchanged
`authorizeFailedChatRunRetryWake`. The controller lease is checked on
each attempt, as before.

## Verification

- `cd server && npx vitest run
src/services/chat-channels/failed-chat-retry-source-stop-wait.test.ts`:
10 tests pass. They use real lease rows and the real
`prepareStoppedSandboxForRecovery`,
`restoreSandboxStopAfterFailedRestart` and
`taskChatRetryExecutionStopped`.
- Red check: with the helper changed to make one attempt only (the old
behavior), the wait test fails with the incident's refusal: `promise
rejected "Error: This chat request can't be repeated…" { status: 409,
details: { code: 'chat_failed_run_retry_not_authorized' } }`. The abort
test also fails. Then I restored the helper.
- Red check for the review finding: with the old "no pending stop, so
refuse" line, the new test that commits the stop receipt between the
refusal and the pending-stop read fails with the old refusal. It passes
with the final read.
- A first version of that fix read the authorization again after every
refusal. CI showed this was wrong: `durable-chat-wakeup.test.ts` counts
authorization calls, and a budget-hold case in
`chat-channels.integration.test.ts` changed outcome. I reproduced both
failures locally with that version. The current version rechecks only
when a stop was pending, and those tests pass:
`durable-chat-wakeup.test.ts` (51), the CI-failing chat integration
cases by name (73), the retry-filtered chat integration run (261), and
the focused suites (300).
- Safety cases in the same file: refusal at the deadline when the
sandbox stays held; immediate refusal when no stop is in progress (an
active lease whose executor can still run); no wait for a reasoned
refusal, a selector mismatch, or another error; no wait for a run that
is not an exact replay; the wait stops when the run is stopped.
- Focused existing suites pass: `authorization-run-read.test.ts`,
`wake-queue/application/use-cases.test.ts`,
`__tests__/durable-chat-wakeup.test.ts`, `task-retry.test.ts`,
`budget-resume-continuation.test.ts` (299 tests in total, including the
new file).
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` selects almost the full suite because `heartbeat.ts` is
a shared module. I did not run that locally. Required CI supplies that
coverage.
- I did not do a live staging reproduction. The incident timing cannot
be produced on demand. The test reproduces the lease state that the
refusal saw.

## Risks

- A repeated request can now hold its own prepared sandbox for up to 3
more minutes before it fails. This happens only while a stop of the
failed run's sandbox is in progress.
- The wait can not let a second executor run. It changes only when the
check runs, and every attempt uses the unchanged authority.
- If workspace recovery takes longer than 3 minutes, the run is refused
as before.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
needed: the code and tests describe the behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 08:47:32 -08:00
1 parent dce4d47f60
commit 88469bd2bc
3 files changed
+258 -2

No files matched your search

@@ -0,0 +1,161 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, issues } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { conflict } from "../../errors.js";
import { FailedChatRunRetryAuthorizationError } from "../durable-chat-wakeup.js";
import { remoteTerminationReceipt } from "../remote-execution-termination.js";
import { prepareStoppedSandboxForRecovery, restoreSandboxStopAfterFailedRestart } from "../sandbox-stop-and-retain.js";
import { taskChatRetryExecutionStopped } from "./task-retry-authority.js";
import { awaitFailedChatRetrySourceStop, failedChatRetrySourceStopPending } from "./failed-chat-retry-source-stop-wait.js";
describe("exact chat replay while workspace recovery restarts the failed run's sandbox", () => {
let postgres: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
postgres = await startEmbeddedPostgresTestDatabase("chat-retry-source-stop-");
db = createDb(postgres.connectionString);
}, 30_000);
afterAll(async () => { await db?.$client.end(); await postgres?.cleanup(); });
/** A failed native run whose remote sandbox was stopped with a bound receipt. */
async function stoppedSource() {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Chat retry", issuePrefix: `C${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Reviewer", adapterType: "paperclip_runner", status: "idle" });
await db.insert(issues).values({ id: issueId, companyId, title: "Review", status: "blocked", assigneeAgentId: agentId });
const [source] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, status: "failed",
runtimeMode: "native", nativeIssueId: issueId, nativePhase: "terminal_failure",
errorCode: "native_continuation_requires_reconciliation", contextSnapshot: { issueId }, finishedAt: new Date() }).returning();
const [environment] = await db.insert(environments).values({ name: runId, driver: "sandbox" }).returning();
const identity = { id: randomUUID(), companyId, heartbeatRunId: runId, provider: "fake-sandbox", providerLeaseId: randomUUID() };
const [lease] = await db.insert(environmentLeases).values({ ...identity, issueId, environmentId: environment.id,
status: "released", cleanupStatus: "success", releasedAt: new Date(), leasePolicy: "retain_on_failure",
metadata: { pluginId: "fake-plugin", remoteCwd: "/work", remoteExecutionTermination:
remoteTerminationReceipt(identity, { providerLeaseId: identity.providerLeaseId, state: "stopped" }) },
}).returning();
return { source: source!, lease: lease!, companyId };
}
/** Workspace recovery suspends the stop receipt and restarts the sandbox to save its files. */
async function recoveryRestart(lease: typeof environmentLeases.$inferSelect) {
const pending = await prepareStoppedSandboxForRecovery(db, lease);
expect(pending).not.toBeNull();
return pending!;
}
async function restop(pending: typeof environmentLeases.$inferSelect) {
const attemptId = String(pending.metadata?.pendingCleanupAttemptId);
expect(await restoreSandboxStopAfterFailedRestart(db, pending, { attemptId })).not.toBeNull();
}
/** The containment part of the retry authority, as assertFailedNativeRetryState applies it. */
const authorize = (sourceRunId: string) => async () => {
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, sourceRunId));
if (!await taskChatRetryExecutionStopped(db, run!))
throw conflict("This chat request can't be repeated exactly as it was sent.", { code: "chat_failed_run_retry_not_authorized" });
return true;
};
it("loses the source's containment proof while recovery holds the sandbox, then regains it", async () => {
const { source, lease, companyId } = await stoppedSource();
expect(await taskChatRetryExecutionStopped(db, source)).toBe(true);
expect(await failedChatRetrySourceStopPending(db, companyId, source.id)).toBe(false);
const pending = await recoveryRestart(lease);
// This is the state the replay's dispatch recheck saw in the incident.
expect(await taskChatRetryExecutionStopped(db, source)).toBe(false);
expect(await failedChatRetrySourceStopPending(db, companyId, source.id)).toBe(true);
await restop(pending);
expect(await taskChatRetryExecutionStopped(db, source)).toBe(true);
expect(await failedChatRetrySourceStopPending(db, companyId, source.id)).toBe(false);
});
it("waits for the sandbox to stop again instead of failing the replay", async () => {
const { source, lease, companyId } = await stoppedSource();
const pending = await recoveryRestart(lease);
await expect(authorize(source.id)()).rejects.toMatchObject({ details: { code: "chat_failed_run_retry_not_authorized" } });
const onWait = vi.fn();
const attempt = vi.fn(authorize(source.id));
const restopped = new Promise<void>(resolve => setTimeout(() => void restop(pending).then(resolve), 150));
await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id,
signal: new AbortController().signal, onWait, pollMs: 25 }, attempt)).resolves.toBe(true);
await restopped;
expect(onWait).toHaveBeenCalledTimes(1);
expect(attempt.mock.calls.length).toBeGreaterThan(1);
});
it("checks once more when the stop settles between the refusal and the pending-stop read", async () => {
const { source, lease, companyId } = await stoppedSource();
const pending = await recoveryRestart(lease);
const check = authorize(source.id);
let first = true;
// The refused transaction has released its locks; recovery then commits the stop receipt.
const attempt = vi.fn(async () => {
if (!first) return check();
first = false;
await check().catch(async error => { await restop(pending); throw error; });
throw new Error("expected the first attempt to be refused");
});
const onWait = vi.fn();
await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id,
signal: new AbortController().signal, onWait, pollMs: 25 }, attempt)).resolves.toBe(true);
expect(attempt).toHaveBeenCalledTimes(2);
expect(onWait).not.toHaveBeenCalled();
});
it("refuses at the deadline when the sandbox stays held", async () => {
const { source, lease, companyId } = await stoppedSource();
await recoveryRestart(lease);
await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id,
signal: new AbortController().signal, timeoutMs: 100, pollMs: 25 }, authorize(source.id)))
.rejects.toMatchObject({ details: { code: "chat_failed_run_retry_not_authorized" } });
});
it("refuses without waiting when no stop of the source sandbox is in progress", async () => {
const { source, companyId } = await stoppedSource();
// An unreleased lease without a stop intent: the old executor may still be running.
await db.update(environmentLeases).set({ status: "active", cleanupStatus: null, releasedAt: null, metadata: { pluginId: "fake-plugin" } })
.where(eq(environmentLeases.heartbeatRunId, source.id));
const attempt = vi.fn(authorize(source.id));
const onWait = vi.fn();
await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id,
signal: new AbortController().signal, onWait, pollMs: 25 }, attempt)).rejects.toMatchObject({ status: 409 });
expect(attempt).toHaveBeenCalledTimes(1);
expect(onWait).not.toHaveBeenCalled();
});
it.each([
["a reasoned refusal", () => conflict("saved", { code: "chat_failed_run_retry_not_authorized", reason: "prior_result_publication" })],
["a mismatched retry selector", () => new FailedChatRunRetryAuthorizationError()],
["another error", () => new Error("database unavailable")],
])("does not wait on %s", async (_label, error) => {
const { source, lease, companyId } = await stoppedSource();
await recoveryRestart(lease);
const attempt = vi.fn(async () => { throw error(); });
await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id,
signal: new AbortController().signal, pollMs: 25 }, attempt)).rejects.toBeInstanceOf(Error);
expect(attempt).toHaveBeenCalledTimes(1);
});
it("does not wait when the run is not an exact replay of a failed run", async () => {
const { source, lease, companyId } = await stoppedSource();
await recoveryRestart(lease);
const attempt = vi.fn(authorize(source.id));
await expect(awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: null,
signal: new AbortController().signal, pollMs: 25 }, attempt)).rejects.toMatchObject({ status: 409 });
expect(attempt).toHaveBeenCalledTimes(1);
});
it("stops waiting when the run is stopped", async () => {
const { source, lease, companyId } = await stoppedSource();
await recoveryRestart(lease);
const control = new AbortController();
const waiting = awaitFailedChatRetrySourceStop(db, { companyId, sourceRunId: source.id,
signal: control.signal, pollMs: 10_000, onWait: () => control.abort(new Error("stopped")) }, authorize(source.id));
await expect(waiting).rejects.toThrow("stopped");
});
});
@@ -0,0 +1,85 @@
import { and, eq } from "drizzle-orm";
import { environmentLeases, type Db } from "@paperclipai/db";
import { HttpError } from "../../errors.js";
import { FailedChatRunRetryAuthorizationError } from "../durable-chat-wakeup.js";
import { readStopOnlyCleanup } from "../sandbox-stop-and-retain.js";
/** Long enough for workspace recovery to save files and stop the sandbox again. */
export const FAILED_CHAT_RETRY_SOURCE_STOP_WAIT_MS = 3 * 60_000;
export const FAILED_CHAT_RETRY_SOURCE_STOP_POLL_MS = 5_000;
/** The containment refusal carries no reason. Reasoned refusals (an accepted
* result, a prior publication or interaction, source authorization) and a
* mismatched retry selector do not change while a sandbox stops. */
function isSourceContainmentRefusal(error: unknown) {
if (!(error instanceof HttpError) || error instanceof FailedChatRunRetryAuthorizationError ||
error.status !== 409) return false;
const details = error.details as Record<string, unknown> | undefined;
return details?.code === "chat_failed_run_retry_not_authorized" && details.reason === undefined;
}
/** A stop of the failed run's sandbox is in progress. Workspace recovery holds
* one while it restarts the sandbox to save its files, and removes the stop
* receipt until it stops it again. This only permits a wait: containment is
* still decided by the unchanged retry authority on each attempt. */
export async function failedChatRetrySourceStopPending(db: Db, companyId: string, sourceRunId: string) {
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, sourceRunId),
));
return leases.some(lease => lease.provider !== "local" && lease.status === "pending_cleanup" &&
readStopOnlyCleanup(lease) !== null);
}
function delay(ms: number, signal: AbortSignal) {
return new Promise<void>((resolve, reject) => {
if (signal.aborted) return reject(signal.reason);
const onAbort = () => { clearTimeout(timer); reject(signal.reason); };
const timer = setTimeout(() => { signal.removeEventListener("abort", onAbort); resolve(); }, ms);
signal.addEventListener("abort", onAbort, { once: true });
});
}
/** Re-run an exact chat replay's authorization while the failed run's sandbox
* stop settles, instead of failing the replay on a transient missing receipt.
* The wait is bounded and stops with the run; the last refusal is returned.
* The pending stop is read before and after each refused attempt. A stop can
* settle after the refused transaction releases its locks, so a stop seen
* before the attempt earns one more attempt even when it is gone after. A
* refusal with no pending stop on either side is returned at once. */
export async function awaitFailedChatRetrySourceStop<T>(db: Db, input: {
companyId: string;
sourceRunId: string | null;
signal: AbortSignal;
onWait?: () => Promise<void> | void;
timeoutMs?: number;
pollMs?: number;
}, attempt: () => Promise<T>): Promise<T> {
const sourceRunId = input.sourceRunId;
if (!sourceRunId) return attempt();
const deadline = Date.now() + (input.timeoutMs ?? FAILED_CHAT_RETRY_SOURCE_STOP_WAIT_MS);
let announced = false;
let settledRetry = false;
for (;;) {
const pendingBefore = await failedChatRetrySourceStopPending(db, input.companyId, sourceRunId);
let pendingAfter: boolean;
try {
return await attempt();
} catch (error) {
if (!isSourceContainmentRefusal(error)) throw error;
pendingAfter = await failedChatRetrySourceStopPending(db, input.companyId, sourceRunId);
if ((!pendingBefore && !pendingAfter) || Date.now() >= deadline) throw error;
}
// The stop settled during the refused attempt: read the authority again
// now, once. A later flip waits for the normal poll interval.
if (!pendingAfter && !settledRetry) {
settledRetry = true;
continue;
}
if (!announced) {
announced = true;
await input.onWait?.();
}
await delay(Math.min(input.pollMs ?? FAILED_CHAT_RETRY_SOURCE_STOP_POLL_MS, Math.max(0, deadline - Date.now())),
input.signal);
}
}
+12 -2
View File
@@ -314,6 +314,7 @@ import {
watchLegacyControllerLease,
} from "./legacy-controller-lease.js";
import { retryChatControlAdmission } from "./chat-control-admission-retry.js";
import { awaitFailedChatRetrySourceStop } from "./chat-channels/failed-chat-retry-source-stop-wait.js";
import {
@@ -2268,8 +2269,17 @@ export function heartbeatService(
}
const runtime = await timePreparation("runtime_state", () => ensureRuntimeState(agent));
const context = parseObject(run.contextSnapshot);
// Workspace recovery can restart the failed run's sandbox to save its
// files while this replay prepares. Wait for that stop to settle; the
// retry authority still decides each attempt.
const authorizeFailedChatRetryExecution = (stage?: "dispatching") =>
retryChatControlAdmission(async () => {
awaitFailedChatRetrySourceStop(db, {
companyId: run.companyId,
sourceRunId: Object.hasOwn(context, "chatFailedRunRetry") ? readNonEmptyString(context.retryOfRunId) : null,
signal: executionControl.controller.signal,
onWait: () => appendRunEvent(run, { eventType: "lifecycle", stream: "system", level: "info",
message: "Waiting for the failed run's sandbox to stop before repeating the chat request" }),
}, () => retryChatControlAdmission(async () => {
executionControl.controller.signal.throwIfAborted();
await controllerLease.assertOwned(stage);
return db.transaction((tx) =>
@@ -2282,7 +2292,7 @@ export function heartbeatService(
runId: run.id,
contextSnapshot: context,
}));
});
}));
const isFailedChatRunRetry = await timePreparation("chat_retry_admission", () => authorizeFailedChatRetryExecution());
// Never adopt a chat-execution attestation supplied in a wake payload.
// Reviewed chat turns rebuild it from the current durable owner below.