Files
PaperClipAI/server/src
88469bd2bc fix(retry): wait for the failed run's sandbox to stop before repeating its chat request (#22423)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task from a connected chat (for example a GitHub pull request
review) can fail. Retry then repeats the exact failed chat request in a
new run
> - Before that run starts the agent, it checks again that the failed
run's executor is stopped. For a remote sandbox, the proof is a stop
receipt on the sandbox lease
> - Workspace recovery can restart the failed run's sandbox at the same
time, to save its files. While it does this, the lease has no stop
receipt
> - The check then refused the repeated request. The run failed for
good, although recovery stopped the sandbox again a few seconds later
> - This pull request makes the repeated request wait while a stop of
that sandbox is in progress, and then ask the retry authority again
> - The benefit is that a correct Retry is not lost because of a short
overlap with workspace recovery

## Linked Issues or Issue Description

Refs #22414. That PR fixes the next step of the same incident: the
system replacement run that copied the consumed retry selector. This PR
fixes the first refusal. The two PRs change different files and are
independent.

**What happened?**

On a hosted staging deploy, a GitHub review reply task failed
(`native_continuation_requires_reconciliation`). Its Daytona sandbox
went into workspace recovery. A Retry started a new run that repeats the
exact chat request, in a fresh isolated workspace. The run passed
admission and its first execution check. After about 14 seconds of
workspace preparation, the `dispatching` recheck refused it with 409
`chat_failed_run_retry_not_authorized` ("This chat request can't be
repeated exactly as it was sent."). No provider work started. The
recovery reported "Saved work recovered" 12 seconds later. The GitHub
reply request was not answered.

**Expected behavior**

The repeated request waits while the failed run's sandbox stops again,
and then continues. If the stop does not settle, the request is refused
as before.

**Steps to reproduce**

1. Make a native chat task fail on a remote sandbox with a recorded stop
receipt.
2. Start workspace recovery for that run.
`prepareStoppedSandboxForRecovery` moves the lease to `pending_cleanup`
and removes `remoteExecutionTermination` while the sandbox restarts.
3. Press Retry during that interval.
`taskChatRetryExecutionStopped(source)` is false, so
`assertFailedNativeRetryState` refuses the run at the `dispatching`
recheck.

The new test does steps 1 and 2 with the real lease transitions on an
embedded Postgres database.

**Paperclip version or commit**

`201abacd6ee5` (the staging release in the incident).

## What Changed

- Add
`server/src/services/chat-channels/failed-chat-retry-source-stop-wait.ts`:
- `failedChatRetrySourceStopPending()` is true when a non-local lease of
the failed run is `pending_cleanup` with a valid stop-only cleanup
intent (`readStopOnlyCleanup`). Workspace recovery holds such an intent
while it restarts and stops the sandbox.
- `awaitFailedChatRetrySourceStop()` runs the authorization again every
5 seconds, for up to 3 minutes, while that stop is pending. It reads the
pending stop before and after each refused attempt. If a stop was
pending before the attempt but is gone after it, the stop settled while
the refused transaction held no locks, so it reads the authorization
once more at once. If no stop is pending before or after, it returns the
refusal after one attempt, exactly as before. It waits only for the
reason-less containment refusal. It does not wait for a reasoned
refusal, a mismatched retry selector
(`FailedChatRunRetryAuthorizationError`), or any other error. It stops
when the run is stopped, and it returns the last refusal at the
deadline.
- In `heartbeat.ts`, both execution-phase checks of an exact replay (the
first check and the `dispatching` recheck) use this wait. The run log
records one line when the wait starts.

The wait grants no authority. Each attempt still runs the unchanged
`authorizeFailedChatRunRetryWake`. The controller lease is checked on
each attempt, as before.

## Verification

- `cd server && npx vitest run
src/services/chat-channels/failed-chat-retry-source-stop-wait.test.ts`:
10 tests pass. They use real lease rows and the real
`prepareStoppedSandboxForRecovery`,
`restoreSandboxStopAfterFailedRestart` and
`taskChatRetryExecutionStopped`.
- Red check: with the helper changed to make one attempt only (the old
behavior), the wait test fails with the incident's refusal: `promise
rejected "Error: This chat request can't be repeated…" { status: 409,
details: { code: 'chat_failed_run_retry_not_authorized' } }`. The abort
test also fails. Then I restored the helper.
- Red check for the review finding: with the old "no pending stop, so
refuse" line, the new test that commits the stop receipt between the
refusal and the pending-stop read fails with the old refusal. It passes
with the final read.
- A first version of that fix read the authorization again after every
refusal. CI showed this was wrong: `durable-chat-wakeup.test.ts` counts
authorization calls, and a budget-hold case in
`chat-channels.integration.test.ts` changed outcome. I reproduced both
failures locally with that version. The current version rechecks only
when a stop was pending, and those tests pass:
`durable-chat-wakeup.test.ts` (51), the CI-failing chat integration
cases by name (73), the retry-filtered chat integration run (261), and
the focused suites (300).
- Safety cases in the same file: refusal at the deadline when the
sandbox stays held; immediate refusal when no stop is in progress (an
active lease whose executor can still run); no wait for a reasoned
refusal, a selector mismatch, or another error; no wait for a run that
is not an exact replay; the wait stops when the run is stopped.
- Focused existing suites pass: `authorization-run-read.test.ts`,
`wake-queue/application/use-cases.test.ts`,
`__tests__/durable-chat-wakeup.test.ts`, `task-retry.test.ts`,
`budget-resume-continuation.test.ts` (299 tests in total, including the
new file).
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` selects almost the full suite because `heartbeat.ts` is
a shared module. I did not run that locally. Required CI supplies that
coverage.
- I did not do a live staging reproduction. The incident timing cannot
be produced on demand. The test reproduces the lease state that the
refusal saw.

## Risks

- A repeated request can now hold its own prepared sandbox for up to 3
more minutes before it fails. This happens only while a stop of the
failed run's sandbox is in progress.
- The wait can not let a second executor run. It changes only when the
check runs, and every attempt uses the unchanged authority.
- If workspace recovery takes longer than 3 minutes, the run is refused
as before.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
needed: the code and tests describe the behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 08:47:32 -08:00
..
…
…
…
…
…