mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task from a connected chat (for example a GitHub pull request review) can fail. Retry then repeats the exact failed chat request in a new run > - Before that run starts the agent, it checks again that the failed run's executor is stopped. For a remote sandbox, the proof is a stop receipt on the sandbox lease > - Workspace recovery can restart the failed run's sandbox at the same time, to save its files. While it does this, the lease has no stop receipt > - The check then refused the repeated request. The run failed for good, although recovery stopped the sandbox again a few seconds later > - This pull request makes the repeated request wait while a stop of that sandbox is in progress, and then ask the retry authority again > - The benefit is that a correct Retry is not lost because of a short overlap with workspace recovery ## Linked Issues or Issue Description Refs #22414. That PR fixes the next step of the same incident: the system replacement run that copied the consumed retry selector. This PR fixes the first refusal. The two PRs change different files and are independent. **What happened?** On a hosted staging deploy, a GitHub review reply task failed (`native_continuation_requires_reconciliation`). Its Daytona sandbox went into workspace recovery. A Retry started a new run that repeats the exact chat request, in a fresh isolated workspace. The run passed admission and its first execution check. After about 14 seconds of workspace preparation, the `dispatching` recheck refused it with 409 `chat_failed_run_retry_not_authorized` ("This chat request can't be repeated exactly as it was sent."). No provider work started. The recovery reported "Saved work recovered" 12 seconds later. The GitHub reply request was not answered. **Expected behavior** The repeated request waits while the failed run's sandbox stops again, and then continues. If the stop does not settle, the request is refused as before. **Steps to reproduce** 1. Make a native chat task fail on a remote sandbox with a recorded stop receipt. 2. Start workspace recovery for that run. `prepareStoppedSandboxForRecovery` moves the lease to `pending_cleanup` and removes `remoteExecutionTermination` while the sandbox restarts. 3. Press Retry during that interval. `taskChatRetryExecutionStopped(source)` is false, so `assertFailedNativeRetryState` refuses the run at the `dispatching` recheck. The new test does steps 1 and 2 with the real lease transitions on an embedded Postgres database. **Paperclip version or commit** `201abacd6ee5` (the staging release in the incident). ## What Changed - Add `server/src/services/chat-channels/failed-chat-retry-source-stop-wait.ts`: - `failedChatRetrySourceStopPending()` is true when a non-local lease of the failed run is `pending_cleanup` with a valid stop-only cleanup intent (`readStopOnlyCleanup`). Workspace recovery holds such an intent while it restarts and stops the sandbox. - `awaitFailedChatRetrySourceStop()` runs the authorization again every 5 seconds, for up to 3 minutes, while that stop is pending. It reads the pending stop before and after each refused attempt. If a stop was pending before the attempt but is gone after it, the stop settled while the refused transaction held no locks, so it reads the authorization once more at once. If no stop is pending before or after, it returns the refusal after one attempt, exactly as before. It waits only for the reason-less containment refusal. It does not wait for a reasoned refusal, a mismatched retry selector (`FailedChatRunRetryAuthorizationError`), or any other error. It stops when the run is stopped, and it returns the last refusal at the deadline. - In `heartbeat.ts`, both execution-phase checks of an exact replay (the first check and the `dispatching` recheck) use this wait. The run log records one line when the wait starts. The wait grants no authority. Each attempt still runs the unchanged `authorizeFailedChatRunRetryWake`. The controller lease is checked on each attempt, as before. ## Verification - `cd server && npx vitest run src/services/chat-channels/failed-chat-retry-source-stop-wait.test.ts`: 10 tests pass. They use real lease rows and the real `prepareStoppedSandboxForRecovery`, `restoreSandboxStopAfterFailedRestart` and `taskChatRetryExecutionStopped`. - Red check: with the helper changed to make one attempt only (the old behavior), the wait test fails with the incident's refusal: `promise rejected "Error: This chat request can't be repeated…" { status: 409, details: { code: 'chat_failed_run_retry_not_authorized' } }`. The abort test also fails. Then I restored the helper. - Red check for the review finding: with the old "no pending stop, so refuse" line, the new test that commits the stop receipt between the refusal and the pending-stop read fails with the old refusal. It passes with the final read. - A first version of that fix read the authorization again after every refusal. CI showed this was wrong: `durable-chat-wakeup.test.ts` counts authorization calls, and a budget-hold case in `chat-channels.integration.test.ts` changed outcome. I reproduced both failures locally with that version. The current version rechecks only when a stop was pending, and those tests pass: `durable-chat-wakeup.test.ts` (51), the CI-failing chat integration cases by name (73), the retry-filtered chat integration run (261), and the focused suites (300). - Safety cases in the same file: refusal at the deadline when the sandbox stays held; immediate refusal when no stop is in progress (an active lease whose executor can still run); no wait for a reasoned refusal, a selector mismatch, or another error; no wait for a run that is not an exact replay; the wait stops when the run is stopped. - Focused existing suites pass: `authorization-run-read.test.ts`, `wake-queue/application/use-cases.test.ts`, `__tests__/durable-chat-wakeup.test.ts`, `task-retry.test.ts`, `budget-resume-continuation.test.ts` (299 tests in total, including the new file). - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify` selects almost the full suite because `heartbeat.ts` is a shared module. I did not run that locally. Required CI supplies that coverage. - I did not do a live staging reproduction. The incident timing cannot be produced on demand. The test reproduces the lease state that the refusal saw. ## Risks - A repeated request can now hold its own prepared sandbox for up to 3 more minutes before it fails. This happens only while a stop of the failed run's sandbox is in progress. - The wait can not let a second executor run. It changes only when the check runs, and every attempt uses the unchanged authority. - If workspace recovery takes longer than 3 minutes, the run is refused as before. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (not needed: the code and tests describe the behavior) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>