mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm sandbox keeps an agent's Runner alive between runs, so the next run on the task starts fast > - When a run ends, the server closes the idle warm Runner in-process. A separate SQL worker stops the sandbox when its warm window ends (#21410, merged today) > - Both steps start at the same moment. The sandbox stop wins, so the idle close fails. The failed owner stays registered, and the next run throws `native_instruction_materialization_handoff_unavailable` before any provider work > - Recovery must just work: a follow-up run must never fail because a predecessor's warm copy is gone > - This pull request lets the idle close finish before the sandbox stops, prepares instructions fresh when the warm copy cannot be handed off, and releases an owner whose sandbox the host already gave up > - The benefit is that automatic follow-up runs start normally after a warm window ends ## Linked Issues or Issue Description Refs #21472 (base of this stacked PR), #21410 (introduced the warm-window stop). **What happened?** In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three automatic follow-up runs failed before any provider work with `native_instruction_materialization_handoff_unavailable`: run `5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a release swap. In each case, the predecessor's idle close had failed first with `provider_state_unreadable` / "Daytona sandbox lease … is no longer active". The cause: the warm-window stop from #21410 stopped the sandbox at the same moment that the idle close started. For the first run's predecessor, the close was requested at 20:30:34 and the sandbox got SIGTERM at 20:30:37.8. **What did you expect?** The follow-up run prepares its instructions fresh and continues. The warm-window stop does not interrupt the host's own idle close. **Steps to reproduce** 1. Run a warm Daytona task until it ends, so its warm window starts. 2. Let the warm window end while the idle timer closes the Runner. 3. Start a follow-up run on the same task. It fails with `native_instruction_materialization_handoff_unavailable`. Two of the three tasks then ended stranded. That downstream part (the claim-gate cancellation with `execution_reconciliation_required` and the stranded notice) is owned and fixed by another session. It is **not** in this PR. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts` `claimWarmNativeInstructionCopy`: after it retires a stale warm owner whose copy cannot be adopted, it returns `null`. The caller then prepares instructions fresh. Before, it threw `native_instruction_materialization_handoff_unavailable`. - `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window. The in-process idle close (close grace plus the 30 s proof read) finishes first. - `native-session-executor.ts` idle timer: when an idle close fails, it reads the lease rows for that sandbox. If every row is no longer `active` or `retained` (the host gave the sandbox up), it releases the in-process owner. The next run then prepares fresh. - `doc/execution-semantics.md`: one paragraph about the stop margin and fresh preparation. ## Verification Red → green (each test fails on the base `35ba76d0ef` and passes on this branch): - `native-session-executor.test.ts` › "hands off collection before probing or retiring a warm owner: unavailable" — red: throws `handoff_unavailable`; green: returns `null` (fresh preparation). - `native-session-executor.test.ts` › "forgets a warm owner whose idle close failed only once the host gave up its sandbox: pending_cleanup / released / retained" — red on all three; green: the owner is released for `pending_cleanup` and `released`, and kept (retired by the successor) for `retained`. - `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the warm runner time to finish before stopping its sandbox" — red: the sandbox stops 3 s after the window; green: it stops only after the margin. Two existing tests ("keeps a sandbox still inside its warm window, and wakes when the window ends", "reconciles a warm sandbox left running across a restart") now expect the deadline with the margin. Local runs on this branch: - `server`: `tsc --noEmit` clean. - `vitest run src/__tests__/warm-sandbox-expiry.test.ts src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed. - `vitest run src/services/native-runtime/native-session-executor.test.ts -t "hands off collection before probing or retiring a warm owner|forgets a warm owner whose idle close failed"`: 10/10 passed. Not run: a Daytona wave with this change. ## Risks - Warm sandboxes now stop up to 2 minutes later than the configured window. This is a small cost increase per warm sandbox. I chose a fixed margin over an in-process "is a close running?" check because the stop is pure SQL. The margin works when two servers run at the same time (the server that runs the stop may not be the server that owns the Runner), and it needs no busy-loop or cross-server signal. - The margin does not cover a re-armed idle timer or a close that takes longer than 2 minutes. Fresh preparation (fix 1) and the owner release (fix 3) cover those cases. - Not run on Daytona. The evidence is from the wave above and the unit tests. - In the wave, a `restore_failed` copy-back onto the just-restarted sandbox was also seen. I did not find its root cause. It is not changed here. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>