mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in remote sandboxes (for example Daytona) through reusable environment leases > - A warm run ends with its lease `retained` and its sandbox still running, so the next turn on the task starts warm > - After the warm window, only the provider's own idle auto-stop stopped that sandbox. Paperclip never stopped it > - On staging, idle Daytona sandboxes get provider activity about every 10 minutes. With a 12 minute auto-stop, they never stop and keep billing > - This pull request makes Paperclip stop a warm sandbox itself when its warm window ends and no live run holds it > - The benefit is that idle sandboxes stop on time, while warm turns inside the window stay warm ## Linked Issues or Issue Description **What happened?** Daytona sandboxes for ended warm runs stay in state `started` long after their warm window. On 2026-11-13 the staging Daytona org had 111 started sandboxes. One group of 23 idle sandboxes (auto-stop 12 min) got new provider activity exactly 10 minutes after their last use, so Daytona never auto-stopped them. **Expected behavior** Paperclip stops a sandbox when its warm window ends and no live run holds it. Daytona auto-stop stays only a backstop. **Steps to reproduce** 1. Run a warm `paperclip_runner` task on a reusable sandbox environment. 2. Let the run succeed. The lease becomes `retained` and the sandbox keeps running. 3. Wait past the warm window. Nothing in Paperclip stops the sandbox; it runs until the provider auto-stop fires, or forever if activity resets the provider idle clock. **Deployment mode** Hosted (Paperclip Cloud staging) with the Daytona sandbox provider plugin. **Additional context** Plugin Runner task leases (`driver: "plugin"`) are out of scope: their release path does not use the stop-only contract, so this change leaves them to their own cleanup. ## What Changed - New `server/src/services/warm-sandbox-expiry.ts` owns warm sandbox expiry: - The warm window is the run's resolved runner idle timeout (`resolveWarmRunnerIdleTimeoutMsForRun`). The window starts at the lease release (`lastUsedAt`). - Plugin Runner task leases (`driver: "plugin"`) are excluded. Their cleanup releases through the plugin, not the verified stop-only contract, so expiry could free the task's files. - A lease saved before this change has no recorded window. It uses the longest window a run can configure (24h), so it is never stopped early. - A candidate is a `retained` `reuse_by_environment` lease whose run has ended. It is held (never stopped, never a deadline) when another lease holds the same sandbox, a run on the task is `running` (it may be resuming this sandbox), or an operation is in progress on the lease (stop intent, export resume, restore recovery, cleanup claim, successor/supersede markers). - Due sandboxes go through the existing stop-only contract (`prepareSupersededWarmSandboxStop`). The cleanup sweep stops the exact sandbox at the provider. The lease then ends `released` as stopped and kept, which the next run on the task resumes. - Acquisition (plugin and built-in sandbox paths) records the run's warm window on the lease (`warmSandboxIdleTimeoutMs`). - `nextEnvironmentCleanupAt` includes the earliest warm window end. The environment cleanup worker sleeps until that deadline. It also wakes on every lease release and runs at boot, so a sandbox left running across a restart or deploy is reconciled. There is no interval scan. - `recoveredOrphanFirstAttemptSql` treats a warm-window stop as due at once, so the first stop does not wait for the 5 minute retry backoff. - The pending-cleanup sweep hands due warm sandboxes to the stop-only cleanup before it reads its rows, so one pass stops them. - Only this instance's own lease rows are judged. Nothing lists provider resources, so a sandbox of another company or environment is never touched. ## Verification - `pnpm exec vitest run src/__tests__/warm-sandbox-expiry.test.ts` (server): 9 passed. - Inside the warm window: kept, and the worker deadline is exactly the window end. - Past the window: stop intent recorded, first stop due now. - Through the cleanup sweep: the provider stop is dispatched once; the lease ends `released`, stopped and kept. - A live run's lease on the same sandbox: kept, and no deadline. - A run on the task is starting: kept. - The lease's own run is still running: kept. - A plugin Runner task lease past its window: left to its own cleanup, and no deadline. - Orphan after a restart: a lease retained 35h ago without a recorded window is stopped on the first (boot) pass; one retained 1h ago is kept with a deadline at 24h. - Red/green: with the wiring removed (sweep step, deadline, first-attempt marker), 4 of the 8 tests fail (window deadline, past-window stop, sweep stop, restart reconcile). With the change, all 8 pass. - CI on this PR covers the broad suites for the final revision. A local `pnpm verify` selected a very wide set (the change touches `heartbeat/recovery.ts`) and hit 30 s timeouts in unrelated suites under machine load, so I stopped it and rely on CI. - After deploy, I check the staging Daytona org: started counts for ended warm runs drop, and live or in-window sandboxes stay started. ## Risks - A run that starts on the task in the moment between the deadline check and the stop could resume a sandbox that is being stopped. A `running` run on the task holds the sandbox, and the stop intent is a compare-and-swap on the exact retained row, so this window is small. The same race already exists for the retained-orphan settle. - The first boot after deploy stops every retained warm sandbox that is older than 24h (or past its recorded window). This is the intended cleanup. It makes one stop call per lease, 25 leases per pass. - The stop-only contract disables Daytona auto-delete on the stopped sandbox (existing behavior of that contract). Daytona auto-archive still moves it to cold storage. The retained-orphan settle destroys it when the task no longer uses it. - The deadline query reads `retained` leases without a dedicated index, the same as the existing pending-cleanup deadline. It runs only when the cleanup worker wakes. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, Daytona API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>