Files
PaperClipAI/server
DottaandClaude Opus 5.5 24302a09ad fix(environments): stop warm sandboxes once their warm window ends (#21410)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes (for example Daytona) through
reusable environment leases
> - A warm run ends with its lease `retained` and its sandbox still
running, so the next turn on the task starts warm
> - After the warm window, only the provider's own idle auto-stop
stopped that sandbox. Paperclip never stopped it
> - On staging, idle Daytona sandboxes get provider activity about every
10 minutes. With a 12 minute auto-stop, they never stop and keep billing
> - This pull request makes Paperclip stop a warm sandbox itself when
its warm window ends and no live run holds it
> - The benefit is that idle sandboxes stop on time, while warm turns
inside the window stay warm

## Linked Issues or Issue Description

**What happened?**
Daytona sandboxes for ended warm runs stay in state `started` long after
their warm window. On 2026-11-13 the staging Daytona org had 111 started
sandboxes. One group of 23 idle sandboxes (auto-stop 12 min) got new
provider activity exactly 10 minutes after their last use, so Daytona
never auto-stopped them.

**Expected behavior**
Paperclip stops a sandbox when its warm window ends and no live run
holds it. Daytona auto-stop stays only a backstop.

**Steps to reproduce**
1. Run a warm `paperclip_runner` task on a reusable sandbox environment.
2. Let the run succeed. The lease becomes `retained` and the sandbox
keeps running.
3. Wait past the warm window. Nothing in Paperclip stops the sandbox; it
runs until the provider auto-stop fires, or forever if activity resets
the provider idle clock.

**Deployment mode**
Hosted (Paperclip Cloud staging) with the Daytona sandbox provider
plugin.

**Additional context**
Plugin Runner task leases (`driver: "plugin"`) are out of scope: their
release path does not use the stop-only contract, so this change leaves
them to their own cleanup.

## What Changed

- New `server/src/services/warm-sandbox-expiry.ts` owns warm sandbox
expiry:
- The warm window is the run's resolved runner idle timeout
(`resolveWarmRunnerIdleTimeoutMsForRun`). The window starts at the lease
release (`lastUsedAt`).
- Plugin Runner task leases (`driver: "plugin"`) are excluded. Their
cleanup releases through the plugin, not the verified stop-only
contract, so expiry could free the task's files.
- A lease saved before this change has no recorded window. It uses the
longest window a run can configure (24h), so it is never stopped early.
- A candidate is a `retained` `reuse_by_environment` lease whose run has
ended. It is held (never stopped, never a deadline) when another lease
holds the same sandbox, a run on the task is `running` (it may be
resuming this sandbox), or an operation is in progress on the lease
(stop intent, export resume, restore recovery, cleanup claim,
successor/supersede markers).
- Due sandboxes go through the existing stop-only contract
(`prepareSupersededWarmSandboxStop`). The cleanup sweep stops the exact
sandbox at the provider. The lease then ends `released` as stopped and
kept, which the next run on the task resumes.
- Acquisition (plugin and built-in sandbox paths) records the run's warm
window on the lease (`warmSandboxIdleTimeoutMs`).
- `nextEnvironmentCleanupAt` includes the earliest warm window end. The
environment cleanup worker sleeps until that deadline. It also wakes on
every lease release and runs at boot, so a sandbox left running across a
restart or deploy is reconciled. There is no interval scan.
- `recoveredOrphanFirstAttemptSql` treats a warm-window stop as due at
once, so the first stop does not wait for the 5 minute retry backoff.
- The pending-cleanup sweep hands due warm sandboxes to the stop-only
cleanup before it reads its rows, so one pass stops them.
- Only this instance's own lease rows are judged. Nothing lists provider
resources, so a sandbox of another company or environment is never
touched.

## Verification

- `pnpm exec vitest run src/__tests__/warm-sandbox-expiry.test.ts`
(server): 9 passed.
- Inside the warm window: kept, and the worker deadline is exactly the
window end.
  - Past the window: stop intent recorded, first stop due now.
- Through the cleanup sweep: the provider stop is dispatched once; the
lease ends `released`, stopped and kept.
  - A live run's lease on the same sandbox: kept, and no deadline.
  - A run on the task is starting: kept.
  - The lease's own run is still running: kept.
- A plugin Runner task lease past its window: left to its own cleanup,
and no deadline.
- Orphan after a restart: a lease retained 35h ago without a recorded
window is stopped on the first (boot) pass; one retained 1h ago is kept
with a deadline at 24h.
- Red/green: with the wiring removed (sweep step, deadline,
first-attempt marker), 4 of the 8 tests fail (window deadline,
past-window stop, sweep stop, restart reconcile). With the change, all 8
pass.
- CI on this PR covers the broad suites for the final revision. A local
`pnpm verify` selected a very wide set (the change touches
`heartbeat/recovery.ts`) and hit 30 s timeouts in unrelated suites under
machine load, so I stopped it and rely on CI.
- After deploy, I check the staging Daytona org: started counts for
ended warm runs drop, and live or in-window sandboxes stay started.

## Risks

- A run that starts on the task in the moment between the deadline check
and the stop could resume a sandbox that is being stopped. A `running`
run on the task holds the sandbox, and the stop intent is a
compare-and-swap on the exact retained row, so this window is small. The
same race already exists for the retained-orphan settle.
- The first boot after deploy stops every retained warm sandbox that is
older than 24h (or past its recorded window). This is the intended
cleanup. It makes one stop call per lease, 25 leases per pass.
- The stop-only contract disables Daytona auto-delete on the stopped
sandbox (existing behavior of that contract). Daytona auto-archive still
moves it to cold storage. The retained-orphan settle destroys it when
the task no longer uses it.
- The deadline query reads `retained` leases without a dedicated index,
the same as the existing pending-cleanup deadline. It runs only when the
cleanup worker wakes.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, Daytona API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:47:43 -06:00
..