Files
PaperClipAI/server
DottaandClaude Opus 5.5 b03ea621c6 fix(environments): retry a teardown interrupted by server death at once (#19749)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a remote run, Paperclip destroys its provider sandbox. The
teardown first claims the lease with a 15-minute ownership fence, so no
other cleanup runs at the same time
> - #19696 made restart recovery end a fence when the claiming server
process is proven to have exited. It only does that for claims that
record the claimer
> - The run-release teardown and the stop-only teardown did not record
the claimer
> - When the server died during a run-release teardown, the sandbox
therefore stayed up until the 15-minute fence expired
> - This pull request records the claimer on those claims, and stops a
teardown attempt lost to a dead server from counting as a failed attempt
> - The benefit is that a restart during sandbox teardown destroys the
sandbox at the next startup sweep

## Linked Issues or Issue Description

**What happened?**
In a Daytona restart chaos wave, a run finished at 05:34:48. Its release
claimed the lease and called the provider, and a SIGKILL stopped the
server 2 seconds later. After restart the lease stayed `pending_cleanup`
with an in-flight claim. The sandbox was destroyed at 05:49:49, when the
15-minute claim fence expired. There was no provider error.

**Expected behavior**
A teardown interrupted by the server's death is retried by the next
startup sweep. It is not counted as a failed attempt. A real provider
error keeps the normal backoff.

**Steps to reproduce**
1. Release a run's sandbox lease with destroy, and hold the provider
call.
2. Stop the server during the provider call.
3. Restart. On master, the sweep skips the lease until the fence expires
15 minutes after the claim.

Refs #19696.

## What Changed

- `environment-runtime.ts`: the run-release teardown claim
(`destroyClaimedSandboxLease`) and the stop-only teardown claim save
`pendingCleanupController`, bound to the claim's attempt id, the same
way sweep claims do since #19696.
- `heartbeat/recovery.ts`: sweep claims mark that they counted an
attempt. When the sweep ends the fence of a lost claim, it gives that
attempt back, so a teardown lost to a dead server does not move toward
the cap or the long cooldown. Claims from other paths did not count an
attempt, so nothing changes for them.
- Workspace-repair and export-retry fences are unchanged. They protect
workspace work, not teardown.

## Verification

- Red, then green:
- `environment-runtime.test.ts` → "retries a run-release teardown at the
next sweep after the releasing server dies". The provider call is held
during `releaseRunLeases(..., "destroy")`, the claim is marked as from
an exited server, and a restarted sweep runs. On master the claim has no
controller record and the sweep skips the lease. With the fix, the sweep
destroys it at once and records attempt 1.
- `heartbeat-pending-cleanup-sweep.test.ts` → "does not count a sweep
attempt lost to a dead server as a failed attempt". On master the retry
is recorded as attempt 4; with the fix it is attempt 3 again.
- `environment-runtime`, `heartbeat-pending-cleanup-sweep`,
`sandbox-creation-operations` and `environment-routes` pass (367 tests).
`tsc --noEmit` for `server` passes.

## Risks

- The proof that a claimer exited is the same-PID-space proof from
#19696. A live, foreign or unidentified claimer keeps its fence.
- Giving back a counted attempt applies only after that proof. A
teardown that fails with a provider error still counts and backs off.
- No schema change. The new fields live in existing JSON metadata.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 01:08:11 -06:00
..