## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Operators delete a sandbox environment, optionally with consent to
destroy its sandboxes (`DELETE
/api/environments/:id?destroyReusableSandboxLeases=true`)
> - A sandbox whose teardown failed or was interrupted is kept as a
`pending_cleanup` lease, and it blocks the delete
> - The cleanup sweep retries such a lease only after a cooldown of up
to 5 minutes
> - A consented delete therefore returned 409 `pending_sandbox_cleanup`
for minutes. A caller with a shorter deadline gave up, and the
environment and its sandbox stayed behind
> - This pull request makes the consented delete run one teardown
attempt for the environment's pending cleanups now, then re-check
> - The benefit is that a consented environment delete completes as soon
as the provider teardown succeeds
## Linked Issues or Issue Description
**What happened?**
After a restart chaos wave, the harness deleted the wave's Daytona
environment with `destroyReusableSandboxLeases=true`. The server
returned 409 `pending_sandbox_cleanup` every 3 seconds for the whole
120-second client deadline. A run lease had entered `pending_cleanup`
after its run failed, and the sweep's retry cooldown had not passed. The
environment was not deleted.
**Expected behavior**
With explicit consent, the delete tears down the environment's own
pending sandbox cleanups and completes when that teardown succeeds. A
teardown that fails, or one already in flight elsewhere, still blocks
the delete.
**Steps to reproduce**
1. Leave a sandbox lease of an environment in `pending_cleanup` with a
future retry time.
2. `DELETE /api/environments/:id?destroyReusableSandboxLeases=true`.
3. On master, the response is 409 `pending_sandbox_cleanup` until the
cooldown ends.
Refs #19696 (restart recovery for creation and teardown claims; this
builds on it).
## What Changed
- `heartbeat/recovery.ts`: `sweepPendingCleanupLeases` accepts
`environmentRetry: { environmentId, actorId }`. Like the existing
run-scoped explicit retry, it bypasses the cooldown for one attempt,
never takes over an in-flight attempt, and logs
`environment_lease.cleanup_retried` with reason `environment_deleted`.
It touches only that environment's leases.
- `routes/environments.ts`: when the delete has consent and the only
blockers are pending cleanup and reusable leases, it runs that retry
first, then re-reads the blast radius. The existing reusable-lease
destroy and the final guard are unchanged. The new dependency is a
narrow, optional `retryPendingEnvironmentCleanup` option; without it,
behavior is unchanged.
- `app.ts`: wires the option to the existing heartbeat service instance.
## Verification
- Red, then green:
- `environment-routes.test.ts`: "runs the environment's pending sandbox
cleanup now and deletes with explicit consent" and "keeps rejecting when
the consented pending cleanup attempt does not finish" fail before and
pass after. "does not run pending sandbox cleanup without consent or
past another blocker" passes in both.
- `heartbeat-pending-cleanup-sweep.test.ts`: "runs one attempt for a
deleted environment's cleanup during its cooldown, scoped to that
environment" fails before and passes after. It checks that another
environment's lease and an in-flight claim are untouched, and that the
activity log is written.
- `sandbox-creation-operations.test.ts`: "reclaims an issue-bound
creation after its run failed and the issue finished on a later run"
covers another chaos finding. A creation intent bound to an issue was
interrupted by SIGKILL 0.5 s after it committed, its run ended
`process_lost`, and a later run finished the issue. The test fails on
the code before #19696 and passes on master, so #19696 already covers
this case. The test is kept as a regression.
- After rebasing on master, `environment-routes`,
`heartbeat-pending-cleanup-sweep`, `agent-runner-qualification-routes`
and `sandbox-creation-operations` pass (192 tests). `tsc --noEmit`
reports no errors in the changed files.
## Risks
- A consented delete now makes provider teardown calls for pending
cleanups. Those sandboxes were already queued for destruction, and the
operator consented to destroy the environment's sandboxes.
- One explicit attempt counts toward the lease's attempt cap, the same
as a user Retry.
- The delete request now waits for those teardown attempts. Each attempt
uses the existing plugin timeouts.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, test execution). Extended reasoning
enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Server tests that need a real PostgreSQL process must use
./helpers/embedded-postgres.ts instead of constructing embedded-postgres
directly.
The shared helper creates a throwaway data directory and a reserved-safe
loopback port for each test database. This protects the live Paperclip
control-plane Postgres from server vitest runs; see PAP-2033 for the incident
that introduced this guard.
