Files
PaperClipAI/doc/idle-environment-cleanup.md
T
Nicky LeachandClaude Opus 5.5 717aae060a fix(environments): let same-run recovery create a replacement once its sandbox is proven stopped (#21223)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute on environment leases, and `createOwnedSandbox`
records each sandbox creation so that a lost reply never creates a
second sandbox
> - The record key is company, environment, run and request key. The
plugin request key is the environment fingerprint, so it is the same for
every attempt of one run
> - A same-run recovery can need a new environment after the first lease
is closed with successful cleanup and a termination receipt
> - The old record is still `attached` to the closed lease, so the
recovery fails with "Sandbox creation has already transferred ownership"
before the provider is called
> - This pull request lets the creation fence accept the same stop proof
that recovery already requires. A creation record whose sandbox is
proven stopped becomes resolved, and a resolved record no longer holds
its key
> - The benefit is that same-run recovery gets one replacement sandbox,
while unconfirmed cleanup still blocks a second sandbox

## Linked Issues or Issue Description

Fixes #21214

Refs #20755. That pull request fixes the same failure with a different
rule. It retries under a new key generation (`<key>#2`, `#3`, ...) when
the recorded lease has ended, and it accepts any ended status, including
`failed`, without a termination receipt. A lease whose cleanup is not
confirmed can then produce a second sandbox while the first one can
still run. This pull request keeps the original key and requires the
same stop proof that Core uses elsewhere
(`hasRemoteTerminationReceipt`). It does not add a key generation.

## What Changed

- `server/src/services/sandbox-creation-operations.ts`: when
`createOwnedSandbox` finds the key held by an `attached` operation whose
lease is not active, it checks the same stop proof that same-run
recovery requires before it acquires:
`hasRemoteTerminationReceipt(resolveRemoteTerminationLease(lease))`, for
the sandbox that this operation created (`creationProviderLeaseId`, or
the lease's `providerLeaseId`).
- With that proof, it runs the same admission transaction one more time.
That transaction first marks the old operation `resolved` (`WHERE id AND
state = 'attached' AND lease_id`), and then inserts the new operation.
  - Without that proof, it throws exactly as before.
- The check covers every source of a stop receipt: `releaseLease`, the
Runner `complete` operation (which writes receipts with direct SQL for
reusable leases), stop-and-retain settlement, and the receipt that
`resolveRemoteTerminationLease` derives for an expired reusable lease.
- A task replacement can leave an earlier operation attached to a lease
that now holds another sandbox. That operation is not proven and keeps
its key.
- Concurrent recoveries serialize on the row update. The second one
updates nothing, finds the first one's unresolved operation through the
partial index, and replays or refuses as before. There is one provider
call.
- Migration `0475_sandbox_creation_owner_unresolved.sql` recreates
`sandbox_creation_operations_owner_idx` under the same name as a partial
unique index (`state <> 'resolved'`). The drop and the create are in one
migration transaction. The Drizzle schema matches.
- The conflict lookup in `createOwnedSandbox` ignores resolved rows. An
active attached lease is still replayed. Pending and
`needs_reconciliation` rows still refuse. `releaseLease` does not
change.
- `doc/DATABASE.md` and `doc/idle-environment-cleanup.md` describe the
operation lifecycle.

## Verification

- `pnpm --filter @paperclipai/db generate`
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`
- `vitest run src/migration-snapshot-drift.test.ts` (db)
- `vitest run src/__tests__/sandbox-creation-operations.test.ts`: 48/48.
With the service change reverted, the 5 replacement tests fail.
- The 9 other server test files that read `sandbox_creation_operations`,
plus `environment-runtime.test.ts`: 564/564
- `pnpm check:migrations` passes

New tests in `sandbox-creation-operations.test.ts`:

- A stop receipt from `releaseLease` (sandbox driver, and the plugin
driver through `acquireRunLease` in the #21214 sequence), from the
`complete` operation's direct write on a retained reusable lease, and
derived only from the destroyed successor of an expired reusable lease
each let the next creation resolve the old operation and create exactly
one replacement. A retry replays that replacement. These tests fail
before the fix.
- Failed cleanup, success without a receipt, a receipt for another lease
or another provider task, a retained lease, and pending cleanup each
still throw "transferred ownership" with no provider call.
- An earlier task's operation that a task replacement left attached
keeps its key.
- Two concurrent recoveries make one provider call. This test fails
before the fix.
- Reapplying migration 0418 keeps the partial index.

I did not run `pnpm verify` or the full suite locally. CI runs them.

## Risks

- Migration: the drop and the rebuild of the index take an exclusive
lock on `sandbox_creation_operations`. The table has one small row for
each sandbox creation, so the lock is short.
- Rolling deploy: the migration is marked `online_compatible`. The
previous release uses an untargeted `ON CONFLICT DO NOTHING`, which also
matches a partial unique index. Its conflict lookup has no state filter,
so it can find a resolved row and throw. That fails closed and never
creates a second sandbox.
- A legacy runner-task operation without `creationProviderLeaseId`
matches any lease it is attached to. Runner-task operations record this
field before they attach, so this is only a theoretical gap.
- An operation stays `attached` until the next creation with the same
key checks it. This is the same as before when no recovery comes.
- Stop-and-retain settlement is not tested directly. The shared receipt
check covers it.
- The fence is unchanged for every case without a stop receipt.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents) and extended thinking. One sub-agent
wrote the change. A second session reviewed it before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:24:34 -08:00

13 KiB

Idle environment cleanup

Pending lease cleanup and historical resource verification use the existing transaction signals, delivery coordinator, and in-process scheduler. They no longer run unconditionally from the server maintenance tick. The run reaper performs its inline pending-cleanup sweep only after actually recovering a run or an orphaned active lease. Explicit user retry keeps its existing direct path.

Work and ownership

Duty Truth Trigger and next deadline Quiet condition
Failed teardown Existing environment_leases rows in pending_cleanup Commit signal; later of row backoff and saved retry/ownership deadline No pending lease or deferred orphan
Interrupted acquisition sandbox_creation_operations committed before provider creation Transaction signal; saved acquisition deadline; five-minute backoff after recovery failure Normal lease or one claimed cleanup lease owns the resource; unsupported uncertainty requires operator reconciliation
Historical provider verification Existing failed ephemeral leases and finished custom-image setups without matching cleanup proof Commit signal; five-minute retry while an unverified candidate remains No unverified candidates, including retained verified history

environment-cleanup-state.ts owns the deadline query and the shared safe reader for retry/ownership metadata. Cleanup claims and the scheduler use that same reader, including malformed-number handling and the existing maximum horizon. Startup retains the immediate scan, but never bypasses a saved in-flight claim or retry cooldown. Subsequent scans respect the five-minute row backoff. Claims, provider readiness, Stop authority, workspace retention, copyback, receipt validation, and escalation rules stay with their current owners.

environment-cleanup-work.ts registers two independent workers. Their generation counters preserve a wake until successful reconciliation, including a first write during drain, changes during a sweep, and failure followed by drain. The existing heartbeat interval bounds pending-cleanup starts after bursts of writes. Historical verification has a separate minimum five-minute spacing, including commit-triggered passes, and does not delay server readiness. No timer remains once there is no obligation. A self-generated write can require a final reconciliation before quiet. A failed sweep retains a retry. Shutdown unsubscribes local buffer wakes; the delivery coordinator stops subscriptions/timers and joins active work.

Producer coverage

The shared writeEnvironmentCleanup helper registers and awaits intent before a write in a transaction. Existing transaction owners call notifyEnvironmentCleanup(tx) directly. Nested calls retain outer transaction settlement and uncertainty handling through the shared DB implementation.

  • environments.ts: orphan insertion, release, metadata replacement, and run-wide release. These stores cover normal provider teardown and failed acquisition.
  • environment-runtime.ts: reusable sandbox cleanup claims and their settlement, and Stop/export cleanup claims. Renewal of an already active claim retains its bounded timer; it cannot create new pending work.
  • heartbeat/recovery.ts and heartbeat/run-control.ts: orphan-active recovery, retry settlement, and explicit recovery of historical leases.
  • environment-task-runtime.ts: unconfirmed plugin task termination and plugin replacement creations. Replacement intent commits before the provider call.
  • sandbox-creation-operations.ts: admission, atomic lease transfer, and bounded recovery. An unavailable database stops admission before external creation.
  • sandbox-stop-and-retain.ts: persisted Stop intent and its settlement.
  • native-workspace-export-resume.ts and native-workspace-export-retry.ts: terminal-export cleanup, resumed-export cleanup claims, and settlement.
  • workspace-recovery.ts: verified copyback schedules destruction.
  • native-workspace-sync.ts and the native session executor's backup stamp: writes that can invalidate an existing historical row-version proof.
  • environment-custom-images.ts: provider state, terminal state, and promotion. Initial active setup insertion is not a historical verification candidate.

Other lease metadata writes neither create pending cleanup nor change the row-version/provider identity of a matching historical proof. Changes to an already pending claim remain covered by its existing deadline. Deletion removes work; an already scheduled reconciliation observes that removal. Independent SQL writers and another server process are not covered by these process-local signals.

Creation ownership

A creation operation retains company, original environment, run, request identity, provider, and nonsecret configuration references. Its environment FK uses SET NULL; company/run/task identifiers remain after those records are removed. One operation owns each company/environment/run/request until it is resolved. A retry of an attached operation returns its active lease. Pending or unresolved ownership never authorizes another create. An attached operation resolves when its lease is proven stopped, checked at the next creation with the same key: the closed lease must carry the stop proof same-run recovery requires (remoteExecutionHasStopped), a termination receipt for the sandbox this operation created, whether release, the Runner complete operation, or a destroyed reusable successor supplied it. That creation resolves the operation and records its replacement in one transaction, so concurrent recoveries make one provider call. A recovered cleanup operation resolves when its cleanup lease settles. Failed cleanup, success without a receipt, a receipt for another sandbox, a retained lease, and pending cleanup keep the operation and its key. Normal success transfers operation ownership and the lease in one transaction. Current company binding and run authority are checked again at attachment. Declared secret-reference fields retain only their references and nonsecret value hashes. Recovery refuses lookup under a changed credential value; it records manual reconciliation instead of treating a missing resource in a new account as proof. A missing, deleted, disabled, or archived saved secret, a missing or inactive saved version, or an explicitly disabled/coming-soon provider vault also records manual reconciliation before provider lookup. Admin-disabled credentials require operator reconciliation; recovery does not reactivate or use them. Temporary secret-provider or database failures retain the pending operation and its retry deadline.

Recovery never resumes partially prepared execution. It asks a capable provider to inspect the stable creation identity, validates its ownership envelope, and transfers one cleanup lease under the operation lock. The existing claimed cleanup worker owns deletion and confirms resolution. Failed lookup/transfer records a bounded retry and permits other operations to proceed. An attached operation cannot transfer to cleanup. Pending creation blocks instance sleep until its obligation transfers.

Daytona version 0.1.10 declares creationOperationVersion 1. It uses the committed operation UUID in its existing creation name and labels. Recovery uses an explicit creationOperation request on the existing read-only inspect hook and strict failed-creation deletion path. Normal lease inspection still uses its recorded resource ID; creationOperationId metadata is provenance only. A missing name alone cannot prove a late allocation will never appear. Missing identity or an unsupported provider records needs_reconciliation and an operation-scoped warning. A deleted company also records manual reconciliation before credential lookup or cleanup transfer. A company lock fences deletion during the transfer transaction. That manual diagnostic has no automatic timer and does not prevent idle. The row must be reconciled before retrying the same request. Other providers do not yet claim automatic creation lookup. Generic plugin operations retain identity only, including a returned task resource ID before expiry checks or lease attachment. A failed or ambiguous attachment keeps that known ID for reconciliation; it does not authorize unconditional release. These operations do not copy credentials, resolved configuration, or provider payloads into the operation record. A lost provider reply can still leave the resource ID unknown.

Legacy deferred orphan buffers

Runtimes that hold an unwritten orphan register their flusher in a registry scoped to the shared database handle. Registration wakes the worker without contacting the database. A sweep flushes registered runtimes as well as its own startup spool; this reaches a producer runtime's buffer even when it differs from the heartbeat's runtime. The registry retains only runtimes with unresolved buffer ownership.

Unreadable directories, records, and malformed records retain a retry; a missing spool directory is an empty spool. A spool read failure still allows other runtimes to flush their buffers and persisted leases to clean up, then fails the sweep so recovery stays scheduled. Each concrete cleanup pass reloads existing legacy spool entries, including entries beyond buffer capacity. Normal acquisition and plugin task replacement do not append new fallback files. Retained entries remain available for the separate cutover inventory. No empty-spool filesystem or database polling is introduced. Existing limits on a simultaneous database, spool, and buffer-capacity failure remain; this is not a new durable queue.

Verification and boundaries

The PostgreSQL worker tests exercise quiet configured state and verified history, real store activation, persisted backoff, failed teardown and restart, nested commit/rollback, drain/resume, cross-runtime deferred work, failed spool reads, shutdown joining, provider uncertainty, and historical pagination. Existing runtime, reaper, Stop, custom-image, and workspace tests cover the underlying cleanup authority and provider contracts.

The runtime fixture now gives each simulated instance its own DB handle and spool. Its journal-write failure injection targets the transaction boundary rather than the direct update that moved inside that transaction. Fake-clock worker tests wait for PostgreSQL I/O and distinguish an executing sweep from a retained next wake.

This does not migrate discovery of orphaned active leases or legacy shutdown workspace settlement from the broader run-recovery loop. Custom-image active session expiry also remains on its existing tick; only historical verification moves here. Other maintenance duties, external probes, and connected transports still affect idle traffic. No Cloud alarm coverage or host-sleep exemption changes. These tests prove the changed workers become quiet and recover; they do not prove deployed suspension or fleet savings.

Custom-image setup expiry

Setup expiry uses a separate custom-image-expiry worker. The custom-image service owns the active-status predicate, the earliest saved expiresAt, and provider cancellation. Startup registers the worker in the existing enabled heartbeat housekeeping path; the maintenance tick no longer calls its sweep.

All supported setup producers live in environment-custom-images.ts: initial creation, provider refresh, cancellation/failure status, and promotion. They register intent in their owning transaction. A provider refresh can move the deadline earlier; a terminal transition removes it. Historical verification metadata does not change active expiry eligibility and does not need this signal. Environment deletion can only remove obligations; an already scheduled pass observes the removal. Independent SQL and other DB roots are outside the existing process-local notification contract.

No active dated session means no recurring reads or provider calls. Overdue batches and failures retain a bounded retry, while first writes and failed reconciliation survive drain. A missing expiry retains the existing behavior (no inferred expiration); sleep safety still treats unsupported live resources conservatively. Provider cancellation, failure state, and historical resource verification keep their existing semantics. No Cloud alarm coverage changes.

Historical retry test observation

The historical verification worker intentionally does not delay server readiness. Its provider sweep finishes before the following database pending-state read registers a retry deadline. The regression test now waits for that future deadline before inspecting it; sweep completion alone is not scheduler settlement. The runtime worker, retry interval, and readiness contract are unchanged.

A controlled 150ms delay before the real pending-state query reproduced the current CI failure: the immediate assertion observed a null deadline. The corrected assertion passed with the same delay. The temporary delay was removed, and all 14 cleanup tests passed against an isolated PostgreSQL database. The existing external test-database hook is carried at the stack foundation so intermediate branches can use this same verification path.

An initial attempt at the foundation could not start embedded PostgreSQL because that earlier branch lacked the external fixture hook. A broader fixture wait for all idle holds to clear then failed three cases that legitimately retain pending cleanup obligations. That wait was removed. Only the historical deadline assertion changed; its requirement is now a deadline strictly in the future. The complete suite passes with the original readiness and idle-hold behavior preserved.