## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs execute on environment leases, and `createOwnedSandbox` records each sandbox creation so that a lost reply never creates a second sandbox > - The record key is company, environment, run and request key. The plugin request key is the environment fingerprint, so it is the same for every attempt of one run > - A same-run recovery can need a new environment after the first lease is closed with successful cleanup and a termination receipt > - The old record is still `attached` to the closed lease, so the recovery fails with "Sandbox creation has already transferred ownership" before the provider is called > - This pull request lets the creation fence accept the same stop proof that recovery already requires. A creation record whose sandbox is proven stopped becomes resolved, and a resolved record no longer holds its key > - The benefit is that same-run recovery gets one replacement sandbox, while unconfirmed cleanup still blocks a second sandbox ## Linked Issues or Issue Description Fixes #21214 Refs #20755. That pull request fixes the same failure with a different rule. It retries under a new key generation (`<key>#2`, `#3`, ...) when the recorded lease has ended, and it accepts any ended status, including `failed`, without a termination receipt. A lease whose cleanup is not confirmed can then produce a second sandbox while the first one can still run. This pull request keeps the original key and requires the same stop proof that Core uses elsewhere (`hasRemoteTerminationReceipt`). It does not add a key generation. ## What Changed - `server/src/services/sandbox-creation-operations.ts`: when `createOwnedSandbox` finds the key held by an `attached` operation whose lease is not active, it checks the same stop proof that same-run recovery requires before it acquires: `hasRemoteTerminationReceipt(resolveRemoteTerminationLease(lease))`, for the sandbox that this operation created (`creationProviderLeaseId`, or the lease's `providerLeaseId`). - With that proof, it runs the same admission transaction one more time. That transaction first marks the old operation `resolved` (`WHERE id AND state = 'attached' AND lease_id`), and then inserts the new operation. - Without that proof, it throws exactly as before. - The check covers every source of a stop receipt: `releaseLease`, the Runner `complete` operation (which writes receipts with direct SQL for reusable leases), stop-and-retain settlement, and the receipt that `resolveRemoteTerminationLease` derives for an expired reusable lease. - A task replacement can leave an earlier operation attached to a lease that now holds another sandbox. That operation is not proven and keeps its key. - Concurrent recoveries serialize on the row update. The second one updates nothing, finds the first one's unresolved operation through the partial index, and replays or refuses as before. There is one provider call. - Migration `0475_sandbox_creation_owner_unresolved.sql` recreates `sandbox_creation_operations_owner_idx` under the same name as a partial unique index (`state <> 'resolved'`). The drop and the create are in one migration transaction. The Drizzle schema matches. - The conflict lookup in `createOwnedSandbox` ignores resolved rows. An active attached lease is still replayed. Pending and `needs_reconciliation` rows still refuse. `releaseLease` does not change. - `doc/DATABASE.md` and `doc/idle-environment-cleanup.md` describe the operation lifecycle. ## Verification - `pnpm --filter @paperclipai/db generate` - `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter @paperclipai/server typecheck` - `vitest run src/migration-snapshot-drift.test.ts` (db) - `vitest run src/__tests__/sandbox-creation-operations.test.ts`: 48/48. With the service change reverted, the 5 replacement tests fail. - The 9 other server test files that read `sandbox_creation_operations`, plus `environment-runtime.test.ts`: 564/564 - `pnpm check:migrations` passes New tests in `sandbox-creation-operations.test.ts`: - A stop receipt from `releaseLease` (sandbox driver, and the plugin driver through `acquireRunLease` in the #21214 sequence), from the `complete` operation's direct write on a retained reusable lease, and derived only from the destroyed successor of an expired reusable lease each let the next creation resolve the old operation and create exactly one replacement. A retry replays that replacement. These tests fail before the fix. - Failed cleanup, success without a receipt, a receipt for another lease or another provider task, a retained lease, and pending cleanup each still throw "transferred ownership" with no provider call. - An earlier task's operation that a task replacement left attached keeps its key. - Two concurrent recoveries make one provider call. This test fails before the fix. - Reapplying migration 0418 keeps the partial index. I did not run `pnpm verify` or the full suite locally. CI runs them. ## Risks - Migration: the drop and the rebuild of the index take an exclusive lock on `sandbox_creation_operations`. The table has one small row for each sandbox creation, so the lock is short. - Rolling deploy: the migration is marked `online_compatible`. The previous release uses an untargeted `ON CONFLICT DO NOTHING`, which also matches a partial unique index. Its conflict lookup has no state filter, so it can find a resolved row and throw. That fails closed and never creates a second sandbox. - A legacy runner-task operation without `creationProviderLeaseId` matches any lease it is attached to. Runner-task operations record this field before they attach, so this is only a theoretical gap. - An operation stays `attached` until the next creation with the same key checks it. This is the same as before when no recovery comes. - Stop-and-retain settlement is not tested directly. The shared receipt check covers it. - The fence is unchanged for every case without a stop receipt. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, sub-agents) and extended thinking. One sub-agent wrote the change. A second session reviewed it before this pull request. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
13 KiB
Idle environment cleanup
Pending lease cleanup and historical resource verification use the existing transaction signals, delivery coordinator, and in-process scheduler. They no longer run unconditionally from the server maintenance tick. The run reaper performs its inline pending-cleanup sweep only after actually recovering a run or an orphaned active lease. Explicit user retry keeps its existing direct path.
Work and ownership
| Duty | Truth | Trigger and next deadline | Quiet condition |
|---|---|---|---|
| Failed teardown | Existing environment_leases rows in pending_cleanup |
Commit signal; later of row backoff and saved retry/ownership deadline | No pending lease or deferred orphan |
| Interrupted acquisition | sandbox_creation_operations committed before provider creation |
Transaction signal; saved acquisition deadline; five-minute backoff after recovery failure | Normal lease or one claimed cleanup lease owns the resource; unsupported uncertainty requires operator reconciliation |
| Historical provider verification | Existing failed ephemeral leases and finished custom-image setups without matching cleanup proof | Commit signal; five-minute retry while an unverified candidate remains | No unverified candidates, including retained verified history |
environment-cleanup-state.ts owns the deadline query and the shared safe reader
for retry/ownership metadata. Cleanup claims and the scheduler use that same
reader, including malformed-number handling and the existing maximum horizon.
Startup retains the immediate scan, but never bypasses a saved in-flight claim or
retry cooldown. Subsequent scans respect the five-minute row backoff. Claims,
provider readiness, Stop authority, workspace retention, copyback, receipt
validation, and escalation rules stay with their current owners.
environment-cleanup-work.ts registers two independent workers. Their generation
counters preserve a wake until successful reconciliation, including a first write
during drain, changes during a sweep, and failure followed by drain. The existing
heartbeat interval bounds pending-cleanup starts after bursts of writes. Historical
verification has a separate minimum five-minute spacing, including commit-triggered
passes, and does not delay server readiness. No timer remains once
there is no obligation. A self-generated write can require a final reconciliation
before quiet. A failed sweep retains a retry. Shutdown unsubscribes local buffer
wakes; the delivery coordinator stops subscriptions/timers and joins active work.
Producer coverage
The shared writeEnvironmentCleanup helper registers and awaits intent before a
write in a transaction. Existing transaction owners call
notifyEnvironmentCleanup(tx) directly. Nested calls retain outer transaction
settlement and uncertainty handling through the shared DB implementation.
environments.ts: orphan insertion, release, metadata replacement, and run-wide release. These stores cover normal provider teardown and failed acquisition.environment-runtime.ts: reusable sandbox cleanup claims and their settlement, and Stop/export cleanup claims. Renewal of an already active claim retains its bounded timer; it cannot create new pending work.heartbeat/recovery.tsandheartbeat/run-control.ts: orphan-active recovery, retry settlement, and explicit recovery of historical leases.environment-task-runtime.ts: unconfirmed plugin task termination and plugin replacement creations. Replacement intent commits before the provider call.sandbox-creation-operations.ts: admission, atomic lease transfer, and bounded recovery. An unavailable database stops admission before external creation.sandbox-stop-and-retain.ts: persisted Stop intent and its settlement.native-workspace-export-resume.tsandnative-workspace-export-retry.ts: terminal-export cleanup, resumed-export cleanup claims, and settlement.workspace-recovery.ts: verified copyback schedules destruction.native-workspace-sync.tsand the native session executor's backup stamp: writes that can invalidate an existing historical row-version proof.environment-custom-images.ts: provider state, terminal state, and promotion. Initial active setup insertion is not a historical verification candidate.
Other lease metadata writes neither create pending cleanup nor change the row-version/provider identity of a matching historical proof. Changes to an already pending claim remain covered by its existing deadline. Deletion removes work; an already scheduled reconciliation observes that removal. Independent SQL writers and another server process are not covered by these process-local signals.
Creation ownership
A creation operation retains company, original environment, run, request identity,
provider, and nonsecret configuration references. Its environment FK uses SET NULL;
company/run/task identifiers remain after those records are removed. One operation
owns each company/environment/run/request until it is resolved. A retry of an
attached operation returns its active lease. Pending or unresolved ownership never
authorizes another create. An attached operation resolves when its lease is proven
stopped, checked at the next creation with the same key: the closed lease must
carry the stop proof same-run recovery requires (remoteExecutionHasStopped), a
termination receipt for the sandbox this operation created, whether release, the
Runner complete operation, or a destroyed reusable successor supplied it. That
creation resolves the operation and records its replacement in one transaction,
so concurrent recoveries make one provider call. A recovered cleanup operation
resolves when its cleanup lease settles. Failed cleanup, success without a
receipt, a receipt for another sandbox, a retained lease, and pending cleanup keep
the operation and its key.
Normal success transfers operation ownership and the lease in one transaction.
Current company binding and run authority are checked again at attachment.
Declared secret-reference fields retain only their references and nonsecret value
hashes. Recovery refuses lookup under a changed credential value; it records manual
reconciliation instead of treating a missing resource in a new account as proof.
A missing, deleted, disabled, or archived saved secret, a missing or inactive saved version,
or an explicitly disabled/coming-soon provider vault also records manual
reconciliation before provider lookup. Admin-disabled credentials require operator
reconciliation; recovery does not reactivate or use them. Temporary secret-provider
or database failures retain the pending operation and its retry deadline.
Recovery never resumes partially prepared execution. It asks a capable provider to inspect the stable creation identity, validates its ownership envelope, and transfers one cleanup lease under the operation lock. The existing claimed cleanup worker owns deletion and confirms resolution. Failed lookup/transfer records a bounded retry and permits other operations to proceed. An attached operation cannot transfer to cleanup. Pending creation blocks instance sleep until its obligation transfers.
Daytona version 0.1.10 declares creationOperationVersion 1. It uses the committed operation UUID in its existing creation name and labels. Recovery uses an explicit creationOperation request on the existing read-only inspect hook and strict failed-creation deletion path. Normal lease inspection still uses its recorded resource ID; creationOperationId metadata is provenance only. A missing name alone cannot prove a late allocation will never appear. Missing identity or an unsupported provider records needs_reconciliation and an operation-scoped warning. A deleted company also records manual reconciliation before credential lookup or cleanup transfer. A company lock fences deletion during the transfer transaction. That manual diagnostic has no automatic timer and does not prevent idle. The row must be reconciled before retrying the same request. Other providers do not yet claim automatic creation lookup. Generic plugin operations retain identity only, including a returned task resource ID before expiry checks or lease attachment. A failed or ambiguous attachment keeps that known ID for reconciliation; it does not authorize unconditional release. These operations do not copy credentials, resolved configuration, or provider payloads into the operation record. A lost provider reply can still leave the resource ID unknown.
Legacy deferred orphan buffers
Runtimes that hold an unwritten orphan register their flusher in a registry scoped to the shared database handle. Registration wakes the worker without contacting the database. A sweep flushes registered runtimes as well as its own startup spool; this reaches a producer runtime's buffer even when it differs from the heartbeat's runtime. The registry retains only runtimes with unresolved buffer ownership.
Unreadable directories, records, and malformed records retain a retry; a missing spool directory is an empty spool. A spool read failure still allows other runtimes to flush their buffers and persisted leases to clean up, then fails the sweep so recovery stays scheduled. Each concrete cleanup pass reloads existing legacy spool entries, including entries beyond buffer capacity. Normal acquisition and plugin task replacement do not append new fallback files. Retained entries remain available for the separate cutover inventory. No empty-spool filesystem or database polling is introduced. Existing limits on a simultaneous database, spool, and buffer-capacity failure remain; this is not a new durable queue.
Verification and boundaries
The PostgreSQL worker tests exercise quiet configured state and verified history, real store activation, persisted backoff, failed teardown and restart, nested commit/rollback, drain/resume, cross-runtime deferred work, failed spool reads, shutdown joining, provider uncertainty, and historical pagination. Existing runtime, reaper, Stop, custom-image, and workspace tests cover the underlying cleanup authority and provider contracts.
The runtime fixture now gives each simulated instance its own DB handle and spool. Its journal-write failure injection targets the transaction boundary rather than the direct update that moved inside that transaction. Fake-clock worker tests wait for PostgreSQL I/O and distinguish an executing sweep from a retained next wake.
This does not migrate discovery of orphaned active leases or legacy shutdown workspace settlement from the broader run-recovery loop. Custom-image active session expiry also remains on its existing tick; only historical verification moves here. Other maintenance duties, external probes, and connected transports still affect idle traffic. No Cloud alarm coverage or host-sleep exemption changes. These tests prove the changed workers become quiet and recover; they do not prove deployed suspension or fleet savings.
Custom-image setup expiry
Setup expiry uses a separate custom-image-expiry worker. The custom-image
service owns the active-status predicate, the earliest saved expiresAt, and
provider cancellation. Startup registers the worker in the existing enabled
heartbeat housekeeping path; the maintenance tick no longer calls its sweep.
All supported setup producers live in environment-custom-images.ts: initial
creation, provider refresh, cancellation/failure status, and promotion. They
register intent in their owning transaction. A provider refresh can move the
deadline earlier; a terminal transition removes it. Historical verification
metadata does not change active expiry eligibility and does not need this signal.
Environment deletion can only remove obligations; an already scheduled pass
observes the removal. Independent SQL and other DB roots are outside the existing
process-local notification contract.
No active dated session means no recurring reads or provider calls. Overdue batches and failures retain a bounded retry, while first writes and failed reconciliation survive drain. A missing expiry retains the existing behavior (no inferred expiration); sleep safety still treats unsupported live resources conservatively. Provider cancellation, failure state, and historical resource verification keep their existing semantics. No Cloud alarm coverage changes.
Historical retry test observation
The historical verification worker intentionally does not delay server readiness. Its provider sweep finishes before the following database pending-state read registers a retry deadline. The regression test now waits for that future deadline before inspecting it; sweep completion alone is not scheduler settlement. The runtime worker, retry interval, and readiness contract are unchanged.
A controlled 150ms delay before the real pending-state query reproduced the current CI failure: the immediate assertion observed a null deadline. The corrected assertion passed with the same delay. The temporary delay was removed, and all 14 cleanup tests passed against an isolated PostgreSQL database. The existing external test-database hook is carried at the stack foundation so intermediate branches can use this same verification path.
An initial attempt at the foundation could not start embedded PostgreSQL because that earlier branch lacked the external fixture hook. A broader fixture wait for all idle holds to clear then failed three cases that legitimately retain pending cleanup obligations. That wait was removed. Only the historical deadline assertion changed; its requirement is now a deadline strictly in the future. The complete suite passes with the original readiness and idle-hold behavior preserved.