PaperClipAI/doc/idle-environment-cleanup.md

# Idle environment cleanup

Pending lease cleanup and historical resource verification use the existing
transaction signals, delivery coordinator, and in-process scheduler. They no
longer run unconditionally from the server maintenance tick. The run reaper
performs its inline pending-cleanup sweep only after actually recovering a run
or an orphaned active lease. Explicit user retry keeps its existing direct path.

## Work and ownership

| Duty | Truth | Trigger and next deadline | Quiet condition |
| --- | --- | --- | --- |
| Failed teardown | Existing `environment_leases` rows in `pending_cleanup` | Commit signal; later of row backoff and saved retry/ownership deadline | No pending lease or deferred orphan |
| Failed acquisition before a DB row exists | Existing local orphan spool and the producing runtime's buffer | Local buffer registration; five-minute retry while buffered; startup spool load | Durable row owns recovery and spool copy is removed |
| Historical provider verification | Existing failed ephemeral leases and finished custom-image setups without matching cleanup proof | Commit signal; five-minute retry while an unverified candidate remains | No unverified candidates, including retained verified history |

`environment-cleanup-state.ts` owns the deadline query and the shared safe reader
for retry/ownership metadata. Cleanup claims and the scheduler use that same
reader, including malformed-number handling and the existing maximum horizon.
Startup retains the immediate scan, but never bypasses a saved in-flight claim or
retry cooldown. Subsequent scans respect the five-minute row backoff. Claims,
provider readiness, Stop authority, workspace retention, copyback, receipt
validation, and escalation rules stay with their current owners.

`environment-cleanup-work.ts` registers two independent workers. Their generation
counters preserve a wake until successful reconciliation, including a first write
during drain, changes during a sweep, and failure followed by drain. The existing
heartbeat interval bounds pending-cleanup starts after bursts of writes. Historical
verification has a separate minimum five-minute spacing, including commit-triggered
passes, and does not delay server readiness. No timer remains once
there is no obligation. A self-generated write can require a final reconciliation
before quiet. A failed sweep retains a retry. Shutdown unsubscribes local buffer
wakes; the delivery coordinator stops subscriptions/timers and joins active work.

## Producer coverage

The shared `writeEnvironmentCleanup` helper registers and awaits intent before a
write in a transaction. Existing transaction owners call
`notifyEnvironmentCleanup(tx)` directly. Nested calls retain outer transaction
settlement and uncertainty handling through the shared DB implementation.

- `environments.ts`: orphan insertion, release, metadata replacement, and run-wide
  release. These stores cover normal provider teardown and failed acquisition.
- `environment-runtime.ts`: reusable sandbox cleanup claims and their settlement,
  and Stop/export cleanup claims. Renewal of an already active claim retains its
  bounded timer; it cannot create new pending work.
- `heartbeat/recovery.ts` and `heartbeat/run-control.ts`: orphan-active recovery,
  retry settlement, and explicit recovery of historical leases.
- `environment-task-runtime.ts`: unconfirmed plugin task termination and plugin
  replacement orphans, including a registered deferred flusher when the row
  cannot be written. The flusher retains an idle hold until the row is durable.
- `sandbox-stop-and-retain.ts`: persisted Stop intent and its settlement.
- `native-workspace-export-resume.ts` and `native-workspace-export-retry.ts`:
  terminal-export cleanup, resumed-export cleanup claims, and settlement.
- `workspace-recovery.ts`: verified copyback schedules destruction.
- `native-workspace-sync.ts` and the native session executor's backup stamp:
  writes that can invalidate an existing historical row-version proof.
- `environment-custom-images.ts`: provider state, terminal state, and promotion.
  Initial active setup insertion is not a historical verification candidate.

Other lease metadata writes neither create pending cleanup nor change the
row-version/provider identity of a matching historical proof. Changes to an
already pending claim remain covered by its existing deadline. Deletion removes
work; an already scheduled reconciliation observes that removal. Independent SQL
writers and another server process are not covered by these process-local signals.

## Deferred orphan buffers

Runtimes that hold an unwritten orphan register their flusher in a registry scoped
to the shared database handle. Registration wakes the worker without contacting
the database. A sweep flushes registered runtimes as well as its own startup spool;
this reaches a producer runtime's buffer even when it differs from the heartbeat's
runtime. The registry retains only runtimes with unresolved buffer ownership.

Unreadable directories, records, and malformed records retain a retry; a missing
spool directory is an empty spool. A spool read failure still allows other runtimes to flush their buffers and
persisted leases to clean up, then fails the sweep so recovery stays scheduled.
Each concrete cleanup pass reloads the spool, including entries appended by
plugin task replacement after startup and entries beyond buffer capacity.
No empty-spool filesystem or database polling is introduced. Existing limits on a simultaneous database,
spool, and buffer-capacity failure remain; this is not a new durable queue.

## Verification and boundaries

The PostgreSQL worker tests exercise quiet configured state and verified history,
real store activation, persisted backoff, failed teardown and restart, nested
commit/rollback, drain/resume, cross-runtime deferred work, failed spool reads,
shutdown joining, provider uncertainty, and historical pagination. Existing
runtime, reaper, Stop, custom-image, and workspace tests cover the underlying
cleanup authority and provider contracts.

The runtime fixture now gives each simulated instance its own DB handle and spool.
Its journal-write failure injection targets the transaction boundary rather than
the direct update that moved inside that transaction. Fake-clock worker tests wait
for PostgreSQL I/O and distinguish an executing sweep from a retained next wake.

This does not migrate discovery of orphaned active leases or legacy shutdown
workspace settlement from the broader run-recovery loop. Custom-image *active
session expiry* also remains on its existing tick; only historical verification
moves here. Other maintenance duties, external probes, and connected transports
still affect idle traffic. No Cloud alarm coverage or host-sleep exemption changes.
These tests prove the changed workers become quiet and recover; they do not prove
deployed suspension or fleet savings.

## Custom-image setup expiry

Setup expiry uses a separate `custom-image-expiry` worker. The custom-image
service owns the active-status predicate, the earliest saved `expiresAt`, and
provider cancellation. Startup registers the worker in the existing enabled
heartbeat housekeeping path; the maintenance tick no longer calls its sweep.

All supported setup producers live in `environment-custom-images.ts`: initial
creation, provider refresh, cancellation/failure status, and promotion. They
register intent in their owning transaction. A provider refresh can move the
deadline earlier; a terminal transition removes it. Historical verification
metadata does not change active expiry eligibility and does not need this signal.
Environment deletion can only remove obligations; an already scheduled pass
observes the removal. Independent SQL and other DB roots are outside the existing
process-local notification contract.

No active dated session means no recurring reads or provider calls. Overdue
batches and failures retain a bounded retry, while first writes and failed
reconciliation survive drain. A missing expiry retains the existing behavior
(no inferred expiration); sleep safety still treats unsupported live resources
conservatively. Provider cancellation, failure state, and historical resource
verification keep their existing semantics. No Cloud alarm coverage changes.

## Historical retry test observation

The historical verification worker intentionally does not delay server readiness.
Its provider sweep finishes before the following database pending-state read
registers a retry deadline. The regression test now waits for that future
deadline before inspecting it; sweep completion alone is not scheduler settlement.
The runtime worker, retry interval, and readiness contract are unchanged.

A controlled 150ms delay before the real pending-state query reproduced the
current CI failure: the immediate assertion observed a null deadline. The
corrected assertion passed with the same delay. The temporary delay was removed,
and all 14 cleanup tests passed against an isolated PostgreSQL database. The
existing external test-database hook is carried at the stack foundation so
intermediate branches can use this same verification path.

An initial attempt at the foundation could not start embedded PostgreSQL because
that earlier branch lacked the external fixture hook. A broader fixture wait for
all idle holds to clear then failed three cases that legitimately retain pending
cleanup obligations. That wait was removed. Only the historical deadline assertion
changed; its requirement is now a deadline strictly in the future. The complete
suite passes with the original readiness and idle-hold behavior preserved.