PaperClipAI/doc/idle-sleep-safety.md

Automatic idle sleep safety

GET /api/instance/task-drain?idleSleepSafety=1&ownerId=<owner> adds an instance-wide idleSleepSafety report for instance administrators and signed Cloud control callers. Ordinary task drains keep their existing behavior. Company members and agents cannot read this report.

Hosting protocol

1. Hold external admission at the hosting layer. Queue user requests for wake or abort sleep when activity arrives. Do not send traffic into an instance that is being stopped.
2. Send POST /api/instance/task-drain with { "purpose": "idle", "ttlMs": 120000 }. Idle holds require a TTL from 5 to 300 seconds. Save the returned ownerId and expiresAt. An existing drain returns 409. An unsupported purpose must leave automatic sleep disabled.
3. Poll the report with that exact owner. Only { "version": 1, "backgroundWork": "none" } permits ordinary idle sleep. The optional routine protocol below requires a durable host alarm as well. Require the same ownerId, draining: true, zero run/wake counters, and enough remaining TTL for the provider stop. present, unknown, missing fields, or errors must keep the instance awake. Do not interpret quiescent alone as authorization.
4. Revalidate the same hold immediately before stopping the instance. Keep external admission closed through the provider stop. Never use an old report after a restart, expired hold, changed owner, or failed validation.
5. If sleep is abandoned, send DELETE /api/instance/task-drain?ownerId=<owner>. A missing or stale owner cannot release another controller's idle hold. The TTL clears an abandoned hold if the controller loses its response.

An idle hold rejects new tenant HTTP requests and WebSocket upgrades with 503 and Retry-After: 1. Health checks and authenticated task-drain control remain available. Ordinary deploy drains do not enable this HTTP gate. The host must handle retry/wake at its ingress before enabling automatic sleep. Health and control mutations remain tracked through completion, including signed bootstrap writes. Only the task-drain read skips request tracking. Its authentication finishes under a separate work token before entering the report, so it counts concurrent writes without blocking on its own scan.

A matched static UI file served by the admitted, handler-tracked application finishes its request tracking when its response ends or closes, including a match found after the client has already disconnected. Unfinished middleware promises remain tracked. Routers share this application scope. A mounted Express application has hidden handler promises, so it keeps the existing conservative hold after an aborted download until the application itself completes the response. An application without installed handler tracking keeps the same conservative hold. Missing files continue through the normal fallback and error handlers; they do not exempt other GET requests or callback-only application work.

Budget and agent lifecycle work

Budget accounting commits its charge, admission block, and enforcement generation before returning to its caller. Cancellation delivery runs separately through the delivery coordinator, using the existing policy version pair and committed run Stop records. It can finish already-admitted stops during drain, but cannot start another paid turn. A provider still stopping, an unacknowledged Stop, a pending policy generation, and unresolved accounting remain sleep blockers.

Agent lifecycle uses the same transaction-aware coordinator. Hire approval, configuration invalidation, manual commands, and computed agent budget holds register work before their transaction writes. Company status transitions register inherited-hold projection; changing a company name, description, or logo does not. Projection selects only agents whose company pause/archive hold differs from the committed company state, in pages of 100 under one company accounting transaction per page. Pause/archive transactions also persist pausing operations for agents with cancellable runs, reading only those involved agents in batches. Immediate reactivation can remove an inherited hold but cannot erase an owed Stop; new provider work waits for lifecycle readiness. Archive activity counts describe requested pauses and cancellations, not completed provider termination. Agent and project budget changes never enumerate company peers.

Startup recovers pending lifecycle operations and inherited-hold mismatches. Thereafter notifications and stored retry/lease deadlines drive work. Stable ready, paused, rejected, and terminated history does not trigger recurring policy scans. Pending phases and inherited-hold mismatches block sleep. Lifecycle execution can verify configurations or resume queued work, so it waits while admission is draining or the instance is in warm standby; its durable obligation is retained. A future in-process retry timer does not grant permission to sleep.

These notifications cover the existing single-active-writer ownership model. Independent SQL writers and additional processes need an explicit wake protocol; process-local notifications do not provide cross-replica delivery. Worker quiet alone is not evidence that the host or database actually suspended.

Scheduled routine wake protocol

Historical failed ephemeral leases and terminal custom-image setup sessions can become idle only after the host observes deletion of their exact provider resource. Cloud checks bounded batches on already-awake instances, outside the readiness path. The optional environmentInspectLease plugin RPC is read-only: it must never reconnect, start, stop, or destroy an allocation. Older workers without this method remain conservative. Daytona accepts its SDK's typed not-found response; authentication and transport failures are not deletion proof. Setup checks require the recorded provider-config fingerprint; lease checks use the recorded plugin and scoped config, never a replacement binding. For legacy secret references without a version manifest, only an unrotated initial version with a matching resolved-value hash establishes credential identity. Rotated/mutable credentials and ambient Daytona keys defer proof; not-found under another account cannot establish historical cleanup. Inspection also rejects an ambient API endpoint override when no endpoint was recorded. The manager permits at most two outstanding inspection RPCs across workers. Caller timeouts retain their slots until the actual worker reply or exit; timeouts cannot accumulate additional provider reads in later sweeps.

Host-owned verification metadata binds the company, environment, resource, provider handle, optional run, and exact row version. Concurrent resume, replacement, cleanup, or configuration changes invalidate verification. Provider-supplied metadata cannot supply this marker. Inspection preserves failure outcomes and history and does not create a run-continuation receipt. Normal setup cancellation and timeout preserve the host's recorded config fingerprint and RPC company scope for later read-only verification. Retained allocations, unfinished runs/accounting/retries, export intents and live setup credentials continue to block sleep.

Pending human-only connection setup can wait asleep when no active enabled connection of the requested purpose exists. Available access reblocks sleep so the existing reconciliation path can run; answering a setup card continues to use the normal admission and wake path. The delivery worker uses the same passive-wait proof to release its local admission hold. On an already-awake instance it keeps its existing five-second maintenance cadence, without creating a wake alarm. This catches newly available access even when its write does not notify the connection queue. Pending real deliveries and their downstream wake requests still block sleep. An owned drain pauses passive-only maintenance before it admits tracked work, so a slow pre-sleep backup can finish with an unchanged generation. New producer notifications still preempt the maintenance timer and invalidate that backup.

An unconfigured Slack draft can also wait for browser setup when its matching channel connection is disabled and still a draft. It must have no provider identity, prior activation/activity, or setup error. Partially configured, enabled, foreign, and other-provider endpoints remain blockers; accepted chat deliveries retain their independent checks. An optional Slack slash-command name is configuration only and does not create background work. Other setup fields still retain their blockers.

Cloud also reconciles missed routine issue-status hooks in bounded batches on awake instances. Exact company/routine/issue/run ownership and row locks are required. The normal routine service records completion or transient failure from the current issue status; a resumed issue can use its existing continuation rules. This repairs stale handoff bookkeeping without exempting unprocessed routine runs or consuming deferred comment requests. Lock contention and retryable transaction failures defer only that candidate; other repairs continue, and later sweeps retry the deferred row.

Verified signed Cloud control callers may opt into GET /api/instance/task-drain?idleSleepSafety=1&scheduledWake=1&ownerId=<owner>. Tenant tokens, browser instance administrators, local implicit actors, and agents cannot request this extension. Ordinary reports retain their existing behavior.

If all other background-work checks pass, active routines may return idleSleepSafety: { version: 2, backgroundWork: "none", scheduledWake }. The proof has version: 1, kind: "routines", a SHA-256 generation, and the earliest nextWakeAt across every company. Only enabled, unarchived schedule triggers with valid cron, timezone and future next_run_at qualify. Unsupported triggers, setup work, due work, unknown states and more than 10,000 active triggers keep the instance awake. Inactive triggers do not create obligations. Active routines without enabled triggers have no timed work. Definitions are read in the same repeatable-read transaction as the other durable checks.

The host must commit one durable alarm for the exact stack under its live sleep lease before stopping compute. It must then re-read the same owned report and require the exact generation and deadline. Generation binds the process, hold owner and routine/trigger catalog, so restart or schedule edits invalidate it. No routine definitions or tenant identifiers appear in the proof. Failure to persist or validate the alarm abandons sleep. Cloud acknowledges persistence inside its controller; this report alone never authorizes scheduled sleep.

Cloud must start waking early enough for measured startup latency and retry failed or missed wakes across controller restart. After wake, the existing routine scheduler owns execution, atomic claims and catch-up rules. Local scheduling is unchanged for self-hosted instances and whenever Cloud declines sleep. This first extension covers routines only. Agent timers, plugin jobs, retries, integrations and unsupported queues retain their existing blockers. All mutations must enter through the host admission/wake path. Independent database writers are outside this single-process protocol.

Verify a full scheduled sleep/wake/execution cycle on staging before enabling this host capability. The tests cover ownership, future deadlines, changed generations and conservative fallback.

Passive issues and scheduled monitors

Signed Cloud control callers can request scheduledWake=2 to include issue monitoring alongside routines. Earlier hosts requesting scheduledWake=1 retain the routine-only rules, including the conservative open-issue check. Browser administrators and tenant credentials cannot request either extension.

With the second capability, backlog tasks, unassigned todo tasks, explicit blocked tasks and human-owned tasks may be passive records. Their presence alone does not require compute. An assigned onboarding first task is also passive with an idle assignee before its first run, user-authored comment or user-resolved opening card. This uses the same awaiting-user predicate as recovery; a seeded greeting does not start work. User input still enters through the host wake path. Execution locks, automated pending review, conversation state, unknown state and dependency repair remain blockers. A blocked task whose blockers have all finished stays awake for reconciliation. Enabled task watchdogs and unsupported recovery keep their existing checks. Incoming changes still require the hosting admission and wake path.

An active issue monitor needs a saved future monitor_next_check_at matching its validated execution policy, a company-scoped agent, no outstanding dispatch claim, and remaining attempts and timeout. Due, malformed, quota-recovery, unowned and inconsistent monitors keep the workspace awake. Inspection does not clear schedules, advance attempts or dispatch an agent.

The combined report uses version: 2 and a scheduled proof with version: 1 and kind: "routines_and_issue_monitors". Its deadline is the earliest required routine or monitor time across all companies. Its generation binds the current process, idle owner and inspected catalogs. Hosts must acknowledge this known proof kind, persist the alarm and validate the unchanged owned report before stopping compute. Existing hosts reject the new kind. Monitor dispatch and startup catch-up continue through the existing scheduler and atomic claims; self-hosted scheduling is unchanged. This capability does not cover all future work in the instance: every other safety check continues to apply.

Retry history

Retry promotion keeps scheduled_retry_at on a run for inspection. That date is consumed history when the run is terminal, has a finish time at or after the retry date, and has no accounting work pending. The idle report uses the same proof for failed ephemeral leases, coalesced wakes, and acknowledged question responses. A live retry child still blocks through the instance-wide run check. Unfinished runs, retry dates after the finish time, and pending accounting stay awake. The inspection does not clear dates or change retry scheduling.

Retained local workspaces

Local directories for primary projects and Git worktrees do not require an awake process by themselves. The report permits their retained rows only in known modes and states, with no runtime configuration, service state, commands, pending reopen, or cleanup already eligible. Saved environment references alone are passive. Shared workspaces with a project workspace remain blockers because they can inherit runtime configuration. Unfamiliar providers, strategies, configuration and malformed metadata also remain blockers.

This does not delete files, alter retention, or clear workspace records. Future cleanup eligibility is an earliest allowed time; eligible cleanup can wait for the next wake. Due cleanup, unfinished operations, runtime leases and runtime services still keep the instance awake through their separate checks. The host must keep workspace storage attached across sleep and preserve its backup policy.

Accepted work and cleanup

HTTP admission is installed before body parsing, authentication, webhooks and MCP ingress. The Express adapter tracks handler promises in nested routers and error middleware. Sending a response or losing the client does not complete an async handler. A callback-only handler that abandons its response without ending it remains a sleep blocker. An async route that settles after a disconnect releases its request token; normal cancellation does not permanently block sleep. New routes must be registered before this adapter is installed. Fire-and-forget work must have durable queue state or use trackIdleWork; a response is never a substitute for tracking that work. Accepted live-event WebSocket authentication remains counted until its promise settles, including after the raw socket closes. Detached startup reconciliation for built-in agents, managed homes and persisted runtime services also remains counted after readiness until its writes settle.

Scheduler work already in flight remains counted until its promise settles. Idle holds pause new scheduler admissions. Database backup promises remain counted through success or failure. Configured periodic backups block sleep by default. The opt-in checkpoint mode below preserves a final backup instead of running the timer while asleep. A generation counter invalidates a scan if tracked work both starts and finishes during inspection. Reports remain unknown until startup recovery and HTTP tracking are installed.

Terminal workspace recovery also registers a local sleep probe. Incomplete startup pages and failed recovery remain blockers until a successful pass. Active cleanup and unresolved producer transactions retain their admission holds. Cooldown and inspection-backoff timers are maintenance hints over retained domain rows. They can wait for the next wake and are restored at startup; they do not create external alarm coverage or exempt other persisted work from this report.

Backup checkpoints for idle instances

PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED=1 opts an instance with automatic backups enabled into this mode. Keep its database and persistent backup volume attached across sleep. All writers must use the instance's admission protocol; this mode cannot cover independent writers to the same database. On managed instances, checkpoint-enabled safety reads require a verified signed Cloud control actor. Tenant instance-admin elevation does not grant platform backup authority. Self-hosted instance administrators retain operator access.

Each owned safety read first checks local work, plugin drains and durable work. Known local blockers return before opening a database transaction. If every other check passes, the server takes a fresh backup with the existing backup engine and the retention policy from Instance Settings. It reads the full gzip archive, checks its trailer and nonempty content, and syncs the archive and directory before returning none. The final pre-stop revalidation takes another fresh backup. Reads can therefore take longer; the host must bound its request and still leave enough lease time for a confirmed provider stop.

There is no cached backup exemption. Failed, concurrent or untracked backups, other work during the scan, changed owners and expired holds cannot authorize sleep. Client timeouts do not clear the backup's in-flight token. Slow backups may retain an awake instance until a later sleep attempt.

A durable .idle-backup-wake-required marker is written before the dump. No recurring duplicate archives are produced while the instance is asleep. The last pre-sleep archive and existing history stay on persistent storage; no retention pruning runs during sleep. On restart, the server attempts a fresh backup before enabling idle eligibility, then resumes the normal interval. Only a verified new backup clears the marker. Failure keeps it for the next scheduled retry. Disabling automatic backups explicitly still disables this startup backup. Disabling only checkpoint mode restores the sleep blocker and continues to honor an existing restart marker.

This does not replace filesystem backups, encryption-key recovery or off-provider disaster recovery. Validate restoration of a staging checkpoint before enabling the flag on customer instances.

The report inspects accounting and sandbox-cleanup spools directly. Any entry, including malformed JSON, temporary files and failed write probes, blocks sleep. Read errors and invalid spool paths return unknown. Recovery readers that skip bad records cannot certify an empty spool.

An orphan cleanup token starts before its spool write and remains held through buffering, queue splices, flush retries and buffer overflow. It ends only after the database owns cleanup. Queued usage receipts stay counted until capture settles. If receipt capture and the database failure fence both fail, a token continues to block sleep until the failure fence is durably written. A query that updates zero rows does not clear that debt. Unrecoverable cleanup requires operator reconciliation; this API does not discard it to save compute.

Persisted work and limits

The report checks all companies in one bounded read-only database transaction. Queued and orphaned runs, timers, retries, accounting debt, unfinished issues, cleanup, active routines, integrations, external ingress credentials, and plugin work block sleep. Completed ordinary run history does not. Less common work sources conservatively block on any retained records. Active issue watchdogs block sleep even when their watched issue is complete and no review has started. Missing migrations, malformed configuration and database errors produce unknown without exposing SQL or tenant data.

Active runtime_auth connections with purpose ai are saved credentials, not independent workers. Their saved configuration does not block sleep. Login sessions and runs still pass the existing durable checks. Detached subscription observations and quota reads remain counted through their actual completion, including token refresh after a quota response times out. Credential-recovery journals, partial writes and probes block sleep across restart. Empty company directories left after successful recovery do not; invalid paths and read failures remain unknown. Credentials and connection records are retained. Disabled draft connections and saved outbound HTTP MCP connections have no independent listener. They can permit sleep while accepted calls and credential work still pass the other fences. MCP connections with a retained GitHub webhook binding, local stdio processes, channels, and other active transports remain blockers until their inbound work has a supported wake contract.

With the issue-scheduling capability, pending questions, confirmations, checkbox confirmations and task suggestions can be passive waits. Their producer must have finished without retry or accounting debt. The response audience and continuation policy stay unchanged. Unknown pending siblings, foreign-company records and unfinished producers block sleep. A resolved card does not make an otherwise runnable issue passive. The schedule generation includes response identity and audience, including when only a routine owns the next alarm. Merge confirmations remain blockers because GitHub polling can accept them without a human response and does not yet have a durable wake contract.

Known board escalation actions for stranded work, missing disposition and a lost review path can also be passive. They require a blocked source in the same company, board ownership, no automatic wake, no monitor or timeout, and no live execution binding. The action remains visible and unresolved. Other recovery causes and autonomous owners remain blockers.

A failed ephemeral lease is history only with successful cleanup, a release timestamp, and an exact provider termination receipt bound to its company, run, lease and resource. Its run must be finished without retry or accounting debt. Pending cleanup claims and export/stop-and-retain intents still block. Failure labels or cleanup success without that proof do not authorize sleep. Lease history is retained; this check performs no provider teardown or data updates. A released local ephemeral bookkeeping lease needs no remote termination receipt when its exact run has finished and both its provider and environment are local. It must have no provider resource reference or pending cleanup, export or retention intent. Unknown drivers and remote resources still block.

Finished login-session history permits sleep only after its promotion claim is cleared and any retained provider reference has a matching released, expired, or verified failed ephemeral lease with successful cleanup and a release timestamp in the same company and environment. The failed lease must satisfy the same termination and run checks above. Missing cleanup evidence, cleanup_pending, and live login flows remain blockers. A finished coalesced wake is history only when its exact company/agent's linked run has finished without retry or accounting debt. Image-setup history still blocks: promotion can succeed before teardown is confirmed, so a terminal label does not prove that its provider resource is gone.

Completed question-answer deliveries permit sleep only after acknowledgement and completion of their exact target run in the same company, without retry or accounting debt. A fallback without a target run requires its exact question's finished durable wake receipt in the same company. A coalesced fallback also requires its exact company/agent's linked run to finish without retry or accounting debt. The question-history index covers every wake status. Pending deliveries, missing completion evidence, and unfinished handoffs still block. Failed deliveries without a pending target are history; other work checks still apply.

Processed connector-event receipts with a processing timestamp are idempotency history. Delivered connection-intent outboxes also permit sleep once their exact accepted wakes finish. A system-retired connection card parked behind a pending response keeps its undelivered outbox intact; the normal response path resumes delivery. Pending event application, unfinished wakes and all other execution checks still block. None of these exceptions fabricates completion or removes a durable receipt.

Enabled plugins require the current worker's verified idle drain. A version label does not prove that arbitrary worker code has no background activity. No plugin approval or configuration bypass is supported. Scheduled and externally triggered work needs a durable host wake mechanism before it can permit sleep.

This is a single-process admission protocol. A restart clears the owner. The host must serialize sleep with deploy/wake operations and bind its final check to the instance it stops. An unexpected restart after the final check can still race an external provider stop; a process-local report cannot fence that remote operation by itself. Multiple replicas require shared fencing. The protocol does not make existing in-memory-only debt survive an unplanned process crash.

Related public PR #13413 adds runtime-service controller requirements and uses the same purpose: "idle" / ownerId vocabulary. This report conservatively blocks all retained runtime-service state instead of approving such services.

Muse maintenance

Muse maintenance uses the shared delivery coordinator. Startup reconstructs existing work and marks incomplete receiver evidence once. Pairing, OAuth grant attachment/revocation/replay, qualification changes and new assignments notify the coordinator before their transaction writes. Nested writes wake it only after the outer transaction settles; uncertain outcomes retain the normal hold.

Unrevoked qualifications schedule their saved expiration time. An instance with no Muse bindings, or with idle bindings and no qualification deadline, performs no recurring maintenance SQL after startup. Failed reconciliation retries in five seconds. Offered, claimed, accepted and not-yet-finalized settled assignments retain that five-second recovery cadence until settlement; this covers terminal run writers even when their direct reconciliation path fails. Retained finalized assignment history does not keep that cadence running. Direct SQL or independent process writers require explicit integration; startup still reconstructs state.

Warm standby prevents maintenance SQL. An idle drain pauses ordinary dispatch and passive deadlines through the shared coordinator. Startup and notifications received during drain retain a memory-only wake until a sweep can run. The wait does not query the database. Producer uncertainty and outstanding assignment recovery retain its existing admission rules. Shutdown awaits the worker before flushing the receiver. The receiver's separate batching timer remains; an empty buffer performs no database work. Existing Muse durable sleep blockers, external worker stop evidence, and ingress requirements remain unchanged. This change establishes database quiet for inactive Muse, not host sleep eligibility for a configured Muse connection.