From baa94bbea58919bcc6cf247a708f04e6e1002679 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 9 Oct 2026 17:25:02 -0700 Subject: [PATCH] Fix idle sleep checks for finished login and wake history (#15761) ## Thinking Path > - Paperclip manages AI agent work and keeps a durable record of each run. > - Automatic idle sleep must wait for accepted work and provider cleanup. > - The current sleep check treats every saved login session as active work. > - It also treats every coalesced wake request as pending work. > - These records can remain after their work and cleanup finish. > - This pull request checks the cleanup and linked-run evidence before it treats these records as history. > - The change lets otherwise idle instances sleep and keeps uncertain work awake. ## Linked Issues or Issue Description **What happened?** A finished login session blocks idle sleep after its provider resource has been removed. A consumed coalesced wake also blocks sleep after its linked run finishes. **Expected behavior** Finished history should permit sleep when cleanup or run completion is confirmed. Active work, missing evidence, retry debt, and pending accounting must still block sleep. **Steps to reproduce** 1. Create a finished login session with a retained provider reference. 2. Record a successful release for the same resource, company, and environment. 3. Check idle sleep safety under an owned, quiet admission hold. 4. The current check reports background work. The new check treats the session as history. 5. Repeat with a finished coalesced wake linked to a finished run in the same company and agent. **Paperclip version or commit** Reproduced on master at 98e3cec78e8. Related work: #15522, #15599, #15629. A search of open and merged idle-sleep PRs found no duplicate. ## What Changed - Permit finished login history only after its promotion claim is cleared. A retained resource reference needs successful cleanup, a release timestamp, and the same company and environment. Conflicting cleanup evidence still blocks sleep. - Permit a finished coalesced wake only after its exact linked run finishes without a retry or pending cost accounting. - Keep the lease UUID column unchanged. Convert only valid UUID-shaped session references so both existing lease indexes remain usable. - Add real PostgreSQL tests for safe history and missing, active, or conflicting evidence. Cover text references, UUID references, and conflicting internal/external matches. - Document these checks. Image-setup history still blocks sleep because its terminal state does not confirm cleanup. ## Verification - `pnpm exec vitest run server/src/__tests__/idle-sleep-safety.test.ts`: 84 tests passed, including the PostgreSQL cases. - `pnpm -r typecheck`: passed. `pnpm --filter @paperclipai/server typecheck` also passed after the index fix. - Full default test coverage was exercised through the supported shard runner. The initial run was not green. Fixture-permission, local PostgreSQL startup, socket, route timeout, and Git stress failures passed controlled rechecks with normal permissions, one database worker, or isolated runs. - All route suites, UI/CLI suites, and all shared, database, adapter, and SDK projects were exercised. One damaged-session recovery test in `native-session-resume.test.ts` still fails and also reproduces on unmodified base `98e3cec78e8`. Full-suite green status is not claimed. - `pnpm build`: passed in an isolated checkout of the initial commit. The build and full Paperclip CI also passed on final head `d61b489c26b`: https://github.com/paperclipai/paperclip/actions/runs/38005908398. Both required branch-protection checks passed. The optional Superagent security scan is still running and has not returned findings. - A local PostgreSQL query-plan probe used 10,000 lease rows and 200 finished login rows with mixed internal and external references. The actual sleep query used both the primary-key and provider-reference indexes. Execution took 1.63 ms in this synthetic fixture. - A read-only comparison on a controlled staging instance found login history as the current check's sole durable-work blocker. The new query found no durable-work blockers. This comparison did not authorize or perform provider shutdown. - The current owned sleep controller refused sleep and released its hold. The instance remained healthy. Existing backup archives were unchanged. ## Risks - Missing schema, missing cleanup evidence, stale claims, and unfinished linked runs fail closed. - The change preserves history and provider references. It does not delete records or release resources. - Image setup, other durable work, plugin draining, in-flight work, and backup checks retain their current behavior. - No schema migration or new API is required. ## Model Used - OpenAI Codex, GPT-6. Used repository inspection, code execution, and test tools. The exact deployment model identifier and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the tests for this change locally and they pass. The broader local baseline failure is described in Verification. - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-sleep-safety.md | 9 ++ .../src/__tests__/idle-sleep-safety.test.ts | 129 +++++++++++++++++- server/src/services/idle-sleep-safety.ts | 37 ++++- 3 files changed, 171 insertions(+), 4 deletions(-) diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index 71bd5a549e2..30622581e24 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -120,6 +120,15 @@ block sleep even when their watched issue is complete and no review has started. Missing migrations, malformed configuration and database errors produce unknown without exposing SQL or tenant data. +Finished login-session history permits sleep only after its promotion claim is +cleared and any retained provider reference has a matching released or expired +lease with successful cleanup and a release timestamp in the same company and +environment. Missing cleanup evidence, `cleanup_pending`, and live login flows +remain blockers. A finished coalesced wake is history only when its exact +company/agent's linked run has finished without retry or accounting debt. +Image-setup history still blocks: promotion can succeed before teardown is +confirmed, so a terminal label does not prove that its provider resource is gone. + Every enabled plugin blocks sleep. A version label does not prove that arbitrary worker code has no background activity. No plugin approval or configuration bypass is supported. Scheduled and externally triggered work needs a durable diff --git a/server/src/__tests__/idle-sleep-safety.test.ts b/server/src/__tests__/idle-sleep-safety.test.ts index 66a023dd9b9..718d33ce534 100644 --- a/server/src/__tests__/idle-sleep-safety.test.ts +++ b/server/src/__tests__/idle-sleep-safety.test.ts @@ -178,7 +178,7 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp const companyId = randomUUID(); const agentId = randomUUID(); const runId = randomUUID(); - await db.insert(companies).values({ id: companyId, name: "Idle test", issuePrefix: "IDLE" }); + await db.insert(companies).values({ id: companyId, name: "Idle test", issuePrefix: `T${companyId.slice(0, 6).toUpperCase()}` }); await db.insert(agents).values({ id: agentId, companyId, name: "On-demand agent", role: "engineer", status: "idle", adapterType: "process" }); await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", status: "succeeded" }); return { companyId, agentId, runId }; @@ -246,6 +246,44 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp expect(await read()).toEqual(present); }); + it("allows consumed coalesced wakes only after their linked run finishes", async () => { + const { companyId, agentId, runId } = await seed(); + await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, runId)); + await db.insert(agentWakeupRequests).values({ + companyId, agentId, source: "assignment", status: "coalesced", runId, finishedAt: new Date(now), + }); + expect(await read()).toEqual(present); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date(now) }) + .where(eq(heartbeatRuns.id, runId)); + expect(await read()).toEqual(none); + }); + + it.each(["unfinished", "missing_run", "wrong_company", "wrong_agent", "unfinished_run", "retry", "accounting"])( + "keeps a coalesced wake awake with %s evidence", async kind => { + const { companyId, agentId, runId } = await seed(); + await db.update(heartbeatRuns).set({ finishedAt: new Date(now) }).where(eq(heartbeatRuns.id, runId)); + let linkedRunId = runId; + if (kind === "missing_run") linkedRunId = randomUUID(); + if (kind === "wrong_company") { + linkedRunId = (await seed()).runId; + await db.update(heartbeatRuns).set({ finishedAt: new Date(now) }).where(eq(heartbeatRuns.id, linkedRunId)); + } + if (kind === "wrong_agent") { + const otherAgentId = randomUUID(); + await db.insert(agents).values({ id: otherAgentId, companyId, name: "Other agent", role: "engineer", status: "idle", adapterType: "process" }); + await db.update(heartbeatRuns).set({ agentId: otherAgentId }).where(eq(heartbeatRuns.id, runId)); + } + if (kind === "unfinished_run") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, runId)); + if (kind === "retry") await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now + 1000) }).where(eq(heartbeatRuns.id, runId)); + if (kind === "accounting") await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, runId)); + await db.insert(agentWakeupRequests).values({ + companyId, agentId, source: "assignment", status: "coalesced", runId: linkedRunId, + finishedAt: kind === "unfinished" ? null : new Date(now), + }); + expect(await read()).toEqual(present); + }, + ); + it("blocks an active routine without waiting for its next due time", async () => { const { companyId } = await seed(); await db.insert(routines).values({ companyId, title: "Tomorrow", status: "active" }); @@ -268,6 +306,95 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp expect(await read()).toEqual(present); }); + async function seedFinishedLogin() { + const { companyId } = await seed(); + const [environment] = await db.insert(environments).values({ name: "Finished login fixture" }).returning(); + const [lease] = await db.insert(environmentLeases).values({ + companyId, environmentId: environment!.id, providerLeaseId: "fixture-login-resource", + status: "released", cleanupStatus: "success", releasedAt: new Date(now), + }).returning(); + const [session] = await db.insert(adapterAuthSessions).values({ + companyId, environmentId: environment!.id, adapterType: "codex_local", + startedByUserId: "fixture-user", publicSessionId: "finished-fixture-session", + status: "authenticated", finishedAt: new Date(now), providerLeaseId: lease!.providerLeaseId, + }).returning(); + return { companyId, environment: environment!, lease: lease!, session: session! }; + } + + it.each(["authenticated", "completed", "failed", "timed_out", "cancelled"] as const)( + "permits finished %s login history after confirmed provider cleanup", async status => { + const { session } = await seedFinishedLogin(); + await db.update(adapterAuthSessions).set({ status }).where(eq(adapterAuthSessions.id, session.id)); + expect(await read()).toEqual(none); + }, + ); + + it.each(["canonical", "uppercase"])("accepts a %s cleanup reference to the internal lease id", async format => { + const { session, lease } = await seedFinishedLogin(); + await db.update(adapterAuthSessions).set({ providerLeaseId: format === "uppercase" ? lease.id.toUpperCase() : lease.id }) + .where(eq(adapterAuthSessions.id, session.id)); + expect(await read()).toEqual(none); + }); + + it.each(["fixture-provider-id", "00000000-0000-0000-0000-00000000000z", randomUUID()])( + "accepts cleaned external provider reference %s without an unsafe UUID cast", async providerLeaseId => { + const { session, lease } = await seedFinishedLogin(); + await db.update(environmentLeases).set({ providerLeaseId }).where(eq(environmentLeases.id, lease.id)); + await db.update(adapterAuthSessions).set({ providerLeaseId }).where(eq(adapterAuthSessions.id, session.id)); + expect(await read()).toEqual(none); + }, + ); + + it("keeps a UUID-shaped external reference awake when an internal lease conflicts", async () => { + const { companyId, environment, session, lease } = await seedFinishedLogin(); + const conflictingId = randomUUID(); + await db.update(environmentLeases).set({ providerLeaseId: conflictingId }).where(eq(environmentLeases.id, lease.id)); + await db.update(adapterAuthSessions).set({ providerLeaseId: conflictingId }).where(eq(adapterAuthSessions.id, session.id)); + await db.insert(environmentLeases).values({ + id: conflictingId, companyId, environmentId: environment.id, + status: "released", cleanupStatus: "failed", releasedAt: new Date(now), + }); + expect(await read()).toEqual(present); + await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, conflictingId)); + expect(await read()).toEqual(none); + }); + + it("permits terminal login history without a provider resource", async () => { + const { session } = await seedFinishedLogin(); + await db.update(adapterAuthSessions).set({ providerLeaseId: null }).where(eq(adapterAuthSessions.id, session.id)); + expect(await read()).toEqual(none); + }); + + it.each(["cleanup_pending", "starting", "promoting", "awaiting_code", "submitting", "stored"] as const)( + "retains %s login work even with an old successful cleanup receipt", async status => { + const { session } = await seedFinishedLogin(); + await db.update(adapterAuthSessions).set({ status }).where(eq(adapterAuthSessions.id, session.id)); + expect(await read()).toEqual(present); + }, + ); + + it.each(["unfinished", "promotion_claim", "missing_lease", "wrong_company", "wrong_environment", "cleanup_failed", "cleanup_unknown", "unreleased", "conflicting_lease"])( + "retains terminal login history with %s cleanup evidence", async kind => { + const { companyId, session, lease, environment } = await seedFinishedLogin(); + if (kind === "unfinished") await db.update(adapterAuthSessions).set({ finishedAt: null }).where(eq(adapterAuthSessions.id, session.id)); + if (kind === "promotion_claim") await db.update(adapterAuthSessions).set({ promotionExpiresAt: new Date(now + 1000) }).where(eq(adapterAuthSessions.id, session.id)); + if (kind === "missing_lease") await db.delete(environmentLeases).where(eq(environmentLeases.id, lease.id)); + if (kind === "wrong_company") await db.update(environmentLeases).set({ companyId: (await seed()).companyId }).where(eq(environmentLeases.id, lease.id)); + if (kind === "wrong_environment") { + const [other] = await db.insert(environments).values({ name: "Other environment", driver: "sandbox" }).returning(); + await db.update(environmentLeases).set({ environmentId: other!.id }).where(eq(environmentLeases.id, lease.id)); + } + if (kind === "cleanup_failed") await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, lease.id)); + if (kind === "cleanup_unknown") await db.update(environmentLeases).set({ cleanupStatus: null }).where(eq(environmentLeases.id, lease.id)); + if (kind === "unreleased") await db.update(environmentLeases).set({ releasedAt: null }).where(eq(environmentLeases.id, lease.id)); + if (kind === "conflicting_lease") await db.insert(environmentLeases).values({ + companyId, environmentId: environment.id, providerLeaseId: lease.providerLeaseId, + status: "released", cleanupStatus: "failed", releasedAt: new Date(now), + }); + expect(await read()).toEqual(present); + }, + ); + it("reports a pending secret proposal with future expiry", async () => { const { companyId, agentId, runId } = await seed(); await db.insert(companySecretProposals).values({ diff --git a/server/src/services/idle-sleep-safety.ts b/server/src/services/idle-sleep-safety.ts index a8c7a80e13b..4f2059f3bb4 100644 --- a/server/src/services/idle-sleep-safety.ts +++ b/server/src/services/idle-sleep-safety.ts @@ -45,8 +45,16 @@ const WORK_CHECKS = [ `SELECT 1 FROM agent_api_keys WHERE revoked_at IS NULL`, `SELECT 1 FROM board_api_keys WHERE revoked_at IS NULL AND (expires_at IS NULL OR expires_at > now())`, - `SELECT 1 FROM agent_wakeup_requests WHERE - status NOT IN ('completed', 'failed', 'cancelled', 'skipped', 'timed_out')`, + // Coalescing consumes a request into another run. Retain it as work until + // both the request and that exact company/agent's run have finished. + `SELECT 1 FROM agent_wakeup_requests w WHERE + w.status NOT IN ('completed', 'failed', 'cancelled', 'skipped', 'timed_out', 'coalesced') + OR (w.status = 'coalesced' AND (w.finished_at IS NULL OR NOT EXISTS ( + SELECT 1 FROM heartbeat_runs r WHERE r.id = w.run_id + AND r.company_id = w.company_id AND r.agent_id = w.agent_id + AND r.status IN ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted') + AND r.finished_at IS NOT NULL AND r.scheduled_retry_at IS NULL + AND r.cost_accounting_pending = false)))`, `SELECT 1 FROM issues WHERE status NOT IN ('done', 'cancelled')`, // A completed issue can still need its first watchdog review, including a // retry after immediate evaluation failed before creating a review/run. @@ -62,6 +70,29 @@ const WORK_CHECKS = [ `SELECT 1 FROM issue_recovery_actions WHERE status NOT IN ('resolved', 'cancelled')`, `SELECT 1 FROM status_cards WHERE archived_at IS NULL`, `SELECT 1 FROM external_objects WHERE NOT is_terminal OR next_refresh_at IS NOT NULL`, + // Login history keeps its provider reference after teardown. A terminal + // label alone is insufficient: require a finished, unclaimed session and + // positive cleanup evidence for its resource in the same company/environment. + // Cast only UUID-shaped session references, leaving both lease indexes usable. + `SELECT 1 FROM adapter_auth_sessions s WHERE + s.status NOT IN ('authenticated', 'completed', 'failed', 'timed_out', 'cancelled') + OR s.finished_at IS NULL OR s.promotion_expires_at IS NOT NULL + OR (s.provider_lease_id IS NOT NULL AND NOT EXISTS ( + SELECT 1 FROM environment_leases l WHERE l.company_id = s.company_id + AND l.environment_id = s.environment_id + AND (l.provider_lease_id = s.provider_lease_id OR l.id = CASE + WHEN s.provider_lease_id ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' + THEN s.provider_lease_id::uuid END) + AND l.status IN ('released', 'expired') AND l.cleanup_status = 'success' + AND l.released_at IS NOT NULL)) + OR (s.provider_lease_id IS NOT NULL AND EXISTS ( + SELECT 1 FROM environment_leases l WHERE l.company_id = s.company_id + AND l.environment_id = s.environment_id + AND (l.provider_lease_id = s.provider_lease_id OR l.id = CASE + WHEN s.provider_lease_id ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' + THEN s.provider_lease_id::uuid END) + AND (l.status IN ('released', 'expired') AND l.cleanup_status = 'success' + AND l.released_at IS NOT NULL) IS NOT TRUE))`, // These less common work sources fail closed on any retained state. Their // terminal-state exceptions can be added with tests for the owning service. ...[ @@ -73,7 +104,7 @@ const WORK_CHECKS = [ "native_run_finalizations", "company_transfer_runs", "decisions", "decision_effect_executions", "decision_archive_notification_outbox", "browser_use_sessions", "browser_use_runs", "browser_use_browsers", "environment_custom_image_setup_sessions", "feedback_exports", - "adapter_auth_sessions", "company_secret_proposals", "execution_workspaces", + "company_secret_proposals", "execution_workspaces", "mcp_oauth_grants", "mcp_mutation_receipts", "mcp_event_subscriptions", "mcp_event_deliveries", "mcp_attachment_uploads", "dot_agent_bindings", "dot_runner_assignments", "dot_runner_operations", "dot_mailbox_items",