From 4b59e2a2ecc67e98b336b8eb4c09959681672d54 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 11 Oct 2026 10:01:57 -0700 Subject: [PATCH] fix(server): allow idle sleep with saved AI auth and verified lease history (#15897) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Automatic idle sleep must preserve accepted work, cleanup and recovery. > - Saved AI login connections are configuration without independent workers. > - Failed ephemeral leases can remain after their resources have stopped. > - Detached subscription observations and quota refreshes can outlive a response. > - This pull request distinguishes inert configuration and verified history from unfinished work. > - The benefit is safe idle eligibility while keeping credentials and audit history. ## Linked Issues or Issue Description **What happened?** An active saved AI login blocks idle sleep after its login and agent runs finish. A failed ephemeral lease also blocks after successful cleanup and provider termination. The connection guard currently hides detached quota and subscription work that can outlive the request that started it. **Expected behavior** An idle instance can sleep with saved AI credentials and verified failed-lease history. Accepted refresh work, recovery journals, unfinished login sessions, pending cleanup and run accounting must still block. **Steps to reproduce** 1. Save an AI connection with purpose `ai`, transport `runtime_auth` and status `active`. 2. Finish login and all agent work, then acquire an owned idle drain. 3. Read the idle safety report. The saved connection alone reports background work. 4. Repeat with a failed ephemeral lease whose exact provider termination and finished run are recorded. Its history also reports background work. **Paperclip version or commit** Based on `0bbd586a2f6`, after updates from master. **Deployment mode** Single-process managed or self-hosted server using the owned idle-drain protocol. Related work: #15522, #15761 and #15851. Searched open and closed pull requests for related idle-sleep changes. The open accounting recovery work in #15879 is separate; this change does not settle or discard accounting debt. ## What Changed - Exempt active saved AI runtime-auth configuration from the connection sleep check. Other connection types and unknown states keep their existing checks. - Track detached subscription discovery and provider observations until their actual promises settle. - Track underlying quota work through an HTTP timeout, including an OAuth exchange and the save of its replacement credential. - Inspect the credential-recovery directory without reading credential bytes. Journals, probes, temporary files, unexpected entries and read failures keep sleep blocked. Empty company directories left by successful recovery permit sleep. - Exempt a failed ephemeral lease only after successful cleanup, a release timestamp, an exact company/run/lease/resource termination receipt, and a finished run without retry or accounting debt. Export and cleanup intents still block. - Use the same failed-lease proof when checking finished login-session cleanup. - Add PostgreSQL and filesystem regression tests and document the behavior. ## Verification - Four focused suites passed: 186 tests covering durable guard decisions, local recovery, detached subscription work and quota timeouts. - `pnpm build` passed. - `pnpm -r typecheck` passed. - `pnpm --filter @paperclipai/server build` and a final server TypeScript check passed after the directory-scanner cleanup. - The final directory-scan recheck passed all 18 tests. - The original-base `pnpm test:run` attempt found three local failures. It was stopped before completion to update the branch after two master changes. All three also reproduce on unchanged master `e733bddc45e`: two Git default-branch fixtures in `workspace-runtime.test.ts` and one native runner recovery fixture in `native-session-resume.test.ts`. - Both Git fixtures pass with a command-scoped `init.defaultBranch=master` setting. The native fixture fails with the same result on both branches. No global Git setting was changed. - All CI gates passed on the final head `ff63497d9bef40f5f00567bc729fb17536cfe136`: [complete CI run](https://github.com/paperclipai/paperclip/actions/runs/38156312722). The only skipped checks are the two path-gated Storybook jobs. - Greptile Apex scored the final head 5/5 with no actionable findings and no unresolved review threads. The branch has no merge conflicts with master. - After the master update, `pnpm build` and `pnpm -r typecheck` passed again. All four focused suites passed again: 186 tests. - Staging deployment and live sleep/wake verification are pending the Core merge and the verified private Cloud build. No production rollout has been started. ## Risks - The failed-lease exception is deliberately narrow. Older records without exact termination evidence or a complete run receipt continue to block. - Pending OAuth recovery blocks sleep until it is recovered. No credential, history, cleanup obligation or accounting receipt is deleted. - The existing owned drain, ingress fence, local-work generation checks, plugin drain and backup checkpoint remain required. - No schema migration or provider mutation is introduced. Roll back to the previous application release to restore the conservative guard behavior. ## Model Used OpenAI GPT-6 (Codex), with repository inspection, code editing, test execution and read-only provider analysis. No additional model version or context-window size was supplied by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no unresolved comments - [x] I will address all reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/idle-sleep-safety.md | 27 +++- .../__tests__/company-quota-windows.test.ts | 30 ++++- server/src/__tests__/idle-local-work.test.ts | 53 +++++++- .../src/__tests__/idle-sleep-safety.test.ts | 120 +++++++++++++++++- .../__tests__/subscription-refresh.test.ts | 35 +++++ server/src/services/idle-local-work.ts | 28 +++- server/src/services/idle-sleep-safety.ts | 40 +++++- .../src/services/quota-credential-recovery.ts | 4 +- server/src/services/quota-windows.ts | 7 +- server/src/services/subscription-refresh.ts | 9 +- 10 files changed, 333 insertions(+), 20 deletions(-) diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index b4bf960ce5..e006680d1a 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -113,17 +113,36 @@ operator reconciliation; this API does not discard it to save compute. The report checks all companies in one bounded read-only database transaction. Queued and orphaned runs, timers, retries, accounting debt, unfinished issues, -cleanup, active routines, integrations, external API credentials, and plugin +cleanup, active routines, integrations, external ingress credentials, and plugin work block sleep. Completed ordinary run history does not. Less common work sources conservatively block on any retained records. Active issue watchdogs block sleep even when their watched issue is complete and no review has started. Missing migrations, malformed configuration and database errors produce unknown without exposing SQL or tenant data. +Active `runtime_auth` connections with purpose `ai` are saved credentials, not +independent workers. Their saved configuration does not block sleep. Login +sessions and runs still pass the existing durable checks. Detached subscription +observations and quota reads remain counted through their actual completion, +including token refresh after a quota response times out. Credential-recovery +journals, partial writes and probes block sleep across restart. Empty company +directories left after successful recovery do not; invalid paths and read +failures remain unknown. Credentials and connection records are retained. +MCP, chat and other connections still block until they have a wake contract. + +A failed ephemeral lease is history only with successful cleanup, a release +timestamp, and an exact provider termination receipt bound to its company, run, +lease and resource. Its run must be finished without retry or accounting debt. +Pending cleanup claims and export/stop-and-retain intents still block. Failure +labels or cleanup success without that proof do not authorize sleep. Lease +history is retained; this check performs no provider teardown or data updates. + Finished login-session history permits sleep only after its promotion claim is -cleared and any retained provider reference has a matching released or expired -lease with successful cleanup and a release timestamp in the same company and -environment. Missing cleanup evidence, `cleanup_pending`, and live login flows +cleared and any retained provider reference has a matching released, expired, +or verified failed ephemeral lease with successful cleanup and a release +timestamp in the same company and environment. The failed lease must satisfy +the same termination and run checks above. Missing cleanup evidence, +`cleanup_pending`, and live login flows remain blockers. A finished coalesced wake is history only when its exact company/agent's linked run has finished without retry or accounting debt. Image-setup history still blocks: promotion can succeed before teardown is diff --git a/server/src/__tests__/company-quota-windows.test.ts b/server/src/__tests__/company-quota-windows.test.ts index 646f821657..38d0ef238a 100644 --- a/server/src/__tests__/company-quota-windows.test.ts +++ b/server/src/__tests__/company-quota-windows.test.ts @@ -25,6 +25,7 @@ vi.mock("@paperclipai/adapter-claude-local/server", () => ({ fetchClaudeQuota: mocks.claude, })); import { fetchCompanyQuotaWindows } from "../services/quota-windows.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; let revision = 1; const db = { @@ -192,7 +193,8 @@ describe("connected account quotas", () => { it("bounds a stalled credential lookup and preserves its account identity", async () => { vi.useFakeTimers(); mocks.accounts.mockResolvedValue([account("stalled")]); - mocks.credential.mockReturnValue(new Promise(() => {})); + let finish!: (value: string) => void; + mocks.credential.mockReturnValue(new Promise(resolve => { finish = resolve; })); const pending = fetchCompanyQuotaWindows(db, "company-8", "user-8"); await vi.advanceTimersByTimeAsync(20_001); const [result] = await pending; @@ -203,6 +205,32 @@ describe("connected account quotas", () => { }); expect(result.accountKey).toBeTruthy(); expect(result.error).not.toContain("timed out"); + finish(JSON.stringify({ accessToken: "fixture" })); + await vi.advanceTimersByTimeAsync(0); + }); + + it("retains the actual credential refresh after its HTTP quota deadline expires", async () => { + vi.useFakeTimers(); + const before = idleWorkSnapshot(); + mocks.accounts.mockResolvedValue([account("tracked-refresh")]); + mocks.codex.mockRejectedValueOnce(new Error("401")); + let finish!: (value: string) => void; + mocks.refresh.mockReturnValue(new Promise(resolve => { finish = resolve; })); + const pending = fetchCompanyQuotaWindows(db, "tracked-company", "tracked-user"); + try { + await vi.advanceTimersByTimeAsync(0); + expect(mocks.refresh).toHaveBeenCalledOnce(); + expect(idleWorkSnapshot().active).toBe(before.active + 1); + await vi.advanceTimersByTimeAsync(20_001); + expect((await pending)[0].ok).toBe(false); + expect(idleWorkSnapshot().active).toBe(before.active + 1); + } finally { + finish(JSON.stringify({ accessToken: "fixture" })); + await vi.advanceTimersByTimeAsync(0); + await pending; + } + expect(idleWorkSnapshot().active).toBe(before.active); + expect(idleWorkSnapshot().generation).toBe(before.generation + 2); }); it("does not start a provider request when a credential resolves after its deadline", async () => { vi.useFakeTimers(); diff --git a/server/src/__tests__/idle-local-work.test.ts b/server/src/__tests__/idle-local-work.test.ts index 666c58d0b2..9728c97562 100644 --- a/server/src/__tests__/idle-local-work.test.ts +++ b/server/src/__tests__/idle-local-work.test.ts @@ -2,7 +2,7 @@ import { promises as fs } from "node:fs"; import os from "node:os"; import path from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { inspectIdleSpool, readIdleLocalWork, markIdleIngressTracked, markIdleStartupComplete, registerIdleSpoolDirectory } from "../services/idle-local-work.js"; +import { inspectIdleSpool, inspectIdleQuotaRecovery, idleQuotaRecoveryPath, readIdleLocalWork, markIdleIngressTracked, markIdleStartupComplete, registerIdleSpoolDirectory } from "../services/idle-local-work.js"; let root: string; beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "idle-spool-")); }); @@ -51,3 +51,54 @@ describe("idle cleanup inspection", () => { }); }); + +describe("idle credential recovery inspection", () => { + const company = "7238bc31-f3ca-42ba-b487-4122e77b8867"; + it("allows missing roots and empty company directories after successful recovery", async () => { + expect(await inspectIdleQuotaRecovery(path.join(root, "missing"))).toBe("none"); + expect(await inspectIdleQuotaRecovery(root)).toBe("none"); + await fs.mkdir(path.join(root, company)); + expect(await inspectIdleQuotaRecovery(root)).toBe("none"); + }); + it.each(["receipt.json", "malformed.json", "partial.tmp", "write.probe", "unexpected"])( + "retains every pending company entry without reading its contents: %s", async name => { + await fs.mkdir(path.join(root, company)); + const file = path.join(root, company, name); + await fs.writeFile(file, "unreadable recovery fixture"); + const read = vi.spyOn(fs, "readFile"); + expect(await inspectIdleQuotaRecovery(root)).toBe("present"); + expect(read).not.toHaveBeenCalled(); + await fs.rm(file); + expect(await inspectIdleQuotaRecovery(root)).toBe("none"); + }, + ); + it("retains unexpected root entries, nested directories and directory symlinks", async () => { + await fs.mkdir(path.join(root, "unexpected")); + expect(await inspectIdleQuotaRecovery(root)).toBe("present"); + await fs.rm(path.join(root, "unexpected"), { recursive: true }); + await fs.symlink(path.join(root, "missing"), path.join(root, company)); + expect(await inspectIdleQuotaRecovery(root)).toBe("present"); + await fs.rm(path.join(root, company)); + await fs.mkdir(path.join(root, company, "nested"), { recursive: true }); + expect(await inspectIdleQuotaRecovery(root)).toBe("present"); + }); + it("fails closed on invalid roots and directory read errors", async () => { + const file = path.join(root, "file"), link = path.join(root, "link"); + await fs.writeFile(file, ""); await fs.symlink(root, link); + expect(await inspectIdleQuotaRecovery(file)).toBe("unknown"); + expect(await inspectIdleQuotaRecovery(link)).toBe("unknown"); + vi.spyOn(fs, "opendir").mockRejectedValue(Object.assign(new Error("private path"), { code: "EACCES" })); + expect(await inspectIdleQuotaRecovery(root)).toBe("unknown"); + }); + it("keeps persisted credential recovery awake after the request and process work have ended", async () => { + vi.stubEnv("PAPERCLIP_HOME", root); + markIdleIngressTracked(); markIdleStartupComplete({ scheduledBackups: false }); + const directory = path.join(idleQuotaRecoveryPath(), company); + await fs.mkdir(directory, { recursive: true }); + const file = path.join(directory, "pending.json"); + await fs.writeFile(file, "encrypted journal fixture"); + expect(await readIdleLocalWork()).toBe("present"); + await fs.rm(file); + expect(await readIdleLocalWork()).toBe("none"); + }); +}); diff --git a/server/src/__tests__/idle-sleep-safety.test.ts b/server/src/__tests__/idle-sleep-safety.test.ts index f8e1524107..4b7d731972 100644 --- a/server/src/__tests__/idle-sleep-safety.test.ts +++ b/server/src/__tests__/idle-sleep-safety.test.ts @@ -6,7 +6,7 @@ import { agentInstructionWorkingCopies, agents, agentApiKeys, agentWakeupRequests, companies, companySecretProposals, createDb, adapterAuthSessions, environments, environmentLeases, executionWorkspaces, heartbeatRuns, issues, issueWatchdogs, issueThreadInteractions, issueQuestionResponseDeliveries, - projects, routines, type Db, + projects, routines, toolApplications, toolConnections, type Db, } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js"; @@ -443,6 +443,124 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp expect(await read()).toEqual(present); }); + async function seedAiConnection() { + const seeded = await seed(); + const [application] = await db.insert(toolApplications).values({ + companyId: seeded.companyId, name: "AI fixture", type: "rest_api", + }).returning(); + const [connection] = await db.insert(toolConnections).values({ + companyId: seeded.companyId, applicationId: application!.id, + name: "Saved AI login", uid: randomUUID(), connectionPurpose: "ai", + transport: "runtime_auth", status: "active", enabled: true, + }).returning(); + return { ...seeded, connection: connection! }; + } + + it("allows saved AI credentials while preserving their connection", async () => { + const { connection } = await seedAiConnection(); + expect(await read()).toEqual(none); + expect(await db.select().from(toolConnections).where(eq(toolConnections.id, connection.id))) + .toMatchObject([{ status: "active", enabled: true }]); + }); + + it.each(["draft", "active_login", "pending_run", "accepted_refresh"])( + "keeps AI connections awake for %s work", async kind => { + const { companyId, runId, connection } = await seedAiConnection(); + if (kind === "draft") await db.update(toolConnections).set({ status: "draft" }).where(eq(toolConnections.id, connection.id)); + if (kind === "pending_run") await db.update(heartbeatRuns).set({ status: "queued" }).where(eq(heartbeatRuns.id, runId)); + if (kind === "active_login") { + const [environment] = await db.insert(environments).values({ name: "Active AI login" }).returning(); + await db.insert(adapterAuthSessions).values({ + companyId, environmentId: environment!.id, connectionId: connection.id, + adapterType: "codex_local", startedByUserId: "fixture-user", + publicSessionId: randomUUID(), status: "waiting_for_user", + }); + } + const finish = kind === "accepted_refresh" ? beginIdleTrackedWork() : undefined; + try { expect(await read()).toEqual(kind === "accepted_refresh" ? unknown : present); } + finally { finish?.(); } + }, + ); + + it("retains remote MCP connections alongside inert AI credentials", async () => { + const { companyId, connection } = await seedAiConnection(); + await db.insert(toolConnections).values({ + companyId, applicationId: connection.applicationId, name: "Remote tools", uid: randomUUID(), + connectionPurpose: "tool", transport: "mcp_remote", status: "active", enabled: true, + }); + expect(await read()).toEqual(present); + }); + + async function seedFinishedFailedLease() { + const seeded = await seed(); + await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(now) }) + .where(eq(heartbeatRuns.id, seeded.runId)); + const [environment] = await db.insert(environments).values({ name: "Failed turn environment", driver: "sandbox" }).returning(); + const leaseId = randomUUID(); + const receipt = { + schema: "paperclip.remote-termination.v1", companyId: seeded.companyId, + runId: seeded.runId, leaseId, provider: "plugin:demo:sandbox", + providerLeaseId: "fixture-resource", state: "destroyed", confirmedAt: new Date(now).toISOString(), + }; + const [lease] = await db.insert(environmentLeases).values({ + id: leaseId, companyId: seeded.companyId, environmentId: environment!.id, + heartbeatRunId: seeded.runId, status: "failed", leasePolicy: "ephemeral", + provider: receipt.provider, providerLeaseId: receipt.providerLeaseId, + acquiredAt: new Date(now - 1000), releasedAt: new Date(now), cleanupStatus: "success", + metadata: { remoteExecutionTermination: receipt }, + }).returning(); + return { ...seeded, environment: environment!, lease: lease!, receipt }; + } + + it.each(["destroyed", "stopped"])("allows failed lease history after exact provider %s proof", async state => { + const { lease, receipt } = await seedFinishedFailedLease(); + await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: { ...receipt, state } } }) + .where(eq(environmentLeases.id, lease.id)); + expect(await read()).toEqual(none); + expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id))) + .toMatchObject([{ status: "failed", cleanupStatus: "success" }]); + }); + + it.each([ + "cleanup_failed", "cleanup_unknown", "unreleased", "retained_policy", "reusable_policy", + "active", "pending_cleanup", "missing_receipt", "wrong_company", "wrong_run", "wrong_lease", + "wrong_provider", "wrong_resource", "not_stopped", "missing_run", "unfinished_run", + "retry", "accounting", "cleanup_in_flight", "pending_export", "pending_retain", + ])("retains failed leases with %s evidence", async kind => { + const { lease, receipt, runId } = await seedFinishedFailedLease(); + const change: Partial = {}; + if (kind === "cleanup_failed") change.cleanupStatus = "failed"; + if (kind === "cleanup_unknown") change.cleanupStatus = null; + if (kind === "unreleased") change.releasedAt = null; + if (kind === "retained_policy") change.leasePolicy = "retain_on_failure"; + if (kind === "reusable_policy") change.leasePolicy = "reuse_by_environment"; + if (kind === "active" || kind === "pending_cleanup") change.status = kind; + if (kind === "missing_receipt") change.metadata = {}; + const fields: Record = { wrong_company: "companyId", wrong_run: "runId", wrong_lease: "leaseId", wrong_provider: "provider", wrong_resource: "providerLeaseId", not_stopped: "state" }; + if (fields[kind]) change.metadata = { remoteExecutionTermination: { ...receipt, [fields[kind]]: "wrong" } }; + if (kind === "missing_run") change.heartbeatRunId = null; + if (kind === "unfinished_run") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, runId)); + if (kind === "retry") await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now + 1000) }).where(eq(heartbeatRuns.id, runId)); + if (kind === "accounting") await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, runId)); + if (kind === "cleanup_in_flight") change.metadata = { remoteExecutionTermination: receipt, pendingCleanupInFlight: true }; + if (kind === "pending_export") change.metadata = { remoteExecutionTermination: receipt, nativeWorkspaceExportResume: {} }; + if (kind === "pending_retain") change.metadata = { remoteExecutionTermination: receipt, sandboxStopAndRetain: {} }; + if (Object.keys(change).length) await db.update(environmentLeases).set(change).where(eq(environmentLeases.id, lease.id)); + expect(await read()).toEqual(present); + }); + + it("uses the same failed-lease cleanup proof for finished login history", async () => { + const { companyId, environment, lease } = await seedFinishedFailedLease(); + await db.insert(adapterAuthSessions).values({ + companyId, environmentId: environment.id, adapterType: "codex_local", startedByUserId: "fixture-user", + publicSessionId: randomUUID(), providerLeaseId: lease.providerLeaseId, + status: "authenticated", finishedAt: new Date(now), + }); + expect(await read()).toEqual(none); + await db.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, lease.id)); + expect(await read()).toEqual(present); + }); + it("reports a login sandbox awaiting cleanup", async () => { const { companyId } = await seed(); const [environment] = await db.insert(environments).values({ name: "Login fixture" }).returning(); diff --git a/server/src/__tests__/subscription-refresh.test.ts b/server/src/__tests__/subscription-refresh.test.ts index abe237d87f..cb2f4bddc4 100644 --- a/server/src/__tests__/subscription-refresh.test.ts +++ b/server/src/__tests__/subscription-refresh.test.ts @@ -2,6 +2,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; import type { Db } from "@paperclipai/db"; import { refreshCompanySubscriptions, refreshSubscriptionConnection } from "../services/subscription-refresh.js"; import type { SubscriptionConnection } from "../services/subscriptions.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; const mocks = vi.hoisted(() => ({ accounts: vi.fn(), credential: vi.fn(), register: vi.fn(), probe: vi.fn() })); vi.mock("../services/ai-connections.js", () => ({ aiConnectionService: () => ({ subscriptionAccounts: mocks.accounts, credential: mocks.credential }) })); @@ -28,6 +29,40 @@ beforeEach(() => { }); describe("subscription discovery capacity", () => { + it("tracks detached provider refresh through its final database write", async () => { + const db = database(), probe = deferred<{ credentialKey: string }>(), save = deferred(); + const before = idleWorkSnapshot(); + mocks.probe.mockReturnValue(probe.promise); + mocks.register.mockReturnValue(save.promise); + const task = refreshSubscriptionConnection(db, input("tracked"), "tracked"); + try { + await vi.waitFor(() => expect(mocks.probe).toHaveBeenCalledOnce()); + expect(idleWorkSnapshot().active).toBe(before.active + 1); + expect(refreshSubscriptionConnection(db, input("tracked"), "tracked")).toBe(task); + expect(idleWorkSnapshot().active).toBe(before.active + 1); + probe.resolve({ credentialKey: "fixture-key" }); + await vi.waitFor(() => expect(mocks.register).toHaveBeenCalledOnce()); + expect(idleWorkSnapshot().active).toBe(before.active + 1); + } finally { + probe.resolve({ credentialKey: "fixture-key" }); save.resolve("tracked"); await task; + } + expect(idleWorkSnapshot().active).toBe(before.active); + expect(idleWorkSnapshot().generation).toBe(before.generation + 2); + }); + + it("tracks detached discovery while it waits for account credentials", async () => { + const db = database(), accounts = deferred<[]>(); + const before = idleWorkSnapshot(); + mocks.accounts.mockReturnValue(accounts.promise); + const task = refreshCompanySubscriptions(db, "tracked-company", "alice"); + try { + expect(idleWorkSnapshot().active).toBe(before.active + 1); + expect(refreshCompanySubscriptions(db, "tracked-company", "alice")).toBe(task); + expect(idleWorkSnapshot().active).toBe(before.active + 1); + } finally { accounts.resolve([]); await task; } + expect(idleWorkSnapshot().active).toBe(before.active); + }); + it.each([false, true])("waits for a provider slot without dropping accepted discovery (provider fails: %s)", async (fails) => { const db = database(); const gates = Array.from({ length: 5 }, () => deferred()); diff --git a/server/src/services/idle-local-work.ts b/server/src/services/idle-local-work.ts index 2d54ad9030..0338a66330 100644 --- a/server/src/services/idle-local-work.ts +++ b/server/src/services/idle-local-work.ts @@ -9,6 +9,7 @@ export function registerIdleSpoolDirectory(directory: string): void { spoolDirec export const idleAccountingSpoolPath = () => path.join(resolvePaperclipInstanceRoot(), "accounting-receipts"); export const idleOrphanSpoolPath = () => process.env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR ?? path.join(resolvePaperclipInstanceRoot(), "data", "sandbox-orphan-cleanup"); +export const idleQuotaRecoveryPath = () => path.join(resolvePaperclipInstanceRoot(), "quota-credential-recovery"); let startupComplete = false; let ingressTracked = false; let scheduledBackups = false; @@ -33,13 +34,38 @@ export async function inspectIdleSpool(directory: string): Promise { + try { + const info = await fs.lstat(directory); + if (!info.isDirectory() || info.isSymbolicLink()) return "unknown"; + const dir = await fs.opendir(directory); + try { + for (let entry = await dir.read(); entry !== null; entry = await dir.read()) { + if (!entry.isDirectory() || entry.isSymbolicLink() + || !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(entry.name)) return "present"; + const state = await inspectIdleSpool(path.join(directory, entry.name)); + if (state !== "none") return state; + } + return "none"; + } finally { await dir.close(); } + } catch (error) { + return (error as NodeJS.ErrnoException).code === "ENOENT" ? "none" : "unknown"; + } +} + export async function readIdleLocalWork(options: { backupCheckpoint?: boolean } = {}): Promise { if (!startupComplete || !ingressTracked) return "unknown"; // This exemption belongs only to a scan which will finish a new backup // under the same admission hold before it can return an empty report. if ((scheduledBackups && !options.backupCheckpoint) || idleWorkSnapshot().active !== 0) return "present"; const directories = new Set([...spoolDirectories, idleAccountingSpoolPath(), idleOrphanSpoolPath()]); - const results = await Promise.all([...directories].map(inspectIdleSpool)); + const results = await Promise.all([ + ...[...directories].map(inspectIdleSpool), + inspectIdleQuotaRecovery(idleQuotaRecoveryPath()), + ]); if (results.includes("unknown")) return "unknown"; return results.includes("present") ? "present" : "none"; } diff --git a/server/src/services/idle-sleep-safety.ts b/server/src/services/idle-sleep-safety.ts index 3e637a8c0a..4d1a5f6196 100644 --- a/server/src/services/idle-sleep-safety.ts +++ b/server/src/services/idle-sleep-safety.ts @@ -22,6 +22,32 @@ export type InspectIdlePlugins = (hold: { ownerId: string; expiresAt: number }) pluginIds: string[]; }>; +// A failed turn may have successfully torn down its ephemeral allocation. A +// cleanup label alone is not provider termination proof. Require the exact +// host-owned receipt and finished run, and retain all cleanup/export intents. +// Aliases are fixed at the call sites, never supplied by a client. +function finishedFailedLease(alias: string): string { + const receipt = `${alias}.metadata->'remoteExecutionTermination'`; + return `${alias}.status = 'failed' AND ${alias}.lease_policy = 'ephemeral' + AND ${alias}.cleanup_status = 'success' AND ${alias}.released_at IS NOT NULL + AND COALESCE(${alias}.metadata->>'pendingCleanupInFlight', 'false') = 'false' + AND NOT (COALESCE(${alias}.metadata, '{}'::jsonb) ?| ARRAY['nativeWorkspaceExportResume', 'sandboxStopAndRetain']) + AND ${alias}.provider IS NOT NULL AND ${alias}.provider <> 'local' + AND ${alias}.provider_lease_id IS NOT NULL AND ${alias}.provider_lease_id <> '' + AND ${receipt}->>'schema' = 'paperclip.remote-termination.v1' + AND ${receipt}->>'companyId' = ${alias}.company_id::text + AND ${receipt}->>'runId' = ${alias}.heartbeat_run_id::text + AND ${receipt}->>'leaseId' = ${alias}.id::text + AND ${receipt}->>'provider' = ${alias}.provider + AND ${receipt}->>'providerLeaseId' = ${alias}.provider_lease_id + AND ${receipt}->>'state' IN ('stopped', 'destroyed') + AND EXISTS (SELECT 1 FROM heartbeat_runs r WHERE r.id = ${alias}.heartbeat_run_id + AND r.company_id = ${alias}.company_id + AND r.status IN ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted') + AND r.finished_at IS NOT NULL AND r.scheduled_retry_at IS NULL + AND r.cost_accounting_pending = false)`; +} + // These are deliberately conservative. Completed agent runs and workspace // operations are history, not reasons to keep an otherwise idle instance up. // Other background features remain awake until their work and inbound events @@ -60,13 +86,18 @@ const WORK_CHECKS = [ // retry after immediate evaluation failed before creating a review/run. `SELECT 1 FROM issue_watchdogs WHERE status <> 'disabled'`, `SELECT 1 FROM workspace_operations WHERE status NOT IN ('succeeded', 'failed', 'cancelled')`, - `SELECT 1 FROM environment_leases WHERE status NOT IN ('released', 'expired')`, + `SELECT 1 FROM environment_leases l WHERE status NOT IN ('released', 'expired') + AND (${finishedFailedLease("l")}) IS NOT TRUE`, `SELECT 1 FROM routines WHERE status NOT IN ('paused', 'archived')`, `SELECT 1 FROM routine_runs WHERE status NOT IN ('completed', 'failed', 'skipped', 'cancelled')`, `SELECT 1 FROM plugin_jobs WHERE status <> 'paused'`, `SELECT 1 FROM plugin_job_runs WHERE status NOT IN ('succeeded', 'failed', 'cancelled')`, `SELECT 1 FROM plugin_webhook_deliveries WHERE status NOT IN ('succeeded', 'failed')`, - `SELECT 1 FROM tool_connections WHERE status <> 'archived'`, + // Saved AI credentials have no independent worker. Login sessions, accepted + // quota/refresh work and recovery journals have their own checks. Other + // integrations still need an ingress/scheduler wake contract. + `SELECT 1 FROM tool_connections WHERE status <> 'archived' + AND NOT (status = 'active' AND connection_purpose = 'ai' AND transport = 'runtime_auth')`, `SELECT 1 FROM issue_recovery_actions WHERE status NOT IN ('resolved', 'cancelled')`, `SELECT 1 FROM status_cards WHERE archived_at IS NULL`, `SELECT 1 FROM external_objects WHERE NOT is_terminal OR next_refresh_at IS NOT NULL`, @@ -83,7 +114,8 @@ const WORK_CHECKS = [ AND (l.provider_lease_id = s.provider_lease_id OR l.id = CASE WHEN s.provider_lease_id ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' THEN s.provider_lease_id::uuid END) - AND l.status IN ('released', 'expired') AND l.cleanup_status = 'success' + AND (l.status IN ('released', 'expired') OR (${finishedFailedLease("l")})) + AND l.cleanup_status = 'success' AND l.released_at IS NOT NULL)) OR (s.provider_lease_id IS NOT NULL AND EXISTS ( SELECT 1 FROM environment_leases l WHERE l.company_id = s.company_id @@ -91,7 +123,7 @@ const WORK_CHECKS = [ AND (l.provider_lease_id = s.provider_lease_id OR l.id = CASE WHEN s.provider_lease_id ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' THEN s.provider_lease_id::uuid END) - AND (l.status IN ('released', 'expired') AND l.cleanup_status = 'success' + AND ((l.status IN ('released', 'expired') OR (${finishedFailedLease("l")})) AND l.cleanup_status = 'success' AND l.released_at IS NOT NULL) IS NOT TRUE))`, // Answer delivery history is not work once its acknowledged handoff has // finished. Keep pending deliveries, unfinished target runs, and missing or diff --git a/server/src/services/quota-credential-recovery.ts b/server/src/services/quota-credential-recovery.ts index d666641c05..583ae23524 100644 --- a/server/src/services/quota-credential-recovery.ts +++ b/server/src/services/quota-credential-recovery.ts @@ -2,9 +2,9 @@ import { createHash, randomUUID } from "node:crypto"; import { promises as fs } from "node:fs"; import path from "node:path"; import { z } from "zod"; -import { resolvePaperclipInstanceRoot } from "../home-paths.js"; import { localEncryptedProvider } from "../secrets/local-encrypted-provider.js"; import { ensureDurableDirectory, syncDirectory } from "../lib/durable-directory.js"; +import { idleQuotaRecoveryPath } from "./idle-local-work.js"; const recoverySchema = z.object({ companyId: z.string().uuid(), grantId: z.string().uuid(), connectionId: z.string().uuid(), secretId: z.string().uuid(), @@ -13,7 +13,7 @@ const recoverySchema = z.object({ type Recovery = z.infer; export const quotaCredentialHash = (value: string) => createHash("sha256").update(value).digest("hex"); export const quotaCredentialRecoveryPath = (companyId: string, grantId: string) => - path.join(resolvePaperclipInstanceRoot(), "quota-credential-recovery", companyId, `${grantId}.json`); + path.join(idleQuotaRecoveryPath(), companyId, `${grantId}.json`); /** Called only while holding the company credential mutation lock. The local * encrypted provider uses the instance secrets master key, including when the diff --git a/server/src/services/quota-windows.ts b/server/src/services/quota-windows.ts index 2d363969b2..a31b2153fd 100644 --- a/server/src/services/quota-windows.ts +++ b/server/src/services/quota-windows.ts @@ -8,6 +8,7 @@ import { companySecrets, companySecretVersions, userSecretDefinitions, type Db } import { fetchCodexQuota } from "@paperclipai/adapter-codex-local/server"; import { fetchClaudeQuota } from "@paperclipai/adapter-claude-local/server"; import { aiConnectionService } from "./ai-connections.js"; +import { trackIdleWork } from "./task-admission.js"; const accountRequests = new Map }>(); const publicError = "Subscription quota is currently unavailable. Check usage with your provider."; @@ -51,7 +52,9 @@ export async function fetchCompanyQuotaWindows(db: Db, companyId: string, userId let pending = accountRequests.get(key); if (!pending || pending.expires <= Date.now()) { const controller = new AbortController(); - const result = (async (): Promise => { + // The HTTP deadline can expire while an OAuth exchange still needs to + // save its consumed token. Track the actual work, not its timeout wrapper. + const result = trackIdleWork((async (): Promise => { let credentialResolved = false; try { const value = await service.credential(row); @@ -104,7 +107,7 @@ export async function fetchCompanyQuotaWindows(db: Db, companyId: string, userId return { ...base, ok: false, windows: [], error: publicError, errorFamily }; } - })(); + })()); const bounded: Promise = withQuotaTimeout(base.provider, result, () => controller.abort()).then(quota => { const observation = { ...base, ...quota, ...(quota.ok ? {} : { error: publicError }) }; if (observation.ok && observation.accountKey !== accountKey) { diff --git a/server/src/services/subscription-refresh.ts b/server/src/services/subscription-refresh.ts index cc46f8333c..24322d099e 100644 --- a/server/src/services/subscription-refresh.ts +++ b/server/src/services/subscription-refresh.ts @@ -5,6 +5,7 @@ import { subscriptionService, type SubscriptionConnection } from "./subscription import { probeSubscriptionIdentity } from "./subscription-identity.js"; import { logger } from "../middleware/logger.js"; import { tooManyRequests } from "../errors.js"; +import { trackIdleWork } from "./task-admission.js"; const jobs = new WeakMap>>(); const discovery = new WeakMap>>(); @@ -19,7 +20,7 @@ export function refreshSubscriptionConnection(db: Db, input: SubscriptionConnect // Bound provider concurrency across all companies on this instance. Dashboard // discovery waits below; opportunistic run observations remain due when busy. if (active.size >= 4) return Promise.resolve(); - const task = (async () => { + const task = trackIdleWork((async () => { const [claimed] = await db.update(aiSubscriptions).set({ lastCheckedAt: new Date() }).where(and( eq(aiSubscriptions.companyId, input.companyId), eq(aiSubscriptions.id, id), or(isNull(aiSubscriptions.lastCheckedAt), lt(aiSubscriptions.lastCheckedAt, new Date(Date.now() - REFRESH_MS))), @@ -38,7 +39,7 @@ export function refreshSubscriptionConnection(db: Db, input: SubscriptionConnect })().catch(() => { // No raw provider/credential errors are logged or returned to a user. logger.warn({ companyId: input.companyId, subscriptionId: id }, "Subscription observation could not be saved"); - }).finally(() => active!.delete(id)); + }).finally(() => active!.delete(id))); active.set(id, task); return task; } @@ -50,7 +51,7 @@ export function refreshCompanySubscriptions(db: Db, companyId: string, userId: s const existing = active.get(key); if (existing) return existing; if (active.size >= 20) throw tooManyRequests("Subscription account checks are busy. Try again shortly."); - const task = (async () => { + const task = trackIdleWork((async () => { const service = aiConnectionService(db); // Uses the existing credential audience. Being a company admin does not // authorize probing another member's personal credentials. @@ -75,7 +76,7 @@ export function refreshCompanySubscriptions(db: Db, companyId: string, userId: s } } })().catch(() => logger.warn({ companyId }, "Subscription discovery unavailable")) - .finally(() => active!.delete(key)); + .finally(() => active!.delete(key))); active.set(key, task); return task; }