fix(server): allow idle sleep with saved AI auth and verified lease history (#15897)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Automatic idle sleep must preserve accepted work, cleanup and
recovery.
> - Saved AI login connections are configuration without independent
workers.
> - Failed ephemeral leases can remain after their resources have
stopped.
> - Detached subscription observations and quota refreshes can outlive a
response.
> - This pull request distinguishes inert configuration and verified
history from unfinished work.
> - The benefit is safe idle eligibility while keeping credentials and
audit history.

## Linked Issues or Issue Description

**What happened?**

An active saved AI login blocks idle sleep after its login and agent
runs finish. A failed ephemeral lease also blocks after successful
cleanup and provider termination. The connection guard currently hides
detached quota and subscription work that can outlive the request that
started it.

**Expected behavior**

An idle instance can sleep with saved AI credentials and verified
failed-lease history. Accepted refresh work, recovery journals,
unfinished login sessions, pending cleanup and run accounting must still
block.

**Steps to reproduce**

1. Save an AI connection with purpose `ai`, transport `runtime_auth` and
status `active`.
2. Finish login and all agent work, then acquire an owned idle drain.
3. Read the idle safety report. The saved connection alone reports
background work.
4. Repeat with a failed ephemeral lease whose exact provider termination
and finished run are recorded. Its history also reports background work.

**Paperclip version or commit**

Based on `0bbd586a2f6`, after updates from master.

**Deployment mode**

Single-process managed or self-hosted server using the owned idle-drain
protocol.

Related work: #15522, #15761 and #15851. Searched open and closed pull
requests for related idle-sleep changes. The open accounting recovery
work in #15879 is separate; this change does not settle or discard
accounting debt.

## What Changed

- Exempt active saved AI runtime-auth configuration from the connection
sleep check. Other connection types and unknown states keep their
existing checks.
- Track detached subscription discovery and provider observations until
their actual promises settle.
- Track underlying quota work through an HTTP timeout, including an
OAuth exchange and the save of its replacement credential.
- Inspect the credential-recovery directory without reading credential
bytes. Journals, probes, temporary files, unexpected entries and read
failures keep sleep blocked. Empty company directories left by
successful recovery permit sleep.
- Exempt a failed ephemeral lease only after successful cleanup, a
release timestamp, an exact company/run/lease/resource termination
receipt, and a finished run without retry or accounting debt. Export and
cleanup intents still block.
- Use the same failed-lease proof when checking finished login-session
cleanup.
- Add PostgreSQL and filesystem regression tests and document the
behavior.

## Verification

- Four focused suites passed: 186 tests covering durable guard
decisions, local recovery, detached subscription work and quota
timeouts.
- `pnpm build` passed.
- `pnpm -r typecheck` passed.
- `pnpm --filter @paperclipai/server build` and a final server
TypeScript check passed after the directory-scanner cleanup.
- The final directory-scan recheck passed all 18 tests.
- The original-base `pnpm test:run` attempt found three local failures.
It was stopped before completion to update the branch after two master
changes. All three also reproduce on unchanged master `e733bddc45e`: two
Git default-branch fixtures in `workspace-runtime.test.ts` and one
native runner recovery fixture in `native-session-resume.test.ts`.
- Both Git fixtures pass with a command-scoped
`init.defaultBranch=master` setting. The native fixture fails with the
same result on both branches. No global Git setting was changed.
- All CI gates passed on the final head
`ff63497d9bef40f5f00567bc729fb17536cfe136`: [complete CI
run](https://github.com/paperclipai/paperclip/actions/runs/38156312722).
The only skipped checks are the two path-gated Storybook jobs.
- Greptile Apex scored the final head 5/5 with no actionable findings
and no unresolved review threads. The branch has no merge conflicts with
master.
- After the master update, `pnpm build` and `pnpm -r typecheck` passed
again. All four focused suites passed again: 186 tests.
- Staging deployment and live sleep/wake verification are pending the
Core merge and the verified private Cloud build. No production rollout
has been started.

## Risks

- The failed-lease exception is deliberately narrow. Older records
without exact termination evidence or a complete run receipt continue to
block.
- Pending OAuth recovery blocks sleep until it is recovered. No
credential, history, cleanup obligation or accounting receipt is
deleted.
- The existing owned drain, ingress fence, local-work generation checks,
plugin drain and backup checkpoint remain required.
- No schema migration or provider mutation is introduced. Roll back to
the previous application release to restore the conservative guard
behavior.

## Model Used

OpenAI GPT-6 (Codex), with repository inspection, code editing, test
execution and read-only provider analysis. No additional model version
or context-window size was supplied by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no unresolved comments
- [x] I will address all reviewer comments before requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-10-11 10:01:57 -07:00
1 parent 0bbd586a2f
commit 4b59e2a2ec
10 files changed
+333 -20

No files matched your search

+23 -4
View File
@@ -113,17 +113,36 @@ operator reconciliation; this API does not discard it to save compute.
The report checks all companies in one bounded read-only database transaction.
Queued and orphaned runs, timers, retries, accounting debt, unfinished issues,
cleanup, active routines, integrations, external API credentials, and plugin
cleanup, active routines, integrations, external ingress credentials, and plugin
work block sleep. Completed ordinary run history does not. Less common work
sources conservatively block on any retained records. Active issue watchdogs
block sleep even when their watched issue is complete and no review has started.
Missing migrations, malformed configuration and database errors produce unknown
without exposing SQL or tenant data.
Active `runtime_auth` connections with purpose `ai` are saved credentials, not
independent workers. Their saved configuration does not block sleep. Login
sessions and runs still pass the existing durable checks. Detached subscription
observations and quota reads remain counted through their actual completion,
including token refresh after a quota response times out. Credential-recovery
journals, partial writes and probes block sleep across restart. Empty company
directories left after successful recovery do not; invalid paths and read
failures remain unknown. Credentials and connection records are retained.
MCP, chat and other connections still block until they have a wake contract.
A failed ephemeral lease is history only with successful cleanup, a release
timestamp, and an exact provider termination receipt bound to its company, run,
lease and resource. Its run must be finished without retry or accounting debt.
Pending cleanup claims and export/stop-and-retain intents still block. Failure
labels or cleanup success without that proof do not authorize sleep. Lease
history is retained; this check performs no provider teardown or data updates.
Finished login-session history permits sleep only after its promotion claim is
cleared and any retained provider reference has a matching released or expired
lease with successful cleanup and a release timestamp in the same company and
environment. Missing cleanup evidence, `cleanup_pending`, and live login flows
cleared and any retained provider reference has a matching released, expired,
or verified failed ephemeral lease with successful cleanup and a release
timestamp in the same company and environment. The failed lease must satisfy
the same termination and run checks above. Missing cleanup evidence,
`cleanup_pending`, and live login flows
remain blockers. A finished coalesced wake is history only when its exact
company/agent's linked run has finished without retry or accounting debt.
Image-setup history still blocks: promotion can succeed before teardown is
@@ -25,6 +25,7 @@ vi.mock("@paperclipai/adapter-claude-local/server", () => ({
fetchClaudeQuota: mocks.claude,
}));
import { fetchCompanyQuotaWindows } from "../services/quota-windows.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
let revision = 1;
const db = {
@@ -192,7 +193,8 @@ describe("connected account quotas", () => {
it("bounds a stalled credential lookup and preserves its account identity", async () => {
vi.useFakeTimers();
mocks.accounts.mockResolvedValue([account("stalled")]);
mocks.credential.mockReturnValue(new Promise(() => {}));
let finish!: (value: string) => void;
mocks.credential.mockReturnValue(new Promise<string>(resolve => { finish = resolve; }));
const pending = fetchCompanyQuotaWindows(db, "company-8", "user-8");
await vi.advanceTimersByTimeAsync(20_001);
const [result] = await pending;
@@ -203,6 +205,32 @@ describe("connected account quotas", () => {
});
expect(result.accountKey).toBeTruthy();
expect(result.error).not.toContain("timed out");
finish(JSON.stringify({ accessToken: "fixture" }));
await vi.advanceTimersByTimeAsync(0);
});
it("retains the actual credential refresh after its HTTP quota deadline expires", async () => {
vi.useFakeTimers();
const before = idleWorkSnapshot();
mocks.accounts.mockResolvedValue([account("tracked-refresh")]);
mocks.codex.mockRejectedValueOnce(new Error("401"));
let finish!: (value: string) => void;
mocks.refresh.mockReturnValue(new Promise<string>(resolve => { finish = resolve; }));
const pending = fetchCompanyQuotaWindows(db, "tracked-company", "tracked-user");
try {
await vi.advanceTimersByTimeAsync(0);
expect(mocks.refresh).toHaveBeenCalledOnce();
expect(idleWorkSnapshot().active).toBe(before.active + 1);
await vi.advanceTimersByTimeAsync(20_001);
expect((await pending)[0].ok).toBe(false);
expect(idleWorkSnapshot().active).toBe(before.active + 1);
} finally {
finish(JSON.stringify({ accessToken: "fixture" }));
await vi.advanceTimersByTimeAsync(0);
await pending;
}
expect(idleWorkSnapshot().active).toBe(before.active);
expect(idleWorkSnapshot().generation).toBe(before.generation + 2);
});
it("does not start a provider request when a credential resolves after its deadline", async () => {
vi.useFakeTimers();
+52 -1
View File
@@ -2,7 +2,7 @@ import { promises as fs } from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { inspectIdleSpool, readIdleLocalWork, markIdleIngressTracked, markIdleStartupComplete, registerIdleSpoolDirectory } from "../services/idle-local-work.js";
import { inspectIdleSpool, inspectIdleQuotaRecovery, idleQuotaRecoveryPath, readIdleLocalWork, markIdleIngressTracked, markIdleStartupComplete, registerIdleSpoolDirectory } from "../services/idle-local-work.js";
let root: string;
beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "idle-spool-")); });
@@ -51,3 +51,54 @@ describe("idle cleanup inspection", () => {
});
});
describe("idle credential recovery inspection", () => {
const company = "7238bc31-f3ca-42ba-b487-4122e77b8867";
it("allows missing roots and empty company directories after successful recovery", async () => {
expect(await inspectIdleQuotaRecovery(path.join(root, "missing"))).toBe("none");
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
await fs.mkdir(path.join(root, company));
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
});
it.each(["receipt.json", "malformed.json", "partial.tmp", "write.probe", "unexpected"])(
"retains every pending company entry without reading its contents: %s", async name => {
await fs.mkdir(path.join(root, company));
const file = path.join(root, company, name);
await fs.writeFile(file, "unreadable recovery fixture");
const read = vi.spyOn(fs, "readFile");
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
expect(read).not.toHaveBeenCalled();
await fs.rm(file);
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
},
);
it("retains unexpected root entries, nested directories and directory symlinks", async () => {
await fs.mkdir(path.join(root, "unexpected"));
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
await fs.rm(path.join(root, "unexpected"), { recursive: true });
await fs.symlink(path.join(root, "missing"), path.join(root, company));
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
await fs.rm(path.join(root, company));
await fs.mkdir(path.join(root, company, "nested"), { recursive: true });
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
});
it("fails closed on invalid roots and directory read errors", async () => {
const file = path.join(root, "file"), link = path.join(root, "link");
await fs.writeFile(file, ""); await fs.symlink(root, link);
expect(await inspectIdleQuotaRecovery(file)).toBe("unknown");
expect(await inspectIdleQuotaRecovery(link)).toBe("unknown");
vi.spyOn(fs, "opendir").mockRejectedValue(Object.assign(new Error("private path"), { code: "EACCES" }));
expect(await inspectIdleQuotaRecovery(root)).toBe("unknown");
});
it("keeps persisted credential recovery awake after the request and process work have ended", async () => {
vi.stubEnv("PAPERCLIP_HOME", root);
markIdleIngressTracked(); markIdleStartupComplete({ scheduledBackups: false });
const directory = path.join(idleQuotaRecoveryPath(), company);
await fs.mkdir(directory, { recursive: true });
const file = path.join(directory, "pending.json");
await fs.writeFile(file, "encrypted journal fixture");
expect(await readIdleLocalWork()).toBe("present");
await fs.rm(file);
expect(await readIdleLocalWork()).toBe("none");
});
});
+119 -1
View File
@@ -6,7 +6,7 @@ import {
agentInstructionWorkingCopies, agents, agentApiKeys, agentWakeupRequests, companies, companySecretProposals, createDb,
adapterAuthSessions, environments, environmentLeases, executionWorkspaces,
heartbeatRuns, issues, issueWatchdogs, issueThreadInteractions, issueQuestionResponseDeliveries,
projects, routines, type Db,
projects, routines, toolApplications, toolConnections, type Db,
} from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";
@@ -443,6 +443,124 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp
expect(await read()).toEqual(present);
});
async function seedAiConnection() {
const seeded = await seed();
const [application] = await db.insert(toolApplications).values({
companyId: seeded.companyId, name: "AI fixture", type: "rest_api",
}).returning();
const [connection] = await db.insert(toolConnections).values({
companyId: seeded.companyId, applicationId: application!.id,
name: "Saved AI login", uid: randomUUID(), connectionPurpose: "ai",
transport: "runtime_auth", status: "active", enabled: true,
}).returning();
return { ...seeded, connection: connection! };
}
it("allows saved AI credentials while preserving their connection", async () => {
const { connection } = await seedAiConnection();
expect(await read()).toEqual(none);
expect(await db.select().from(toolConnections).where(eq(toolConnections.id, connection.id)))
.toMatchObject([{ status: "active", enabled: true }]);
});
it.each(["draft", "active_login", "pending_run", "accepted_refresh"])(
"keeps AI connections awake for %s work", async kind => {
const { companyId, runId, connection } = await seedAiConnection();
if (kind === "draft") await db.update(toolConnections).set({ status: "draft" }).where(eq(toolConnections.id, connection.id));
if (kind === "pending_run") await db.update(heartbeatRuns).set({ status: "queued" }).where(eq(heartbeatRuns.id, runId));
if (kind === "active_login") {
const [environment] = await db.insert(environments).values({ name: "Active AI login" }).returning();
await db.insert(adapterAuthSessions).values({
companyId, environmentId: environment!.id, connectionId: connection.id,
adapterType: "codex_local", startedByUserId: "fixture-user",
publicSessionId: randomUUID(), status: "waiting_for_user",
});
}
const finish = kind === "accepted_refresh" ? beginIdleTrackedWork() : undefined;
try { expect(await read()).toEqual(kind === "accepted_refresh" ? unknown : present); }
finally { finish?.(); }
},
);
it("retains remote MCP connections alongside inert AI credentials", async () => {
const { companyId, connection } = await seedAiConnection();
await db.insert(toolConnections).values({
companyId, applicationId: connection.applicationId, name: "Remote tools", uid: randomUUID(),
connectionPurpose: "tool", transport: "mcp_remote", status: "active", enabled: true,
});
expect(await read()).toEqual(present);
});
async function seedFinishedFailedLease() {
const seeded = await seed();
await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(now) })
.where(eq(heartbeatRuns.id, seeded.runId));
const [environment] = await db.insert(environments).values({ name: "Failed turn environment", driver: "sandbox" }).returning();
const leaseId = randomUUID();
const receipt = {
schema: "paperclip.remote-termination.v1", companyId: seeded.companyId,
runId: seeded.runId, leaseId, provider: "plugin:demo:sandbox",
providerLeaseId: "fixture-resource", state: "destroyed", confirmedAt: new Date(now).toISOString(),
};
const [lease] = await db.insert(environmentLeases).values({
id: leaseId, companyId: seeded.companyId, environmentId: environment!.id,
heartbeatRunId: seeded.runId, status: "failed", leasePolicy: "ephemeral",
provider: receipt.provider, providerLeaseId: receipt.providerLeaseId,
acquiredAt: new Date(now - 1000), releasedAt: new Date(now), cleanupStatus: "success",
metadata: { remoteExecutionTermination: receipt },
}).returning();
return { ...seeded, environment: environment!, lease: lease!, receipt };
}
it.each(["destroyed", "stopped"])("allows failed lease history after exact provider %s proof", async state => {
const { lease, receipt } = await seedFinishedFailedLease();
await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: { ...receipt, state } } })
.where(eq(environmentLeases.id, lease.id));
expect(await read()).toEqual(none);
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)))
.toMatchObject([{ status: "failed", cleanupStatus: "success" }]);
});
it.each([
"cleanup_failed", "cleanup_unknown", "unreleased", "retained_policy", "reusable_policy",
"active", "pending_cleanup", "missing_receipt", "wrong_company", "wrong_run", "wrong_lease",
"wrong_provider", "wrong_resource", "not_stopped", "missing_run", "unfinished_run",
"retry", "accounting", "cleanup_in_flight", "pending_export", "pending_retain",
])("retains failed leases with %s evidence", async kind => {
const { lease, receipt, runId } = await seedFinishedFailedLease();
const change: Partial<typeof environmentLeases.$inferInsert> = {};
if (kind === "cleanup_failed") change.cleanupStatus = "failed";
if (kind === "cleanup_unknown") change.cleanupStatus = null;
if (kind === "unreleased") change.releasedAt = null;
if (kind === "retained_policy") change.leasePolicy = "retain_on_failure";
if (kind === "reusable_policy") change.leasePolicy = "reuse_by_environment";
if (kind === "active" || kind === "pending_cleanup") change.status = kind;
if (kind === "missing_receipt") change.metadata = {};
const fields: Record<string, string> = { wrong_company: "companyId", wrong_run: "runId", wrong_lease: "leaseId", wrong_provider: "provider", wrong_resource: "providerLeaseId", not_stopped: "state" };
if (fields[kind]) change.metadata = { remoteExecutionTermination: { ...receipt, [fields[kind]]: "wrong" } };
if (kind === "missing_run") change.heartbeatRunId = null;
if (kind === "unfinished_run") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, runId));
if (kind === "retry") await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now + 1000) }).where(eq(heartbeatRuns.id, runId));
if (kind === "accounting") await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, runId));
if (kind === "cleanup_in_flight") change.metadata = { remoteExecutionTermination: receipt, pendingCleanupInFlight: true };
if (kind === "pending_export") change.metadata = { remoteExecutionTermination: receipt, nativeWorkspaceExportResume: {} };
if (kind === "pending_retain") change.metadata = { remoteExecutionTermination: receipt, sandboxStopAndRetain: {} };
if (Object.keys(change).length) await db.update(environmentLeases).set(change).where(eq(environmentLeases.id, lease.id));
expect(await read()).toEqual(present);
});
it("uses the same failed-lease cleanup proof for finished login history", async () => {
const { companyId, environment, lease } = await seedFinishedFailedLease();
await db.insert(adapterAuthSessions).values({
companyId, environmentId: environment.id, adapterType: "codex_local", startedByUserId: "fixture-user",
publicSessionId: randomUUID(), providerLeaseId: lease.providerLeaseId,
status: "authenticated", finishedAt: new Date(now),
});
expect(await read()).toEqual(none);
await db.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, lease.id));
expect(await read()).toEqual(present);
});
it("reports a login sandbox awaiting cleanup", async () => {
const { companyId } = await seed();
const [environment] = await db.insert(environments).values({ name: "Login fixture" }).returning();
@@ -2,6 +2,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest";
import type { Db } from "@paperclipai/db";
import { refreshCompanySubscriptions, refreshSubscriptionConnection } from "../services/subscription-refresh.js";
import type { SubscriptionConnection } from "../services/subscriptions.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
const mocks = vi.hoisted(() => ({ accounts: vi.fn(), credential: vi.fn(), register: vi.fn(), probe: vi.fn() }));
vi.mock("../services/ai-connections.js", () => ({ aiConnectionService: () => ({ subscriptionAccounts: mocks.accounts, credential: mocks.credential }) }));
@@ -28,6 +29,40 @@ beforeEach(() => {
});
describe("subscription discovery capacity", () => {
it("tracks detached provider refresh through its final database write", async () => {
const db = database(), probe = deferred<{ credentialKey: string }>(), save = deferred<string>();
const before = idleWorkSnapshot();
mocks.probe.mockReturnValue(probe.promise);
mocks.register.mockReturnValue(save.promise);
const task = refreshSubscriptionConnection(db, input("tracked"), "tracked");
try {
await vi.waitFor(() => expect(mocks.probe).toHaveBeenCalledOnce());
expect(idleWorkSnapshot().active).toBe(before.active + 1);
expect(refreshSubscriptionConnection(db, input("tracked"), "tracked")).toBe(task);
expect(idleWorkSnapshot().active).toBe(before.active + 1);
probe.resolve({ credentialKey: "fixture-key" });
await vi.waitFor(() => expect(mocks.register).toHaveBeenCalledOnce());
expect(idleWorkSnapshot().active).toBe(before.active + 1);
} finally {
probe.resolve({ credentialKey: "fixture-key" }); save.resolve("tracked"); await task;
}
expect(idleWorkSnapshot().active).toBe(before.active);
expect(idleWorkSnapshot().generation).toBe(before.generation + 2);
});
it("tracks detached discovery while it waits for account credentials", async () => {
const db = database(), accounts = deferred<[]>();
const before = idleWorkSnapshot();
mocks.accounts.mockReturnValue(accounts.promise);
const task = refreshCompanySubscriptions(db, "tracked-company", "alice");
try {
expect(idleWorkSnapshot().active).toBe(before.active + 1);
expect(refreshCompanySubscriptions(db, "tracked-company", "alice")).toBe(task);
expect(idleWorkSnapshot().active).toBe(before.active + 1);
} finally { accounts.resolve([]); await task; }
expect(idleWorkSnapshot().active).toBe(before.active);
});
it.each([false, true])("waits for a provider slot without dropping accepted discovery (provider fails: %s)", async (fails) => {
const db = database();
const gates = Array.from({ length: 5 }, () => deferred<void>());
+27 -1
View File
@@ -9,6 +9,7 @@ export function registerIdleSpoolDirectory(directory: string): void { spoolDirec
export const idleAccountingSpoolPath = () => path.join(resolvePaperclipInstanceRoot(), "accounting-receipts");
export const idleOrphanSpoolPath = () => process.env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR ??
path.join(resolvePaperclipInstanceRoot(), "data", "sandbox-orphan-cleanup");
export const idleQuotaRecoveryPath = () => path.join(resolvePaperclipInstanceRoot(), "quota-credential-recovery");
let startupComplete = false;
let ingressTracked = false;
let scheduledBackups = false;
@@ -33,13 +34,38 @@ export async function inspectIdleSpool(directory: string): Promise<IdleLocalWork
}
}
/** Credential journals are grouped by company. Successful recovery leaves its
* empty company directory behind; every file, probe or unexpected entry still
* blocks. Never decrypt journals or follow directory symlinks during a scan. */
export async function inspectIdleQuotaRecovery(directory: string): Promise<IdleLocalWork> {
try {
const info = await fs.lstat(directory);
if (!info.isDirectory() || info.isSymbolicLink()) return "unknown";
const dir = await fs.opendir(directory);
try {
for (let entry = await dir.read(); entry !== null; entry = await dir.read()) {
if (!entry.isDirectory() || entry.isSymbolicLink()
|| !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(entry.name)) return "present";
const state = await inspectIdleSpool(path.join(directory, entry.name));
if (state !== "none") return state;
}
return "none";
} finally { await dir.close(); }
} catch (error) {
return (error as NodeJS.ErrnoException).code === "ENOENT" ? "none" : "unknown";
}
}
export async function readIdleLocalWork(options: { backupCheckpoint?: boolean } = {}): Promise<IdleLocalWork> {
if (!startupComplete || !ingressTracked) return "unknown";
// This exemption belongs only to a scan which will finish a new backup
// under the same admission hold before it can return an empty report.
if ((scheduledBackups && !options.backupCheckpoint) || idleWorkSnapshot().active !== 0) return "present";
const directories = new Set([...spoolDirectories, idleAccountingSpoolPath(), idleOrphanSpoolPath()]);
const results = await Promise.all([...directories].map(inspectIdleSpool));
const results = await Promise.all([
...[...directories].map(inspectIdleSpool),
inspectIdleQuotaRecovery(idleQuotaRecoveryPath()),
]);
if (results.includes("unknown")) return "unknown";
return results.includes("present") ? "present" : "none";
}
+36 -4
View File
@@ -22,6 +22,32 @@ export type InspectIdlePlugins = (hold: { ownerId: string; expiresAt: number })
pluginIds: string[];
}>;
// A failed turn may have successfully torn down its ephemeral allocation. A
// cleanup label alone is not provider termination proof. Require the exact
// host-owned receipt and finished run, and retain all cleanup/export intents.
// Aliases are fixed at the call sites, never supplied by a client.
function finishedFailedLease(alias: string): string {
const receipt = `${alias}.metadata->'remoteExecutionTermination'`;
return `${alias}.status = 'failed' AND ${alias}.lease_policy = 'ephemeral'
AND ${alias}.cleanup_status = 'success' AND ${alias}.released_at IS NOT NULL
AND COALESCE(${alias}.metadata->>'pendingCleanupInFlight', 'false') = 'false'
AND NOT (COALESCE(${alias}.metadata, '{}'::jsonb) ?| ARRAY['nativeWorkspaceExportResume', 'sandboxStopAndRetain'])
AND ${alias}.provider IS NOT NULL AND ${alias}.provider <> 'local'
AND ${alias}.provider_lease_id IS NOT NULL AND ${alias}.provider_lease_id <> ''
AND ${receipt}->>'schema' = 'paperclip.remote-termination.v1'
AND ${receipt}->>'companyId' = ${alias}.company_id::text
AND ${receipt}->>'runId' = ${alias}.heartbeat_run_id::text
AND ${receipt}->>'leaseId' = ${alias}.id::text
AND ${receipt}->>'provider' = ${alias}.provider
AND ${receipt}->>'providerLeaseId' = ${alias}.provider_lease_id
AND ${receipt}->>'state' IN ('stopped', 'destroyed')
AND EXISTS (SELECT 1 FROM heartbeat_runs r WHERE r.id = ${alias}.heartbeat_run_id
AND r.company_id = ${alias}.company_id
AND r.status IN ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted')
AND r.finished_at IS NOT NULL AND r.scheduled_retry_at IS NULL
AND r.cost_accounting_pending = false)`;
}
// These are deliberately conservative. Completed agent runs and workspace
// operations are history, not reasons to keep an otherwise idle instance up.
// Other background features remain awake until their work and inbound events
@@ -60,13 +86,18 @@ const WORK_CHECKS = [
// retry after immediate evaluation failed before creating a review/run.
`SELECT 1 FROM issue_watchdogs WHERE status <> 'disabled'`,
`SELECT 1 FROM workspace_operations WHERE status NOT IN ('succeeded', 'failed', 'cancelled')`,
`SELECT 1 FROM environment_leases WHERE status NOT IN ('released', 'expired')`,
`SELECT 1 FROM environment_leases l WHERE status NOT IN ('released', 'expired')
AND (${finishedFailedLease("l")}) IS NOT TRUE`,
`SELECT 1 FROM routines WHERE status NOT IN ('paused', 'archived')`,
`SELECT 1 FROM routine_runs WHERE status NOT IN ('completed', 'failed', 'skipped', 'cancelled')`,
`SELECT 1 FROM plugin_jobs WHERE status <> 'paused'`,
`SELECT 1 FROM plugin_job_runs WHERE status NOT IN ('succeeded', 'failed', 'cancelled')`,
`SELECT 1 FROM plugin_webhook_deliveries WHERE status NOT IN ('succeeded', 'failed')`,
`SELECT 1 FROM tool_connections WHERE status <> 'archived'`,
// Saved AI credentials have no independent worker. Login sessions, accepted
// quota/refresh work and recovery journals have their own checks. Other
// integrations still need an ingress/scheduler wake contract.
`SELECT 1 FROM tool_connections WHERE status <> 'archived'
AND NOT (status = 'active' AND connection_purpose = 'ai' AND transport = 'runtime_auth')`,
`SELECT 1 FROM issue_recovery_actions WHERE status NOT IN ('resolved', 'cancelled')`,
`SELECT 1 FROM status_cards WHERE archived_at IS NULL`,
`SELECT 1 FROM external_objects WHERE NOT is_terminal OR next_refresh_at IS NOT NULL`,
@@ -83,7 +114,8 @@ const WORK_CHECKS = [
AND (l.provider_lease_id = s.provider_lease_id OR l.id = CASE
WHEN s.provider_lease_id ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
THEN s.provider_lease_id::uuid END)
AND l.status IN ('released', 'expired') AND l.cleanup_status = 'success'
AND (l.status IN ('released', 'expired') OR (${finishedFailedLease("l")}))
AND l.cleanup_status = 'success'
AND l.released_at IS NOT NULL))
OR (s.provider_lease_id IS NOT NULL AND EXISTS (
SELECT 1 FROM environment_leases l WHERE l.company_id = s.company_id
@@ -91,7 +123,7 @@ const WORK_CHECKS = [
AND (l.provider_lease_id = s.provider_lease_id OR l.id = CASE
WHEN s.provider_lease_id ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
THEN s.provider_lease_id::uuid END)
AND (l.status IN ('released', 'expired') AND l.cleanup_status = 'success'
AND ((l.status IN ('released', 'expired') OR (${finishedFailedLease("l")})) AND l.cleanup_status = 'success'
AND l.released_at IS NOT NULL) IS NOT TRUE))`,
// Answer delivery history is not work once its acknowledged handoff has
// finished. Keep pending deliveries, unfinished target runs, and missing or
@@ -2,9 +2,9 @@ import { createHash, randomUUID } from "node:crypto";
import { promises as fs } from "node:fs";
import path from "node:path";
import { z } from "zod";
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
import { localEncryptedProvider } from "../secrets/local-encrypted-provider.js";
import { ensureDurableDirectory, syncDirectory } from "../lib/durable-directory.js";
import { idleQuotaRecoveryPath } from "./idle-local-work.js";
const recoverySchema = z.object({
companyId: z.string().uuid(), grantId: z.string().uuid(), connectionId: z.string().uuid(), secretId: z.string().uuid(),
@@ -13,7 +13,7 @@ const recoverySchema = z.object({
type Recovery = z.infer<typeof recoverySchema>;
export const quotaCredentialHash = (value: string) => createHash("sha256").update(value).digest("hex");
export const quotaCredentialRecoveryPath = (companyId: string, grantId: string) =>
path.join(resolvePaperclipInstanceRoot(), "quota-credential-recovery", companyId, `${grantId}.json`);
path.join(idleQuotaRecoveryPath(), companyId, `${grantId}.json`);
/** Called only while holding the company credential mutation lock. The local
* encrypted provider uses the instance secrets master key, including when the
+5 -2
View File
@@ -8,6 +8,7 @@ import { companySecrets, companySecretVersions, userSecretDefinitions, type Db }
import { fetchCodexQuota } from "@paperclipai/adapter-codex-local/server";
import { fetchClaudeQuota } from "@paperclipai/adapter-claude-local/server";
import { aiConnectionService } from "./ai-connections.js";
import { trackIdleWork } from "./task-admission.js";
const accountRequests = new Map<string, { expires: number; result: Promise<ProviderQuotaResult> }>();
const publicError = "Subscription quota is currently unavailable. Check usage with your provider.";
@@ -51,7 +52,9 @@ export async function fetchCompanyQuotaWindows(db: Db, companyId: string, userId
let pending = accountRequests.get(key);
if (!pending || pending.expires <= Date.now()) {
const controller = new AbortController();
const result = (async (): Promise<ProviderQuotaResult> => {
// The HTTP deadline can expire while an OAuth exchange still needs to
// save its consumed token. Track the actual work, not its timeout wrapper.
const result = trackIdleWork((async (): Promise<ProviderQuotaResult> => {
let credentialResolved = false;
try {
const value = await service.credential(row);
@@ -104,7 +107,7 @@ export async function fetchCompanyQuotaWindows(db: Db, companyId: string, userId
return { ...base, ok: false, windows: [], error: publicError,
errorFamily };
}
})();
})());
const bounded: Promise<ProviderQuotaResult> = withQuotaTimeout(base.provider, result, () => controller.abort()).then(quota => {
const observation = { ...base, ...quota, ...(quota.ok ? {} : { error: publicError }) };
if (observation.ok && observation.accountKey !== accountKey) {
+5 -4
View File
@@ -5,6 +5,7 @@ import { subscriptionService, type SubscriptionConnection } from "./subscription
import { probeSubscriptionIdentity } from "./subscription-identity.js";
import { logger } from "../middleware/logger.js";
import { tooManyRequests } from "../errors.js";
import { trackIdleWork } from "./task-admission.js";
const jobs = new WeakMap<Db, Map<string, Promise<void>>>();
const discovery = new WeakMap<Db, Map<string, Promise<void>>>();
@@ -19,7 +20,7 @@ export function refreshSubscriptionConnection(db: Db, input: SubscriptionConnect
// Bound provider concurrency across all companies on this instance. Dashboard
// discovery waits below; opportunistic run observations remain due when busy.
if (active.size >= 4) return Promise.resolve();
const task = (async () => {
const task = trackIdleWork((async () => {
const [claimed] = await db.update(aiSubscriptions).set({ lastCheckedAt: new Date() }).where(and(
eq(aiSubscriptions.companyId, input.companyId), eq(aiSubscriptions.id, id),
or(isNull(aiSubscriptions.lastCheckedAt), lt(aiSubscriptions.lastCheckedAt, new Date(Date.now() - REFRESH_MS))),
@@ -38,7 +39,7 @@ export function refreshSubscriptionConnection(db: Db, input: SubscriptionConnect
})().catch(() => {
// No raw provider/credential errors are logged or returned to a user.
logger.warn({ companyId: input.companyId, subscriptionId: id }, "Subscription observation could not be saved");
}).finally(() => active!.delete(id));
}).finally(() => active!.delete(id)));
active.set(id, task);
return task;
}
@@ -50,7 +51,7 @@ export function refreshCompanySubscriptions(db: Db, companyId: string, userId: s
const existing = active.get(key);
if (existing) return existing;
if (active.size >= 20) throw tooManyRequests("Subscription account checks are busy. Try again shortly.");
const task = (async () => {
const task = trackIdleWork((async () => {
const service = aiConnectionService(db);
// Uses the existing credential audience. Being a company admin does not
// authorize probing another member's personal credentials.
@@ -75,7 +76,7 @@ export function refreshCompanySubscriptions(db: Db, companyId: string, userId: s
}
}
})().catch(() => logger.warn({ companyId }, "Subscription discovery unavailable"))
.finally(() => active!.delete(key));
.finally(() => active!.delete(key)));
active.set(key, task);
return task;
}