Files
PaperClipAI/server/src/services/quota-windows.ts
T
Devin FoleyandPaperclip 4b59e2a2ec fix(server): allow idle sleep with saved AI auth and verified lease history (#15897)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Automatic idle sleep must preserve accepted work, cleanup and
recovery.
> - Saved AI login connections are configuration without independent
workers.
> - Failed ephemeral leases can remain after their resources have
stopped.
> - Detached subscription observations and quota refreshes can outlive a
response.
> - This pull request distinguishes inert configuration and verified
history from unfinished work.
> - The benefit is safe idle eligibility while keeping credentials and
audit history.

## Linked Issues or Issue Description

**What happened?**

An active saved AI login blocks idle sleep after its login and agent
runs finish. A failed ephemeral lease also blocks after successful
cleanup and provider termination. The connection guard currently hides
detached quota and subscription work that can outlive the request that
started it.

**Expected behavior**

An idle instance can sleep with saved AI credentials and verified
failed-lease history. Accepted refresh work, recovery journals,
unfinished login sessions, pending cleanup and run accounting must still
block.

**Steps to reproduce**

1. Save an AI connection with purpose `ai`, transport `runtime_auth` and
status `active`.
2. Finish login and all agent work, then acquire an owned idle drain.
3. Read the idle safety report. The saved connection alone reports
background work.
4. Repeat with a failed ephemeral lease whose exact provider termination
and finished run are recorded. Its history also reports background work.

**Paperclip version or commit**

Based on `0bbd586a2f6`, after updates from master.

**Deployment mode**

Single-process managed or self-hosted server using the owned idle-drain
protocol.

Related work: #15522, #15761 and #15851. Searched open and closed pull
requests for related idle-sleep changes. The open accounting recovery
work in #15879 is separate; this change does not settle or discard
accounting debt.

## What Changed

- Exempt active saved AI runtime-auth configuration from the connection
sleep check. Other connection types and unknown states keep their
existing checks.
- Track detached subscription discovery and provider observations until
their actual promises settle.
- Track underlying quota work through an HTTP timeout, including an
OAuth exchange and the save of its replacement credential.
- Inspect the credential-recovery directory without reading credential
bytes. Journals, probes, temporary files, unexpected entries and read
failures keep sleep blocked. Empty company directories left by
successful recovery permit sleep.
- Exempt a failed ephemeral lease only after successful cleanup, a
release timestamp, an exact company/run/lease/resource termination
receipt, and a finished run without retry or accounting debt. Export and
cleanup intents still block.
- Use the same failed-lease proof when checking finished login-session
cleanup.
- Add PostgreSQL and filesystem regression tests and document the
behavior.

## Verification

- Four focused suites passed: 186 tests covering durable guard
decisions, local recovery, detached subscription work and quota
timeouts.
- `pnpm build` passed.
- `pnpm -r typecheck` passed.
- `pnpm --filter @paperclipai/server build` and a final server
TypeScript check passed after the directory-scanner cleanup.
- The final directory-scan recheck passed all 18 tests.
- The original-base `pnpm test:run` attempt found three local failures.
It was stopped before completion to update the branch after two master
changes. All three also reproduce on unchanged master `e733bddc45e`: two
Git default-branch fixtures in `workspace-runtime.test.ts` and one
native runner recovery fixture in `native-session-resume.test.ts`.
- Both Git fixtures pass with a command-scoped
`init.defaultBranch=master` setting. The native fixture fails with the
same result on both branches. No global Git setting was changed.
- All CI gates passed on the final head
`ff63497d9bef40f5f00567bc729fb17536cfe136`: [complete CI
run](https://github.com/paperclipai/paperclip/actions/runs/38156312722).
The only skipped checks are the two path-gated Storybook jobs.
- Greptile Apex scored the final head 5/5 with no actionable findings
and no unresolved review threads. The branch has no merge conflicts with
master.
- After the master update, `pnpm build` and `pnpm -r typecheck` passed
again. All four focused suites passed again: 186 tests.
- Staging deployment and live sleep/wake verification are pending the
Core merge and the verified private Cloud build. No production rollout
has been started.

## Risks

- The failed-lease exception is deliberately narrow. Older records
without exact termination evidence or a complete run receipt continue to
block.
- Pending OAuth recovery blocks sleep until it is recovered. No
credential, history, cleanup obligation or accounting receipt is
deleted.
- The existing owned drain, ingress fence, local-work generation checks,
plugin drain and backup checkpoint remain required.
- No schema migration or provider mutation is introduced. Roll back to
the previous application release to restore the conservative guard
behavior.

## Model Used

OpenAI GPT-6 (Codex), with repository inspection, code editing, test
execution and read-only provider analysis. No additional model version
or context-window size was supplied by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no unresolved comments
- [x] I will address all reviewer comments before requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-11 10:01:57 -07:00

206 lines
12 KiB
TypeScript

import type { ProviderQuotaResult } from "@paperclipai/shared";
import { redactDiagnosticText } from "@paperclipai/adapter-utils";
import { listServerAdapters } from "../adapters/registry.js";
import { logger } from "../middleware/logger.js";
import { createHash } from "node:crypto";
import { eq, inArray, and } from "drizzle-orm";
import { companySecrets, companySecretVersions, userSecretDefinitions, type Db } from "@paperclipai/db";
import { fetchCodexQuota } from "@paperclipai/adapter-codex-local/server";
import { fetchClaudeQuota } from "@paperclipai/adapter-claude-local/server";
import { aiConnectionService } from "./ai-connections.js";
import { trackIdleWork } from "./task-admission.js";
const accountRequests = new Map<string, { expires: number; result: Promise<ProviderQuotaResult> }>();
const publicError = "Subscription quota is currently unavailable. Check usage with your provider.";
/** Managed connections use their own credentials regardless of where agents run.
* Never silently substitute the control-plane host's login for a remote account. */
export async function fetchCompanyQuotaWindows(db: Db, companyId: string, userId: string): Promise<ProviderQuotaResult[]> {
const service = aiConnectionService(db);
const accounts = await service.quotaAccounts(companyId, userId);
if (!accounts.length) {
return ["anthropic", "openai"].map(provider => ({ provider, ok: false,
accountKey: `unconnected:${companyId}:${provider}`, source: "managed-connection",
errorFamily: "credentials_unavailable", error: "Connect a subscription account in AI connections to view its quota.", windows: [] }));
}
// Resolve revision metadata before consulting the cache, so rotation and
// revocation cannot serve windows associated with an old credential.
const secretIds = accounts.flatMap(row => row.grant.credentialSecretRefs.map(ref => ref.secretId));
const readRevisions = async (ids: string[]) => ids.length ? await db.select({ id: companySecrets.id, latestVersion: companySecrets.latestVersion, status: companySecrets.status, versionStatus: companySecretVersions.status, revokedAt: companySecretVersions.revokedAt, definitionId: companySecrets.userSecretDefinitionId, definitionStatus: userSecretDefinitions.status, definitionDeletedAt: userSecretDefinitions.deletedAt })
.from(companySecrets).leftJoin(companySecretVersions, and(eq(companySecretVersions.secretId, companySecrets.id), eq(companySecretVersions.version, companySecrets.latestVersion))).leftJoin(userSecretDefinitions, and(eq(userSecretDefinitions.id, companySecrets.userSecretDefinitionId), eq(userSecretDefinitions.companyId, companyId))).where(and(eq(companySecrets.companyId, companyId), inArray(companySecrets.id, ids))) : [];
const revisions = await readRevisions(secretIds);
const identity = (row: (typeof accounts)[number], metadata: typeof revisions) => ({
provider: row.summary.provider,
accountKey: createHash("sha256").update(JSON.stringify([companyId, row.connection.id, row.grant.id,
row.connection.updatedAt, row.grant.updatedAt,
row.grant.credentialSecretRefs.map(ref => [ref.secretId, metadata.find(r => r.id === ref.secretId)]),
])).digest("hex"),
accountLabel: row.summary.name, source: "managed-connection",
});
async function readAccount(row: (typeof accounts)[number]): Promise<ProviderQuotaResult> {
let base = identity(row, revisions);
const { accountKey } = base;
const definitionUnavailable = row.grant.credentialSecretRefs.some(ref => {
const revision = revisions.find(value => value.id === ref.secretId);
return revision && (revision.status !== "active" || revision.versionStatus === "disabled" || revision.revokedAt != null
|| (revision.definitionId && (revision.definitionStatus !== "active" || revision.definitionDeletedAt !== null)));
});
if (row.summary.status !== "connected" || definitionUnavailable) {
return { ...base, ok: false, errorFamily: "credentials_unavailable", error: publicError, windows: [] };
}
const key = `${userId}:${accountKey}`;
let pending = accountRequests.get(key);
if (!pending || pending.expires <= Date.now()) {
const controller = new AbortController();
// The HTTP deadline can expire while an OAuth exchange still needs to
// save its consumed token. Track the actual work, not its timeout wrapper.
const result = trackIdleWork((async (): Promise<ProviderQuotaResult> => {
let credentialResolved = false;
try {
const value = await service.credential(row);
controller.signal.throwIfAborted();
credentialResolved = true;
let windows;
if (base.provider === "openai") {
let auth;
try { auth = JSON.parse(value); } catch { throw new Error("credentials_unavailable"); }
const token = auth.tokens?.access_token ?? auth.accessToken;
if (typeof token !== "string" || !token) {
return { ...base, ok: false, errorFamily: "credentials_unavailable", error: publicError, windows: [] };
}
const read = (auth: { tokens?: { access_token?: string; account_id?: string | null }; accessToken?: string; accountId?: string | null }) =>
fetchCodexQuota(auth.tokens?.access_token ?? auth.accessToken!, auth.tokens?.account_id ?? auth.accountId ?? null, controller.signal);
try { windows = await read(auth); }
catch (error) {
if (!(error instanceof Error) || !/\b401\b/.test(error.message)) throw error;
auth = JSON.parse(await service.refreshQuotaCredential(row, value, controller.signal));
// Rotation changes both the grant and secret revision. Return and
// cache the observation under the identity the next poll will see.
const refreshed = (await service.quotaAccounts(companyId, userId)).find(candidate =>
candidate.connection.id === row.connection.id && candidate.grant.id === row.grant.id);
if (!refreshed || refreshed.summary.status !== "connected") throw new Error("credentials_unavailable");
base = identity(refreshed, await readRevisions(refreshed.grant.credentialSecretRefs.map(ref => ref.secretId)));
// A reconnect can race the completed refresh. Bind the value and
// cache identity to one revision, rather than label old quota with
// a newer account's key. A concurrent change defers this poll.
const refreshedValue = await service.credential(refreshed);
const verified = (await service.quotaAccounts(companyId, userId)).find(candidate =>
candidate.connection.id === row.connection.id && candidate.grant.id === row.grant.id);
if (!verified || verified.summary.status !== "connected") throw new Error("credentials_unavailable");
const verifiedIdentity = identity(verified, await readRevisions(verified.grant.credentialSecretRefs.map(ref => ref.secretId)));
if (verifiedIdentity.accountKey !== base.accountKey) throw new Error("provider_unavailable");
auth = JSON.parse(refreshedValue);
controller.signal.throwIfAborted();
windows = await read(auth);
}
} else {
windows = await fetchClaudeQuota(value, controller.signal);
}
return { ...base, ok: true, windows, capturedAt: new Date().toISOString() };
} catch (error) {
const message = error instanceof Error ? error.message : "Quota unavailable";
const missing = message === "credentials_unavailable";
const invalid = message === "authentication_required" || (credentialResolved && /\b401\b|refresh_token_(?:reused|expired|invalidated)/i.test(message));
const errorFamily = missing ? "credentials_unavailable" : invalid ? "authentication_required"
: credentialResolved && /\b403\b/.test(message) ? "permission_denied" : "provider_unavailable";
logger.warn({ companyId, accountKey, provider: base.provider, authenticationFailed: invalid }, "Connected account quota unavailable");
return { ...base, ok: false, windows: [], error: publicError,
errorFamily };
}
})());
const bounded: Promise<ProviderQuotaResult> = withQuotaTimeout(base.provider, result, () => controller.abort()).then(quota => {
const observation = { ...base, ...quota, ...(quota.ok ? {} : { error: publicError }) };
if (observation.ok && observation.accountKey !== accountKey) {
if (accountRequests.get(key)?.result === bounded) accountRequests.delete(key);
if (accountRequests.size >= 500) accountRequests.delete(accountRequests.keys().next().value!);
accountRequests.set(`${userId}:${observation.accountKey}`, { result: Promise.resolve(observation), expires: Date.now() + 30_000 });
}
return observation;
});
pending = { result: bounded, expires: Date.now() + 30_000 };
if (accountRequests.size >= 500) accountRequests.delete(accountRequests.keys().next().value!);
accountRequests.set(key, pending);
}
return pending.result;
}
const results: ProviderQuotaResult[] = new Array(accounts.length);
let next = 0;
await Promise.all(Array.from({ length: Math.min(4, accounts.length) }, async () => {
while (next < accounts.length) {
const index = next++;
results[index] = await readAccount(accounts[index]);
}
}));
return results;
}
const QUOTA_PROVIDER_TIMEOUT_MS = 20_000;
function providerSlugForAdapterType(type: string): string {
switch (type) {
case "claude_local":
return "anthropic";
case "codex_local":
return "openai";
default:
return type;
}
}
/**
* Asks each registered adapter for its provider quota windows and aggregates the results.
* Adapters that don't implement getQuotaWindows() are silently skipped.
* Individual adapter failures are caught and returned as error results rather than
* letting one provider's outage block the entire response.
*/
export async function fetchAllQuotaWindows(): Promise<ProviderQuotaResult[]> {
const adapters = listServerAdapters().filter((a) => a.getQuotaWindows != null);
const settled = await Promise.allSettled(
adapters.map((adapter) => withQuotaTimeout(adapter.type, Promise.resolve().then(() => adapter.getQuotaWindows!()))),
);
return settled.map((result, i) => {
const adapterType = adapters[i]!.type;
const quota: ProviderQuotaResult = result.status === "fulfilled" ? result.value : {
provider: providerSlugForAdapterType(adapterType),
ok: false,
error: String(result.reason),
windows: [],
};
if (quota.ok) return { ...quota, error: undefined };
logger.warn({
adapterType,
errorFamily: quota.errorFamily,
diagnostic: redactDiagnosticText(quota.error ?? "Quota probe failed").slice(0, 2_000),
}, "Provider subscription quota unavailable");
return { ...quota, error: "Subscription quota is currently unavailable. Check usage with your provider." };
});
}
async function withQuotaTimeout(
adapterType: string,
task: Promise<ProviderQuotaResult>,
onTimeout?: () => void,
): Promise<ProviderQuotaResult> {
let timeoutId: NodeJS.Timeout | null = null;
try {
return await Promise.race([
task,
new Promise<ProviderQuotaResult>((resolve) => {
timeoutId = setTimeout(() => {
onTimeout?.();
resolve({
provider: providerSlugForAdapterType(adapterType),
ok: false,
error: `quota polling timed out after ${Math.round(QUOTA_PROVIDER_TIMEOUT_MS / 1000)}s`,
windows: [],
});
}, QUOTA_PROVIDER_TIMEOUT_MS);
}),
]);
} finally {
if (timeoutId) clearTimeout(timeoutId);
}
}