Files
PaperClipAI/server/src/services/subscription-refresh.ts
T
Devin FoleyandPaperclip 4b59e2a2ec fix(server): allow idle sleep with saved AI auth and verified lease history (#15897)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Automatic idle sleep must preserve accepted work, cleanup and
recovery.
> - Saved AI login connections are configuration without independent
workers.
> - Failed ephemeral leases can remain after their resources have
stopped.
> - Detached subscription observations and quota refreshes can outlive a
response.
> - This pull request distinguishes inert configuration and verified
history from unfinished work.
> - The benefit is safe idle eligibility while keeping credentials and
audit history.

## Linked Issues or Issue Description

**What happened?**

An active saved AI login blocks idle sleep after its login and agent
runs finish. A failed ephemeral lease also blocks after successful
cleanup and provider termination. The connection guard currently hides
detached quota and subscription work that can outlive the request that
started it.

**Expected behavior**

An idle instance can sleep with saved AI credentials and verified
failed-lease history. Accepted refresh work, recovery journals,
unfinished login sessions, pending cleanup and run accounting must still
block.

**Steps to reproduce**

1. Save an AI connection with purpose `ai`, transport `runtime_auth` and
status `active`.
2. Finish login and all agent work, then acquire an owned idle drain.
3. Read the idle safety report. The saved connection alone reports
background work.
4. Repeat with a failed ephemeral lease whose exact provider termination
and finished run are recorded. Its history also reports background work.

**Paperclip version or commit**

Based on `0bbd586a2f6`, after updates from master.

**Deployment mode**

Single-process managed or self-hosted server using the owned idle-drain
protocol.

Related work: #15522, #15761 and #15851. Searched open and closed pull
requests for related idle-sleep changes. The open accounting recovery
work in #15879 is separate; this change does not settle or discard
accounting debt.

## What Changed

- Exempt active saved AI runtime-auth configuration from the connection
sleep check. Other connection types and unknown states keep their
existing checks.
- Track detached subscription discovery and provider observations until
their actual promises settle.
- Track underlying quota work through an HTTP timeout, including an
OAuth exchange and the save of its replacement credential.
- Inspect the credential-recovery directory without reading credential
bytes. Journals, probes, temporary files, unexpected entries and read
failures keep sleep blocked. Empty company directories left by
successful recovery permit sleep.
- Exempt a failed ephemeral lease only after successful cleanup, a
release timestamp, an exact company/run/lease/resource termination
receipt, and a finished run without retry or accounting debt. Export and
cleanup intents still block.
- Use the same failed-lease proof when checking finished login-session
cleanup.
- Add PostgreSQL and filesystem regression tests and document the
behavior.

## Verification

- Four focused suites passed: 186 tests covering durable guard
decisions, local recovery, detached subscription work and quota
timeouts.
- `pnpm build` passed.
- `pnpm -r typecheck` passed.
- `pnpm --filter @paperclipai/server build` and a final server
TypeScript check passed after the directory-scanner cleanup.
- The final directory-scan recheck passed all 18 tests.
- The original-base `pnpm test:run` attempt found three local failures.
It was stopped before completion to update the branch after two master
changes. All three also reproduce on unchanged master `e733bddc45e`: two
Git default-branch fixtures in `workspace-runtime.test.ts` and one
native runner recovery fixture in `native-session-resume.test.ts`.
- Both Git fixtures pass with a command-scoped
`init.defaultBranch=master` setting. The native fixture fails with the
same result on both branches. No global Git setting was changed.
- All CI gates passed on the final head
`ff63497d9bef40f5f00567bc729fb17536cfe136`: [complete CI
run](https://github.com/paperclipai/paperclip/actions/runs/38156312722).
The only skipped checks are the two path-gated Storybook jobs.
- Greptile Apex scored the final head 5/5 with no actionable findings
and no unresolved review threads. The branch has no merge conflicts with
master.
- After the master update, `pnpm build` and `pnpm -r typecheck` passed
again. All four focused suites passed again: 186 tests.
- Staging deployment and live sleep/wake verification are pending the
Core merge and the verified private Cloud build. No production rollout
has been started.

## Risks

- The failed-lease exception is deliberately narrow. Older records
without exact termination evidence or a complete run receipt continue to
block.
- Pending OAuth recovery blocks sleep until it is recovered. No
credential, history, cleanup obligation or accounting receipt is
deleted.
- The existing owned drain, ingress fence, local-work generation checks,
plugin drain and backup checkpoint remain required.
- No schema migration or provider mutation is introduced. Roll back to
the previous application release to restore the conservative guard
behavior.

## Model Used

OpenAI GPT-6 (Codex), with repository inspection, code editing, test
execution and read-only provider analysis. No additional model version
or context-window size was supplied by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no unresolved comments
- [x] I will address all reviewer comments before requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-11 10:01:57 -07:00

83 lines
4.7 KiB
TypeScript

import { and, eq, lt, or, isNull } from "drizzle-orm";
import { aiSubscriptions, aiSubscriptionConnections, type Db } from "@paperclipai/db";
import { aiConnectionService } from "./ai-connections.js";
import { subscriptionService, type SubscriptionConnection } from "./subscriptions.js";
import { probeSubscriptionIdentity } from "./subscription-identity.js";
import { logger } from "../middleware/logger.js";
import { tooManyRequests } from "../errors.js";
import { trackIdleWork } from "./task-admission.js";
const jobs = new WeakMap<Db, Map<string, Promise<void>>>();
const discovery = new WeakMap<Db, Map<string, Promise<void>>>();
const REFRESH_MS = 6 * 60 * 60 * 1000;
/** Cached metadata refresh only. Never exchanges tokens or blocks a run/page on a provider. */
export function refreshSubscriptionConnection(db: Db, input: SubscriptionConnection, id: string, request: typeof fetch = fetch): Promise<void> {
let active = jobs.get(db);
if (!active) jobs.set(db, active = new Map());
const existing = active.get(id);
if (existing) return existing;
// Bound provider concurrency across all companies on this instance. Dashboard
// discovery waits below; opportunistic run observations remain due when busy.
if (active.size >= 4) return Promise.resolve();
const task = trackIdleWork((async () => {
const [claimed] = await db.update(aiSubscriptions).set({ lastCheckedAt: new Date() }).where(and(
eq(aiSubscriptions.companyId, input.companyId), eq(aiSubscriptions.id, id),
or(isNull(aiSubscriptions.lastCheckedAt), lt(aiSubscriptions.lastCheckedAt, new Date(Date.now() - REFRESH_MS))),
)).returning({ id: aiSubscriptions.id });
if (!claimed) return;
try {
const observation = await probeSubscriptionIdentity(input.companyId, input.provider, input.credential, request);
// A later execution/reconnect may now use another account. Never replace
// that binding with a slow response for the previous credential.
const [binding] = await db.select().from(aiSubscriptionConnections).where(and(eq(aiSubscriptionConnections.companyId, input.companyId), eq(aiSubscriptionConnections.grantId, input.grantId)));
if (binding?.credentialKey !== observation.credentialKey) return;
await subscriptionService(db).register(input, observation);
} catch {
await db.update(aiSubscriptions).set({ refreshStatus: "unavailable" }).where(and(eq(aiSubscriptions.companyId, input.companyId), eq(aiSubscriptions.id, id)));
}
})().catch(() => {
// No raw provider/credential errors are logged or returned to a user.
logger.warn({ companyId: input.companyId, subscriptionId: id }, "Subscription observation could not be saved");
}).finally(() => active!.delete(id)));
active.set(id, task);
return task;
}
export function refreshCompanySubscriptions(db: Db, companyId: string, userId: string): Promise<void> {
let active = discovery.get(db);
if (!active) discovery.set(db, active = new Map());
const key = `${companyId}:${userId}`;
const existing = active.get(key);
if (existing) return existing;
if (active.size >= 20) throw tooManyRequests("Subscription account checks are busy. Try again shortly.");
const task = trackIdleWork((async () => {
const service = aiConnectionService(db);
// Uses the existing credential audience. Being a company admin does not
// authorize probing another member's personal credentials.
for (const row of await service.subscriptionAccounts(companyId, userId)) {
if (row.summary.status !== "connected") continue;
try {
const credential = await service.credential(row);
const input: SubscriptionConnection = { companyId, connectionId: row.connection.id, grantId: row.grant.id,
provider: row.summary.provider, name: row.connection.name, ownerUserId: row.grant.subjectUserId, credential };
const id = await subscriptionService(db).register(input);
// Accepted discovery must not silently skip an account. At most twenty
// discovery tasks can wait here; never create an unbounded provider queue.
// Recheck after each completion because another discovery may take a slot.
let providers = jobs.get(db);
while (providers && providers.size >= 4 && !providers.has(id)) {
await Promise.race(providers.values());
providers = jobs.get(db);
}
await refreshSubscriptionConnection(db, input, id);
} catch {
logger.warn({ companyId, connectionId: row.connection.id }, "Subscription discovery unavailable for a connection");
}
}
})().catch(() => logger.warn({ companyId }, "Subscription discovery unavailable"))
.finally(() => active!.delete(key)));
active.set(key, task);
return task;
}