mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-15 03:51:22 +02:00
fix(server): allow idle sleep with saved AI auth and verified lease history (#15897)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Automatic idle sleep must preserve accepted work, cleanup and recovery. > - Saved AI login connections are configuration without independent workers. > - Failed ephemeral leases can remain after their resources have stopped. > - Detached subscription observations and quota refreshes can outlive a response. > - This pull request distinguishes inert configuration and verified history from unfinished work. > - The benefit is safe idle eligibility while keeping credentials and audit history. ## Linked Issues or Issue Description **What happened?** An active saved AI login blocks idle sleep after its login and agent runs finish. A failed ephemeral lease also blocks after successful cleanup and provider termination. The connection guard currently hides detached quota and subscription work that can outlive the request that started it. **Expected behavior** An idle instance can sleep with saved AI credentials and verified failed-lease history. Accepted refresh work, recovery journals, unfinished login sessions, pending cleanup and run accounting must still block. **Steps to reproduce** 1. Save an AI connection with purpose `ai`, transport `runtime_auth` and status `active`. 2. Finish login and all agent work, then acquire an owned idle drain. 3. Read the idle safety report. The saved connection alone reports background work. 4. Repeat with a failed ephemeral lease whose exact provider termination and finished run are recorded. Its history also reports background work. **Paperclip version or commit** Based on `0bbd586a2f6`, after updates from master. **Deployment mode** Single-process managed or self-hosted server using the owned idle-drain protocol. Related work: #15522, #15761 and #15851. Searched open and closed pull requests for related idle-sleep changes. The open accounting recovery work in #15879 is separate; this change does not settle or discard accounting debt. ## What Changed - Exempt active saved AI runtime-auth configuration from the connection sleep check. Other connection types and unknown states keep their existing checks. - Track detached subscription discovery and provider observations until their actual promises settle. - Track underlying quota work through an HTTP timeout, including an OAuth exchange and the save of its replacement credential. - Inspect the credential-recovery directory without reading credential bytes. Journals, probes, temporary files, unexpected entries and read failures keep sleep blocked. Empty company directories left by successful recovery permit sleep. - Exempt a failed ephemeral lease only after successful cleanup, a release timestamp, an exact company/run/lease/resource termination receipt, and a finished run without retry or accounting debt. Export and cleanup intents still block. - Use the same failed-lease proof when checking finished login-session cleanup. - Add PostgreSQL and filesystem regression tests and document the behavior. ## Verification - Four focused suites passed: 186 tests covering durable guard decisions, local recovery, detached subscription work and quota timeouts. - `pnpm build` passed. - `pnpm -r typecheck` passed. - `pnpm --filter @paperclipai/server build` and a final server TypeScript check passed after the directory-scanner cleanup. - The final directory-scan recheck passed all 18 tests. - The original-base `pnpm test:run` attempt found three local failures. It was stopped before completion to update the branch after two master changes. All three also reproduce on unchanged master `e733bddc45e`: two Git default-branch fixtures in `workspace-runtime.test.ts` and one native runner recovery fixture in `native-session-resume.test.ts`. - Both Git fixtures pass with a command-scoped `init.defaultBranch=master` setting. The native fixture fails with the same result on both branches. No global Git setting was changed. - All CI gates passed on the final head `ff63497d9bef40f5f00567bc729fb17536cfe136`: [complete CI run](https://github.com/paperclipai/paperclip/actions/runs/38156312722). The only skipped checks are the two path-gated Storybook jobs. - Greptile Apex scored the final head 5/5 with no actionable findings and no unresolved review threads. The branch has no merge conflicts with master. - After the master update, `pnpm build` and `pnpm -r typecheck` passed again. All four focused suites passed again: 186 tests. - Staging deployment and live sleep/wake verification are pending the Core merge and the verified private Cloud build. No production rollout has been started. ## Risks - The failed-lease exception is deliberately narrow. Older records without exact termination evidence or a complete run receipt continue to block. - Pending OAuth recovery blocks sleep until it is recovered. No credential, history, cleanup obligation or accounting receipt is deleted. - The existing owned drain, ingress fence, local-work generation checks, plugin drain and backup checkpoint remain required. - No schema migration or provider mutation is introduced. Roll back to the previous application release to restore the conservative guard behavior. ## Model Used OpenAI GPT-6 (Codex), with repository inspection, code editing, test execution and read-only provider analysis. No additional model version or context-window size was supplied by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no unresolved comments - [x] I will address all reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
0bbd586a2f
commit
4b59e2a2ec
10 files changed
+333
-20
No files matched your search
@@ -113,17 +113,36 @@ operator reconciliation; this API does not discard it to save compute.
|
||||
|
||||
The report checks all companies in one bounded read-only database transaction.
|
||||
Queued and orphaned runs, timers, retries, accounting debt, unfinished issues,
|
||||
cleanup, active routines, integrations, external API credentials, and plugin
|
||||
cleanup, active routines, integrations, external ingress credentials, and plugin
|
||||
work block sleep. Completed ordinary run history does not. Less common work
|
||||
sources conservatively block on any retained records. Active issue watchdogs
|
||||
block sleep even when their watched issue is complete and no review has started.
|
||||
Missing migrations, malformed configuration and database errors produce unknown
|
||||
without exposing SQL or tenant data.
|
||||
|
||||
Active `runtime_auth` connections with purpose `ai` are saved credentials, not
|
||||
independent workers. Their saved configuration does not block sleep. Login
|
||||
sessions and runs still pass the existing durable checks. Detached subscription
|
||||
observations and quota reads remain counted through their actual completion,
|
||||
including token refresh after a quota response times out. Credential-recovery
|
||||
journals, partial writes and probes block sleep across restart. Empty company
|
||||
directories left after successful recovery do not; invalid paths and read
|
||||
failures remain unknown. Credentials and connection records are retained.
|
||||
MCP, chat and other connections still block until they have a wake contract.
|
||||
|
||||
A failed ephemeral lease is history only with successful cleanup, a release
|
||||
timestamp, and an exact provider termination receipt bound to its company, run,
|
||||
lease and resource. Its run must be finished without retry or accounting debt.
|
||||
Pending cleanup claims and export/stop-and-retain intents still block. Failure
|
||||
labels or cleanup success without that proof do not authorize sleep. Lease
|
||||
history is retained; this check performs no provider teardown or data updates.
|
||||
|
||||
Finished login-session history permits sleep only after its promotion claim is
|
||||
cleared and any retained provider reference has a matching released or expired
|
||||
lease with successful cleanup and a release timestamp in the same company and
|
||||
environment. Missing cleanup evidence, `cleanup_pending`, and live login flows
|
||||
cleared and any retained provider reference has a matching released, expired,
|
||||
or verified failed ephemeral lease with successful cleanup and a release
|
||||
timestamp in the same company and environment. The failed lease must satisfy
|
||||
the same termination and run checks above. Missing cleanup evidence,
|
||||
`cleanup_pending`, and live login flows
|
||||
remain blockers. A finished coalesced wake is history only when its exact
|
||||
company/agent's linked run has finished without retry or accounting debt.
|
||||
Image-setup history still blocks: promotion can succeed before teardown is
|
||||
|
||||
@@ -25,6 +25,7 @@ vi.mock("@paperclipai/adapter-claude-local/server", () => ({
|
||||
fetchClaudeQuota: mocks.claude,
|
||||
}));
|
||||
import { fetchCompanyQuotaWindows } from "../services/quota-windows.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
|
||||
let revision = 1;
|
||||
const db = {
|
||||
@@ -192,7 +193,8 @@ describe("connected account quotas", () => {
|
||||
it("bounds a stalled credential lookup and preserves its account identity", async () => {
|
||||
vi.useFakeTimers();
|
||||
mocks.accounts.mockResolvedValue([account("stalled")]);
|
||||
mocks.credential.mockReturnValue(new Promise(() => {}));
|
||||
let finish!: (value: string) => void;
|
||||
mocks.credential.mockReturnValue(new Promise<string>(resolve => { finish = resolve; }));
|
||||
const pending = fetchCompanyQuotaWindows(db, "company-8", "user-8");
|
||||
await vi.advanceTimersByTimeAsync(20_001);
|
||||
const [result] = await pending;
|
||||
@@ -203,6 +205,32 @@ describe("connected account quotas", () => {
|
||||
});
|
||||
expect(result.accountKey).toBeTruthy();
|
||||
expect(result.error).not.toContain("timed out");
|
||||
finish(JSON.stringify({ accessToken: "fixture" }));
|
||||
await vi.advanceTimersByTimeAsync(0);
|
||||
});
|
||||
|
||||
it("retains the actual credential refresh after its HTTP quota deadline expires", async () => {
|
||||
vi.useFakeTimers();
|
||||
const before = idleWorkSnapshot();
|
||||
mocks.accounts.mockResolvedValue([account("tracked-refresh")]);
|
||||
mocks.codex.mockRejectedValueOnce(new Error("401"));
|
||||
let finish!: (value: string) => void;
|
||||
mocks.refresh.mockReturnValue(new Promise<string>(resolve => { finish = resolve; }));
|
||||
const pending = fetchCompanyQuotaWindows(db, "tracked-company", "tracked-user");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(0);
|
||||
expect(mocks.refresh).toHaveBeenCalledOnce();
|
||||
expect(idleWorkSnapshot().active).toBe(before.active + 1);
|
||||
await vi.advanceTimersByTimeAsync(20_001);
|
||||
expect((await pending)[0].ok).toBe(false);
|
||||
expect(idleWorkSnapshot().active).toBe(before.active + 1);
|
||||
} finally {
|
||||
finish(JSON.stringify({ accessToken: "fixture" }));
|
||||
await vi.advanceTimersByTimeAsync(0);
|
||||
await pending;
|
||||
}
|
||||
expect(idleWorkSnapshot().active).toBe(before.active);
|
||||
expect(idleWorkSnapshot().generation).toBe(before.generation + 2);
|
||||
});
|
||||
it("does not start a provider request when a credential resolves after its deadline", async () => {
|
||||
vi.useFakeTimers();
|
||||
|
||||
@@ -2,7 +2,7 @@ import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { inspectIdleSpool, readIdleLocalWork, markIdleIngressTracked, markIdleStartupComplete, registerIdleSpoolDirectory } from "../services/idle-local-work.js";
|
||||
import { inspectIdleSpool, inspectIdleQuotaRecovery, idleQuotaRecoveryPath, readIdleLocalWork, markIdleIngressTracked, markIdleStartupComplete, registerIdleSpoolDirectory } from "../services/idle-local-work.js";
|
||||
|
||||
let root: string;
|
||||
beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "idle-spool-")); });
|
||||
@@ -51,3 +51,54 @@ describe("idle cleanup inspection", () => {
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
describe("idle credential recovery inspection", () => {
|
||||
const company = "7238bc31-f3ca-42ba-b487-4122e77b8867";
|
||||
it("allows missing roots and empty company directories after successful recovery", async () => {
|
||||
expect(await inspectIdleQuotaRecovery(path.join(root, "missing"))).toBe("none");
|
||||
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
|
||||
await fs.mkdir(path.join(root, company));
|
||||
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
|
||||
});
|
||||
it.each(["receipt.json", "malformed.json", "partial.tmp", "write.probe", "unexpected"])(
|
||||
"retains every pending company entry without reading its contents: %s", async name => {
|
||||
await fs.mkdir(path.join(root, company));
|
||||
const file = path.join(root, company, name);
|
||||
await fs.writeFile(file, "unreadable recovery fixture");
|
||||
const read = vi.spyOn(fs, "readFile");
|
||||
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
|
||||
expect(read).not.toHaveBeenCalled();
|
||||
await fs.rm(file);
|
||||
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
|
||||
},
|
||||
);
|
||||
it("retains unexpected root entries, nested directories and directory symlinks", async () => {
|
||||
await fs.mkdir(path.join(root, "unexpected"));
|
||||
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
|
||||
await fs.rm(path.join(root, "unexpected"), { recursive: true });
|
||||
await fs.symlink(path.join(root, "missing"), path.join(root, company));
|
||||
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
|
||||
await fs.rm(path.join(root, company));
|
||||
await fs.mkdir(path.join(root, company, "nested"), { recursive: true });
|
||||
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
|
||||
});
|
||||
it("fails closed on invalid roots and directory read errors", async () => {
|
||||
const file = path.join(root, "file"), link = path.join(root, "link");
|
||||
await fs.writeFile(file, ""); await fs.symlink(root, link);
|
||||
expect(await inspectIdleQuotaRecovery(file)).toBe("unknown");
|
||||
expect(await inspectIdleQuotaRecovery(link)).toBe("unknown");
|
||||
vi.spyOn(fs, "opendir").mockRejectedValue(Object.assign(new Error("private path"), { code: "EACCES" }));
|
||||
expect(await inspectIdleQuotaRecovery(root)).toBe("unknown");
|
||||
});
|
||||
it("keeps persisted credential recovery awake after the request and process work have ended", async () => {
|
||||
vi.stubEnv("PAPERCLIP_HOME", root);
|
||||
markIdleIngressTracked(); markIdleStartupComplete({ scheduledBackups: false });
|
||||
const directory = path.join(idleQuotaRecoveryPath(), company);
|
||||
await fs.mkdir(directory, { recursive: true });
|
||||
const file = path.join(directory, "pending.json");
|
||||
await fs.writeFile(file, "encrypted journal fixture");
|
||||
expect(await readIdleLocalWork()).toBe("present");
|
||||
await fs.rm(file);
|
||||
expect(await readIdleLocalWork()).toBe("none");
|
||||
});
|
||||
});
|
||||
@@ -6,7 +6,7 @@ import {
|
||||
agentInstructionWorkingCopies, agents, agentApiKeys, agentWakeupRequests, companies, companySecretProposals, createDb,
|
||||
adapterAuthSessions, environments, environmentLeases, executionWorkspaces,
|
||||
heartbeatRuns, issues, issueWatchdogs, issueThreadInteractions, issueQuestionResponseDeliveries,
|
||||
projects, routines, type Db,
|
||||
projects, routines, toolApplications, toolConnections, type Db,
|
||||
} from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";
|
||||
@@ -443,6 +443,124 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
async function seedAiConnection() {
|
||||
const seeded = await seed();
|
||||
const [application] = await db.insert(toolApplications).values({
|
||||
companyId: seeded.companyId, name: "AI fixture", type: "rest_api",
|
||||
}).returning();
|
||||
const [connection] = await db.insert(toolConnections).values({
|
||||
companyId: seeded.companyId, applicationId: application!.id,
|
||||
name: "Saved AI login", uid: randomUUID(), connectionPurpose: "ai",
|
||||
transport: "runtime_auth", status: "active", enabled: true,
|
||||
}).returning();
|
||||
return { ...seeded, connection: connection! };
|
||||
}
|
||||
|
||||
it("allows saved AI credentials while preserving their connection", async () => {
|
||||
const { connection } = await seedAiConnection();
|
||||
expect(await read()).toEqual(none);
|
||||
expect(await db.select().from(toolConnections).where(eq(toolConnections.id, connection.id)))
|
||||
.toMatchObject([{ status: "active", enabled: true }]);
|
||||
});
|
||||
|
||||
it.each(["draft", "active_login", "pending_run", "accepted_refresh"])(
|
||||
"keeps AI connections awake for %s work", async kind => {
|
||||
const { companyId, runId, connection } = await seedAiConnection();
|
||||
if (kind === "draft") await db.update(toolConnections).set({ status: "draft" }).where(eq(toolConnections.id, connection.id));
|
||||
if (kind === "pending_run") await db.update(heartbeatRuns).set({ status: "queued" }).where(eq(heartbeatRuns.id, runId));
|
||||
if (kind === "active_login") {
|
||||
const [environment] = await db.insert(environments).values({ name: "Active AI login" }).returning();
|
||||
await db.insert(adapterAuthSessions).values({
|
||||
companyId, environmentId: environment!.id, connectionId: connection.id,
|
||||
adapterType: "codex_local", startedByUserId: "fixture-user",
|
||||
publicSessionId: randomUUID(), status: "waiting_for_user",
|
||||
});
|
||||
}
|
||||
const finish = kind === "accepted_refresh" ? beginIdleTrackedWork() : undefined;
|
||||
try { expect(await read()).toEqual(kind === "accepted_refresh" ? unknown : present); }
|
||||
finally { finish?.(); }
|
||||
},
|
||||
);
|
||||
|
||||
it("retains remote MCP connections alongside inert AI credentials", async () => {
|
||||
const { companyId, connection } = await seedAiConnection();
|
||||
await db.insert(toolConnections).values({
|
||||
companyId, applicationId: connection.applicationId, name: "Remote tools", uid: randomUUID(),
|
||||
connectionPurpose: "tool", transport: "mcp_remote", status: "active", enabled: true,
|
||||
});
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
async function seedFinishedFailedLease() {
|
||||
const seeded = await seed();
|
||||
await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(now) })
|
||||
.where(eq(heartbeatRuns.id, seeded.runId));
|
||||
const [environment] = await db.insert(environments).values({ name: "Failed turn environment", driver: "sandbox" }).returning();
|
||||
const leaseId = randomUUID();
|
||||
const receipt = {
|
||||
schema: "paperclip.remote-termination.v1", companyId: seeded.companyId,
|
||||
runId: seeded.runId, leaseId, provider: "plugin:demo:sandbox",
|
||||
providerLeaseId: "fixture-resource", state: "destroyed", confirmedAt: new Date(now).toISOString(),
|
||||
};
|
||||
const [lease] = await db.insert(environmentLeases).values({
|
||||
id: leaseId, companyId: seeded.companyId, environmentId: environment!.id,
|
||||
heartbeatRunId: seeded.runId, status: "failed", leasePolicy: "ephemeral",
|
||||
provider: receipt.provider, providerLeaseId: receipt.providerLeaseId,
|
||||
acquiredAt: new Date(now - 1000), releasedAt: new Date(now), cleanupStatus: "success",
|
||||
metadata: { remoteExecutionTermination: receipt },
|
||||
}).returning();
|
||||
return { ...seeded, environment: environment!, lease: lease!, receipt };
|
||||
}
|
||||
|
||||
it.each(["destroyed", "stopped"])("allows failed lease history after exact provider %s proof", async state => {
|
||||
const { lease, receipt } = await seedFinishedFailedLease();
|
||||
await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: { ...receipt, state } } })
|
||||
.where(eq(environmentLeases.id, lease.id));
|
||||
expect(await read()).toEqual(none);
|
||||
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)))
|
||||
.toMatchObject([{ status: "failed", cleanupStatus: "success" }]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
"cleanup_failed", "cleanup_unknown", "unreleased", "retained_policy", "reusable_policy",
|
||||
"active", "pending_cleanup", "missing_receipt", "wrong_company", "wrong_run", "wrong_lease",
|
||||
"wrong_provider", "wrong_resource", "not_stopped", "missing_run", "unfinished_run",
|
||||
"retry", "accounting", "cleanup_in_flight", "pending_export", "pending_retain",
|
||||
])("retains failed leases with %s evidence", async kind => {
|
||||
const { lease, receipt, runId } = await seedFinishedFailedLease();
|
||||
const change: Partial<typeof environmentLeases.$inferInsert> = {};
|
||||
if (kind === "cleanup_failed") change.cleanupStatus = "failed";
|
||||
if (kind === "cleanup_unknown") change.cleanupStatus = null;
|
||||
if (kind === "unreleased") change.releasedAt = null;
|
||||
if (kind === "retained_policy") change.leasePolicy = "retain_on_failure";
|
||||
if (kind === "reusable_policy") change.leasePolicy = "reuse_by_environment";
|
||||
if (kind === "active" || kind === "pending_cleanup") change.status = kind;
|
||||
if (kind === "missing_receipt") change.metadata = {};
|
||||
const fields: Record<string, string> = { wrong_company: "companyId", wrong_run: "runId", wrong_lease: "leaseId", wrong_provider: "provider", wrong_resource: "providerLeaseId", not_stopped: "state" };
|
||||
if (fields[kind]) change.metadata = { remoteExecutionTermination: { ...receipt, [fields[kind]]: "wrong" } };
|
||||
if (kind === "missing_run") change.heartbeatRunId = null;
|
||||
if (kind === "unfinished_run") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, runId));
|
||||
if (kind === "retry") await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now + 1000) }).where(eq(heartbeatRuns.id, runId));
|
||||
if (kind === "accounting") await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, runId));
|
||||
if (kind === "cleanup_in_flight") change.metadata = { remoteExecutionTermination: receipt, pendingCleanupInFlight: true };
|
||||
if (kind === "pending_export") change.metadata = { remoteExecutionTermination: receipt, nativeWorkspaceExportResume: {} };
|
||||
if (kind === "pending_retain") change.metadata = { remoteExecutionTermination: receipt, sandboxStopAndRetain: {} };
|
||||
if (Object.keys(change).length) await db.update(environmentLeases).set(change).where(eq(environmentLeases.id, lease.id));
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
it("uses the same failed-lease cleanup proof for finished login history", async () => {
|
||||
const { companyId, environment, lease } = await seedFinishedFailedLease();
|
||||
await db.insert(adapterAuthSessions).values({
|
||||
companyId, environmentId: environment.id, adapterType: "codex_local", startedByUserId: "fixture-user",
|
||||
publicSessionId: randomUUID(), providerLeaseId: lease.providerLeaseId,
|
||||
status: "authenticated", finishedAt: new Date(now),
|
||||
});
|
||||
expect(await read()).toEqual(none);
|
||||
await db.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, lease.id));
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
it("reports a login sandbox awaiting cleanup", async () => {
|
||||
const { companyId } = await seed();
|
||||
const [environment] = await db.insert(environments).values({ name: "Login fixture" }).returning();
|
||||
|
||||
@@ -2,6 +2,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { refreshCompanySubscriptions, refreshSubscriptionConnection } from "../services/subscription-refresh.js";
|
||||
import type { SubscriptionConnection } from "../services/subscriptions.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
|
||||
const mocks = vi.hoisted(() => ({ accounts: vi.fn(), credential: vi.fn(), register: vi.fn(), probe: vi.fn() }));
|
||||
vi.mock("../services/ai-connections.js", () => ({ aiConnectionService: () => ({ subscriptionAccounts: mocks.accounts, credential: mocks.credential }) }));
|
||||
@@ -28,6 +29,40 @@ beforeEach(() => {
|
||||
});
|
||||
|
||||
describe("subscription discovery capacity", () => {
|
||||
it("tracks detached provider refresh through its final database write", async () => {
|
||||
const db = database(), probe = deferred<{ credentialKey: string }>(), save = deferred<string>();
|
||||
const before = idleWorkSnapshot();
|
||||
mocks.probe.mockReturnValue(probe.promise);
|
||||
mocks.register.mockReturnValue(save.promise);
|
||||
const task = refreshSubscriptionConnection(db, input("tracked"), "tracked");
|
||||
try {
|
||||
await vi.waitFor(() => expect(mocks.probe).toHaveBeenCalledOnce());
|
||||
expect(idleWorkSnapshot().active).toBe(before.active + 1);
|
||||
expect(refreshSubscriptionConnection(db, input("tracked"), "tracked")).toBe(task);
|
||||
expect(idleWorkSnapshot().active).toBe(before.active + 1);
|
||||
probe.resolve({ credentialKey: "fixture-key" });
|
||||
await vi.waitFor(() => expect(mocks.register).toHaveBeenCalledOnce());
|
||||
expect(idleWorkSnapshot().active).toBe(before.active + 1);
|
||||
} finally {
|
||||
probe.resolve({ credentialKey: "fixture-key" }); save.resolve("tracked"); await task;
|
||||
}
|
||||
expect(idleWorkSnapshot().active).toBe(before.active);
|
||||
expect(idleWorkSnapshot().generation).toBe(before.generation + 2);
|
||||
});
|
||||
|
||||
it("tracks detached discovery while it waits for account credentials", async () => {
|
||||
const db = database(), accounts = deferred<[]>();
|
||||
const before = idleWorkSnapshot();
|
||||
mocks.accounts.mockReturnValue(accounts.promise);
|
||||
const task = refreshCompanySubscriptions(db, "tracked-company", "alice");
|
||||
try {
|
||||
expect(idleWorkSnapshot().active).toBe(before.active + 1);
|
||||
expect(refreshCompanySubscriptions(db, "tracked-company", "alice")).toBe(task);
|
||||
expect(idleWorkSnapshot().active).toBe(before.active + 1);
|
||||
} finally { accounts.resolve([]); await task; }
|
||||
expect(idleWorkSnapshot().active).toBe(before.active);
|
||||
});
|
||||
|
||||
it.each([false, true])("waits for a provider slot without dropping accepted discovery (provider fails: %s)", async (fails) => {
|
||||
const db = database();
|
||||
const gates = Array.from({ length: 5 }, () => deferred<void>());
|
||||
|
||||
@@ -9,6 +9,7 @@ export function registerIdleSpoolDirectory(directory: string): void { spoolDirec
|
||||
export const idleAccountingSpoolPath = () => path.join(resolvePaperclipInstanceRoot(), "accounting-receipts");
|
||||
export const idleOrphanSpoolPath = () => process.env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR ??
|
||||
path.join(resolvePaperclipInstanceRoot(), "data", "sandbox-orphan-cleanup");
|
||||
export const idleQuotaRecoveryPath = () => path.join(resolvePaperclipInstanceRoot(), "quota-credential-recovery");
|
||||
let startupComplete = false;
|
||||
let ingressTracked = false;
|
||||
let scheduledBackups = false;
|
||||
@@ -33,13 +34,38 @@ export async function inspectIdleSpool(directory: string): Promise<IdleLocalWork
|
||||
}
|
||||
}
|
||||
|
||||
/** Credential journals are grouped by company. Successful recovery leaves its
|
||||
* empty company directory behind; every file, probe or unexpected entry still
|
||||
* blocks. Never decrypt journals or follow directory symlinks during a scan. */
|
||||
export async function inspectIdleQuotaRecovery(directory: string): Promise<IdleLocalWork> {
|
||||
try {
|
||||
const info = await fs.lstat(directory);
|
||||
if (!info.isDirectory() || info.isSymbolicLink()) return "unknown";
|
||||
const dir = await fs.opendir(directory);
|
||||
try {
|
||||
for (let entry = await dir.read(); entry !== null; entry = await dir.read()) {
|
||||
if (!entry.isDirectory() || entry.isSymbolicLink()
|
||||
|| !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(entry.name)) return "present";
|
||||
const state = await inspectIdleSpool(path.join(directory, entry.name));
|
||||
if (state !== "none") return state;
|
||||
}
|
||||
return "none";
|
||||
} finally { await dir.close(); }
|
||||
} catch (error) {
|
||||
return (error as NodeJS.ErrnoException).code === "ENOENT" ? "none" : "unknown";
|
||||
}
|
||||
}
|
||||
|
||||
export async function readIdleLocalWork(options: { backupCheckpoint?: boolean } = {}): Promise<IdleLocalWork> {
|
||||
if (!startupComplete || !ingressTracked) return "unknown";
|
||||
// This exemption belongs only to a scan which will finish a new backup
|
||||
// under the same admission hold before it can return an empty report.
|
||||
if ((scheduledBackups && !options.backupCheckpoint) || idleWorkSnapshot().active !== 0) return "present";
|
||||
const directories = new Set([...spoolDirectories, idleAccountingSpoolPath(), idleOrphanSpoolPath()]);
|
||||
const results = await Promise.all([...directories].map(inspectIdleSpool));
|
||||
const results = await Promise.all([
|
||||
...[...directories].map(inspectIdleSpool),
|
||||
inspectIdleQuotaRecovery(idleQuotaRecoveryPath()),
|
||||
]);
|
||||
if (results.includes("unknown")) return "unknown";
|
||||
return results.includes("present") ? "present" : "none";
|
||||
}
|
||||
@@ -22,6 +22,32 @@ export type InspectIdlePlugins = (hold: { ownerId: string; expiresAt: number })
|
||||
pluginIds: string[];
|
||||
}>;
|
||||
|
||||
// A failed turn may have successfully torn down its ephemeral allocation. A
|
||||
// cleanup label alone is not provider termination proof. Require the exact
|
||||
// host-owned receipt and finished run, and retain all cleanup/export intents.
|
||||
// Aliases are fixed at the call sites, never supplied by a client.
|
||||
function finishedFailedLease(alias: string): string {
|
||||
const receipt = `${alias}.metadata->'remoteExecutionTermination'`;
|
||||
return `${alias}.status = 'failed' AND ${alias}.lease_policy = 'ephemeral'
|
||||
AND ${alias}.cleanup_status = 'success' AND ${alias}.released_at IS NOT NULL
|
||||
AND COALESCE(${alias}.metadata->>'pendingCleanupInFlight', 'false') = 'false'
|
||||
AND NOT (COALESCE(${alias}.metadata, '{}'::jsonb) ?| ARRAY['nativeWorkspaceExportResume', 'sandboxStopAndRetain'])
|
||||
AND ${alias}.provider IS NOT NULL AND ${alias}.provider <> 'local'
|
||||
AND ${alias}.provider_lease_id IS NOT NULL AND ${alias}.provider_lease_id <> ''
|
||||
AND ${receipt}->>'schema' = 'paperclip.remote-termination.v1'
|
||||
AND ${receipt}->>'companyId' = ${alias}.company_id::text
|
||||
AND ${receipt}->>'runId' = ${alias}.heartbeat_run_id::text
|
||||
AND ${receipt}->>'leaseId' = ${alias}.id::text
|
||||
AND ${receipt}->>'provider' = ${alias}.provider
|
||||
AND ${receipt}->>'providerLeaseId' = ${alias}.provider_lease_id
|
||||
AND ${receipt}->>'state' IN ('stopped', 'destroyed')
|
||||
AND EXISTS (SELECT 1 FROM heartbeat_runs r WHERE r.id = ${alias}.heartbeat_run_id
|
||||
AND r.company_id = ${alias}.company_id
|
||||
AND r.status IN ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted')
|
||||
AND r.finished_at IS NOT NULL AND r.scheduled_retry_at IS NULL
|
||||
AND r.cost_accounting_pending = false)`;
|
||||
}
|
||||
|
||||
// These are deliberately conservative. Completed agent runs and workspace
|
||||
// operations are history, not reasons to keep an otherwise idle instance up.
|
||||
// Other background features remain awake until their work and inbound events
|
||||
@@ -60,13 +86,18 @@ const WORK_CHECKS = [
|
||||
// retry after immediate evaluation failed before creating a review/run.
|
||||
`SELECT 1 FROM issue_watchdogs WHERE status <> 'disabled'`,
|
||||
`SELECT 1 FROM workspace_operations WHERE status NOT IN ('succeeded', 'failed', 'cancelled')`,
|
||||
`SELECT 1 FROM environment_leases WHERE status NOT IN ('released', 'expired')`,
|
||||
`SELECT 1 FROM environment_leases l WHERE status NOT IN ('released', 'expired')
|
||||
AND (${finishedFailedLease("l")}) IS NOT TRUE`,
|
||||
`SELECT 1 FROM routines WHERE status NOT IN ('paused', 'archived')`,
|
||||
`SELECT 1 FROM routine_runs WHERE status NOT IN ('completed', 'failed', 'skipped', 'cancelled')`,
|
||||
`SELECT 1 FROM plugin_jobs WHERE status <> 'paused'`,
|
||||
`SELECT 1 FROM plugin_job_runs WHERE status NOT IN ('succeeded', 'failed', 'cancelled')`,
|
||||
`SELECT 1 FROM plugin_webhook_deliveries WHERE status NOT IN ('succeeded', 'failed')`,
|
||||
`SELECT 1 FROM tool_connections WHERE status <> 'archived'`,
|
||||
// Saved AI credentials have no independent worker. Login sessions, accepted
|
||||
// quota/refresh work and recovery journals have their own checks. Other
|
||||
// integrations still need an ingress/scheduler wake contract.
|
||||
`SELECT 1 FROM tool_connections WHERE status <> 'archived'
|
||||
AND NOT (status = 'active' AND connection_purpose = 'ai' AND transport = 'runtime_auth')`,
|
||||
`SELECT 1 FROM issue_recovery_actions WHERE status NOT IN ('resolved', 'cancelled')`,
|
||||
`SELECT 1 FROM status_cards WHERE archived_at IS NULL`,
|
||||
`SELECT 1 FROM external_objects WHERE NOT is_terminal OR next_refresh_at IS NOT NULL`,
|
||||
@@ -83,7 +114,8 @@ const WORK_CHECKS = [
|
||||
AND (l.provider_lease_id = s.provider_lease_id OR l.id = CASE
|
||||
WHEN s.provider_lease_id ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
|
||||
THEN s.provider_lease_id::uuid END)
|
||||
AND l.status IN ('released', 'expired') AND l.cleanup_status = 'success'
|
||||
AND (l.status IN ('released', 'expired') OR (${finishedFailedLease("l")}))
|
||||
AND l.cleanup_status = 'success'
|
||||
AND l.released_at IS NOT NULL))
|
||||
OR (s.provider_lease_id IS NOT NULL AND EXISTS (
|
||||
SELECT 1 FROM environment_leases l WHERE l.company_id = s.company_id
|
||||
@@ -91,7 +123,7 @@ const WORK_CHECKS = [
|
||||
AND (l.provider_lease_id = s.provider_lease_id OR l.id = CASE
|
||||
WHEN s.provider_lease_id ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
|
||||
THEN s.provider_lease_id::uuid END)
|
||||
AND (l.status IN ('released', 'expired') AND l.cleanup_status = 'success'
|
||||
AND ((l.status IN ('released', 'expired') OR (${finishedFailedLease("l")})) AND l.cleanup_status = 'success'
|
||||
AND l.released_at IS NOT NULL) IS NOT TRUE))`,
|
||||
// Answer delivery history is not work once its acknowledged handoff has
|
||||
// finished. Keep pending deliveries, unfinished target runs, and missing or
|
||||
|
||||
@@ -2,9 +2,9 @@ import { createHash, randomUUID } from "node:crypto";
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { z } from "zod";
|
||||
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
|
||||
import { localEncryptedProvider } from "../secrets/local-encrypted-provider.js";
|
||||
import { ensureDurableDirectory, syncDirectory } from "../lib/durable-directory.js";
|
||||
import { idleQuotaRecoveryPath } from "./idle-local-work.js";
|
||||
|
||||
const recoverySchema = z.object({
|
||||
companyId: z.string().uuid(), grantId: z.string().uuid(), connectionId: z.string().uuid(), secretId: z.string().uuid(),
|
||||
@@ -13,7 +13,7 @@ const recoverySchema = z.object({
|
||||
type Recovery = z.infer<typeof recoverySchema>;
|
||||
export const quotaCredentialHash = (value: string) => createHash("sha256").update(value).digest("hex");
|
||||
export const quotaCredentialRecoveryPath = (companyId: string, grantId: string) =>
|
||||
path.join(resolvePaperclipInstanceRoot(), "quota-credential-recovery", companyId, `${grantId}.json`);
|
||||
path.join(idleQuotaRecoveryPath(), companyId, `${grantId}.json`);
|
||||
|
||||
/** Called only while holding the company credential mutation lock. The local
|
||||
* encrypted provider uses the instance secrets master key, including when the
|
||||
|
||||
@@ -8,6 +8,7 @@ import { companySecrets, companySecretVersions, userSecretDefinitions, type Db }
|
||||
import { fetchCodexQuota } from "@paperclipai/adapter-codex-local/server";
|
||||
import { fetchClaudeQuota } from "@paperclipai/adapter-claude-local/server";
|
||||
import { aiConnectionService } from "./ai-connections.js";
|
||||
import { trackIdleWork } from "./task-admission.js";
|
||||
|
||||
const accountRequests = new Map<string, { expires: number; result: Promise<ProviderQuotaResult> }>();
|
||||
const publicError = "Subscription quota is currently unavailable. Check usage with your provider.";
|
||||
@@ -51,7 +52,9 @@ export async function fetchCompanyQuotaWindows(db: Db, companyId: string, userId
|
||||
let pending = accountRequests.get(key);
|
||||
if (!pending || pending.expires <= Date.now()) {
|
||||
const controller = new AbortController();
|
||||
const result = (async (): Promise<ProviderQuotaResult> => {
|
||||
// The HTTP deadline can expire while an OAuth exchange still needs to
|
||||
// save its consumed token. Track the actual work, not its timeout wrapper.
|
||||
const result = trackIdleWork((async (): Promise<ProviderQuotaResult> => {
|
||||
let credentialResolved = false;
|
||||
try {
|
||||
const value = await service.credential(row);
|
||||
@@ -104,7 +107,7 @@ export async function fetchCompanyQuotaWindows(db: Db, companyId: string, userId
|
||||
return { ...base, ok: false, windows: [], error: publicError,
|
||||
errorFamily };
|
||||
}
|
||||
})();
|
||||
})());
|
||||
const bounded: Promise<ProviderQuotaResult> = withQuotaTimeout(base.provider, result, () => controller.abort()).then(quota => {
|
||||
const observation = { ...base, ...quota, ...(quota.ok ? {} : { error: publicError }) };
|
||||
if (observation.ok && observation.accountKey !== accountKey) {
|
||||
|
||||
@@ -5,6 +5,7 @@ import { subscriptionService, type SubscriptionConnection } from "./subscription
|
||||
import { probeSubscriptionIdentity } from "./subscription-identity.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { tooManyRequests } from "../errors.js";
|
||||
import { trackIdleWork } from "./task-admission.js";
|
||||
|
||||
const jobs = new WeakMap<Db, Map<string, Promise<void>>>();
|
||||
const discovery = new WeakMap<Db, Map<string, Promise<void>>>();
|
||||
@@ -19,7 +20,7 @@ export function refreshSubscriptionConnection(db: Db, input: SubscriptionConnect
|
||||
// Bound provider concurrency across all companies on this instance. Dashboard
|
||||
// discovery waits below; opportunistic run observations remain due when busy.
|
||||
if (active.size >= 4) return Promise.resolve();
|
||||
const task = (async () => {
|
||||
const task = trackIdleWork((async () => {
|
||||
const [claimed] = await db.update(aiSubscriptions).set({ lastCheckedAt: new Date() }).where(and(
|
||||
eq(aiSubscriptions.companyId, input.companyId), eq(aiSubscriptions.id, id),
|
||||
or(isNull(aiSubscriptions.lastCheckedAt), lt(aiSubscriptions.lastCheckedAt, new Date(Date.now() - REFRESH_MS))),
|
||||
@@ -38,7 +39,7 @@ export function refreshSubscriptionConnection(db: Db, input: SubscriptionConnect
|
||||
})().catch(() => {
|
||||
// No raw provider/credential errors are logged or returned to a user.
|
||||
logger.warn({ companyId: input.companyId, subscriptionId: id }, "Subscription observation could not be saved");
|
||||
}).finally(() => active!.delete(id));
|
||||
}).finally(() => active!.delete(id)));
|
||||
active.set(id, task);
|
||||
return task;
|
||||
}
|
||||
@@ -50,7 +51,7 @@ export function refreshCompanySubscriptions(db: Db, companyId: string, userId: s
|
||||
const existing = active.get(key);
|
||||
if (existing) return existing;
|
||||
if (active.size >= 20) throw tooManyRequests("Subscription account checks are busy. Try again shortly.");
|
||||
const task = (async () => {
|
||||
const task = trackIdleWork((async () => {
|
||||
const service = aiConnectionService(db);
|
||||
// Uses the existing credential audience. Being a company admin does not
|
||||
// authorize probing another member's personal credentials.
|
||||
@@ -75,7 +76,7 @@ export function refreshCompanySubscriptions(db: Db, companyId: string, userId: s
|
||||
}
|
||||
}
|
||||
})().catch(() => logger.warn({ companyId }, "Subscription discovery unavailable"))
|
||||
.finally(() => active!.delete(key));
|
||||
.finally(() => active!.delete(key)));
|
||||
active.set(key, task);
|
||||
return task;
|
||||
}
|
||||
Reference in new issue
Block a user