mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget enforcement stops agent runs when a budget hard limit is reached. The stop intent is saved in each run's `result_json` until run control delivers it. > - The budget worker, run control, and the idle-sleep check find these pending stops with a predicate on `result_json`. No index supports that predicate. > - Each check therefore reads every heartbeat run and its large result JSON from TOAST. On a hosted database, this statement family was among the most expensive: about 8,000 calls at about 1 s each, about 9,800 s in total. > - Pending stops are rare. The predicate is constant, so a partial index can contain only those rows. > - This pull request adds that partial index. The SQL that the server sends does not change. > - The benefit is that each check reads a few index entries instead of the full table. The database has more capacity for request work. ## Linked Issues or Issue Description **What existing behavior does this improve?** The check for committed budget Stops that are not delivered yet. The budget worker runs it each time it looks for work (`hasPendingBudgetEnforcement`), and again for each delivery pass (`listPendingBudgetCancellationScopes`). Run control and the idle-sleep check use the same predicate. **Subsystem affected** Budget enforcement in the server (`server/src/services/heartbeat/budget-cancellation.ts`, `server/src/services/budgets.ts`, `server/src/services/heartbeat/run-control.ts`, `server/src/services/idle-sleep-safety.ts`) and the `heartbeat_runs` schema in `packages/db`. **Current behavior** The query groups all rows of `heartbeat_runs` and reads `result_json->'startupCancellation'->'budgetEnforcement'` for each row. A limit of 1 does not help, because the group and sort occur before the limit. Cost grows with the run history, not with the number of pending stops. **Proposed behavior** A partial index, `heartbeat_runs_pending_budget_cancellation_idx`, contains only runs with a pending budget Stop. The planner uses it for all four callers. The results are the same. **Reason and benefit** Local SQL measurements show a 300x to 1,000x decrease in time and buffer reads (see Verification). **Breaking changes** None. ## What Changed - Added the partial index `heartbeat_runs_pending_budget_cancellation_idx` on `heartbeat_runs (company_id)`. Its predicate is identical to `pendingBudgetRunCancellationPredicate`. - Added migration `0455_pending_budget_cancellation_index.sql`. It uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, and it is in the list of migrations that run outside a transaction, as `0436` is. - Moved the scoped run-control read and the idle-sleep check into exported builders in `budget-cancellation.ts`, next to the scope list query. The SQL does not change. The callers and the test now share the same statements. - Added a real PostgreSQL regression test. It checks that delivered, malformed, and missing intents are not pending. It also EXPLAINs the statements that the callers send (scope list with limit 1 and 100, scoped read, `EXISTS` check) and requires the new index in each plan. If a person changes a query shape, the server predicate, or the index predicate, the test fails. - Added `scripts/benchmark-budget-cancellation-scan.ts`. It captures the SQL that the real service sends, then compares the plans, timings, and complete rows with and without the index. - Updated `doc/DATABASE.md`. ## Verification Red/green regression test: - Before the migration, the new test failed: the plan was `Seq Scan` with the predicate as a filter. - After the migration: `npx vitest run src/services/heartbeat/budget-cancellation.test.ts` passes (17 tests). - Also passed: `src/__tests__/budgets-service.test.ts`, `src/__tests__/idle-sleep-diagnostics-postgres.test.ts`, `src/services/budget-resume-continuation.test.ts`. - `pnpm --filter @paperclipai/db check:migrations`, and typecheck of `@paperclipai/db` and `@paperclipai/server` pass. Performance and result equality (local SQL measurements only): ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-cancellation-scan.ts --runs=20000 --payload-kib=16 [--reverse] ``` - Fixture: embedded PostgreSQL 18.1. 20,241 runs in 2 companies, with incompressible result JSON of about 16 KiB (table 379 MB). 40 pending stops in 6 scopes, 200 delivered or malformed decoys, and 1 cross-company intent. - The SQL is the same in both treatments. The script toggles only the new index. 2 warmups and 7 samples per query. Rows are compared outside the timed part. Ordered queries must match in order. The scoped run-control read has no `ORDER BY`, so it is compared as a multiset. | Query (caller) | Baseline median | With index median | Shared buffers | | --- | --- | --- | --- | | `listPendingBudgetCancellationScopes(db, 1)` (worker `hasPending`) | 688 ms | 2.1 ms | 243,500 → 1,158 | | `listPendingBudgetCancellationScopes(db)` (delivery) | 799 ms | 1.4 ms | 243,500 → 1,158 | | Scoped pending runs (run control) | 357 ms | 1.1 ms | 121,835 → 322 | | `EXISTS` pending (idle-sleep check) | 440 ms | 0.37 ms | 240,560 → 2 | - Reverse order (index first) gave the same result: 524 ms → 1.6 ms, 464 ms → 1.4 ms, 306 ms → 1.0 ms, 306 ms → 0.42 ms. - Plans change from `Seq Scan` (or a broad bitmap scan) to `Index Scan` or `Index Only Scan` on the new index. - All result rows are equal in every run. Migration duration: - 20,241 runs (379 MB): index build 2.0 s to 3.9 s. - 200,241 runs with 8 KiB result JSON (2.1 GB): index build 5.6 s. With this table size, the four queries change from 1,148 / 879 / 514 / 4,392 ms to 1.1 / 1.2 / 0.85 / 8.3 ms. - The build is concurrent, so it does not block run writes. ## Risks - Migration: the build reads every run's result JSON one time. It is concurrent and resumable (`IF NOT EXISTS`). The executor repairs an invalid concurrent index on retry. - Write cost: run writes evaluate the index predicate. The index contains very few rows, so the extra cost is small. - Predicate drift: if a later change edits only one of the two predicates, the planner cannot use the index. The new regression test detects this. - The hosted timings are not measured yet. The local numbers come from a synthetic fixture. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
510 lines
34 KiB
TypeScript
510 lines
34 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { spawn } from "node:child_process";
|
|
import { once } from "node:events";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { agents, agentWakeupRequests, browserUseRuns, browserUseSessions, budgetPolicies, companies, connectionGrants, costEvents, environmentLeases, heartbeatRuns, issues, projects, toolApplications, toolConnections } from "@paperclipai/db";
|
|
import { eq, inArray, sql } from "drizzle-orm";
|
|
import { describe, expect, it, vi } from "vitest";
|
|
import { runningProcesses } from "../../adapters/index.js";
|
|
import { terminateLocalService } from "../local-service-supervisor.js";
|
|
import { persistUsageReceipt } from "../usage-receipts.js";
|
|
import { registerBudgetEnforcementWork } from "../budget-enforcement-worker.js";
|
|
import { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
|
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../delivery-work-notifications.js";
|
|
import { PROCESS_START_REQUESTED } from "../native-local-process-stop.js";
|
|
import { createPostgresRunDispatchAdapter } from "../../modules/run-dispatch/adapters/postgres.js";
|
|
import { createHeartbeatRunControl } from "./run-control.js";
|
|
import {
|
|
listPendingBudgetCancellationScopes,
|
|
pendingBudgetCancellationScopesQuery,
|
|
pendingBudgetRunCancellationExistsQuery,
|
|
pendingBudgetRunCancellationPredicate,
|
|
pendingBudgetRunIdsForScopeQuery,
|
|
} from "./budget-cancellation.js";
|
|
import { callbacks, runControlPostgresSupport, setupRunControlDatabase } from "./__tests__/run-control-fixtures.js";
|
|
|
|
vi.mock("../live-events.js", () => ({ publishLiveEvent: vi.fn() }));
|
|
vi.mock("../local-service-supervisor.js", async (importOriginal) => ({
|
|
...await importOriginal<typeof import("../local-service-supervisor.js")>(),
|
|
terminateLocalService: vi.fn(async () => undefined),
|
|
}));
|
|
|
|
describe.skipIf(!runControlPostgresSupport.supported)("scoped budget cancellation", () => {
|
|
let db: Db;
|
|
const { fixture, liveRun } = setupRunControlDatabase("paperclip-heartbeat-budget-cancellation", value => { db = value; });
|
|
|
|
it("rechecks stale budget enforcement before cancelling runs or queued wakes", async () => {
|
|
const { company, agent } = await fixture();
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status: "queued", runtimeMode: "legacy" }).returning();
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: company.id, agentId: agent.id, source: "assignment", status: "queued" }).returning();
|
|
const deps = callbacks(db);
|
|
await createHeartbeatRunControl(db, deps).cancelBudgetScopeWork({ companyId: company.id, scopeType: "agent", scopeId: agent.id, enforcement: { policyId: randomUUID(), version: 1 } });
|
|
expect((await deps.getRun(run.id))?.status).toBe("queued");
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0].status).toBe("queued");
|
|
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
|
|
expect(deps.finalizeAgentStatus).not.toHaveBeenCalled();
|
|
expect(terminateLocalService).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("does not apply an earlier enforcement generation after a budget increase", async () => {
|
|
const { company, agent } = await fixture();
|
|
const [policy] = await db.insert(budgetPolicies).values({ companyId: company.id,
|
|
scopeType: "agent", scopeId: agent.id, windowKind: "lifetime", amount: 10_000,
|
|
enforcementVersion: 2, enforcementDeliveredVersion: 1 }).returning();
|
|
const run = await liveRun(company.id, agent.id);
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: company.id, agentId: agent.id,
|
|
source: "assignment", status: "queued" }).returning();
|
|
await createHeartbeatRunControl(db, callbacks(db)).cancelBudgetScopeWork({ companyId: company.id,
|
|
scopeType: "agent", scopeId: agent.id, enforcement: { policyId: policy.id, version: 1 } });
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toMatchObject({ status: "running", resultJson: null });
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0].status).toBe("queued");
|
|
expect(terminateLocalService).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("fences every affected run and queued wake before bounded provider delivery", async () => {
|
|
const { company, agent } = await fixture();
|
|
const runs = await db.insert(heartbeatRuns).values(Array.from({ length: 5 }, () => ({
|
|
companyId: company.id, agentId: agent.id, status: "running", runtimeMode: "legacy", runtimeModeResolvedAt: new Date(),
|
|
}))).returning();
|
|
const [unrelatedAgent] = await db.insert(agents).values({ companyId: company.id, name: "Unrelated" }).returning();
|
|
const unrelated = await db.insert(heartbeatRuns).values({ companyId: company.id,
|
|
agentId: unrelatedAgent.id, status: "running", runtimeMode: "legacy" }).returning().then(rows => rows[0]);
|
|
const wakes = await db.insert(agentWakeupRequests).values([
|
|
{ companyId: company.id, agentId: agent.id, source: "assignment", status: "queued" },
|
|
{ companyId: company.id, agentId: agent.id, source: "assignment", status: "deferred_issue_execution" },
|
|
{ companyId: company.id, agentId: unrelatedAgent.id, source: "assignment", status: "queued" },
|
|
]).returning();
|
|
runs.forEach((run, i) => runningProcesses.set(run.id, {
|
|
child: { pid: 425000 + i } as never, graceSec: 4, processGroupId: 425000 + i,
|
|
}));
|
|
const releases = new Map<number, () => void>();
|
|
let active = 0, maximumActive = 0, releaseAll = false;
|
|
vi.mocked(terminateLocalService).mockImplementation(async input => {
|
|
active += 1;
|
|
maximumActive = Math.max(maximumActive, active);
|
|
if (!releaseAll) await new Promise<void>(resolve => releases.set(input.pid, resolve));
|
|
active -= 1;
|
|
});
|
|
const deps = callbacks(db);
|
|
const delivery = createHeartbeatRunControl(db, deps).cancelBudgetScopeWork({ companyId: company.id,
|
|
scopeType: "agent", scopeId: agent.id });
|
|
try {
|
|
await vi.waitFor(() => expect(releases.size).toBe(4));
|
|
const persisted = await db.select().from(heartbeatRuns).where(inArray(heartbeatRuns.id, runs.map(run => run.id)));
|
|
expect(persisted.every(run => run.status === "running" && Boolean(run.resultJson?.startupCancellation))).toBe(true);
|
|
const pendingWakes = await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, wakes.map(wake => wake.id)));
|
|
expect(wakes.map(wake => pendingWakes.find(row => row.id === wake.id)?.status)).toEqual(["cancelled", "cancelled", "queued"]);
|
|
expect((await deps.getRun(unrelated.id))?.resultJson).toBeNull();
|
|
// A slow first Stop must not stop another slot from making progress.
|
|
const firstPid = releases.keys().next().value!;
|
|
const otherPid = [...releases.keys()].find(pid => pid !== firstPid)!;
|
|
releases.get(otherPid)!();
|
|
await vi.waitFor(() => expect(releases.size).toBe(5));
|
|
expect(maximumActive).toBe(4);
|
|
expect(active).toBe(4);
|
|
} finally {
|
|
releaseAll = true;
|
|
releases.forEach(release => release());
|
|
await delivery;
|
|
}
|
|
expect((await db.select().from(heartbeatRuns).where(inArray(heartbeatRuns.id, runs.map(run => run.id))))
|
|
.every(run => run.status === "cancelled" && (run.resultJson?.executionCancellation as { state?: string })?.state === "acknowledged")).toBe(true);
|
|
expect((await deps.getRun(unrelated.id))?.status).toBe("running");
|
|
});
|
|
|
|
it("wakes scoped Stop delivery from a partial usage overrun without waiting for adapter termination", async () => {
|
|
const { company, agent } = await fixture();
|
|
const [project, otherProject] = await db.insert(projects).values([
|
|
{ companyId: company.id, name: "Over budget" }, { companyId: company.id, name: "Still admitted" },
|
|
]).returning();
|
|
const sourceId = randomUUID();
|
|
const run = await liveRun(company.id, agent.id);
|
|
await db.update(heartbeatRuns).set({ contextSnapshot: { projectId: project.id }, costAccountingPending: true,
|
|
usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0,
|
|
accountingReceiptReady: false, ledgerScope: { projectId: project.id } },
|
|
}).where(eq(heartbeatRuns.id, run.id));
|
|
const [queued, unrelated] = await db.insert(heartbeatRuns).values([
|
|
{ companyId: company.id, agentId: agent.id, status: "queued", runtimeMode: "legacy", contextSnapshot: { projectId: project.id } },
|
|
{ companyId: company.id, agentId: agent.id, status: "running", runtimeMode: "legacy", contextSnapshot: { projectId: otherProject.id } },
|
|
]).returning();
|
|
const wakes = await db.insert(agentWakeupRequests).values([
|
|
{ companyId: company.id, agentId: agent.id, source: "assignment", status: "queued", payload: { projectId: project.id } },
|
|
{ companyId: company.id, agentId: agent.id, source: "assignment", status: "deferred_issue_execution", payload: { projectId: project.id } },
|
|
{ companyId: company.id, agentId: agent.id, source: "assignment", status: "queued", payload: { projectId: otherProject.id } },
|
|
]).returning();
|
|
const [policy] = await db.insert(budgetPolicies).values({ companyId: company.id, scopeType: "project",
|
|
scopeId: project.id, windowKind: "lifetime", amount: 100, notifyEnabled: false }).returning();
|
|
let releaseTermination!: () => void;
|
|
const termination = new Promise<void>(resolve => { releaseTermination = resolve; });
|
|
vi.mocked(terminateLocalService).mockImplementation(async () => termination);
|
|
const deps = callbacks(db);
|
|
const control = createHeartbeatRunControl(db, deps);
|
|
const errors = vi.fn();
|
|
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: errors });
|
|
const worker = registerBudgetEnforcementWork(db, coordinator, { cancelWorkForScope: control.cancelBudgetScopeWork });
|
|
try {
|
|
// Drain startup recovery first: only the receipt's committed notification
|
|
// may wake the idle worker below. Do not call wake or deliver explicitly.
|
|
await worker.ready;
|
|
expect(terminateLocalService).not.toHaveBeenCalled();
|
|
expect(coordinator.nextWakeAt()).toBeNull();
|
|
const startedAt = performance.now();
|
|
await persistUsageReceipt(db, { schema: "paperclip/accounting-receipt/v1", id: randomUUID(),
|
|
companyId: company.id, runId: run.id, sourceId, sequence: 1, receivedAt: new Date().toISOString(),
|
|
adapterType: agent.adapterType, receipt: { complete: false, usageBasis: "per_run", costUsd: 1.1,
|
|
provider: "test", biller: "test", model: "budget-stop-fixture", billingType: "metered_api", costStatus: "unpriced" },
|
|
});
|
|
const receiptReturnedAt = performance.now();
|
|
await vi.waitFor(async () => {
|
|
const persisted = await deps.getRun(run.id);
|
|
expect(persisted).toMatchObject({ status: "running", usageJson: { budgetExposure: { sequence: 1, costCents: "110.0000000" } },
|
|
resultJson: { startupCancellation: { budgetEnforcement: { policyId: policy.id, scopeType: "project", scopeId: project.id } } } });
|
|
expect(terminateLocalService).toHaveBeenCalledTimes(1);
|
|
}, { interval: 10, timeout: 3_000 });
|
|
const fenceObservedAt = performance.now();
|
|
const fenced = await deps.getRun(run.id);
|
|
expect(fenced?.resultJson?.executionCancellation).toBeUndefined();
|
|
expect((await deps.getRun(queued.id))?.resultJson?.startupCancellation).toBeTruthy();
|
|
const persistedWakes = await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, wakes.map(wake => wake.id)));
|
|
expect(wakes.map(wake => persistedWakes.find(row => row.id === wake.id)?.status)).toEqual(["cancelled", "cancelled", "queued"]);
|
|
const [pending] = await db.select().from(budgetPolicies).where(eq(budgetPolicies.id, policy.id));
|
|
expect(pending.enforcementVersion).toBeGreaterThan(pending.enforcementDeliveredVersion);
|
|
|
|
const requestStartedAt = performance.now();
|
|
let responseTimeout: ReturnType<typeof setTimeout> | undefined;
|
|
try {
|
|
const rows = await Promise.race([
|
|
db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, unrelated.id)),
|
|
new Promise<never>((_, reject) => { responseTimeout = setTimeout(() => reject(new Error("Unrelated database request blocked by Stop delivery")), 1_000); }),
|
|
]);
|
|
expect(rows[0]).toMatchObject({ status: "running", resultJson: null });
|
|
} finally { clearTimeout(responseTimeout); }
|
|
const unrelatedRequestMs = performance.now() - requestStartedAt;
|
|
const terminationReleasedAt = performance.now();
|
|
releaseTermination();
|
|
await vi.waitFor(async () => {
|
|
expect(await deps.getRun(run.id)).toMatchObject({ status: "cancelled", resultJson: { executionCancellation: { state: "acknowledged" } } });
|
|
const [delivered] = await db.select().from(budgetPolicies).where(eq(budgetPolicies.id, policy.id));
|
|
expect(delivered.enforcementDeliveredVersion).toBe(delivered.enforcementVersion);
|
|
}, { interval: 10, timeout: 3_000 });
|
|
expect((await deps.getRun(unrelated.id))?.status).toBe("running");
|
|
expect(errors).not.toHaveBeenCalled();
|
|
console.info("Partial usage budget Stop timing", { receiptMs: Math.round(receiptReturnedAt - startedAt),
|
|
receiptReturnToFenceMs: Math.round(fenceObservedAt - receiptReturnedAt), unrelatedRequestMs: Math.round(unrelatedRequestMs),
|
|
controlledTerminationWaitMs: Math.round(terminationReleasedAt - fenceObservedAt),
|
|
terminationReleaseToAcknowledgementMs: Math.round(performance.now() - terminationReleasedAt) });
|
|
} finally {
|
|
releaseTermination();
|
|
await coordinator.stop();
|
|
}
|
|
});
|
|
|
|
it("drains other provider stops after a failed delivery and retries the unfinished run", async () => {
|
|
const { company, agent } = await fixture();
|
|
const failed = await liveRun(company.id, agent.id);
|
|
const [other] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
|
|
status: "running", runtimeMode: "legacy", runtimeModeResolvedAt: new Date() }).returning();
|
|
runningProcesses.set(other.id, { child: { pid: 434343 } as never, graceSec: 4, processGroupId: 434343 });
|
|
vi.mocked(terminateLocalService).mockImplementation(async input => {
|
|
if (input.pid === 424242) throw new Error("provider unreachable");
|
|
});
|
|
const deps = callbacks(db);
|
|
const control = createHeartbeatRunControl(db, deps);
|
|
const scope = { companyId: company.id, scopeType: "agent" as const, scopeId: agent.id };
|
|
await expect(control.cancelBudgetScopeWork(scope)).rejects.toThrow("Budget cancellation delivery remains pending");
|
|
expect((await deps.getRun(other.id))?.status).toBe("cancelled");
|
|
expect(await deps.getRun(failed.id)).toMatchObject({ status: "running", resultJson: { startupCancellation: expect.any(Object) } });
|
|
expect(runningProcesses.has(failed.id)).toBe(true);
|
|
vi.mocked(terminateLocalService).mockResolvedValue(undefined);
|
|
await control.cancelBudgetScopeWork(scope);
|
|
expect(await deps.getRun(failed.id)).toMatchObject({ status: "cancelled", resultJson: { executionCancellation: { state: "acknowledged" } } });
|
|
});
|
|
|
|
it("shares four Stop slots across independent scopes and service instances", async () => {
|
|
const { company, agent } = await fixture();
|
|
const [otherAgent] = await db.insert(agents).values({ companyId: company.id, name: "Another scope" }).returning();
|
|
const runs = await db.insert(heartbeatRuns).values([agent.id, otherAgent.id].flatMap(agentId =>
|
|
Array.from({ length: 3 }, () => ({ companyId: company.id, agentId,
|
|
status: "running", runtimeMode: "legacy", runtimeModeResolvedAt: new Date() })))).returning();
|
|
runs.forEach((run, i) => runningProcesses.set(run.id, {
|
|
child: { pid: 435000 + i } as never, graceSec: 4, processGroupId: 435000 + i,
|
|
}));
|
|
let active = 0, maximumActive = 0, releaseAll = false;
|
|
const releases = new Map<number, () => void>();
|
|
vi.mocked(terminateLocalService).mockImplementation(async input => {
|
|
active += 1;
|
|
maximumActive = Math.max(maximumActive, active);
|
|
if (!releaseAll) await new Promise<void>(resolve => releases.set(input.pid, resolve));
|
|
active -= 1;
|
|
});
|
|
const deliveries = [agent, otherAgent].map(selected => createHeartbeatRunControl(db, callbacks(db))
|
|
.cancelBudgetScopeWork({ companyId: company.id, scopeType: "agent", scopeId: selected.id }));
|
|
try {
|
|
await vi.waitFor(() => expect(releases.size).toBe(4));
|
|
expect(maximumActive).toBe(4);
|
|
expect((await db.select().from(heartbeatRuns).where(inArray(heartbeatRuns.id, runs.map(run => run.id))))
|
|
.every(run => Boolean(run.resultJson?.startupCancellation))).toBe(true);
|
|
releases.values().next().value!();
|
|
await vi.waitFor(() => expect(releases.size).toBe(5));
|
|
expect(maximumActive).toBe(4);
|
|
} finally {
|
|
releaseAll = true;
|
|
releases.forEach(release => release());
|
|
await Promise.all(deliveries);
|
|
}
|
|
});
|
|
|
|
it("blocks retry promotion after the budget batch commits but before Stop delivery", async () => {
|
|
const { company, agent } = await fixture();
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
|
|
status: "scheduled_retry", runtimeMode: "legacy", runtimeModeResolvedAt: new Date(),
|
|
scheduledRetryAt: new Date(), scheduledRetryReason: "transient_failure" }).returning();
|
|
const deps = callbacks(db);
|
|
const getAgent = deps.getAgent.getMockImplementation()!;
|
|
let entered!: () => void, release!: () => void;
|
|
const deliveryEntered = new Promise<void>(resolve => { entered = resolve; });
|
|
const continueDelivery = new Promise<void>(resolve => { release = resolve; });
|
|
deps.getAgent.mockImplementationOnce(async id => { entered(); await continueDelivery; return getAgent(id); });
|
|
const cancellation = createHeartbeatRunControl(db, deps).cancelBudgetScopeWork({ companyId: company.id,
|
|
scopeType: "agent", scopeId: agent.id });
|
|
try {
|
|
await deliveryEntered;
|
|
expect(await createPostgresRunDispatchAdapter(db).promoteOrCancelDueRetry({
|
|
companyId: company.id, runId: run.id, now: new Date(),
|
|
})).toMatchObject({ outcome: "not_promoted" });
|
|
expect(await deps.getRun(run.id)).toMatchObject({ status: "scheduled_retry",
|
|
resultJson: { startupCancellation: expect.any(Object) } });
|
|
} finally {
|
|
release();
|
|
await cancellation;
|
|
}
|
|
expect(await deps.getRun(run.id)).toMatchObject({ status: "cancelled" });
|
|
});
|
|
|
|
it("recovers a committed Stop after its policy was deleted without fencing newly admitted work", async () => {
|
|
const { company, agent } = await fixture();
|
|
const oldRun = await liveRun(company.id, agent.id);
|
|
const [newRun] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
|
|
status: "queued", runtimeMode: "legacy" }).returning();
|
|
const intent = { companyId: company.id, scopeType: "agent", scopeId: agent.id,
|
|
policyId: randomUUID(), version: 1 };
|
|
await db.update(heartbeatRuns).set({ resultJson: { startupCancellation: {
|
|
requestedAt: new Date().toISOString(), budgetEnforcement: intent,
|
|
} } }).where(eq(heartbeatRuns.id, oldRun.id));
|
|
const scopes = await listPendingBudgetCancellationScopes(db);
|
|
expect(scopes).toHaveLength(1);
|
|
const deps = callbacks(db);
|
|
await createHeartbeatRunControl(db, deps).cancelBudgetScopeWork(scopes[0]);
|
|
expect(await deps.getRun(oldRun.id)).toMatchObject({ status: "cancelled",
|
|
resultJson: { startupCancellation: { budgetEnforcement: { deliveredAt: expect.any(String) } } } });
|
|
expect(await deps.getRun(newRun.id)).toMatchObject({ status: "queued", resultJson: null });
|
|
expect(await listPendingBudgetCancellationScopes(db)).toEqual([]);
|
|
});
|
|
|
|
it("retains a terminal budget Stop until actual stop acknowledgement exists", async () => {
|
|
const { company, agent } = await fixture();
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
|
|
status: "cancelled", runtimeMode: "legacy", startedAt: new Date(), resultJson: {
|
|
startupCancellation: { budgetEnforcement: { companyId: company.id, scopeType: "agent", scopeId: agent.id,
|
|
policyId: randomUUID(), version: 1 } },
|
|
} }).returning();
|
|
const control = createHeartbeatRunControl(db, callbacks(db));
|
|
const [scope] = await listPendingBudgetCancellationScopes(db);
|
|
await expect(control.cancelBudgetScopeWork(scope)).rejects.toThrow("Budget cancellation delivery remains pending");
|
|
expect(await listPendingBudgetCancellationScopes(db)).toHaveLength(1);
|
|
expect(terminateLocalService).not.toHaveBeenCalled();
|
|
await db.update(heartbeatRuns).set({ resultJson: { ...run.resultJson,
|
|
executionCancellation: { state: "acknowledged", acknowledgedAt: new Date().toISOString() },
|
|
} }).where(eq(heartbeatRuns.id, run.id));
|
|
await control.cancelBudgetScopeWork(scope);
|
|
expect(await listPendingBudgetCancellationScopes(db)).toEqual([]);
|
|
});
|
|
|
|
it.each(["local exit", "remote receipt", "unproven", "later launch"] as const)("reconciles normal completion before queued budget Stop using %s evidence", async evidence => {
|
|
const { company, agent } = await fixture();
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
|
|
status: "running", runtimeMode: "native", startedAt: new Date() }).returning();
|
|
const deps = callbacks(db);
|
|
if (evidence === "local exit" || evidence === "later launch") {
|
|
const child = spawn(process.execPath, ["-e", "setTimeout(() => process.exit(0), 50)"], { detached: true, stdio: "ignore" });
|
|
const exited = once(child, "exit");
|
|
try {
|
|
expect(child.pid).toBeGreaterThan(1);
|
|
await deps.persistRunProcessMetadata(run.id, { pid: child.pid!, processGroupId: child.pid!,
|
|
startedAt: new Date().toISOString(), targetKind: "local" });
|
|
await exited;
|
|
} finally { if (child.exitCode === null) child.kill("SIGKILL"); }
|
|
if (evidence === "later launch") await deps.appendRunEvent(run, {
|
|
eventType: PROCESS_START_REQUESTED, stream: "system", level: "info", message: "A later launch invalidated the earlier exit proof",
|
|
});
|
|
}
|
|
if (evidence === "remote receipt") {
|
|
const leaseId = randomUUID(), providerLeaseId = randomUUID();
|
|
await db.insert(environmentLeases).values({ id: leaseId, companyId: company.id, heartbeatRunId: run.id,
|
|
provider: "daytona", providerLeaseId, status: "released", releasedAt: new Date(), cleanupStatus: "success",
|
|
metadata: { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: company.id,
|
|
runId: run.id, leaseId, provider: "daytona", providerLeaseId, state: "stopped", confirmedAt: new Date().toISOString() } },
|
|
});
|
|
}
|
|
const getRun = deps.getRun.getMockImplementation()!;
|
|
deps.getRun.mockImplementationOnce(async id => {
|
|
// Budget delivery persisted its fence, then ordinary native completion
|
|
// won the race before this run acquired a provider Stop delivery slot.
|
|
expect((await getRun(id))?.resultJson?.startupCancellation).toBeTruthy();
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date(), nativePhase: "committed",
|
|
processPid: null, processGroupId: null, processStartedAt: null }).where(eq(heartbeatRuns.id, id));
|
|
return getRun(id);
|
|
});
|
|
const policyId = randomUUID();
|
|
await db.insert(budgetPolicies).values({ id: policyId, companyId: company.id, scopeType: "agent",
|
|
scopeId: agent.id, windowKind: "lifetime", amount: 1, enforcementVersion: 1 });
|
|
await db.insert(costEvents).values({ companyId: company.id, agentId: agent.id, provider: "test",
|
|
model: "overrun", inputTokens: 1, outputTokens: 1, costCents: 2, occurredAt: new Date() });
|
|
const scope = { companyId: company.id, scopeType: "agent" as const, scopeId: agent.id,
|
|
enforcement: { policyId, version: 1 } };
|
|
const control = createHeartbeatRunControl(db, deps);
|
|
const succeeded = evidence === "local exit" || evidence === "remote receipt";
|
|
const delivery = control.cancelBudgetScopeWork(scope);
|
|
if (succeeded) await delivery;
|
|
else await expect(delivery).rejects.toThrow("Budget cancellation delivery remains pending");
|
|
const persisted = await getRun(run.id);
|
|
expect(persisted).toMatchObject({ status: "succeeded", nativePhase: "committed" });
|
|
expect(persisted?.resultJson?.executionCancellation).toBeUndefined();
|
|
const intent = (persisted?.resultJson?.startupCancellation as { budgetEnforcement?: Record<string, unknown> })?.budgetEnforcement;
|
|
if (succeeded) expect(intent).toMatchObject({ deliveredAt: expect.any(String),
|
|
terminationProof: evidence === "local exit" ? "local_process_exit" : "provider_termination_receipt" });
|
|
else expect(intent?.deliveredAt).toBeUndefined();
|
|
expect(await listPendingBudgetCancellationScopes(db)).toHaveLength(succeeded ? 0 : 1);
|
|
expect(terminateLocalService).not.toHaveBeenCalled();
|
|
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("moves an attempted failed Stop behind another pending scope", async () => {
|
|
const { company, agent } = await fixture();
|
|
const [otherAgent] = await db.insert(agents).values({ companyId: company.id, name: "Later recovery" }).returning();
|
|
await db.insert(heartbeatRuns).values([agent, otherAgent].map((selected, index) => ({
|
|
companyId: company.id, agentId: selected.id, status: "cancelled", runtimeMode: "legacy",
|
|
resultJson: { startupCancellation: { requestedAt: `2020-01-0${index + 1}T00:00:00.000Z`,
|
|
budgetEnforcement: { companyId: company.id, scopeType: "agent", scopeId: selected.id,
|
|
policyId: randomUUID(), version: 1 } } },
|
|
})));
|
|
const [first] = await listPendingBudgetCancellationScopes(db, 1);
|
|
expect(first.scopeId).toBe(agent.id);
|
|
await expect(createHeartbeatRunControl(db, callbacks(db)).cancelBudgetScopeWork(first))
|
|
.rejects.toThrow("Budget cancellation delivery remains pending");
|
|
const [next] = await listPendingBudgetCancellationScopes(db, 1);
|
|
expect(next.scopeId).toBe(otherAgent.id);
|
|
const [attempted] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agent.id));
|
|
expect((attempted.resultJson?.startupCancellation as { budgetEnforcement?: { lastAttemptAt?: string } })
|
|
?.budgetEnforcement?.lastAttemptAt).toEqual(expect.any(String));
|
|
});
|
|
|
|
it("limits project budget cancellation by company, effective project, and admission cutoff", async () => {
|
|
const { company, agent } = await fixture();
|
|
const [otherCompany] = await db.insert(companies).values({ name: "Other company", issuePrefix: "OC" }).returning();
|
|
const [otherAgent] = await db.insert(agents).values({ companyId: otherCompany.id, name: "Other company agent" }).returning();
|
|
const projectRows = await db.insert(projects).values([
|
|
{ companyId: company.id, name: "Target" }, { companyId: company.id, name: "Override" },
|
|
]).returning();
|
|
const [issue] = await db.insert(issues).values({ companyId: company.id, projectId: projectRows[0].id, title: "Bound task", issueNumber: 1, identifier: "RC-1" }).returning();
|
|
const old = new Date("2026-01-01T00:00:00Z");
|
|
const cutoff = new Date("2026-01-02T00:00:00Z");
|
|
const wakes = await db.insert(agentWakeupRequests).values([
|
|
{ companyId: company.id, agentId: agent.id, source: "assignment", status: "queued", createdAt: old, payload: { issueId: issue.id } },
|
|
{ companyId: company.id, agentId: agent.id, source: "assignment", status: "queued", createdAt: old, payload: { issueId: issue.id, projectId: projectRows[1].id } },
|
|
{ companyId: company.id, agentId: agent.id, source: "assignment", status: "queued", createdAt: cutoff, payload: { projectId: projectRows[0].id } },
|
|
{ companyId: otherCompany.id, agentId: otherAgent.id, source: "assignment", status: "queued", createdAt: old, payload: { projectId: projectRows[0].id } },
|
|
]).returning();
|
|
await createHeartbeatRunControl(db, callbacks(db)).cancelBudgetScopeWork({ companyId: company.id, scopeType: "project", scopeId: projectRows[0].id, createdBefore: cutoff });
|
|
const persisted = await db.select().from(agentWakeupRequests);
|
|
expect(wakes.map(wake => persisted.find(row => row.id === wake.id)?.status)).toEqual(["cancelled", "queued", "queued", "queued"]);
|
|
});
|
|
|
|
it("fences detached browsers after their parent run finishes using admitted project scope and cutoff", async () => {
|
|
const { company, agent } = await fixture();
|
|
const [project, otherProject] = await db.insert(projects).values([
|
|
{ companyId: company.id, name: "Stopped project" }, { companyId: company.id, name: "Other project" },
|
|
]).returning();
|
|
const [movedIssue, targetIssue] = await db.insert(issues).values([
|
|
{ companyId: company.id, projectId: otherProject.id, title: "Moved after browser admission" },
|
|
{ companyId: company.id, projectId: project.id, title: "Current project" },
|
|
]).returning();
|
|
const [parent] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
|
|
status: "succeeded", finishedAt: new Date(), runtimeMode: "legacy" }).returning();
|
|
const [application] = await db.insert(toolApplications).values({ companyId: company.id, name: "Browser fixture", type: "rest_api" }).returning();
|
|
const [connection] = await db.insert(toolConnections).values({ companyId: company.id, applicationId: application.id,
|
|
name: "Browser fixture", uid: "budget-browser-fixture", transport: "rest_api" }).returning();
|
|
const [grant] = await db.insert(connectionGrants).values({ companyId: company.id, connectionId: connection.id, kind: "organization" }).returning();
|
|
const cutoff = new Date();
|
|
const old = new Date(cutoff.getTime() - 1_000);
|
|
const later = new Date(cutoff.getTime() + 60_000);
|
|
const [admitted, other, legacy, newer, noProject] = await db.insert(browserUseSessions).values([
|
|
{ issueId: movedIssue.id, stopRequested: "cancel" }, { issueId: targetIssue.id },
|
|
{ issueId: targetIssue.id }, { issueId: targetIssue.id, createdAt: later }, { issueId: targetIssue.id },
|
|
].map(values => ({ companyId: company.id, agentId: agent.id, connectionId: connection.id, grantId: grant.id,
|
|
providerSessionId: randomUUID(), status: "running", createdAt: old, nextPollAt: later,
|
|
leaseToken: randomUUID(), leaseUntil: later, error: "retained cleanup context", ...values }))).returning();
|
|
await db.insert(browserUseRuns).values([admitted, other, legacy, noProject].map((session, index) => {
|
|
const invocationId = randomUUID();
|
|
return { companyId: company.id, sessionId: session.id, heartbeatRunId: parent.id, invocationId,
|
|
status: "running", detachedUntil: later, createdAt: old,
|
|
result: index === 2 ? null : { paperclipBudgetHold: { invocationId, amountCents: "10", state: "held",
|
|
projectId: index === 0 ? project.id : index === 1 ? otherProject.id : null } },
|
|
};
|
|
}));
|
|
const wake = vi.fn();
|
|
const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.browser, wake);
|
|
const deps = callbacks(db);
|
|
try {
|
|
await createHeartbeatRunControl(db, deps).cancelBudgetScopeWork({ companyId: company.id,
|
|
scopeType: "project", scopeId: project.id, createdBefore: cutoff });
|
|
const persisted = await db.select().from(browserUseSessions).where(eq(browserUseSessions.companyId, company.id));
|
|
for (const original of [admitted, legacy]) {
|
|
const stopped = persisted.find(session => session.id === original.id)!;
|
|
expect(stopped).toMatchObject({ status: "stopping", stopRequested: "end", providerSessionId: original.providerSessionId,
|
|
leaseToken: original.leaseToken, leaseUntil: original.leaseUntil, error: original.error });
|
|
expect(stopped.nextPollAt.getTime()).toBeLessThan(later.getTime());
|
|
}
|
|
for (const original of [other, newer, noProject]) expect(persisted.find(session => session.id === original.id)).toEqual(original);
|
|
expect(await deps.getRun(parent.id)).toMatchObject({ status: "succeeded", resultJson: null });
|
|
expect(wake).toHaveBeenCalledTimes(1);
|
|
expect(terminateLocalService).not.toHaveBeenCalled();
|
|
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
|
|
} finally { unsubscribe(); }
|
|
});
|
|
|
|
it("reads pending Stops through the partial index and ignores delivered or malformed intents", async () => {
|
|
const { company, agent } = await fixture();
|
|
const intent = { companyId: company.id, scopeType: "agent" as const, scopeId: agent.id, policyId: randomUUID(), version: 1 };
|
|
const [pending] = await db.insert(heartbeatRuns).values([
|
|
{ resultJson: { startupCancellation: { budgetEnforcement: intent } } },
|
|
{ resultJson: { startupCancellation: { budgetEnforcement: { ...intent, deliveredAt: new Date().toISOString() } } } },
|
|
{ resultJson: { startupCancellation: { budgetEnforcement: "legacy" } } },
|
|
{ resultJson: { startupCancellation: { requestedAt: new Date().toISOString() } } },
|
|
{ resultJson: null },
|
|
].map(row => ({ ...row, companyId: company.id, agentId: agent.id, status: "cancelled", runtimeMode: "legacy" as const }))).returning();
|
|
expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(pendingBudgetRunCancellationPredicate()))
|
|
.toEqual([{ id: pending.id }]);
|
|
expect(await listPendingBudgetCancellationScopes(db)).toEqual([{ companyId: company.id, scopeType: "agent", scopeId: agent.id,
|
|
enforcement: { policyId: intent.policyId, version: 1 } }]);
|
|
// The schema index predicate must stay identical to the server predicate.
|
|
// Otherwise every worker pass falls back to reading all run result JSON.
|
|
// EXPLAIN the statements the callers send. This table is tiny, so a
|
|
// sequential scan is cheaper here; with it disabled, the planner can use
|
|
// the partial index only when each statement implies the index predicate.
|
|
const plans = await db.transaction(async tx => {
|
|
await tx.execute(sql`set local enable_seqscan = off`);
|
|
const explained: string[] = [];
|
|
for (const query of [
|
|
pendingBudgetCancellationScopesQuery(db, 1).getSQL(),
|
|
pendingBudgetCancellationScopesQuery(db).getSQL(),
|
|
pendingBudgetRunIdsForScopeQuery(db, intent).getSQL(),
|
|
sql`select exists (${pendingBudgetRunCancellationExistsQuery()})`,
|
|
]) explained.push(JSON.stringify(await tx.execute(sql`explain (format json) ${query}`)));
|
|
return explained;
|
|
});
|
|
expect(plans).toHaveLength(4);
|
|
for (const plan of plans) expect(plan).toContain("heartbeat_runs_pending_budget_cancellation_idx");
|
|
});
|
|
});
|