mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 20:29:18 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The recovery system starts automatic runs: continuations, retries, and sweep replacements > - #21443 added a per-company automatic-run brake. It stops recovery from creating new automatic runs while a company's runs keep failing before they start > - The brake did not hold automatic runs that were already queued. The dispatcher still started them, so a burst of queued recovery runs kept failing in the same way > - This pull request adds the brake's start gate to the queued-run dispatch pass, and wakes a held company at its next try without a polling loop > - The benefit is that a failing company stops spending runs on automatic work, people's work still starts at once, and held work starts again on its own when the brake releases ## Linked Issues or Issue Description Refs #21443 Refs #21433 Refs #21427 Refs #21505 **What happened?** While the automatic-run brake was open for a company, its already-queued automatic runs (continuations and retries) still started. Each one failed before it started, for the same cause that opened the brake. **Expected behavior** A queued automatic run must stay queued while the brake is open. Only the brake's single probe, and then the half-open release rate (2, 4, 8 per window), can start one. A person's work (a Retry, or a continuation of a person's request) must never be held. Held runs must start again on their own, with nobody acting. **Steps to reproduce** 1. Make 20 automatic runs in one company fail before they start, so the brake opens. 2. Queue automatic continuations for that company. 3. Run a dispatch pass. Before this change, the continuations start. ## What Changed - `server/src/services/heartbeat/queue.ts`: the dispatch pass asks `reserveAutomaticRunStart` after the tried/backoff skip and the claim-attempt budget check, and before the budget increment and the full row read. A held run spends no claim attempt, does no row read, and does not enter the declined-claim backoff. If the brake grants the start (its probe or a half-open release start) but the claim then returns null, throws, or finds the row gone, the pass gives the start back, so another automatic run can use it in the same pass. The dispatcher's person-origin answer is passed as is, and it is `undefined` only when the dispatcher did not check origin (an agent with one queued run). Then the brake's own cached person check decides. - `server/src/services/heartbeat/queued-run-line.ts`: `queuedRunDispatchFacts` also returns `personOriginChecked`, so the queue knows when the dispatcher's negative answer is real. - `server/src/services/heartbeat/dispatch-work.ts`: the queued-run dispatch deadline worker adds companies whose next try is due to its scopes, and includes the next try time in its next wake. A held company wakes for its next try without a polling loop. If the brake's bounded wake list had to drop a company, the worker wakes all companies at the earliest dropped time. - `server/src/services/recovery/automatic-run-brake.ts`: when the brake holds a run, it records the company's next try time (probe time or the next release window) in a bounded map. A probe or half-open release grant returns a release that gives the start back with a compare-and-set on its own run or window. It exports `reserveAutomaticRunStart`, `takeDueAutomaticRunWakes`, and `nextAutomaticRunWakeAt`. The brake now uses `isRecoveryRun`, `isPersonOwnedFailureCode`, and `isRestartClassFailureCode` from `continuation-cause.ts` directly. The rules are the same as before; the lazy loader keeps only the imports that would make an import cycle. - `server/src/__tests__/automatic-run-start-gate.test.ts` (new, 11 tests through the real heartbeat queue): bounded queries for 250 held runs over 100 passes, and for 1,500 held runs (past the 1,024-run cache) over 3 passes; a declined probe claim gives the probe back and another automatic run starts in the same pass, and the probe stays with that run on a later pass; an unused half-open start goes back to its window; with 276 held companies (past the 256-company wake list) every company is still woken; a work run behind 250 held runs starts in the same pass; a person's Retry is never held; the only queued run is not held when it continues a person's Retry; exactly one probe starts across two server instances; the release gives 2, 6, 14 cumulative starts over three windows; held runs are not in the declined-claim backoff after the pause; the dispatch worker wakes the company at the next try; and an end-to-end test that a held automatic run stays queued while the brake is open, then starts and succeeds on its own at the next try. - Two small test updates in `automatic-run-brake.test.ts` and `automatic-run-brake-import.test.ts` for the new cache size field and the moved rule. - `server/src/__tests__/automatic-run-brake-recovery-passes.test.ts`: one test for the sweep path from #21505 (continue a blocked task behind a run refused at dispatch). The sweep only records the continuation (no wake, no new run), and `deliverReconciledExecutions`, which the brake already gates, creates the run only after the pause. No second gate is needed on that path. ## Verification - `npx vitest run src/__tests__/automatic-run-start-gate.test.ts`: 11 passed. - Red before for the review fixes: on the first PR head, the 1,500-run cost test made 3,848 queries (bound 130), the declined-probe test started 0 runs (expected 1), and the 276-company wake test missed 20 companies. - Red before: with the `queue.ts` gate line removed, 5 of the 8 tests fail (bounded pass, single probe, release rate, backoff, end-to-end). - Red before for the person case: with the dispatcher's `false` passed through as final, "never holds the only queued run when it continues a person's Retry" fails (0 started). It passes after the change. - `automatic-run-brake.test.ts` 19 passed, `automatic-run-brake-import.test.ts` 1 passed, `automatic-run-brake-recovery-passes.test.ts` 28 passed, `failed-continuation-continues-again.test.ts` 32 passed (after merging master with #21505). - Nearby queue tests (on the final head): `heartbeat-run-dispatch-work` 15, `queued-run-dispatch-stacking` 6, `heartbeat-queued-run-claim-isolation` 10, `assigned-work-stalled-queued-run` 12, all passed. - `server` `tsc --noEmit` is clean. ## Risks - Behavior change only while a company's brake is open or half-open. When the brake is closed, the gate returns from its cache with no query. - The first time the gate sees a non-person automatic run while the brake is on, the brake does one person-origin query for that run. The result is cached per run (bounded to 1,024 runs). - No migration. No API or UI change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
443 lines
30 KiB
TypeScript
443 lines
30 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { eq, inArray, sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { agentWakeupRequests, agents, automaticRunBrakes, companies, createDb, heartbeatRuns } from "@paperclipai/db";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
import {
|
|
AUTOMATIC_RUN_BRAKE, automaticRunOutcomeRules, automaticWorkStatusText, classifyAutomaticRunOutcome, createAutomaticRunBrake,
|
|
isAutomaticRun,
|
|
} from "../services/recovery/automatic-run-brake.js";
|
|
import { taskAutomaticAttempts } from "../services/recovery/task-automatic-attempts.js";
|
|
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
const suite = support.supported ? describe : describe.skip;
|
|
const MINUTE = 60_000;
|
|
|
|
type Run = typeof heartbeatRuns.$inferSelect;
|
|
|
|
suite("automatic-run brake", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
|
|
beforeAll(async () => {
|
|
temporary = await startEmbeddedPostgresTestDatabase("paperclip-automatic-run-brake-");
|
|
db = createDb(temporary.connectionString);
|
|
}, 60_000);
|
|
|
|
afterEach(async () => {
|
|
await db.execute(sql`truncate table automatic_run_brakes, agent_wakeup_requests, heartbeat_runs, agents, companies cascade`);
|
|
}, 120_000);
|
|
|
|
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }, 120_000);
|
|
|
|
async function seedCompany() {
|
|
const companyId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Brake", issuePrefix: `B${companyId.replaceAll("-", "").slice(0, 6)}`,
|
|
requireBoardApprovalForNewAgents: false });
|
|
const agentIds: string[] = [];
|
|
for (let index = 0; index < 2; index += 1) {
|
|
const agentId = randomUUID();
|
|
await db.insert(agents).values({ id: agentId, companyId, name: `Worker ${index}`, role: "engineer", status: "idle",
|
|
adapterType: "codex_local", adapterConfig: {}, runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } }, permissions: {} });
|
|
agentIds.push(agentId);
|
|
}
|
|
return { companyId, agentIds };
|
|
}
|
|
|
|
async function insertRuns(companyId: string, agentId: string, count: number, values: (index: number) => Partial<typeof heartbeatRuns.$inferInsert>) {
|
|
if (count === 0) return [] as Run[];
|
|
return db.insert(heartbeatRuns).values(Array.from({ length: count }, (_, index) => ({ companyId, agentId,
|
|
invocationSource: "automation", triggerDetail: "system", runtimeMode: "legacy",
|
|
contextSnapshot: { wakeReason: "transient_failure_retry", issueId: randomUUID() }, ...values(index) }))).returning();
|
|
}
|
|
|
|
const ago = (ms: number) => new Date(Date.now() - ms);
|
|
|
|
function failedBeforeStarting(companyId: string, agentId: string, count: number, errorCode = "environment_provider_unavailable") {
|
|
return insertRuns(companyId, agentId, count, index => ({ status: "failed", errorCode, executionStage: "preparing",
|
|
createdAt: ago(4 * MINUTE + index), startedAt: ago(3 * MINUTE + index), finishedAt: ago(MINUTE - index * 100),
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }));
|
|
}
|
|
|
|
function interruptedWhileWorking(companyId: string, agentId: string, count: number) {
|
|
const codes = ["controller_lease_lost", "server_shutdown_interrupted", "process_lost", "native_session_interrupted"];
|
|
return insertRuns(companyId, agentId, count, index => ({ status: index % 2 ? "failed" : "interrupted", errorCode: codes[index % codes.length],
|
|
executionStage: "dispatching", lastOutputAt: ago(40_000), createdAt: ago(20 * MINUTE), startedAt: ago(19 * MINUTE),
|
|
finishedAt: ago(30_000 - index * 500), contextSnapshot: { wakeReason: "issue_assigned", issueId: randomUUID() } }));
|
|
}
|
|
|
|
function healthy(companyId: string, agentId: string, count: number, status: "succeeded" | "failed" = "succeeded") {
|
|
return insertRuns(companyId, agentId, count, index => ({ status, errorCode: status === "failed" ? "adapter_failed" : null,
|
|
executionStage: "dispatching", lastOutputAt: ago(10_000), createdAt: ago(5 * MINUTE), startedAt: ago(4 * MINUTE),
|
|
finishedAt: ago(5_000 - index * 10) }));
|
|
}
|
|
|
|
function queuedAutomatic(companyId: string, agentId: string, count: number) {
|
|
return insertRuns(companyId, agentId, count, index => ({ status: "queued", createdAt: ago(30_000 - index),
|
|
contextSnapshot: { wakeReason: "issue_continuation_needed", previousRunId: randomUUID(), issueId: randomUUID() } }));
|
|
}
|
|
|
|
async function forgetRuns(ids: string[]) {
|
|
await db.update(heartbeatRuns).set({ createdAt: sql`created_at - interval '1 hour'`, startedAt: sql`started_at - interval '1 hour'`,
|
|
finishedAt: sql`finished_at - interval '1 hour'` }).where(inArray(heartbeatRuns.id, ids));
|
|
}
|
|
|
|
async function ageBrake(companyId: string, ms: number) {
|
|
const shift = `${ms} milliseconds`;
|
|
await db.update(automaticRunBrakes).set({
|
|
stateChangedAt: sql`state_changed_at - ${shift}::interval`, openedAt: sql`opened_at - ${shift}::interval`,
|
|
nextProbeAt: sql`next_probe_at - ${shift}::interval`, nextCreationAt: sql`next_creation_at - ${shift}::interval`,
|
|
releaseWindowStartedAt: sql`release_window_started_at - ${shift}::interval`,
|
|
}).where(eq(automaticRunBrakes.companyId, companyId));
|
|
}
|
|
|
|
const brakeRow = async (companyId: string) => (await db.select().from(automaticRunBrakes).where(eq(automaticRunBrakes.companyId, companyId)))[0];
|
|
|
|
async function openBrake(companyId: string, agentId: string, brake = createAutomaticRunBrake()) {
|
|
const failures = await failedBeforeStarting(companyId, agentId, 20);
|
|
expect((await brake.refresh(db, companyId)).state).toBe("open");
|
|
return failures;
|
|
}
|
|
|
|
async function heldCount(brake: ReturnType<typeof createAutomaticRunBrake>, runs: Run[], options: { personOrigin?: boolean } = {}) {
|
|
let started = 0;
|
|
for (const run of runs) if (!await brake.holdAutomaticRunStart(db, run, options)) started += 1;
|
|
return started;
|
|
}
|
|
|
|
function countingDb() {
|
|
let queries = 0;
|
|
const counted = new Set(["select", "insert", "update", "delete", "execute"]);
|
|
const proxy = new Proxy(db, {
|
|
get(target, property, receiver) {
|
|
const value = Reflect.get(target, property, receiver);
|
|
if (typeof value !== "function") return value;
|
|
return counted.has(String(property)) ? (...args: unknown[]) => { queries += 1; return value.apply(target, args); } : value.bind(target);
|
|
},
|
|
});
|
|
return { db: proxy as typeof db, queries: () => queries };
|
|
}
|
|
|
|
it("classifies automatic runs and outcomes with the shared person and restart rules", async () => {
|
|
const rules = await automaticRunOutcomeRules();
|
|
expect(isAutomaticRun({ id: "a", companyId: "c", contextSnapshot: { wakeReason: "issue_continuation_needed" } })).toBe(true);
|
|
expect(isAutomaticRun({ id: "a", companyId: "c", scheduledRetryReason: "transient_failure" })).toBe(true);
|
|
expect(isAutomaticRun({ id: "a", companyId: "c", contextSnapshot: { wakeReason: "task_retry", previousRunId: "x" } })).toBe(false);
|
|
expect(isAutomaticRun({ id: "a", companyId: "c", contextSnapshot: { wakeReason: "issue_assigned" } })).toBe(false);
|
|
const base = { status: "failed", runtimeMode: "legacy", startedAt: new Date(), hasOutput: false, providerWorkStarted: null };
|
|
expect(classifyAutomaticRunOutcome({ ...base, errorCode: "environment_provider_unavailable", executionStage: "preparing" }, rules)).toBe("failed_before_starting");
|
|
expect(classifyAutomaticRunOutcome({ ...base, errorCode: "controller_lease_lost", executionStage: "dispatching" }, rules)).toBe("neutral");
|
|
expect(classifyAutomaticRunOutcome({ ...base, errorCode: "process_lost", executionStage: "preparing" }, rules)).toBe("neutral");
|
|
expect(classifyAutomaticRunOutcome({ ...base, errorCode: "budget_hard_stop", executionStage: "preparing" }, rules)).toBe("healthy");
|
|
expect(classifyAutomaticRunOutcome({ ...base, errorCode: "adapter_failed", executionStage: "dispatching" }, rules)).toBe("healthy");
|
|
expect(classifyAutomaticRunOutcome({ ...base, errorCode: "adapter_failed", executionStage: null, providerWorkStarted: "false" }, rules)).toBe("failed_before_starting");
|
|
expect(classifyAutomaticRunOutcome({ ...base, status: "running", errorCode: null, executionStage: "preparing" }, rules)).toBe("neutral");
|
|
expect(classifyAutomaticRunOutcome({ ...base, status: "cancelled", errorCode: "environment_provider_unavailable", executionStage: "preparing" }, rules)).toBe("neutral");
|
|
const preDispatch = { version: 1, providerWorkStarted: false };
|
|
for (const runtimeMode of ["legacy", "native"]) {
|
|
const shape = { ...base, runtimeMode, executionStage: runtimeMode === "legacy" ? "preparing" : null,
|
|
providerWorkStarted: runtimeMode === "legacy" ? "false" : null, nativePreDispatchFailure: runtimeMode === "native" ? preDispatch : undefined };
|
|
for (const errorCode of ["environment_provider_unavailable", "setup_failed", "adapter_failed"]) {
|
|
expect(classifyAutomaticRunOutcome({ ...shape, errorCode }, rules)).toBe("failed_before_starting");
|
|
}
|
|
for (const errorCode of ["controller_lease_lost", "process_lost", "native_session_interrupted", "server_shutdown_interrupted"]) {
|
|
expect(classifyAutomaticRunOutcome({ ...shape, errorCode }, rules)).toBe("neutral");
|
|
}
|
|
const working = { ...base, runtimeMode, executionStage: runtimeMode === "legacy" ? "dispatching" : null, hasOutput: true };
|
|
expect(classifyAutomaticRunOutcome({ ...working, errorCode: "adapter_failed" }, rules)).toBe("healthy");
|
|
expect(classifyAutomaticRunOutcome({ ...working, errorCode: "environment_provider_unavailable" }, rules)).toBe("failed_before_starting");
|
|
}
|
|
});
|
|
|
|
it("opens on native runs that failed before dispatch", async () => {
|
|
const { companyId, agentIds: [agentId] } = await seedCompany();
|
|
await insertRuns(companyId, agentId!, 20, index => ({ runtimeMode: "native", status: "failed",
|
|
errorCode: index % 2 ? "adapter_failed" : "setup_failed", createdAt: ago(4 * MINUTE), startedAt: ago(3 * MINUTE),
|
|
finishedAt: ago(MINUTE - index * 100), nativePhase: "terminal_failure",
|
|
resultJson: { nativePreDispatchFailure: { version: 1, providerWorkStarted: false }, failureCause: index % 2 ? "40P01" : "adapter_reported_error" } }));
|
|
expect((await createAutomaticRunBrake().refresh(db, companyId))).toMatchObject({ state: "open",
|
|
reason: { recentAutomaticRuns: 20, failedBeforeStarting: 20, trigger: "failed_before_starting" } });
|
|
}, 60_000);
|
|
|
|
it("sees failures that finished in the window behind more than 500 newer runs", async () => {
|
|
const { companyId, agentIds: [agentId, otherAgent] } = await seedCompany();
|
|
const failures = await failedBeforeStarting(companyId, agentId!, 20, "setup_failed");
|
|
await db.update(heartbeatRuns).set({ createdAt: sql`now() - interval '90 minutes'`, startedAt: sql`now() - interval '3 minutes'` })
|
|
.where(inArray(heartbeatRuns.id, failures.map(run => run.id)));
|
|
await queuedAutomatic(companyId, otherAgent!, 600);
|
|
await insertRuns(companyId, otherAgent!, 50, () => ({ status: "succeeded", createdAt: ago(MINUTE), contextSnapshot: { wakeReason: "issue_assigned" } }));
|
|
expect((await createAutomaticRunBrake().refresh(db, companyId)).state).toBe("open");
|
|
}, 120_000);
|
|
|
|
it("opens when automatic runs fail before starting, holds all but one probe, and never holds a person's work", async () => {
|
|
const { companyId, agentIds: [agentId, otherAgent] } = await seedCompany();
|
|
const brake = createAutomaticRunBrake();
|
|
await failedBeforeStarting(companyId, agentId!, 20);
|
|
const snapshot = await brake.refresh(db, companyId);
|
|
expect(snapshot).toMatchObject({ state: "open", reason: { recentAutomaticRuns: 20, failedBeforeStarting: 20 } });
|
|
expect(automaticWorkStatusText(snapshot)).toMatch(/^Automatic work is paused for this company since .* because the last 5 automatic runs could not get a sandbox/);
|
|
expect(automaticWorkStatusText(snapshot)).not.toMatch(/brake|probe|_/);
|
|
|
|
const queued = [...await queuedAutomatic(companyId, agentId!, 25), ...await queuedAutomatic(companyId, otherAgent!, 25)];
|
|
const attemptsBefore = await Promise.all(queued.map(run => taskAutomaticAttempts(db, companyId, String(run.contextSnapshot!.issueId), new Date())));
|
|
expect(await heldCount(brake, queued, { personOrigin: false })).toBe(1);
|
|
expect(await heldCount(createAutomaticRunBrake(), queued, { personOrigin: false })).toBe(0);
|
|
const after = await db.select().from(heartbeatRuns).where(inArray(heartbeatRuns.id, queued.map(run => run.id)));
|
|
expect(after.every(run => run.status === "queued" && run.errorCode === null)).toBe(true);
|
|
const attemptsAfter = await Promise.all(queued.map(run => taskAutomaticAttempts(db, companyId, String(run.contextSnapshot!.issueId), new Date())));
|
|
expect(attemptsAfter.map(attempts => attempts.all.length)).toEqual(attemptsBefore.map(attempts => attempts.all.length));
|
|
|
|
const [retry] = await insertRuns(companyId, agentId!, 1, () => ({ status: "queued", contextSnapshot: { wakeReason: "task_retry", previousRunId: queued[0]!.id } }));
|
|
const [assignment] = await insertRuns(companyId, agentId!, 1, () => ({ status: "queued", contextSnapshot: { wakeReason: "issue_assigned" } }));
|
|
const [joinedByPerson] = await insertRuns(companyId, agentId!, 1, () => ({ status: "queued",
|
|
contextSnapshot: { wakeReason: "issue_continuation_needed", previousRunId: randomUUID() } }));
|
|
await db.insert(agentWakeupRequests).values({ companyId, agentId: agentId!, source: "on_demand", triggerDetail: "manual",
|
|
reason: "issue_commented", status: "queued", requestedByActorType: "user", requestedByActorId: "board", runId: joinedByPerson!.id });
|
|
const [continuationOfPersonRetry] = await insertRuns(companyId, agentId!, 1, () => ({ status: "queued",
|
|
contextSnapshot: { wakeReason: "transient_failure_retry", previousRunId: retry!.id } }));
|
|
for (const run of [retry!, assignment!, joinedByPerson!, continuationOfPersonRetry!]) {
|
|
expect(await brake.holdAutomaticRunStart(db, run)).toBe(false);
|
|
}
|
|
expect(await brake.holdAutomaticRunStart(db, queued[3]!, { personOrigin: true })).toBe(false);
|
|
}, 60_000);
|
|
|
|
it("stays closed for mixed outcomes, person-owned failures and task failures after the agent started", async () => {
|
|
const { companyId, agentIds: [agentId] } = await seedCompany();
|
|
const brake = createAutomaticRunBrake();
|
|
expect((await brake.refresh(db, companyId)).state).toBe("closed");
|
|
expect(await brakeRow(companyId)).toBeUndefined();
|
|
await failedBeforeStarting(companyId, agentId!, 8, "setup_failed");
|
|
await healthy(companyId, agentId!, 4, "failed");
|
|
expect((await brake.refresh(db, companyId)).state).toBe("closed");
|
|
await insertRuns(companyId, agentId!, 30, index => ({ status: "failed", errorCode: index % 2 ? "budget_hard_stop" : "claude_auth_required",
|
|
executionStage: "preparing", createdAt: ago(3 * MINUTE), startedAt: ago(2 * MINUTE), finishedAt: ago(MINUTE),
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }));
|
|
await healthy(companyId, agentId!, 30, "failed");
|
|
expect((await brake.refresh(db, companyId)).state).toBe("closed");
|
|
expect(await brakeRow(companyId)).toBeUndefined();
|
|
}, 60_000);
|
|
|
|
it("ignores failures older than the window", async () => {
|
|
const { companyId, agentIds: [agentId] } = await seedCompany();
|
|
const failures = await failedBeforeStarting(companyId, agentId!, 40);
|
|
await db.update(heartbeatRuns).set({ finishedAt: sql`now() - interval '11 minutes'` }).where(inArray(heartbeatRuns.id, failures.map(run => run.id)));
|
|
expect((await createAutomaticRunBrake().refresh(db, companyId)).state).toBe("closed");
|
|
}, 60_000);
|
|
|
|
it("stays closed through a release swap that interrupts working runs, and opens when their continuations cannot get a sandbox", async () => {
|
|
const { companyId, agentIds: [agentId, otherAgent] } = await seedCompany();
|
|
const brake = createAutomaticRunBrake();
|
|
await interruptedWhileWorking(companyId, agentId!, 40);
|
|
expect((await brake.refresh(db, companyId)).state).toBe("closed");
|
|
const continuations = [...await queuedAutomatic(companyId, agentId!, 20), ...await queuedAutomatic(companyId, otherAgent!, 20)];
|
|
expect(await heldCount(createAutomaticRunBrake(), continuations)).toBe(40);
|
|
expect((await brake.reserveCreation(db, companyId, { issueId: randomUUID(), sourceRunId: randomUUID() })).heldUntil).toBeNull();
|
|
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "environment_provider_unavailable", executionStage: "preparing",
|
|
startedAt: sql`now() - interval '20 seconds'`, finishedAt: sql`now() - interval '5 seconds'`,
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } })
|
|
.where(inArray(heartbeatRuns.id, continuations.map(run => run.id)));
|
|
expect((await brake.refresh(db, companyId)).state).toBe("open");
|
|
}, 60_000);
|
|
|
|
it("opens on five sandbox refusals in a row, below the run count", async () => {
|
|
const { companyId, agentIds: [agentId] } = await seedCompany();
|
|
const brake = createAutomaticRunBrake();
|
|
await failedBeforeStarting(companyId, agentId!, 4);
|
|
expect((await brake.refresh(db, companyId)).state).toBe("closed");
|
|
await failedBeforeStarting(companyId, agentId!, 1);
|
|
expect((await brake.refresh(db, companyId)).state).toBe("open");
|
|
}, 60_000);
|
|
|
|
it("waits out a failed probe, then releases at a doubling rate after two probes reach the agent, and closes", async () => {
|
|
const { companyId, agentIds: [agentId, otherAgent] } = await seedCompany();
|
|
const failures = await openBrake(companyId, agentId!);
|
|
await forgetRuns(failures.map(run => run.id));
|
|
const queued = [...await queuedAutomatic(companyId, agentId!, 30), ...await queuedAutomatic(companyId, otherAgent!, 30)];
|
|
const probe = async () => {
|
|
const brake = createAutomaticRunBrake();
|
|
for (const run of queued) if (!await brake.holdAutomaticRunStart(db, run, { personOrigin: false })) return run;
|
|
return null;
|
|
};
|
|
const first = await probe();
|
|
expect(first).not.toBeNull();
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "environment_provider_unavailable", executionStage: "preparing",
|
|
startedAt: sql`now()`, finishedAt: sql`now()`, resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } })
|
|
.where(eq(heartbeatRuns.id, first!.id));
|
|
queued.splice(queued.indexOf(first!), 1);
|
|
expect(await probe()).toBeNull();
|
|
expect((await brakeRow(companyId))).toMatchObject({ state: "open", probeSuccesses: 0, probeRunId: null });
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.probeIntervalMs);
|
|
|
|
for (let success = 0; success < AUTOMATIC_RUN_BRAKE.probeSuccessesToRelease; success += 1) {
|
|
const run = await probe();
|
|
expect(run).not.toBeNull();
|
|
await db.update(heartbeatRuns).set({ status: "running", startedAt: sql`now()`, executionStage: "dispatching" })
|
|
.where(eq(heartbeatRuns.id, run!.id));
|
|
queued.splice(queued.indexOf(run!), 1);
|
|
await createAutomaticRunBrake().refresh(db, companyId);
|
|
}
|
|
expect(await brakeRow(companyId)).toMatchObject({ state: "open", probeSuccesses: 2 });
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.minOpenMs);
|
|
expect((await createAutomaticRunBrake().refresh(db, companyId)).state).toBe("half_open");
|
|
|
|
const startedPerWindow: number[] = [];
|
|
for (let step = 0; step < AUTOMATIC_RUN_BRAKE.releaseSteps; step += 1) {
|
|
const startedNow = await heldCount(createAutomaticRunBrake(), queued, { personOrigin: false })
|
|
+ await heldCount(createAutomaticRunBrake(), queued, { personOrigin: false });
|
|
startedPerWindow.push(startedNow);
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.releaseIntervalMs);
|
|
}
|
|
expect(startedPerWindow).toEqual([2, 4, 8, 16, 32, 64]);
|
|
expect((await createAutomaticRunBrake().refresh(db, companyId)).state).toBe("closed");
|
|
expect(await heldCount(createAutomaticRunBrake(), queued, { personOrigin: false })).toBe(queued.length);
|
|
}, 120_000);
|
|
|
|
it("stays open when runs that started fine are followed by newer failures before starting", async () => {
|
|
const { companyId, agentIds: [agentId] } = await seedCompany();
|
|
const failures = await openBrake(companyId, agentId!);
|
|
await forgetRuns(failures.map(run => run.id));
|
|
await insertRuns(companyId, agentId!, 2, () => ({ status: "running", executionStage: "dispatching", lastOutputAt: ago(80_000),
|
|
startedAt: ago(90_000), createdAt: ago(100_000) }));
|
|
await failedBeforeStarting(companyId, agentId!, 10);
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.minOpenMs);
|
|
expect((await createAutomaticRunBrake().refresh(db, companyId)).state).toBe("open");
|
|
}, 60_000);
|
|
|
|
it("reopens from a gradual release when runs fail before starting again", async () => {
|
|
const { companyId, agentIds: [agentId] } = await seedCompany();
|
|
const failures = await openBrake(companyId, agentId!);
|
|
await forgetRuns(failures.map(run => run.id));
|
|
const recovered = await healthy(companyId, agentId!, 2);
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.minOpenMs);
|
|
expect((await createAutomaticRunBrake().refresh(db, companyId)).state).toBe("half_open");
|
|
await forgetRuns(recovered.map(run => run.id));
|
|
const again = await failedBeforeStarting(companyId, agentId!, 5);
|
|
await db.update(heartbeatRuns).set({ finishedAt: sql`now()` }).where(inArray(heartbeatRuns.id, again.map(run => run.id)));
|
|
expect((await createAutomaticRunBrake().refresh(db, companyId)).state).toBe("open");
|
|
}, 60_000);
|
|
|
|
it("frees the probe slot when the probe is cancelled, deleted or never starts", async () => {
|
|
const { companyId, agentIds: [agentId, otherAgent] } = await seedCompany();
|
|
await openBrake(companyId, agentId!);
|
|
const queued = [...await queuedAutomatic(companyId, agentId!, 3), ...await queuedAutomatic(companyId, otherAgent!, 3)];
|
|
const brake = createAutomaticRunBrake();
|
|
expect(await brake.holdAutomaticRunStart(db, queued[0]!, { personOrigin: false })).toBe(false);
|
|
expect(await heldCount(createAutomaticRunBrake(), queued.slice(1), { personOrigin: false })).toBe(0);
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.probeIntervalMs);
|
|
expect(await createAutomaticRunBrake().holdAutomaticRunStart(db, queued[3]!, { personOrigin: false })).toBe(false);
|
|
expect((await brakeRow(companyId))?.probeRunId).toBe(queued[3]!.id);
|
|
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: sql`now()` }).where(eq(heartbeatRuns.id, queued[3]!.id));
|
|
await createAutomaticRunBrake().refresh(db, companyId);
|
|
expect((await brakeRow(companyId))?.probeRunId).toBeNull();
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.probeIntervalMs);
|
|
expect(await createAutomaticRunBrake().holdAutomaticRunStart(db, queued[4]!, { personOrigin: false })).toBe(false);
|
|
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, queued[4]!.id));
|
|
await createAutomaticRunBrake().refresh(db, companyId);
|
|
expect((await brakeRow(companyId))?.probeRunId).toBeNull();
|
|
}, 60_000);
|
|
|
|
it("lets another run take the probe when the reserved probe never starts within the claim bound", async () => {
|
|
const { companyId, agentIds: [agentId, otherAgent] } = await seedCompany();
|
|
await openBrake(companyId, agentId!);
|
|
const [first] = await queuedAutomatic(companyId, agentId!, 1);
|
|
const [second] = await queuedAutomatic(companyId, otherAgent!, 1);
|
|
expect(await createAutomaticRunBrake().holdAutomaticRunStart(db, first!, { personOrigin: false })).toBe(false);
|
|
expect(await createAutomaticRunBrake().holdAutomaticRunStart(db, second!, { personOrigin: false })).toBe(true);
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.probeClaimBoundMs + 1_000);
|
|
expect(await createAutomaticRunBrake().holdAutomaticRunStart(db, second!, { personOrigin: false })).toBe(false);
|
|
expect((await brakeRow(companyId))?.probeRunId).toBe(second!.id);
|
|
await db.update(heartbeatRuns).set({ status: "running", startedAt: sql`now()` }).where(eq(heartbeatRuns.id, second!.id));
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.probeClaimBoundMs + 1_000);
|
|
expect(await createAutomaticRunBrake().holdAutomaticRunStart(db, first!, { personOrigin: false })).toBe(true);
|
|
}, 60_000);
|
|
|
|
it("closes by itself when no automatic run started or finished for the quiet bound, even with runs waiting", async () => {
|
|
const { companyId, agentIds: [agentId] } = await seedCompany();
|
|
const failures = await openBrake(companyId, agentId!);
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.quietCloseMs - MINUTE);
|
|
expect((await createAutomaticRunBrake().refresh(db, companyId)).state).toBe("open");
|
|
await forgetRuns(failures.map(run => run.id));
|
|
await queuedAutomatic(companyId, agentId!, 5);
|
|
await ageBrake(companyId, 2 * MINUTE);
|
|
expect((await createAutomaticRunBrake().refresh(db, companyId))).toMatchObject({ state: "closed", reason: { trigger: "no_automatic_work" } });
|
|
}, 60_000);
|
|
|
|
it("lets two servers agree on the state and start one probe between them", async () => {
|
|
const { companyId, agentIds: [agentId, otherAgent] } = await seedCompany();
|
|
await failedBeforeStarting(companyId, agentId!, 20);
|
|
const [left, right] = [createAutomaticRunBrake(), createAutomaticRunBrake()];
|
|
const [leftState, rightState] = await Promise.all([left.refresh(db, companyId), right.refresh(db, companyId)]);
|
|
expect([leftState.state, rightState.state]).toEqual(["open", "open"]);
|
|
expect(leftState.stateChangedAt?.getTime()).toBe(rightState.stateChangedAt?.getTime());
|
|
const queued = [...await queuedAutomatic(companyId, agentId!, 25), ...await queuedAutomatic(companyId, otherAgent!, 25)];
|
|
const results = await Promise.all(queued.map((run, index) => (index % 2 ? left : right).holdAutomaticRunStart(db, run, { personOrigin: false })));
|
|
expect(results.filter(held => !held)).toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
it("allows one new automatic run a minute while open and no automatic run is starting, across servers", async () => {
|
|
const { companyId, agentIds: [agentId] } = await seedCompany();
|
|
await openBrake(companyId, agentId!);
|
|
const work = { issueId: randomUUID(), sourceRunId: randomUUID() };
|
|
const [left, right] = [createAutomaticRunBrake(), createAutomaticRunBrake()];
|
|
const decisions = await Promise.all([left, right, left, right].map(async brake => (await brake.reserveCreation(db, companyId, work)).heldUntil));
|
|
expect(decisions.filter(held => held === null)).toHaveLength(1);
|
|
const later = decisions.find(Boolean)!;
|
|
expect(later.getTime()).toBeGreaterThan(Date.now() + 50_000);
|
|
await queuedAutomatic(companyId, agentId!, 3);
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.creationIntervalMs);
|
|
expect((await createAutomaticRunBrake().reserveCreation(db, companyId, work)).heldUntil).toBeNull();
|
|
await insertRuns(companyId, agentId!, 1, () => ({ status: "running", executionStage: "preparing", startedAt: ago(20_000) }));
|
|
await ageBrake(companyId, AUTOMATIC_RUN_BRAKE.creationIntervalMs);
|
|
expect((await createAutomaticRunBrake().reserveCreation(db, companyId, work)).heldUntil).not.toBeNull();
|
|
}, 60_000);
|
|
|
|
it("keeps a pass over hundreds of held runs at a bounded number of queries", async () => {
|
|
const { companyId, agentIds: [agentId, otherAgent] } = await seedCompany();
|
|
await openBrake(companyId, agentId!);
|
|
const queued = [...await queuedAutomatic(companyId, agentId!, 125), ...await queuedAutomatic(companyId, otherAgent!, 125)];
|
|
const counting = countingDb();
|
|
const brake = createAutomaticRunBrake();
|
|
let started = 0;
|
|
for (let pass = 0; pass < 100; pass += 1) {
|
|
for (const run of queued) {
|
|
if (await brake.holdAutomaticRunStart(counting.db, run, { personOrigin: false })) continue;
|
|
started += 1;
|
|
await db.update(heartbeatRuns).set({ status: "running", startedAt: sql`now()` }).where(eq(heartbeatRuns.id, run.id));
|
|
}
|
|
}
|
|
expect(started).toBe(1);
|
|
expect(counting.queries()).toBeLessThanOrEqual(6);
|
|
|
|
const fallback = countingDb();
|
|
const second = createAutomaticRunBrake();
|
|
for (let pass = 0; pass < 100; pass += 1) for (const run of queued) await second.holdAutomaticRunStart(fallback.db, run);
|
|
expect(fallback.queries()).toBeLessThanOrEqual(queued.length + 5);
|
|
}, 120_000);
|
|
|
|
it("reads as closed and logs once when its run rules cannot load", async () => {
|
|
const { companyId, agentIds: [agentId] } = await seedCompany();
|
|
await failedBeforeStarting(companyId, agentId!, 20);
|
|
const loadRules = vi.fn(async () => { throw new Error("module failed to load"); });
|
|
const brake = createAutomaticRunBrake({ loadRules });
|
|
const queued = await queuedAutomatic(companyId, agentId!, 3);
|
|
expect((await brake.refresh(db, companyId)).state).toBe("closed");
|
|
expect((await brake.reserveCreation(db, companyId, { issueId: randomUUID(), sourceRunId: randomUUID() })).heldUntil).toBeNull();
|
|
expect(await heldCount(brake, queued)).toBe(3);
|
|
expect(await brakeRow(companyId)).toBeUndefined();
|
|
}, 60_000);
|
|
|
|
it("keeps its in-process memory bounded across many companies and runs", async () => {
|
|
const brake = createAutomaticRunBrake();
|
|
const { companyId, agentIds: [agentId] } = await seedCompany();
|
|
for (let index = 0; index < AUTOMATIC_RUN_BRAKE.maxCompanies + 100; index += 1) await brake.read(db, randomUUID());
|
|
await openBrake(companyId, agentId!, brake);
|
|
for (let index = 0; index < AUTOMATIC_RUN_BRAKE.maxRuns + 200; index += 1) {
|
|
await brake.holdAutomaticRunStart(db, { id: randomUUID(), companyId, contextSnapshot: { wakeReason: "issue_continuation_needed" } });
|
|
}
|
|
expect(brake.cacheSizes()).toMatchObject({ companies: AUTOMATIC_RUN_BRAKE.maxCompanies, runs: AUTOMATIC_RUN_BRAKE.maxRuns });
|
|
expect(brake.cacheSizes().wakes).toBeLessThanOrEqual(AUTOMATIC_RUN_BRAKE.maxCompanies);
|
|
}, 180_000);
|
|
});
|