mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
perf(runs): read live runs in a fixed number of statements (#21417)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board shows the runs that are live now. The sidebar and the active agents panel read `GET /api/companies/:companyId/live-runs` > - That route checked read access and built the output-silence summary one run at a time, with 8 statements per run > - On a hosted instance this route ran about 6 times per minute with an average of 2.1 s of server time, and its statements and allocations add load to a busy main thread > - This pull request reads the access decisions and the silence state for all runs of the request in one batch > - The benefit is a fixed number of statements per request (15 instead of 8 + 8 per run) and about one quarter of the allocation ## Linked Issues or Issue Description No public issue exists. This is an enhancement. **What existing behavior does this improve?** The live-runs read on the board (`GET /api/companies/:companyId/live-runs`). **Subsystem affected** Server: agent routes, run read authorization, active-run watchdog. **Current behavior** For each run, the route calls `canActorReadHeartbeatRun` (task existence, instance-admin flag, membership, run policy, task privacy) and `buildRunOutputSilence` (two watchdog decision reads and one evaluation-task read). The instance-admin and membership reads repeat the same answer for every run. Measured locally with a statement and allocation harness: | Live runs | Statements per request | Allocated per request | | --- | --- | --- | | 10 | 88 | 10.0 MB | | 50 | 408 | 27.9 MB | | 200 (route returns 50) | 408 | 28.6 MB | **Proposed behavior** The route reads every access decision in one batch and every silence summary in one batch, so the statement count does not grow with the number of runs. **Reason and benefit** Fewer statements and less allocation on a route that the board calls often. **Breaking changes** None. The response is the same. ## What Changed - `server/src/routes/agents.ts`: the live-runs handler decides reads with the existing `getHeartbeatRunReadDecisions` (the run history list already uses it) and builds silence summaries with one `buildRunOutputSilences` call for the readable runs. Unreadable runs are still redacted with `redactHeartbeatRunListRow`, and no silence state is read for them. - Active-run watchdog Postgres adapter: new `findLatestDecisions` and `findOpenStaleRunEvaluations` read all runs of one company in one statement each (`DISTINCT ON`). The single-run `findLatestDecision` and `findOpenStaleRunEvaluation` now call the batch with one run, so both paths use the same query. - Watchdog application: new `createBuildRunOutputSilences` use case groups runs by company. `createBuildRunOutputSilence` now calls it with one run, so the single and batch summaries use one rule. - `recoveryService` and `heartbeatService` expose `buildRunOutputSilences` as pass-throughs. - Tests: new `live-runs-batched-reads.test.ts`; updated watchdog use-case tests and the live-run route unit test. ## Verification - `cd server && npx vitest run src/__tests__/live-runs-batched-reads.test.ts` (embedded Postgres): - The batched read decision equals one `canActorReadHeartbeatRun` call per run for 24 actor kinds (owner, admin, operator, viewer, member, board key, cloud tenant, instance admin, local implicit board, board without membership, agent keys, agent runs on behalf of users, low-trust agents, agent of another company, unauthenticated). The runs mix open tasks, private tasks readable by grant, creation or assignment, private tasks nobody may read, private-project tasks, a hidden task, company-scoped runs, a run whose task was deleted, runs of another company, and runs of another company asked for as this company's. - For every actor that passes the company gate, each live-runs row is either the full row or exactly `redactHeartbeatRunListRow` of it (field by field), as one check per run decides. - Output silence per run equals the single-run reader for: no output yet, active snooze plus a later continue, expired snooze, dismissed false positive, recent output, open evaluation task, done evaluation task, queued run, an evaluation task of another company, and a run that has no rows at all. - The statement count is the same with 19 and 49 live runs for 6 actor kinds. Before the fix this test fails (for example 347 against 107 statements). - `npx vitest run src/modules/active-run-watchdog src/__tests__/heartbeat-active-run-output-watchdog.test.ts src/__tests__/agent-live-run-routes.test.ts src/__tests__/heartbeat-run-privacy-routes.test.ts`: pass. - `npx tsc --noEmit -p server/tsconfig.json` and a test-tree typecheck of the changed tests: pass. `pnpm check:module-boundaries`: pass. - After the fix, measured with the same harness (embedded Postgres, 400 tasks, 40 agents, 5 requests each): | Live runs | Statements per request | Allocated per request | | --- | --- | --- | | 10 | 15 | 6.4 MB | | 50 | 15 | 7.3 MB | | 200 (route returns 50) | 15 | 7.8 MB | ## Risks - This changes an authorization read path. The route now uses the batch helper that the run history list and the live events socket already use. The test compares it with the per-run check for every actor kind in the visibility fixture. - If several open evaluation tasks point at the same run, the batch picks one of them. The single-run read also picked an unordered one before. - No schema, migration, locking or transaction change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
594f9d3ec2
commit
426e8a35dc
10 files changed
+444
-62
No files matched your search
@@ -11,6 +11,7 @@ const mockAgentService = vi.hoisted(() => ({
|
|||||||
const mockHeartbeatService = vi.hoisted(() => ({
|
const mockHeartbeatService = vi.hoisted(() => ({
|
||||||
list: vi.fn(async () => []),
|
list: vi.fn(async () => []),
|
||||||
buildRunOutputSilence: vi.fn(),
|
buildRunOutputSilence: vi.fn(),
|
||||||
|
buildRunOutputSilences: vi.fn(),
|
||||||
decorateActiveRunStatus: vi.fn(),
|
decorateActiveRunStatus: vi.fn(),
|
||||||
getRunIssueSummary: vi.fn(),
|
getRunIssueSummary: vi.fn(),
|
||||||
getActiveRunIssueSummaryForAgent: vi.fn(),
|
getActiveRunIssueSummaryForAgent: vi.fn(),
|
||||||
@@ -344,6 +345,7 @@ describe("agent live run routes", () => {
|
|||||||
});
|
});
|
||||||
mockInstanceSettingsService.listCompanyIds.mockResolvedValue(["company-1"]);
|
mockInstanceSettingsService.listCompanyIds.mockResolvedValue(["company-1"]);
|
||||||
mockHeartbeatService.buildRunOutputSilence.mockResolvedValue(null);
|
mockHeartbeatService.buildRunOutputSilence.mockResolvedValue(null);
|
||||||
|
mockHeartbeatService.buildRunOutputSilences.mockImplementation(async (runs: unknown[]) => runs.map(() => null));
|
||||||
mockHeartbeatService.decorateActiveRunStatus.mockImplementation((run) => ({
|
mockHeartbeatService.decorateActiveRunStatus.mockImplementation((run) => ({
|
||||||
...run,
|
...run,
|
||||||
currentStatusMessage: null,
|
currentStatusMessage: null,
|
||||||
@@ -838,9 +840,9 @@ describe("agent live run routes", () => {
|
|||||||
expect(res.body).toHaveLength(50);
|
expect(res.body).toHaveLength(50);
|
||||||
expect(mockRunSecretRedactionRegistry.redactForRuns).toHaveBeenCalledTimes(1);
|
expect(mockRunSecretRedactionRegistry.redactForRuns).toHaveBeenCalledTimes(1);
|
||||||
expect(mockRunSecretRedactionRegistry.redactForRun).not.toHaveBeenCalled();
|
expect(mockRunSecretRedactionRegistry.redactForRun).not.toHaveBeenCalled();
|
||||||
expect(mockHeartbeatService.buildRunOutputSilence).toHaveBeenCalledTimes(
|
expect(mockHeartbeatService.buildRunOutputSilence).not.toHaveBeenCalled();
|
||||||
50,
|
expect(mockHeartbeatService.buildRunOutputSilences).toHaveBeenCalledTimes(1);
|
||||||
);
|
expect(mockHeartbeatService.buildRunOutputSilences.mock.calls[0]![0]).toHaveLength(50);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("treats explicit zero or invalid live run limit as the capped default", async () => {
|
it("treats explicit zero or invalid live run limit as the capped default", async () => {
|
||||||
|
|||||||
@@ -0,0 +1,245 @@
|
|||||||
|
import { randomUUID } from "node:crypto";
|
||||||
|
import express from "express";
|
||||||
|
import request from "supertest";
|
||||||
|
import { eq } from "drizzle-orm";
|
||||||
|
import { drizzle } from "drizzle-orm/postgres-js";
|
||||||
|
import { afterAll, afterEach, beforeAll, expect, it, vi } from "vitest";
|
||||||
|
import { heartbeatRunWatchdogDecisions, heartbeatRuns, issues, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
|
||||||
|
import { agentRoutes } from "../routes/agents.js";
|
||||||
|
import { errorHandler } from "../middleware/index.js";
|
||||||
|
import { authorizationService, type AuthorizationActor } from "../services/authorization.js";
|
||||||
|
import {
|
||||||
|
canActorReadHeartbeatRun, getHeartbeatRunReadDecisions, redactHeartbeatRunListRow,
|
||||||
|
} from "../services/heartbeat-run-privacy.js";
|
||||||
|
import {
|
||||||
|
ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS, ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS, recoveryService,
|
||||||
|
} from "../services/recovery/service.js";
|
||||||
|
import { describeEmbeddedPostgres, useEmbeddedPostgres } from "./helpers/route-test-harness.js";
|
||||||
|
import {
|
||||||
|
ISSUE_READ_VISIBILITY_ACTORS, seedIssueReadVisibilityFixture, type IssueReadVisibilityActorName,
|
||||||
|
type IssueReadVisibilityFixture,
|
||||||
|
} from "./helpers/issue-read-visibility-fixture.js";
|
||||||
|
|
||||||
|
const NOW = new Date("2026-11-13T12:00:00.000Z");
|
||||||
|
const HOUR = 60 * 60 * 1000;
|
||||||
|
|
||||||
|
type RunBinding = { id: string; companyId: string; scopeKind: "company" | "issue"; issueId: string | null };
|
||||||
|
|
||||||
|
describeEmbeddedPostgres("live runs read in a bounded number of statements", () => {
|
||||||
|
const ctx = useEmbeddedPostgres("paperclip-live-runs-batch-");
|
||||||
|
const oldPrivacy = process.env.PAPERCLIP_ISSUE_PRIVACY_MODE;
|
||||||
|
let f: IssueReadVisibilityFixture;
|
||||||
|
let runs: RunBinding[];
|
||||||
|
let foreignRuns: RunBinding[];
|
||||||
|
let silenceRunIds: Record<"noOutput" | "snoozed" | "expiredSnooze" | "dismissed" | "recent" | "evaluated" | "doneEvaluation" | "queued", string>;
|
||||||
|
|
||||||
|
let order = 0;
|
||||||
|
async function insertRun(input: {
|
||||||
|
companyId: string; agentId: string; issueId?: string | null; scopeKind?: "company" | "issue";
|
||||||
|
status?: string; lastOutputAt?: Date | null; lastOutputSeq?: number; lastOutputStream?: string | null;
|
||||||
|
}) {
|
||||||
|
const id = randomUUID();
|
||||||
|
order += 1;
|
||||||
|
const scopeKind = input.scopeKind ?? (input.issueId ? "issue" : "company");
|
||||||
|
await ctx.db.insert(heartbeatRuns).values({
|
||||||
|
id, companyId: input.companyId, agentId: input.agentId, scopeKind, issueId: input.issueId ?? null,
|
||||||
|
status: input.status ?? "running", startedAt: new Date(NOW.getTime() - 3 * HOUR),
|
||||||
|
createdAt: new Date(NOW.getTime() - 3 * HOUR - order * 1000), lastOutputAt: input.lastOutputAt ?? null,
|
||||||
|
lastOutputSeq: input.lastOutputSeq ?? 0, lastOutputStream: input.lastOutputStream ?? null,
|
||||||
|
usageJson: { inputTokens: 11, cachedInputTokens: 2, outputTokens: 7 }, resultJson: { costUsd: 0.42, summary: "secret summary" },
|
||||||
|
contextSnapshot: input.issueId ? { issueId: input.issueId } : {},
|
||||||
|
});
|
||||||
|
return { id, companyId: input.companyId, scopeKind, issueId: input.issueId ?? null } satisfies RunBinding;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function seedRuns(fixture: IssueReadVisibilityFixture, openTaskRuns: number) {
|
||||||
|
const agentIds = [fixture.readerAgentId, fixture.assigneeAgentId, fixture.lowTrustAgentId];
|
||||||
|
const taskIds = [
|
||||||
|
...Array.from({ length: openTaskRuns }, (_, index) => fixture.publicIds[index % fixture.publicIds.length]!),
|
||||||
|
...fixture.ownedPrivateIds, ...fixture.grantedPrivateIds, ...fixture.assignedPrivateIds, ...fixture.unreadablePrivateIds,
|
||||||
|
...fixture.privateProjectIssueIds, fixture.hiddenId,
|
||||||
|
];
|
||||||
|
const seeded: RunBinding[] = [];
|
||||||
|
for (const [index, issueId] of taskIds.entries()) {
|
||||||
|
seeded.push(await insertRun({ companyId: fixture.companyId, agentId: agentIds[index % agentIds.length]!, issueId }));
|
||||||
|
}
|
||||||
|
seeded.push(await insertRun({ companyId: fixture.companyId, agentId: fixture.readerAgentId }));
|
||||||
|
const doomedTaskId = randomUUID();
|
||||||
|
await ctx.db.insert(issues).values({ id: doomedTaskId, companyId: fixture.companyId, title: "Deleted", status: "todo" });
|
||||||
|
const orphan = await insertRun({ companyId: fixture.companyId, agentId: fixture.assigneeAgentId, issueId: doomedTaskId });
|
||||||
|
await ctx.db.delete(issues).where(eq(issues.id, doomedTaskId));
|
||||||
|
seeded.push({ ...orphan, issueId: null });
|
||||||
|
return seeded;
|
||||||
|
}
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = "enforce";
|
||||||
|
f = await seedIssueReadVisibilityFixture(ctx.db, { publicCount: 12 });
|
||||||
|
runs = await seedRuns(f, 3);
|
||||||
|
foreignRuns = [
|
||||||
|
await insertRun({ companyId: f.foreignCompanyId, agentId: f.foreignAgentId, issueId: f.foreignIds[0]! }),
|
||||||
|
await insertRun({ companyId: f.foreignCompanyId, agentId: f.foreignAgentId, issueId: f.foreignIds[1]! }),
|
||||||
|
await insertRun({ companyId: f.foreignCompanyId, agentId: f.foreignAgentId }),
|
||||||
|
];
|
||||||
|
const silenceRun = (extra: Partial<Parameters<typeof insertRun>[0]>) =>
|
||||||
|
insertRun({ companyId: f.companyId, agentId: f.readerAgentId, ...extra });
|
||||||
|
silenceRunIds = {
|
||||||
|
noOutput: (await silenceRun({})).id,
|
||||||
|
snoozed: (await silenceRun({ lastOutputAt: new Date(NOW.getTime() - 2 * HOUR), lastOutputSeq: 4, lastOutputStream: "stdout" })).id,
|
||||||
|
expiredSnooze: (await silenceRun({ lastOutputAt: new Date(NOW.getTime() - 2 * HOUR), lastOutputSeq: 2, lastOutputStream: "stderr" })).id,
|
||||||
|
dismissed: (await silenceRun({ lastOutputAt: new Date(NOW.getTime() - 5 * HOUR), lastOutputSeq: 9, lastOutputStream: "other" })).id,
|
||||||
|
recent: (await silenceRun({ lastOutputAt: new Date(NOW.getTime() - 10 * 60 * 1000), lastOutputSeq: 1, lastOutputStream: "stdout" })).id,
|
||||||
|
evaluated: (await silenceRun({})).id,
|
||||||
|
doneEvaluation: (await silenceRun({})).id,
|
||||||
|
queued: (await silenceRun({ status: "queued" })).id,
|
||||||
|
};
|
||||||
|
await ctx.db.insert(heartbeatRunWatchdogDecisions).values([
|
||||||
|
{ companyId: f.companyId, runId: silenceRunIds.snoozed, decision: "snooze", snoozedUntil: new Date(NOW.getTime() + HOUR), createdAt: new Date(NOW.getTime() - 2 * HOUR) },
|
||||||
|
{ companyId: f.companyId, runId: silenceRunIds.snoozed, decision: "continue", snoozedUntil: new Date(NOW.getTime() + 2 * HOUR), createdAt: new Date(NOW.getTime() - HOUR) },
|
||||||
|
{ companyId: f.companyId, runId: silenceRunIds.expiredSnooze, decision: "snooze", snoozedUntil: new Date(NOW.getTime() - HOUR) },
|
||||||
|
{ companyId: f.companyId, runId: silenceRunIds.dismissed, decision: "dismissed_false_positive" },
|
||||||
|
{ companyId: f.companyId, runId: silenceRunIds.dismissed, decision: "snooze", snoozedUntil: new Date(NOW.getTime() + HOUR) },
|
||||||
|
]);
|
||||||
|
const evaluationId = randomUUID();
|
||||||
|
await ctx.db.insert(issues).values([
|
||||||
|
{ id: evaluationId, companyId: f.companyId, title: "Evaluate", status: "todo", identifier: "EVAL-1",
|
||||||
|
assigneeAgentId: f.assigneeAgentId, originKind: "stale_active_run_evaluation", originId: silenceRunIds.evaluated },
|
||||||
|
{ companyId: f.companyId, title: "Evaluated before", status: "done", originKind: "stale_active_run_evaluation", originId: silenceRunIds.doneEvaluation },
|
||||||
|
{ companyId: f.foreignCompanyId, title: "Foreign evaluation", status: "todo", originKind: "stale_active_run_evaluation", originId: silenceRunIds.noOutput },
|
||||||
|
]);
|
||||||
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
||||||
|
afterAll(() => {
|
||||||
|
if (oldPrivacy === undefined) delete process.env.PAPERCLIP_ISSUE_PRIVACY_MODE;
|
||||||
|
else process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = oldPrivacy;
|
||||||
|
});
|
||||||
|
afterEach(() => { vi.useRealTimers(); });
|
||||||
|
|
||||||
|
function app(actor: AuthorizationActor, db = ctx.db) {
|
||||||
|
const result = express();
|
||||||
|
result.use(express.json());
|
||||||
|
result.use((req, _res, next) => { req.actor = { ...actor } as Express.Request["actor"]; next(); });
|
||||||
|
result.use("/api", agentRoutes(db, {} as never));
|
||||||
|
result.use(errorHandler);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
function recording() {
|
||||||
|
const statements: string[] = [];
|
||||||
|
const db = drizzle(ctx.db.$client, {
|
||||||
|
schema: ctx.db._.fullSchema, logger: { logQuery: (query) => statements.push(query) },
|
||||||
|
}) as unknown as typeof ctx.db;
|
||||||
|
return { db, statements };
|
||||||
|
}
|
||||||
|
async function liveRuns(actor: AuthorizationActor, companyId = f.companyId, db = ctx.db) {
|
||||||
|
return request(app(actor, db)).get(`/api/companies/${companyId}/live-runs`);
|
||||||
|
}
|
||||||
|
const readEach = (actor: AuthorizationActor, bindings: RunBinding[]) =>
|
||||||
|
Promise.all(bindings.map((run) => canActorReadHeartbeatRun(ctx.db, authorizationService(ctx.db), { ...actor }, run)));
|
||||||
|
|
||||||
|
it.each(ISSUE_READ_VISIBILITY_ACTORS)("decides every run read for %s exactly as one check per run does", async (name) => {
|
||||||
|
const actor = f.actors[name];
|
||||||
|
const bindings = [...runs, ...foreignRuns, ...foreignRuns.map((run) => ({ ...run, companyId: f.companyId }))];
|
||||||
|
const batched = await getHeartbeatRunReadDecisions(ctx.db, authorizationService(ctx.db), { ...actor }, bindings);
|
||||||
|
expect(batched).toEqual(await readEach(actor, bindings));
|
||||||
|
});
|
||||||
|
|
||||||
|
it("covers readable and unreadable runs in the fixture", async () => {
|
||||||
|
const decisions = async (name: IssueReadVisibilityActorName) => readEach(f.actors[name], runs);
|
||||||
|
expect((await decisions("instance admin")).filter(Boolean).length).toBeGreaterThan((await decisions("board member")).filter(Boolean).length);
|
||||||
|
expect(await decisions("board member")).toContain(false);
|
||||||
|
expect(await decisions("board member")).toContain(true);
|
||||||
|
expect(await decisions("agent key")).toContain(false);
|
||||||
|
expect(runs.at(-1)).toMatchObject({ scopeKind: "issue", issueId: null });
|
||||||
|
expect(await readEach(f.actors["instance admin"], [runs.at(-1)!])).toEqual([false]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each(ISSUE_READ_VISIBILITY_ACTORS)("returns %s the live runs one check per run would, redacting the rest field by field", async (name) => {
|
||||||
|
vi.useFakeTimers({ toFake: ["Date"] });
|
||||||
|
vi.setSystemTime(NOW);
|
||||||
|
const actor = f.actors[name];
|
||||||
|
const res = await liveRuns(actor);
|
||||||
|
if (res.status !== 200) {
|
||||||
|
expect([401, 403]).toContain(res.status);
|
||||||
|
for (const run of runs) expect(JSON.stringify(res.body)).not.toContain(run.id);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const full = await liveRuns(f.actors["local implicit board"]);
|
||||||
|
expect(full.status).toBe(200);
|
||||||
|
const fullById = new Map((full.body as Array<Record<string, unknown> & { id: string }>).map((row) => [row.id, row]));
|
||||||
|
const body = res.body as Array<Record<string, unknown> & { id: string; scopeKind?: string }>;
|
||||||
|
expect(body.map((row) => row.id)).toEqual([...fullById.keys()]);
|
||||||
|
const bindings = body.map((row) => {
|
||||||
|
const source = fullById.get(row.id)!;
|
||||||
|
return { id: row.id, companyId: f.companyId, scopeKind: source.scopeKind as "company" | "issue", issueId: (source.issueId as string | null) ?? null };
|
||||||
|
});
|
||||||
|
const expected = await readEach(actor, bindings);
|
||||||
|
for (const [index, row] of body.entries()) {
|
||||||
|
const source = fullById.get(row.id)!;
|
||||||
|
if (expected[index]) {
|
||||||
|
expect(row.redacted).toBeUndefined();
|
||||||
|
expect(row.outputSilence).toEqual(source.outputSilence);
|
||||||
|
expect(row.issueId).toEqual(source.issueId);
|
||||||
|
} else {
|
||||||
|
expect(row).toEqual(JSON.parse(JSON.stringify(redactHeartbeatRunListRow(source))));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
expect(expected).toContain(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("builds each run's output silence as the single-run reader does, including runs with no output yet", async () => {
|
||||||
|
vi.useFakeTimers({ toFake: ["Date"] });
|
||||||
|
vi.setSystemTime(NOW);
|
||||||
|
const res = await liveRuns(f.actors["local implicit board"]);
|
||||||
|
expect(res.status).toBe(200);
|
||||||
|
const byId = new Map((res.body as Array<{ id: string; outputSilence: unknown }>).map((row) => [row.id, row.outputSilence]));
|
||||||
|
const recovery = recoveryService(ctx.db, { enqueueWakeup: vi.fn() } as never);
|
||||||
|
const rows = await ctx.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
|
||||||
|
const single = new Map(await Promise.all(rows.map(async (run) => [run.id, JSON.parse(JSON.stringify(await recovery.buildRunOutputSilence(run, NOW)))] as const)));
|
||||||
|
for (const [id, silence] of byId) {
|
||||||
|
if (silence !== undefined) expect(silence).toEqual(single.get(id));
|
||||||
|
}
|
||||||
|
const thresholds = { suspicionThresholdMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS, criticalThresholdMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS };
|
||||||
|
const silenceOf = (id: string) => single.get(id) as Record<string, unknown>;
|
||||||
|
expect(silenceOf(silenceRunIds.noOutput)).toEqual({
|
||||||
|
lastOutputAt: null, lastOutputSeq: 0, lastOutputStream: null,
|
||||||
|
silenceStartedAt: new Date(NOW.getTime() - 3 * HOUR).toISOString(), silenceAgeMs: 3 * HOUR, level: "critical", ...thresholds,
|
||||||
|
snoozedUntil: null, evaluationIssueId: null, evaluationIssueIdentifier: null, evaluationIssueAssigneeAgentId: null,
|
||||||
|
});
|
||||||
|
expect(silenceOf(silenceRunIds.snoozed)).toMatchObject({
|
||||||
|
lastOutputSeq: 4, lastOutputStream: "stdout", level: "snoozed", snoozedUntil: new Date(NOW.getTime() + 2 * HOUR).toISOString(),
|
||||||
|
});
|
||||||
|
expect(silenceOf(silenceRunIds.expiredSnooze)).toMatchObject({ lastOutputStream: "stderr", level: "critical", snoozedUntil: null });
|
||||||
|
expect(silenceOf(silenceRunIds.dismissed)).toMatchObject({ lastOutputStream: null, level: "not_applicable", snoozedUntil: null });
|
||||||
|
expect(silenceOf(silenceRunIds.recent)).toMatchObject({ level: "suspicious", silenceAgeMs: 10 * 60 * 1000 });
|
||||||
|
expect(silenceOf(silenceRunIds.evaluated)).toMatchObject({ evaluationIssueIdentifier: "EVAL-1", evaluationIssueAssigneeAgentId: f.assigneeAgentId });
|
||||||
|
expect(silenceOf(silenceRunIds.doneEvaluation)).toMatchObject({ evaluationIssueId: null });
|
||||||
|
expect(silenceOf(silenceRunIds.queued)).toMatchObject({ level: "not_applicable", silenceAgeMs: null });
|
||||||
|
|
||||||
|
const batched = await recovery.buildRunOutputSilences([...rows, ...rows.slice(0, 2)], NOW);
|
||||||
|
expect(batched.map((silence) => JSON.parse(JSON.stringify(silence))))
|
||||||
|
.toEqual([...rows, ...rows.slice(0, 2)].map((run) => single.get(run.id)));
|
||||||
|
expect(await recovery.buildRunOutputSilences([], NOW)).toEqual([]);
|
||||||
|
const missing = rows.find((run) => run.id === silenceRunIds.noOutput)!;
|
||||||
|
expect(JSON.parse(JSON.stringify(await recovery.buildRunOutputSilence({ ...missing, id: randomUUID() }, NOW))))
|
||||||
|
.toEqual(silenceOf(silenceRunIds.noOutput));
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
"board owner", "board member", "instance admin", "local implicit board", "agent key", "agent run for operator",
|
||||||
|
] as const)("sends the same number of statements for %s whether 19 or 49 runs are live", async (name) => {
|
||||||
|
const small = await seedIssueReadVisibilityFixture(ctx.db, { publicCount: 12 });
|
||||||
|
const large = await seedIssueReadVisibilityFixture(ctx.db, { publicCount: 12 });
|
||||||
|
const smallRuns = await seedRuns(small, 0);
|
||||||
|
const largeRuns = await seedRuns(large, 30);
|
||||||
|
const actorFor = (fixture: IssueReadVisibilityFixture) => fixture.actors[name];
|
||||||
|
const first = recording();
|
||||||
|
const smallRes = await liveRuns(actorFor(small), small.companyId, first.db);
|
||||||
|
const second = recording();
|
||||||
|
const largeRes = await liveRuns(actorFor(large), large.companyId, second.db);
|
||||||
|
expect(smallRes.status).toBe(200);
|
||||||
|
expect(largeRes.status).toBe(200);
|
||||||
|
expect(smallRes.body).toHaveLength(smallRuns.length + 1);
|
||||||
|
expect(largeRes.body).toHaveLength(largeRuns.length + 1);
|
||||||
|
expect(second.statements.length).toBe(first.statements.length);
|
||||||
|
expect(second.statements.length).toBeLessThanOrEqual(24);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -31,7 +31,7 @@ import {
|
|||||||
import { issueRecoveryActionService } from "../../../services/issue-recovery-actions.js";
|
import { issueRecoveryActionService } from "../../../services/issue-recovery-actions.js";
|
||||||
import { RECOVERY_ORIGIN_KINDS } from "../../../services/recovery/origins.js";
|
import { RECOVERY_ORIGIN_KINDS } from "../../../services/recovery/origins.js";
|
||||||
import { isTerminalIssueStatus } from "../domain/policy.js";
|
import { isTerminalIssueStatus } from "../domain/policy.js";
|
||||||
import type { WatchdogRunReader, WatchdogWriter } from "../application/ports.js";
|
import type { FindLatestDecisionResult, WatchdogRunReader, WatchdogWriter } from "../application/ports.js";
|
||||||
import type {
|
import type {
|
||||||
EvaluationIssueSnapshot,
|
EvaluationIssueSnapshot,
|
||||||
FoldOutcome,
|
FoldOutcome,
|
||||||
@@ -159,10 +159,14 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch
|
|||||||
return row?.pending ?? true;
|
return row?.pending ?? true;
|
||||||
}
|
}
|
||||||
|
|
||||||
async function findLatestDecision(companyId: string, runId: string, now: Date) {
|
async function findLatestDecisions(companyId: string, runIds: readonly string[], now: Date) {
|
||||||
|
const ids = [...new Set(runIds)];
|
||||||
|
const result = new Map<string, FindLatestDecisionResult>();
|
||||||
|
if (ids.length === 0) return result;
|
||||||
const [quietUntilRows, dismissedRows] = await Promise.all([
|
const [quietUntilRows, dismissedRows] = await Promise.all([
|
||||||
db
|
db
|
||||||
.select({
|
.selectDistinctOn([heartbeatRunWatchdogDecisions.runId], {
|
||||||
|
runId: heartbeatRunWatchdogDecisions.runId,
|
||||||
decision: heartbeatRunWatchdogDecisions.decision,
|
decision: heartbeatRunWatchdogDecisions.decision,
|
||||||
snoozedUntil: heartbeatRunWatchdogDecisions.snoozedUntil,
|
snoozedUntil: heartbeatRunWatchdogDecisions.snoozedUntil,
|
||||||
})
|
})
|
||||||
@@ -170,32 +174,39 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch
|
|||||||
.where(
|
.where(
|
||||||
and(
|
and(
|
||||||
eq(heartbeatRunWatchdogDecisions.companyId, companyId),
|
eq(heartbeatRunWatchdogDecisions.companyId, companyId),
|
||||||
eq(heartbeatRunWatchdogDecisions.runId, runId),
|
inArray(heartbeatRunWatchdogDecisions.runId, ids),
|
||||||
inArray(heartbeatRunWatchdogDecisions.decision, ["snooze", "continue"]),
|
inArray(heartbeatRunWatchdogDecisions.decision, ["snooze", "continue"]),
|
||||||
gt(heartbeatRunWatchdogDecisions.snoozedUntil, now),
|
gt(heartbeatRunWatchdogDecisions.snoozedUntil, now),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
.orderBy(desc(heartbeatRunWatchdogDecisions.createdAt))
|
.orderBy(heartbeatRunWatchdogDecisions.runId, desc(heartbeatRunWatchdogDecisions.createdAt)),
|
||||||
.limit(1),
|
|
||||||
db
|
db
|
||||||
.select({ id: heartbeatRunWatchdogDecisions.id })
|
.selectDistinct({ runId: heartbeatRunWatchdogDecisions.runId })
|
||||||
.from(heartbeatRunWatchdogDecisions)
|
.from(heartbeatRunWatchdogDecisions)
|
||||||
.where(
|
.where(
|
||||||
and(
|
and(
|
||||||
eq(heartbeatRunWatchdogDecisions.companyId, companyId),
|
eq(heartbeatRunWatchdogDecisions.companyId, companyId),
|
||||||
eq(heartbeatRunWatchdogDecisions.runId, runId),
|
inArray(heartbeatRunWatchdogDecisions.runId, ids),
|
||||||
eq(heartbeatRunWatchdogDecisions.decision, "dismissed_false_positive"),
|
eq(heartbeatRunWatchdogDecisions.decision, "dismissed_false_positive"),
|
||||||
),
|
),
|
||||||
)
|
),
|
||||||
.limit(1),
|
|
||||||
]);
|
]);
|
||||||
const quietUntilRow = quietUntilRows[0];
|
const quietUntilByRun = new Map(quietUntilRows.map((row) => [row.runId, row]));
|
||||||
return {
|
const dismissed = new Set(dismissedRows.map((row) => row.runId));
|
||||||
dismissedFalsePositive: dismissedRows.length > 0,
|
for (const runId of ids) {
|
||||||
quietUntilDecision: quietUntilRow && quietUntilRow.snoozedUntil
|
const quietUntilRow = quietUntilByRun.get(runId);
|
||||||
? { decision: quietUntilRow.decision as "snooze" | "continue", snoozedUntil: quietUntilRow.snoozedUntil }
|
result.set(runId, {
|
||||||
: null,
|
dismissedFalsePositive: dismissed.has(runId),
|
||||||
};
|
quietUntilDecision: quietUntilRow && quietUntilRow.snoozedUntil
|
||||||
|
? { decision: quietUntilRow.decision as "snooze" | "continue", snoozedUntil: quietUntilRow.snoozedUntil }
|
||||||
|
: null,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function findLatestDecision(companyId: string, runId: string, now: Date): Promise<FindLatestDecisionResult> {
|
||||||
|
return (await findLatestDecisions(companyId, [runId], now)).get(runId)!;
|
||||||
}
|
}
|
||||||
|
|
||||||
function selectEvaluationIssueSnapshot() {
|
function selectEvaluationIssueSnapshot() {
|
||||||
@@ -211,21 +222,29 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
async function findOpenStaleRunEvaluation(companyId: string, runId: string): Promise<EvaluationIssueSnapshot | null> {
|
async function findOpenStaleRunEvaluations(companyId: string, runIds: readonly string[]) {
|
||||||
const [row] = await db
|
const ids = [...new Set(runIds)];
|
||||||
.select(selectEvaluationIssueSnapshot())
|
const result = new Map<string, EvaluationIssueSnapshot>();
|
||||||
|
if (ids.length === 0) return result;
|
||||||
|
const rows = await db
|
||||||
|
.selectDistinctOn([issues.originId], selectEvaluationIssueSnapshot())
|
||||||
.from(issues)
|
.from(issues)
|
||||||
.where(
|
.where(
|
||||||
and(
|
and(
|
||||||
eq(issues.companyId, companyId),
|
eq(issues.companyId, companyId),
|
||||||
eq(issues.originKind, STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND),
|
eq(issues.originKind, STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND),
|
||||||
eq(issues.originId, runId),
|
inArray(issues.originId, ids),
|
||||||
visibleIssueCondition(),
|
visibleIssueCondition(),
|
||||||
notInArray(issues.status, ["done", "cancelled"]),
|
notInArray(issues.status, ["done", "cancelled"]),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
.limit(1);
|
.orderBy(issues.originId);
|
||||||
return row ?? null;
|
for (const row of rows) if (row.originId) result.set(row.originId, row);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function findOpenStaleRunEvaluation(companyId: string, runId: string): Promise<EvaluationIssueSnapshot | null> {
|
||||||
|
return (await findOpenStaleRunEvaluations(companyId, [runId])).get(runId) ?? null;
|
||||||
}
|
}
|
||||||
|
|
||||||
async function findEvaluationIssueById(companyId: string, issueId: string): Promise<EvaluationIssueSnapshot | null> {
|
async function findEvaluationIssueById(companyId: string, issueId: string): Promise<EvaluationIssueSnapshot | null> {
|
||||||
@@ -532,7 +551,9 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch
|
|||||||
findRunForCompany,
|
findRunForCompany,
|
||||||
isRuntimeSettlementPending,
|
isRuntimeSettlementPending,
|
||||||
findLatestDecision,
|
findLatestDecision,
|
||||||
|
findLatestDecisions,
|
||||||
findOpenStaleRunEvaluation,
|
findOpenStaleRunEvaluation,
|
||||||
|
findOpenStaleRunEvaluations,
|
||||||
findEvaluationIssueById,
|
findEvaluationIssueById,
|
||||||
findLatestSameRunTerminalEvidence,
|
findLatestSameRunTerminalEvidence,
|
||||||
findSourceIssue,
|
findSourceIssue,
|
||||||
|
|||||||
@@ -56,7 +56,9 @@ export interface WatchdogRunReader {
|
|||||||
findRunForCompany(companyId: string, runId: string): Promise<RunSnapshot | null>;
|
findRunForCompany(companyId: string, runId: string): Promise<RunSnapshot | null>;
|
||||||
isRuntimeSettlementPending(companyId: string, runId: string): Promise<boolean>;
|
isRuntimeSettlementPending(companyId: string, runId: string): Promise<boolean>;
|
||||||
findLatestDecision(companyId: string, runId: string, now: Date): Promise<FindLatestDecisionResult>;
|
findLatestDecision(companyId: string, runId: string, now: Date): Promise<FindLatestDecisionResult>;
|
||||||
|
findLatestDecisions(companyId: string, runIds: readonly string[], now: Date): Promise<Map<string, FindLatestDecisionResult>>;
|
||||||
findOpenStaleRunEvaluation(companyId: string, runId: string): Promise<EvaluationIssueSnapshot | null>;
|
findOpenStaleRunEvaluation(companyId: string, runId: string): Promise<EvaluationIssueSnapshot | null>;
|
||||||
|
findOpenStaleRunEvaluations(companyId: string, runIds: readonly string[]): Promise<Map<string, EvaluationIssueSnapshot>>;
|
||||||
findLatestSameRunTerminalEvidence(
|
findLatestSameRunTerminalEvidence(
|
||||||
companyId: string,
|
companyId: string,
|
||||||
input: FindLatestSameRunTerminalEvidenceInput,
|
input: FindLatestSameRunTerminalEvidenceInput,
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
import { describe, expect, it } from "vitest";
|
import { describe, expect, it } from "vitest";
|
||||||
import {
|
import {
|
||||||
createBuildRunOutputSilence,
|
createBuildRunOutputSilence,
|
||||||
|
createBuildRunOutputSilences,
|
||||||
createFoldSourceResolvedRun,
|
createFoldSourceResolvedRun,
|
||||||
createRecordWatchdogDecision,
|
createRecordWatchdogDecision,
|
||||||
createScanSilentActiveRuns,
|
createScanSilentActiveRuns,
|
||||||
@@ -35,18 +36,29 @@ describe("application", () => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function makeReader(overrides: Partial<WatchdogRunReader> = {}): WatchdogRunReader {
|
function makeReader(overrides: Partial<WatchdogRunReader> = {}): WatchdogRunReader {
|
||||||
return {
|
const reader: WatchdogRunReader = {
|
||||||
findCandidateSilentRuns: async () => [],
|
findCandidateSilentRuns: async () => [],
|
||||||
findRunForCompany: async () => null,
|
findRunForCompany: async () => null,
|
||||||
isRuntimeSettlementPending: async () => false,
|
isRuntimeSettlementPending: async () => false,
|
||||||
findLatestDecision: async () => ({ quietUntilDecision: null, dismissedFalsePositive: false }),
|
findLatestDecision: async () => ({ quietUntilDecision: null, dismissedFalsePositive: false }),
|
||||||
|
findLatestDecisions: async (companyId, runIds, now) => new Map(await Promise.all(runIds.map(async (runId) =>
|
||||||
|
[runId, await reader.findLatestDecision(companyId, runId, now)] as const))),
|
||||||
findOpenStaleRunEvaluation: async () => null,
|
findOpenStaleRunEvaluation: async () => null,
|
||||||
|
findOpenStaleRunEvaluations: async (companyId, runIds) => {
|
||||||
|
const found = new Map();
|
||||||
|
for (const runId of runIds) {
|
||||||
|
const evaluation = await reader.findOpenStaleRunEvaluation(companyId, runId);
|
||||||
|
if (evaluation) found.set(runId, evaluation);
|
||||||
|
}
|
||||||
|
return found;
|
||||||
|
},
|
||||||
findLatestSameRunTerminalEvidence: async () => null,
|
findLatestSameRunTerminalEvidence: async () => null,
|
||||||
findSourceIssue: async () => null,
|
findSourceIssue: async () => null,
|
||||||
findRunningAgent: async () => null,
|
findRunningAgent: async () => null,
|
||||||
findEvaluationIssueById: async () => null,
|
findEvaluationIssueById: async () => null,
|
||||||
...overrides,
|
...overrides,
|
||||||
};
|
};
|
||||||
|
return reader;
|
||||||
}
|
}
|
||||||
|
|
||||||
describe("createBuildRunOutputSilence", () => {
|
describe("createBuildRunOutputSilence", () => {
|
||||||
@@ -105,6 +117,33 @@ describe("application", () => {
|
|||||||
expect(summary).toMatchObject({ level: "snoozed", snoozedUntil });
|
expect(summary).toMatchObject({ level: "snoozed", snoozedUntil });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("reads decisions and evaluations once per company for a batch, defaulting runs the reader omits", async () => {
|
||||||
|
const now = new Date("2026-01-01T05:00:00.000Z");
|
||||||
|
const calls: string[] = [];
|
||||||
|
const reader = makeReader({
|
||||||
|
findLatestDecisions: async (companyId, runIds) => {
|
||||||
|
calls.push(`decisions ${companyId} ${runIds.join(",")}`);
|
||||||
|
return new Map();
|
||||||
|
},
|
||||||
|
findOpenStaleRunEvaluations: async (companyId, runIds) => {
|
||||||
|
calls.push(`evaluations ${companyId} ${runIds.join(",")}`);
|
||||||
|
return new Map();
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const deps = { reader, suspicionThresholdMs: SUSPICION_THRESHOLD_MS, criticalThresholdMs: CRITICAL_THRESHOLD_MS };
|
||||||
|
const runs = [makeRun({ id: "run-1" }), makeRun({ id: "run-2", companyId: "company-2" }), makeRun({ id: "run-3" })];
|
||||||
|
|
||||||
|
const summaries = await createBuildRunOutputSilences(deps)(runs, now);
|
||||||
|
|
||||||
|
expect(calls.sort()).toEqual([
|
||||||
|
"decisions company-1 run-1,run-3", "decisions company-2 run-2",
|
||||||
|
"evaluations company-1 run-1,run-3", "evaluations company-2 run-2",
|
||||||
|
]);
|
||||||
|
const single = await createBuildRunOutputSilence({ ...deps, reader: makeReader() })(makeRun(), now);
|
||||||
|
expect(summaries).toEqual([single, single, single]);
|
||||||
|
expect(single).toMatchObject({ level: "critical", snoozedUntil: null, evaluationIssueId: null, lastOutputAt: null, lastOutputSeq: 0 });
|
||||||
|
});
|
||||||
|
|
||||||
it("reports not_applicable for a run that is not running", async () => {
|
it("reports not_applicable for a run that is not running", async () => {
|
||||||
const now = new Date("2026-01-01T05:00:00.000Z");
|
const now = new Date("2026-01-01T05:00:00.000Z");
|
||||||
const run = makeRun({ status: "succeeded" });
|
const run = makeRun({ status: "succeeded" });
|
||||||
|
|||||||
@@ -31,43 +31,68 @@ export type BuildRunOutputSilenceRunInput = Pick<
|
|||||||
"id" | "companyId" | "status" | "lastOutputAt" | "lastOutputSeq" | "lastOutputStream" | "processStartedAt" | "startedAt" | "createdAt"
|
"id" | "companyId" | "status" | "lastOutputAt" | "lastOutputSeq" | "lastOutputStream" | "processStartedAt" | "startedAt" | "createdAt"
|
||||||
>;
|
>;
|
||||||
|
|
||||||
|
export function createBuildRunOutputSilences(deps: BuildRunOutputSilenceDeps) {
|
||||||
|
return async function buildRunOutputSilences(
|
||||||
|
runs: readonly BuildRunOutputSilenceRunInput[],
|
||||||
|
now: Date,
|
||||||
|
): Promise<RunOutputSilenceSummary[]> {
|
||||||
|
const runIdsByCompany = new Map<string, string[]>();
|
||||||
|
for (const run of runs) {
|
||||||
|
const runIds = runIdsByCompany.get(run.companyId) ?? [];
|
||||||
|
runIds.push(run.id);
|
||||||
|
runIdsByCompany.set(run.companyId, runIds);
|
||||||
|
}
|
||||||
|
const stateByCompany = new Map(await Promise.all([...runIdsByCompany].map(async ([companyId, runIds]) => {
|
||||||
|
const [decisions, evaluations] = await Promise.all([
|
||||||
|
deps.reader.findLatestDecisions(companyId, runIds, now),
|
||||||
|
deps.reader.findOpenStaleRunEvaluations(companyId, runIds),
|
||||||
|
]);
|
||||||
|
return [companyId, { decisions, evaluations }] as const;
|
||||||
|
})));
|
||||||
|
return runs.map((run) => {
|
||||||
|
const state = stateByCompany.get(run.companyId);
|
||||||
|
const { dismissedFalsePositive, quietUntilDecision } = state?.decisions.get(run.id)
|
||||||
|
?? { dismissedFalsePositive: false, quietUntilDecision: null };
|
||||||
|
const evaluation = state?.evaluations.get(run.id) ?? null;
|
||||||
|
const isRunningRun = run.status === "running";
|
||||||
|
const silenceAgeMsValue = isRunningRun ? silenceAgeMs(run, now) : null;
|
||||||
|
const level = classifySilenceLevel({
|
||||||
|
isRunningRun,
|
||||||
|
silenceAgeMs: silenceAgeMsValue,
|
||||||
|
dismissedFalsePositive,
|
||||||
|
snoozed: Boolean(quietUntilDecision),
|
||||||
|
suspicionThresholdMs: deps.suspicionThresholdMs,
|
||||||
|
criticalThresholdMs: deps.criticalThresholdMs,
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
lastOutputAt: run.lastOutputAt ?? null,
|
||||||
|
lastOutputSeq: run.lastOutputSeq ?? 0,
|
||||||
|
lastOutputStream: run.lastOutputStream === "stdout" || run.lastOutputStream === "stderr"
|
||||||
|
? run.lastOutputStream
|
||||||
|
: null,
|
||||||
|
silenceStartedAt: silenceStartedAt(run),
|
||||||
|
silenceAgeMs: silenceAgeMsValue,
|
||||||
|
level,
|
||||||
|
suspicionThresholdMs: deps.suspicionThresholdMs,
|
||||||
|
criticalThresholdMs: deps.criticalThresholdMs,
|
||||||
|
snoozedUntil: dismissedFalsePositive ? null : quietUntilDecision?.snoozedUntil ?? null,
|
||||||
|
evaluationIssueId: evaluation?.id ?? null,
|
||||||
|
evaluationIssueIdentifier: evaluation?.identifier ?? null,
|
||||||
|
evaluationIssueAssigneeAgentId: evaluation?.assigneeAgentId ?? null,
|
||||||
|
};
|
||||||
|
});
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
export function createBuildRunOutputSilence(deps: BuildRunOutputSilenceDeps) {
|
export function createBuildRunOutputSilence(deps: BuildRunOutputSilenceDeps) {
|
||||||
|
const buildRunOutputSilences = createBuildRunOutputSilences(deps);
|
||||||
return async function buildRunOutputSilence(
|
return async function buildRunOutputSilence(
|
||||||
run: BuildRunOutputSilenceRunInput,
|
run: BuildRunOutputSilenceRunInput,
|
||||||
now: Date,
|
now: Date,
|
||||||
): Promise<RunOutputSilenceSummary> {
|
): Promise<RunOutputSilenceSummary> {
|
||||||
const [decisionState, evaluation] = await Promise.all([
|
const [summary] = await buildRunOutputSilences([run], now);
|
||||||
deps.reader.findLatestDecision(run.companyId, run.id, now),
|
return summary!;
|
||||||
deps.reader.findOpenStaleRunEvaluation(run.companyId, run.id),
|
|
||||||
]);
|
|
||||||
const { dismissedFalsePositive, quietUntilDecision } = decisionState;
|
|
||||||
const isRunningRun = run.status === "running";
|
|
||||||
const silenceAgeMsValue = isRunningRun ? silenceAgeMs(run, now) : null;
|
|
||||||
const level = classifySilenceLevel({
|
|
||||||
isRunningRun,
|
|
||||||
silenceAgeMs: silenceAgeMsValue,
|
|
||||||
dismissedFalsePositive,
|
|
||||||
snoozed: Boolean(quietUntilDecision),
|
|
||||||
suspicionThresholdMs: deps.suspicionThresholdMs,
|
|
||||||
criticalThresholdMs: deps.criticalThresholdMs,
|
|
||||||
});
|
|
||||||
|
|
||||||
return {
|
|
||||||
lastOutputAt: run.lastOutputAt ?? null,
|
|
||||||
lastOutputSeq: run.lastOutputSeq ?? 0,
|
|
||||||
lastOutputStream: run.lastOutputStream === "stdout" || run.lastOutputStream === "stderr"
|
|
||||||
? run.lastOutputStream
|
|
||||||
: null,
|
|
||||||
silenceStartedAt: silenceStartedAt(run),
|
|
||||||
silenceAgeMs: silenceAgeMsValue,
|
|
||||||
level,
|
|
||||||
suspicionThresholdMs: deps.suspicionThresholdMs,
|
|
||||||
criticalThresholdMs: deps.criticalThresholdMs,
|
|
||||||
snoozedUntil: dismissedFalsePositive ? null : quietUntilDecision?.snoozedUntil ?? null,
|
|
||||||
evaluationIssueId: evaluation?.id ?? null,
|
|
||||||
evaluationIssueIdentifier: evaluation?.identifier ?? null,
|
|
||||||
evaluationIssueAssigneeAgentId: evaluation?.assigneeAgentId ?? null,
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import { createPostgresWatchdogAdapter } from "./adapters/postgres.js";
|
|||||||
import { createProcessAdapter } from "./adapters/process.js";
|
import { createProcessAdapter } from "./adapters/process.js";
|
||||||
import {
|
import {
|
||||||
createBuildRunOutputSilence,
|
createBuildRunOutputSilence,
|
||||||
|
createBuildRunOutputSilences,
|
||||||
createFoldSourceResolvedRun,
|
createFoldSourceResolvedRun,
|
||||||
createRecordWatchdogDecision,
|
createRecordWatchdogDecision,
|
||||||
createScanSilentActiveRuns,
|
createScanSilentActiveRuns,
|
||||||
@@ -42,6 +43,11 @@ export function createActiveRunWatchdog(db: Db, config: ActiveRunWatchdogConfig)
|
|||||||
suspicionThresholdMs: config.suspicionThresholdMs,
|
suspicionThresholdMs: config.suspicionThresholdMs,
|
||||||
criticalThresholdMs: config.criticalThresholdMs,
|
criticalThresholdMs: config.criticalThresholdMs,
|
||||||
}),
|
}),
|
||||||
|
buildRunOutputSilences: createBuildRunOutputSilences({
|
||||||
|
reader: postgresAdapter,
|
||||||
|
suspicionThresholdMs: config.suspicionThresholdMs,
|
||||||
|
criticalThresholdMs: config.criticalThresholdMs,
|
||||||
|
}),
|
||||||
scanSilentActiveRuns,
|
scanSilentActiveRuns,
|
||||||
recordWatchdogDecision: createRecordWatchdogDecision({
|
recordWatchdogDecision: createRecordWatchdogDecision({
|
||||||
reader: postgresAdapter,
|
reader: postgresAdapter,
|
||||||
|
|||||||
@@ -7304,13 +7304,17 @@ export function agentRoutes(
|
|||||||
}
|
}
|
||||||
|
|
||||||
const projections = await executionProjectionsForRuns(db, companyId, rows.map(run => run.id));
|
const projections = await executionProjectionsForRuns(db, companyId, rows.map(run => run.id));
|
||||||
res.json(await runRedactions.redactForRuns(companyId, await Promise.all(rows.map(async (run) => !(await actorCanReadRun(req, run)) ? redactHeartbeatRunListRow(run) : ({
|
const readDecisions = await getHeartbeatRunReadDecisions(db, access, req.actor, rows);
|
||||||
|
const readableRuns = rows.filter((_, index) => readDecisions[index]);
|
||||||
|
const silences = await heartbeat.buildRunOutputSilences(readableRuns);
|
||||||
|
const silenceByRun = new Map(readableRuns.map((run, index) => [run.id, silences[index]!]));
|
||||||
|
res.json(await runRedactions.redactForRuns(companyId, rows.map((run, index) => !readDecisions[index] ? redactHeartbeatRunListRow(run) : ({
|
||||||
...heartbeat.decorateActiveRunStatus(run),
|
...heartbeat.decorateActiveRunStatus(run),
|
||||||
agentAppearance: resolveAgentAppearance(run.agentAppearance, run.agentId),
|
agentAppearance: resolveAgentAppearance(run.agentAppearance, run.agentId),
|
||||||
avatarUrl: agentAvatarUrl(resolveAgentAppearance(run.agentAppearance, run.agentId), 512),
|
avatarUrl: agentAvatarUrl(resolveAgentAppearance(run.agentAppearance, run.agentId), 512),
|
||||||
execution: projections.get(run.id) ?? null,
|
execution: projections.get(run.id) ?? null,
|
||||||
outputSilence: await heartbeat.buildRunOutputSilence(run),
|
outputSilence: silenceByRun.get(run.id)!,
|
||||||
}))), req.actor.type === "board" ? getActorInfo(req).actorId : null));
|
})), req.actor.type === "board" ? getActorInfo(req).actorId : null));
|
||||||
});
|
});
|
||||||
|
|
||||||
function readHeartbeatRunId(req: Request): string {
|
function readHeartbeatRunId(req: Request): string {
|
||||||
|
|||||||
@@ -1774,6 +1774,24 @@ export function heartbeatService(
|
|||||||
return recovery.buildRunOutputSilence(run, now);
|
return recovery.buildRunOutputSilence(run, now);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function buildRunOutputSilences(
|
||||||
|
runs: ReadonlyArray<Pick<
|
||||||
|
typeof heartbeatRuns.$inferSelect,
|
||||||
|
| "id"
|
||||||
|
| "companyId"
|
||||||
|
| "status"
|
||||||
|
| "lastOutputAt"
|
||||||
|
| "lastOutputSeq"
|
||||||
|
| "lastOutputStream"
|
||||||
|
| "processStartedAt"
|
||||||
|
| "startedAt"
|
||||||
|
| "createdAt"
|
||||||
|
>>,
|
||||||
|
now = new Date(),
|
||||||
|
) {
|
||||||
|
return recovery.buildRunOutputSilences(runs, now);
|
||||||
|
}
|
||||||
|
|
||||||
async function reconcileResolvedDependencyWakes(opts?: Parameters<typeof recovery.reconcileResolvedDependencyWakeBackstop>[0]) {
|
async function reconcileResolvedDependencyWakes(opts?: Parameters<typeof recovery.reconcileResolvedDependencyWakeBackstop>[0]) {
|
||||||
return recovery.reconcileResolvedDependencyWakeBackstop(opts);
|
return recovery.reconcileResolvedDependencyWakeBackstop(opts);
|
||||||
}
|
}
|
||||||
@@ -4727,6 +4745,7 @@ export function heartbeatService(
|
|||||||
reconcileCostAccounting,
|
reconcileCostAccounting,
|
||||||
|
|
||||||
buildRunOutputSilence,
|
buildRunOutputSilence,
|
||||||
|
buildRunOutputSilences,
|
||||||
|
|
||||||
tickTimers,
|
tickTimers,
|
||||||
tickAgentTimers,
|
tickAgentTimers,
|
||||||
|
|||||||
@@ -2432,6 +2432,24 @@ export function recoveryService(
|
|||||||
return watchdog.buildRunOutputSilence(run, now);
|
return watchdog.buildRunOutputSilence(run, now);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function buildRunOutputSilences(
|
||||||
|
runs: ReadonlyArray<Pick<
|
||||||
|
typeof heartbeatRuns.$inferSelect,
|
||||||
|
| "id"
|
||||||
|
| "companyId"
|
||||||
|
| "status"
|
||||||
|
| "lastOutputAt"
|
||||||
|
| "lastOutputSeq"
|
||||||
|
| "lastOutputStream"
|
||||||
|
| "processStartedAt"
|
||||||
|
| "startedAt"
|
||||||
|
| "createdAt"
|
||||||
|
>>,
|
||||||
|
now = new Date(),
|
||||||
|
): Promise<RunOutputSilenceSummary[]> {
|
||||||
|
return watchdog.buildRunOutputSilences(runs, now);
|
||||||
|
}
|
||||||
|
|
||||||
async function appendRecoveryRunEvent(
|
async function appendRecoveryRunEvent(
|
||||||
run: typeof heartbeatRuns.$inferSelect,
|
run: typeof heartbeatRuns.$inferSelect,
|
||||||
event: {
|
event: {
|
||||||
@@ -6845,6 +6863,7 @@ export function recoveryService(
|
|||||||
return {
|
return {
|
||||||
assignedWork,
|
assignedWork,
|
||||||
buildRunOutputSilence,
|
buildRunOutputSilence,
|
||||||
|
buildRunOutputSilences,
|
||||||
escalateStrandedRecoveryIssueInPlace,
|
escalateStrandedRecoveryIssueInPlace,
|
||||||
escalateStrandedAssignedIssue,
|
escalateStrandedAssignedIssue,
|
||||||
recordWatchdogDecision,
|
recordWatchdogDecision,
|
||||||
|
|||||||
Reference in new issue
Block a user