perf(runs): read live runs in a fixed number of statements (#21417)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board shows the runs that are live now. The sidebar and the
active agents panel read `GET /api/companies/:companyId/live-runs`
> - That route checked read access and built the output-silence summary
one run at a time, with 8 statements per run
> - On a hosted instance this route ran about 6 times per minute with an
average of 2.1 s of server time, and its statements and allocations add
load to a busy main thread
> - This pull request reads the access decisions and the silence state
for all runs of the request in one batch
> - The benefit is a fixed number of statements per request (15 instead
of 8 + 8 per run) and about one quarter of the allocation

## Linked Issues or Issue Description

No public issue exists. This is an enhancement.

**What existing behavior does this improve?**
The live-runs read on the board (`GET
/api/companies/:companyId/live-runs`).

**Subsystem affected**
Server: agent routes, run read authorization, active-run watchdog.

**Current behavior**
For each run, the route calls `canActorReadHeartbeatRun` (task
existence, instance-admin flag, membership, run policy, task privacy)
and `buildRunOutputSilence` (two watchdog decision reads and one
evaluation-task read). The instance-admin and membership reads repeat
the same answer for every run. Measured locally with a statement and
allocation harness:

| Live runs | Statements per request | Allocated per request |
| --- | --- | --- |
| 10 | 88 | 10.0 MB |
| 50 | 408 | 27.9 MB |
| 200 (route returns 50) | 408 | 28.6 MB |

**Proposed behavior**
The route reads every access decision in one batch and every silence
summary in one batch, so the statement count does not grow with the
number of runs.

**Reason and benefit**
Fewer statements and less allocation on a route that the board calls
often.

**Breaking changes**
None. The response is the same.

## What Changed

- `server/src/routes/agents.ts`: the live-runs handler decides reads
with the existing `getHeartbeatRunReadDecisions` (the run history list
already uses it) and builds silence summaries with one
`buildRunOutputSilences` call for the readable runs. Unreadable runs are
still redacted with `redactHeartbeatRunListRow`, and no silence state is
read for them.
- Active-run watchdog Postgres adapter: new `findLatestDecisions` and
`findOpenStaleRunEvaluations` read all runs of one company in one
statement each (`DISTINCT ON`). The single-run `findLatestDecision` and
`findOpenStaleRunEvaluation` now call the batch with one run, so both
paths use the same query.
- Watchdog application: new `createBuildRunOutputSilences` use case
groups runs by company. `createBuildRunOutputSilence` now calls it with
one run, so the single and batch summaries use one rule.
- `recoveryService` and `heartbeatService` expose
`buildRunOutputSilences` as pass-throughs.
- Tests: new `live-runs-batched-reads.test.ts`; updated watchdog
use-case tests and the live-run route unit test.

## Verification

- `cd server && npx vitest run
src/__tests__/live-runs-batched-reads.test.ts` (embedded Postgres):
- The batched read decision equals one `canActorReadHeartbeatRun` call
per run for 24 actor kinds (owner, admin, operator, viewer, member,
board key, cloud tenant, instance admin, local implicit board, board
without membership, agent keys, agent runs on behalf of users, low-trust
agents, agent of another company, unauthenticated). The runs mix open
tasks, private tasks readable by grant, creation or assignment, private
tasks nobody may read, private-project tasks, a hidden task,
company-scoped runs, a run whose task was deleted, runs of another
company, and runs of another company asked for as this company's.
- For every actor that passes the company gate, each live-runs row is
either the full row or exactly `redactHeartbeatRunListRow` of it (field
by field), as one check per run decides.
- Output silence per run equals the single-run reader for: no output
yet, active snooze plus a later continue, expired snooze, dismissed
false positive, recent output, open evaluation task, done evaluation
task, queued run, an evaluation task of another company, and a run that
has no rows at all.
- The statement count is the same with 19 and 49 live runs for 6 actor
kinds. Before the fix this test fails (for example 347 against 107
statements).
- `npx vitest run src/modules/active-run-watchdog
src/__tests__/heartbeat-active-run-output-watchdog.test.ts
src/__tests__/agent-live-run-routes.test.ts
src/__tests__/heartbeat-run-privacy-routes.test.ts`: pass.
- `npx tsc --noEmit -p server/tsconfig.json` and a test-tree typecheck
of the changed tests: pass. `pnpm check:module-boundaries`: pass.
- After the fix, measured with the same harness (embedded Postgres, 400
tasks, 40 agents, 5 requests each):

| Live runs | Statements per request | Allocated per request |
| --- | --- | --- |
| 10 | 15 | 6.4 MB |
| 50 | 15 | 7.3 MB |
| 200 (route returns 50) | 15 | 7.8 MB |

## Risks

- This changes an authorization read path. The route now uses the batch
helper that the run history list and the live events socket already use.
The test compares it with the per-run check for every actor kind in the
visibility fixture.
- If several open evaluation tasks point at the same run, the batch
picks one of them. The single-run read also picked an unordered one
before.
- No schema, migration, locking or transaction change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 11:10:30 -06:00
1 parent 594f9d3ec2
commit 426e8a35dc
10 files changed
+444 -62

No files matched your search

@@ -31,7 +31,7 @@ import {
import { issueRecoveryActionService } from "../../../services/issue-recovery-actions.js";
import { RECOVERY_ORIGIN_KINDS } from "../../../services/recovery/origins.js";
import { isTerminalIssueStatus } from "../domain/policy.js";
import type { WatchdogRunReader, WatchdogWriter } from "../application/ports.js";
import type { FindLatestDecisionResult, WatchdogRunReader, WatchdogWriter } from "../application/ports.js";
import type {
EvaluationIssueSnapshot,
FoldOutcome,
@@ -159,10 +159,14 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch
return row?.pending ?? true;
}
async function findLatestDecision(companyId: string, runId: string, now: Date) {
async function findLatestDecisions(companyId: string, runIds: readonly string[], now: Date) {
const ids = [...new Set(runIds)];
const result = new Map<string, FindLatestDecisionResult>();
if (ids.length === 0) return result;
const [quietUntilRows, dismissedRows] = await Promise.all([
db
.select({
.selectDistinctOn([heartbeatRunWatchdogDecisions.runId], {
runId: heartbeatRunWatchdogDecisions.runId,
decision: heartbeatRunWatchdogDecisions.decision,
snoozedUntil: heartbeatRunWatchdogDecisions.snoozedUntil,
})
@@ -170,32 +174,39 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch
.where(
and(
eq(heartbeatRunWatchdogDecisions.companyId, companyId),
eq(heartbeatRunWatchdogDecisions.runId, runId),
inArray(heartbeatRunWatchdogDecisions.runId, ids),
inArray(heartbeatRunWatchdogDecisions.decision, ["snooze", "continue"]),
gt(heartbeatRunWatchdogDecisions.snoozedUntil, now),
),
)
.orderBy(desc(heartbeatRunWatchdogDecisions.createdAt))
.limit(1),
.orderBy(heartbeatRunWatchdogDecisions.runId, desc(heartbeatRunWatchdogDecisions.createdAt)),
db
.select({ id: heartbeatRunWatchdogDecisions.id })
.selectDistinct({ runId: heartbeatRunWatchdogDecisions.runId })
.from(heartbeatRunWatchdogDecisions)
.where(
and(
eq(heartbeatRunWatchdogDecisions.companyId, companyId),
eq(heartbeatRunWatchdogDecisions.runId, runId),
inArray(heartbeatRunWatchdogDecisions.runId, ids),
eq(heartbeatRunWatchdogDecisions.decision, "dismissed_false_positive"),
),
)
.limit(1),
),
]);
const quietUntilRow = quietUntilRows[0];
return {
dismissedFalsePositive: dismissedRows.length > 0,
quietUntilDecision: quietUntilRow && quietUntilRow.snoozedUntil
? { decision: quietUntilRow.decision as "snooze" | "continue", snoozedUntil: quietUntilRow.snoozedUntil }
: null,
};
const quietUntilByRun = new Map(quietUntilRows.map((row) => [row.runId, row]));
const dismissed = new Set(dismissedRows.map((row) => row.runId));
for (const runId of ids) {
const quietUntilRow = quietUntilByRun.get(runId);
result.set(runId, {
dismissedFalsePositive: dismissed.has(runId),
quietUntilDecision: quietUntilRow && quietUntilRow.snoozedUntil
? { decision: quietUntilRow.decision as "snooze" | "continue", snoozedUntil: quietUntilRow.snoozedUntil }
: null,
});
}
return result;
}
async function findLatestDecision(companyId: string, runId: string, now: Date): Promise<FindLatestDecisionResult> {
return (await findLatestDecisions(companyId, [runId], now)).get(runId)!;
}
function selectEvaluationIssueSnapshot() {
@@ -211,21 +222,29 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch
};
}
async function findOpenStaleRunEvaluation(companyId: string, runId: string): Promise<EvaluationIssueSnapshot | null> {
const [row] = await db
.select(selectEvaluationIssueSnapshot())
async function findOpenStaleRunEvaluations(companyId: string, runIds: readonly string[]) {
const ids = [...new Set(runIds)];
const result = new Map<string, EvaluationIssueSnapshot>();
if (ids.length === 0) return result;
const rows = await db
.selectDistinctOn([issues.originId], selectEvaluationIssueSnapshot())
.from(issues)
.where(
and(
eq(issues.companyId, companyId),
eq(issues.originKind, STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND),
eq(issues.originId, runId),
inArray(issues.originId, ids),
visibleIssueCondition(),
notInArray(issues.status, ["done", "cancelled"]),
),
)
.limit(1);
return row ?? null;
.orderBy(issues.originId);
for (const row of rows) if (row.originId) result.set(row.originId, row);
return result;
}
async function findOpenStaleRunEvaluation(companyId: string, runId: string): Promise<EvaluationIssueSnapshot | null> {
return (await findOpenStaleRunEvaluations(companyId, [runId])).get(runId) ?? null;
}
async function findEvaluationIssueById(companyId: string, issueId: string): Promise<EvaluationIssueSnapshot | null> {
@@ -532,7 +551,9 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch
findRunForCompany,
isRuntimeSettlementPending,
findLatestDecision,
findLatestDecisions,
findOpenStaleRunEvaluation,
findOpenStaleRunEvaluations,
findEvaluationIssueById,
findLatestSameRunTerminalEvidence,
findSourceIssue,
@@ -56,7 +56,9 @@ export interface WatchdogRunReader {
findRunForCompany(companyId: string, runId: string): Promise<RunSnapshot | null>;
isRuntimeSettlementPending(companyId: string, runId: string): Promise<boolean>;
findLatestDecision(companyId: string, runId: string, now: Date): Promise<FindLatestDecisionResult>;
findLatestDecisions(companyId: string, runIds: readonly string[], now: Date): Promise<Map<string, FindLatestDecisionResult>>;
findOpenStaleRunEvaluation(companyId: string, runId: string): Promise<EvaluationIssueSnapshot | null>;
findOpenStaleRunEvaluations(companyId: string, runIds: readonly string[]): Promise<Map<string, EvaluationIssueSnapshot>>;
findLatestSameRunTerminalEvidence(
companyId: string,
input: FindLatestSameRunTerminalEvidenceInput,
@@ -1,6 +1,7 @@
import { describe, expect, it } from "vitest";
import {
createBuildRunOutputSilence,
createBuildRunOutputSilences,
createFoldSourceResolvedRun,
createRecordWatchdogDecision,
createScanSilentActiveRuns,
@@ -35,18 +36,29 @@ describe("application", () => {
}
function makeReader(overrides: Partial<WatchdogRunReader> = {}): WatchdogRunReader {
return {
const reader: WatchdogRunReader = {
findCandidateSilentRuns: async () => [],
findRunForCompany: async () => null,
isRuntimeSettlementPending: async () => false,
findLatestDecision: async () => ({ quietUntilDecision: null, dismissedFalsePositive: false }),
findLatestDecisions: async (companyId, runIds, now) => new Map(await Promise.all(runIds.map(async (runId) =>
[runId, await reader.findLatestDecision(companyId, runId, now)] as const))),
findOpenStaleRunEvaluation: async () => null,
findOpenStaleRunEvaluations: async (companyId, runIds) => {
const found = new Map();
for (const runId of runIds) {
const evaluation = await reader.findOpenStaleRunEvaluation(companyId, runId);
if (evaluation) found.set(runId, evaluation);
}
return found;
},
findLatestSameRunTerminalEvidence: async () => null,
findSourceIssue: async () => null,
findRunningAgent: async () => null,
findEvaluationIssueById: async () => null,
...overrides,
};
return reader;
}
describe("createBuildRunOutputSilence", () => {
@@ -105,6 +117,33 @@ describe("application", () => {
expect(summary).toMatchObject({ level: "snoozed", snoozedUntil });
});
it("reads decisions and evaluations once per company for a batch, defaulting runs the reader omits", async () => {
const now = new Date("2026-01-01T05:00:00.000Z");
const calls: string[] = [];
const reader = makeReader({
findLatestDecisions: async (companyId, runIds) => {
calls.push(`decisions ${companyId} ${runIds.join(",")}`);
return new Map();
},
findOpenStaleRunEvaluations: async (companyId, runIds) => {
calls.push(`evaluations ${companyId} ${runIds.join(",")}`);
return new Map();
},
});
const deps = { reader, suspicionThresholdMs: SUSPICION_THRESHOLD_MS, criticalThresholdMs: CRITICAL_THRESHOLD_MS };
const runs = [makeRun({ id: "run-1" }), makeRun({ id: "run-2", companyId: "company-2" }), makeRun({ id: "run-3" })];
const summaries = await createBuildRunOutputSilences(deps)(runs, now);
expect(calls.sort()).toEqual([
"decisions company-1 run-1,run-3", "decisions company-2 run-2",
"evaluations company-1 run-1,run-3", "evaluations company-2 run-2",
]);
const single = await createBuildRunOutputSilence({ ...deps, reader: makeReader() })(makeRun(), now);
expect(summaries).toEqual([single, single, single]);
expect(single).toMatchObject({ level: "critical", snoozedUntil: null, evaluationIssueId: null, lastOutputAt: null, lastOutputSeq: 0 });
});
it("reports not_applicable for a run that is not running", async () => {
const now = new Date("2026-01-01T05:00:00.000Z");
const run = makeRun({ status: "succeeded" });
@@ -31,43 +31,68 @@ export type BuildRunOutputSilenceRunInput = Pick<
"id" | "companyId" | "status" | "lastOutputAt" | "lastOutputSeq" | "lastOutputStream" | "processStartedAt" | "startedAt" | "createdAt"
>;
export function createBuildRunOutputSilences(deps: BuildRunOutputSilenceDeps) {
return async function buildRunOutputSilences(
runs: readonly BuildRunOutputSilenceRunInput[],
now: Date,
): Promise<RunOutputSilenceSummary[]> {
const runIdsByCompany = new Map<string, string[]>();
for (const run of runs) {
const runIds = runIdsByCompany.get(run.companyId) ?? [];
runIds.push(run.id);
runIdsByCompany.set(run.companyId, runIds);
}
const stateByCompany = new Map(await Promise.all([...runIdsByCompany].map(async ([companyId, runIds]) => {
const [decisions, evaluations] = await Promise.all([
deps.reader.findLatestDecisions(companyId, runIds, now),
deps.reader.findOpenStaleRunEvaluations(companyId, runIds),
]);
return [companyId, { decisions, evaluations }] as const;
})));
return runs.map((run) => {
const state = stateByCompany.get(run.companyId);
const { dismissedFalsePositive, quietUntilDecision } = state?.decisions.get(run.id)
?? { dismissedFalsePositive: false, quietUntilDecision: null };
const evaluation = state?.evaluations.get(run.id) ?? null;
const isRunningRun = run.status === "running";
const silenceAgeMsValue = isRunningRun ? silenceAgeMs(run, now) : null;
const level = classifySilenceLevel({
isRunningRun,
silenceAgeMs: silenceAgeMsValue,
dismissedFalsePositive,
snoozed: Boolean(quietUntilDecision),
suspicionThresholdMs: deps.suspicionThresholdMs,
criticalThresholdMs: deps.criticalThresholdMs,
});
return {
lastOutputAt: run.lastOutputAt ?? null,
lastOutputSeq: run.lastOutputSeq ?? 0,
lastOutputStream: run.lastOutputStream === "stdout" || run.lastOutputStream === "stderr"
? run.lastOutputStream
: null,
silenceStartedAt: silenceStartedAt(run),
silenceAgeMs: silenceAgeMsValue,
level,
suspicionThresholdMs: deps.suspicionThresholdMs,
criticalThresholdMs: deps.criticalThresholdMs,
snoozedUntil: dismissedFalsePositive ? null : quietUntilDecision?.snoozedUntil ?? null,
evaluationIssueId: evaluation?.id ?? null,
evaluationIssueIdentifier: evaluation?.identifier ?? null,
evaluationIssueAssigneeAgentId: evaluation?.assigneeAgentId ?? null,
};
});
};
}
export function createBuildRunOutputSilence(deps: BuildRunOutputSilenceDeps) {
const buildRunOutputSilences = createBuildRunOutputSilences(deps);
return async function buildRunOutputSilence(
run: BuildRunOutputSilenceRunInput,
now: Date,
): Promise<RunOutputSilenceSummary> {
const [decisionState, evaluation] = await Promise.all([
deps.reader.findLatestDecision(run.companyId, run.id, now),
deps.reader.findOpenStaleRunEvaluation(run.companyId, run.id),
]);
const { dismissedFalsePositive, quietUntilDecision } = decisionState;
const isRunningRun = run.status === "running";
const silenceAgeMsValue = isRunningRun ? silenceAgeMs(run, now) : null;
const level = classifySilenceLevel({
isRunningRun,
silenceAgeMs: silenceAgeMsValue,
dismissedFalsePositive,
snoozed: Boolean(quietUntilDecision),
suspicionThresholdMs: deps.suspicionThresholdMs,
criticalThresholdMs: deps.criticalThresholdMs,
});
return {
lastOutputAt: run.lastOutputAt ?? null,
lastOutputSeq: run.lastOutputSeq ?? 0,
lastOutputStream: run.lastOutputStream === "stdout" || run.lastOutputStream === "stderr"
? run.lastOutputStream
: null,
silenceStartedAt: silenceStartedAt(run),
silenceAgeMs: silenceAgeMsValue,
level,
suspicionThresholdMs: deps.suspicionThresholdMs,
criticalThresholdMs: deps.criticalThresholdMs,
snoozedUntil: dismissedFalsePositive ? null : quietUntilDecision?.snoozedUntil ?? null,
evaluationIssueId: evaluation?.id ?? null,
evaluationIssueIdentifier: evaluation?.identifier ?? null,
evaluationIssueAssigneeAgentId: evaluation?.assigneeAgentId ?? null,
};
const [summary] = await buildRunOutputSilences([run], now);
return summary!;
};
}
@@ -4,6 +4,7 @@ import { createPostgresWatchdogAdapter } from "./adapters/postgres.js";
import { createProcessAdapter } from "./adapters/process.js";
import {
createBuildRunOutputSilence,
createBuildRunOutputSilences,
createFoldSourceResolvedRun,
createRecordWatchdogDecision,
createScanSilentActiveRuns,
@@ -42,6 +43,11 @@ export function createActiveRunWatchdog(db: Db, config: ActiveRunWatchdogConfig)
suspicionThresholdMs: config.suspicionThresholdMs,
criticalThresholdMs: config.criticalThresholdMs,
}),
buildRunOutputSilences: createBuildRunOutputSilences({
reader: postgresAdapter,
suspicionThresholdMs: config.suspicionThresholdMs,
criticalThresholdMs: config.criticalThresholdMs,
}),
scanSilentActiveRuns,
recordWatchdogDecision: createRecordWatchdogDecision({
reader: postgresAdapter,