mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(recovery): stop counting a run start as a change to every task of its agent (#21831)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues. The `stranded-assignments` queue checks assigned tasks that have no live run. One check costs about 22 database statements. > - #21729 made the 5-minute backstop re-check only tasks whose own rows changed. A change to the assignee agent's row counts as a change to every task of that agent. #21765 left out the agent's heartbeat writes. > - Execution start in `heartbeat.ts` writes `agents.status = 'running'` and `updated_at`, but not `last_heartbeat_at`. So each run start still makes every task of that agent "changed". The backstop window overlaps the previous one by 5 minutes, so each run start re-checks those tasks in two windows. > - On an almost idle hosted instance with one live run and a new run every few minutes, this gives bursts of about 1,450 checks for about 855 settled tasks, every few minutes. > - This pull request stops counting an agent row whose current status is `running` as a change to its tasks. > - The benefit is no re-checks of settled tasks beside a live run: 540 backstop checks in 30 simulated minutes drop to 6 (only the live task itself), and statements drop from 15,280 to 3,984. ## Linked Issues or Issue Description Refs #21729, #21765, #21795 No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The changed-tasks backstop of the `stranded-assignments` and `assigned-work-invariant` recovery queues (`server/src/services/recovery/changed-tasks.ts`). **Subsystem affected** Server: issue recovery background work. **Current behavior** On staging at near idle (one live native run, about six new runs in 30 minutes), the stranded-check statement bundle runs in bursts of about 10 checks per second for about 2.5 minutes, then stops, then starts again about 4 minutes later. The tasks do not change. The cause: every run start writes the agent row without a heartbeat time. The backstop then loads every open task of that agent as "changed", in two consecutive windows. **Proposed behavior** An agent row counts as a change to its tasks only when the agent is not `running`. A pause, a resume, a hold or a configuration change of an idle agent still counts. A run's own task still gets its run hints and an immediate targeted check. **Reason and benefit** Recovery work becomes proportional to activity, not to the number of tasks held by an agent that just started a run. ## What Changed - `server/src/services/recovery/changed-tasks.ts`: the assignee-agent clause of `changedTaskCondition` now requires `ag.status <> 'running'`. A comment explains why run lifecycle writes are left out. - `server/src/__tests__/issue-recovery-live-run-agent-census.test.ts` (new): a staging-shaped census on the embedded test database with the real recovery workers and a fake clock. It seeds 800 settled in_review tasks with a succeeded latest run (3 workers, about 266 tasks each), 30 in_progress and 30 todo tasks without live runs, and one live native run. The run's agent gets the execution-start write, and the run publishes a run hint every 10 seconds for 30 simulated minutes. It also has three correctness guards: a task of the live agent whose own row changes without a hint is checked within the backstop bound; a resumed agent's task is checked within the backstop bound when the resume sends no hint; and the hourly full pass still checks every settled task while the run stays live. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: a fast test that a run-start write is not a change and a pause during the run is a change. ## Verification Red on `origin/master` (fix reverted), green with the fix: | 30 simulated minutes beside one live run | before | after | | --- | --- | --- | | stranded checks, backstop wake | 540 | 6 | | stranded checks, hint wake (the live task's own run hints) | 178 | 174 | | checks of settled tasks | 534 | 0 | | database statements | 15,280 | 3,984 | | work signals beyond the run's own hints | 0 | 0 | - `npx vitest run src/__tests__/issue-recovery-live-run-agent-census.test.ts` (4 tests): the census fails on master with `expected 534 to be less than or equal to 50` and passes with the fix. The three guards pass. - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts -t "run start"`: fails on master, passes with the fix. - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts src/__tests__/issue-recovery-idle-backstop.test.ts src/__tests__/issue-recovery-activity-hints.test.ts src/__tests__/issue-recovery-signal-fanout.test.ts`: 31 passed, 1 skipped. - `tsc --noEmit -p server/tsconfig.json`: clean. ## Risks - Low risk. The change narrows one clause of the backstop query. No schema change. - If an agent is resumed and then starts a run in the same window, the resume no longer counts through the agent row. A resume already sends its own `pause_lifted` hint (`agent-lifecycle-driver.ts`), and the hourly full pass still covers every task. Before this change, a run finish already hid a resume the same way, because the finish writes a heartbeat time. - A configuration change of an agent while it runs is caught by its next idle write or by the hourly full pass, not by the 5-minute backstop. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8299f7c389
commit
da8cfda42e
3 files changed
+192
No files matched your search
@@ -240,6 +240,21 @@ describeIssueRecovery("issue recovery production producers and deadlines", () =>
|
||||
await state.db.update(agents).set({ status: "idle", updatedAt: new Date(heartbeatAt.getTime() + 1000) }).where(eq(agents.id, f.agentId));
|
||||
expect(await changed()).toEqual(tasks.map(row => row.id).sort());
|
||||
});
|
||||
it("counts an agent's run start as no change to its other tasks, and a pause during the run as a change", async () => {
|
||||
const f = await fixture();
|
||||
const old = new Date(Date.now() - 60 * 60_000);
|
||||
await state.db.update(agents).set({ lastHeartbeatAt: old, updatedAt: old }).where(eq(agents.id, f.agentId));
|
||||
const tasks = await state.db.insert(issues).values(Array.from({ length: 5 }, (_, index) => ({ companyId: f.companyId,
|
||||
title: `Settled ${index}`, status: "in_review", assigneeAgentId: f.agentId, createdAt: old, updatedAt: old }))).returning({ id: issues.id });
|
||||
const since = new Date(Date.now() - 5 * 60_000);
|
||||
const changed = async () => (await state.db.select({ id: issues.id }).from(issues)
|
||||
.where(and(eq(issues.companyId, f.companyId), changedTaskCondition(since)))).map(row => row.id).sort();
|
||||
// The write of execution start in heartbeat.ts: a status flip with no heartbeat time.
|
||||
await state.db.update(agents).set({ status: "running", updatedAt: new Date() }).where(eq(agents.id, f.agentId));
|
||||
expect(await changed()).toEqual([]);
|
||||
await state.db.update(agents).set({ status: "paused", updatedAt: new Date() }).where(eq(agents.id, f.agentId));
|
||||
expect(await changed()).toEqual(tasks.map(row => row.id).sort());
|
||||
});
|
||||
it("resolves a run without a task to no tasks and a missing task to the whole company", async () => {
|
||||
const f = await fixture();
|
||||
const [taskless] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running",
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
import { setImmediate as immediate } from "node:timers/promises";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import { agents, heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import { workSignalDebug } from "@paperclipai/db/work-signal-debug";
|
||||
import { ISSUE_RECOVERY_BACKSTOP_MS, ISSUE_RECOVERY_FULL_BACKSTOP_MS, notifyIssueGraphRecovery }
|
||||
from "../services/issue-recovery-notifications.js";
|
||||
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
|
||||
import { seedRecoveryPopulation } from "./helpers/recovery-population.js";
|
||||
|
||||
vi.hoisted(() => { process.env.PAPERCLIP_DEBUG_WORK_SIGNALS = "stranded-assignments:1,assigned-work-invariant:1"; });
|
||||
|
||||
const MINUTE = 60_000;
|
||||
const TOPICS = new Set(["stranded-assignments", "assigned-work-invariant"]);
|
||||
type Recorded = { at: number; topic: string; name: string; fields: Record<string, unknown> };
|
||||
|
||||
/**
|
||||
* Staging shape: about 860 settled assigned tasks, most of them held by the
|
||||
* same few agents, and one agent whose native run stays live for the whole
|
||||
* window, publishing a run hint every 10 seconds. Nothing about the settled
|
||||
* tasks changes, so the recovery queues must not re-check them.
|
||||
*/
|
||||
describeIssueRecovery("stranded-task checks beside one long live run (census)", () => {
|
||||
const state = issueRecoveryFixture({ meter: true });
|
||||
|
||||
async function advance(ms: number, onTick?: () => Promise<unknown>) {
|
||||
const until = Date.now() + ms;
|
||||
let nextTick = Date.now();
|
||||
while (Date.now() < until) {
|
||||
if (onTick && Date.now() >= nextTick) { await onTick(); nextTick = Date.now() + 10_000; }
|
||||
await vi.advanceTimersByTimeAsync(Math.min(1000, until - Date.now()));
|
||||
await Promise.allSettled([...state.passes]); await immediate();
|
||||
}
|
||||
}
|
||||
|
||||
function record() {
|
||||
const events: Recorded[] = [];
|
||||
vi.spyOn(workSignalDebug, "event").mockImplementation((topic, name, fields = {}) => {
|
||||
if (TOPICS.has(topic)) events.push({ at: Date.now(), topic, name, fields });
|
||||
});
|
||||
vi.spyOn(workSignalDebug, "signal").mockImplementation(topic => { events.push({ at: Date.now(), topic, name: "signal", fields: {} }); });
|
||||
return events;
|
||||
}
|
||||
|
||||
const checksOf = (events: Recorded[], topic: string) => events.filter(event => event.topic === topic && event.name === "check");
|
||||
function summarize(events: Recorded[]) {
|
||||
const byWake = (topic: string) => {
|
||||
const counts: Record<string, number> = {};
|
||||
for (const event of checksOf(events, topic)) counts[String(event.fields.wake)] = (counts[String(event.fields.wake)] ?? 0) + 1;
|
||||
return counts;
|
||||
};
|
||||
return { stranded: checksOf(events, "stranded-assignments").length, assignedWork: checksOf(events, "assigned-work-invariant").length,
|
||||
strandedByWake: byWake("stranded-assignments"), assignedWorkByWake: byWake("assigned-work-invariant") };
|
||||
}
|
||||
|
||||
async function seed() {
|
||||
const f = await state.fixture();
|
||||
// 12 agents leave 3 workers, so each holds about 266 of the settled reviews.
|
||||
const population = await seedRecoveryPopulation(state.db, f.companyId, { agents: 12, inReview: 800, inProgress: 30, todo: 30 });
|
||||
const liveAgentId = population.agentIds[7]!;
|
||||
const [liveTask] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Live native task", status: "in_progress",
|
||||
assigneeAgentId: liveAgentId }).returning({ id: issues.id });
|
||||
// A paused agent with one waiting task, both settled long before the window.
|
||||
const old = new Date(Date.now() - 2 * 60 * MINUTE);
|
||||
const [held] = await state.db.insert(agents).values({ companyId: f.companyId, name: "Held worker", role: "engineer", status: "paused",
|
||||
lifecycleHolds: ["manual"], adapterType: "process", reportsTo: population.agentIds[1], createdAt: old, updatedAt: old,
|
||||
adapterConfig: { command: process.execPath, args: ["-e", "0"], cwd: process.cwd() },
|
||||
runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: true } } }).returning({ id: agents.id });
|
||||
const [waiting] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Waits for its agent", status: "todo",
|
||||
assigneeAgentId: held!.id, createdAt: old, updatedAt: old }).returning({ id: issues.id });
|
||||
return { ...f, population, liveAgentId, liveTaskId: liveTask!.id, heldAgentId: held!.id, waitingTaskId: waiting!.id };
|
||||
}
|
||||
|
||||
async function boot() {
|
||||
const f = await seed();
|
||||
const events = record();
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
||||
await state.start({});
|
||||
await advance(10 * MINUTE);
|
||||
events.length = 0;
|
||||
return { ...f, events };
|
||||
}
|
||||
|
||||
/** What execution start writes (heartbeat.ts): the run, then the agent's
|
||||
* status flip to running without a heartbeat time. */
|
||||
async function startLiveRun(f: Awaited<ReturnType<typeof boot>>) {
|
||||
const [run] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.liveAgentId, status: "running",
|
||||
runtimeMode: "native", startedAt: new Date(), contextSnapshot: { issueId: f.liveTaskId }, nativeIssueId: f.liveTaskId })
|
||||
.returning({ id: heartbeatRuns.id });
|
||||
await state.db.update(issues).set({ executionRunId: run!.id }).where(eq(issues.id, f.liveTaskId));
|
||||
await state.db.update(agents).set({ status: "running", updatedAt: new Date() }).where(eq(agents.id, f.liveAgentId));
|
||||
const runId = run!.id;
|
||||
return {
|
||||
runId,
|
||||
/** A run event: the run row moves and its writers publish a run hint. */
|
||||
tick: () => state.db.transaction(async tx => {
|
||||
await tx.update(heartbeatRuns).set({ updatedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
|
||||
await notifyIssueGraphRecovery(tx, { companyId: f.companyId, runId });
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
it("does not re-check the live agent's settled tasks while its run stays live", async () => {
|
||||
const f = await boot();
|
||||
const live = await startLiveRun(f);
|
||||
let ticks = 0;
|
||||
const before = state.meter.read();
|
||||
await advance(30 * MINUTE, () => { ticks += 1; return live.tick(); });
|
||||
const after = state.meter.read();
|
||||
// Each run event publishes one signal to each of the 7 recovery queues; anything above that is the sweep's own.
|
||||
const signals = f.events.filter(event => event.name === "signal").length;
|
||||
const result = { ...summarize(f.events), statements: after.statements - before.statements, ticks, signals,
|
||||
signalsBeyondRunEvents: signals - ticks * 7 };
|
||||
console.info(`live-run census (30 min): ${JSON.stringify(result)}`);
|
||||
const settled = new Set([...f.population.inReview, ...f.population.inProgress, ...f.population.todo, f.waitingTaskId]);
|
||||
const settledChecks = f.events.filter(event => event.name === "check" && settled.has(String(event.fields.issueId)));
|
||||
// Before the fix: 534 settled checks and 540 backstop checks. After: 0 and 6 (the live task itself).
|
||||
expect(settledChecks.length).toBe(0);
|
||||
expect(result.strandedByWake.backstop ?? 0).toBeLessThanOrEqual(10);
|
||||
// The live task's own run hints: about one check per run event.
|
||||
expect(result.strandedByWake.hint ?? 0).toBeLessThanOrEqual(ticks + 10);
|
||||
expect(result.assignedWork).toBeLessThanOrEqual(10);
|
||||
expect(result.signalsBeyondRunEvents).toBe(0);
|
||||
}, 1_800_000);
|
||||
|
||||
it("still checks a task of the live agent within the backstop bound when its own rows change without a hint", async () => {
|
||||
const f = await boot();
|
||||
const live = await startLiveRun(f);
|
||||
await advance(6 * MINUTE, live.tick);
|
||||
f.events.length = 0;
|
||||
const target = f.population.inReview.find((_, index) => index % 3 === 0)!;
|
||||
const [current] = await state.db.select({ assignee: issues.assigneeAgentId }).from(issues).where(eq(issues.id, target));
|
||||
expect(current!.assignee).toBe(f.liveAgentId);
|
||||
const changedAt = Date.now();
|
||||
// A direct write that sends no hint (for example an edit by an older server).
|
||||
await state.db.update(issues).set({ status: "todo", updatedAt: new Date() }).where(eq(issues.id, target));
|
||||
await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, live.tick);
|
||||
const checked = f.events.find(event => event.topic === "stranded-assignments" && event.name === "check" && event.fields.issueId === target);
|
||||
expect(checked).toBeDefined();
|
||||
expect(checked!.at - changedAt).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE);
|
||||
}, 1_800_000);
|
||||
|
||||
it("still checks a resumed agent's task within the backstop bound when the resume sends no hint", async () => {
|
||||
const f = await boot();
|
||||
const live = await startLiveRun(f);
|
||||
await advance(6 * MINUTE, live.tick);
|
||||
f.events.length = 0;
|
||||
const waiting = f.waitingTaskId;
|
||||
const changedAt = Date.now();
|
||||
await state.db.update(agents).set({ status: "idle", lifecycleHolds: [], updatedAt: new Date() })
|
||||
.where(eq(agents.id, f.heldAgentId));
|
||||
await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, live.tick);
|
||||
const checked = f.events.find(event => event.topic === "stranded-assignments" && event.name === "check" && event.fields.issueId === waiting);
|
||||
expect(checked).toBeDefined();
|
||||
expect(checked!.at - changedAt).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE);
|
||||
}, 1_800_000);
|
||||
|
||||
it("still re-checks every settled task in the hourly full pass", async () => {
|
||||
const f = await boot();
|
||||
const live = await startLiveRun(f);
|
||||
await advance(ISSUE_RECOVERY_FULL_BACKSTOP_MS + 5 * MINUTE, live.tick);
|
||||
const checked = new Set(checksOf(f.events, "stranded-assignments").map(event => String(event.fields.issueId)));
|
||||
const missing = f.population.inReview.filter(id => !checked.has(id));
|
||||
expect(missing).toEqual([]);
|
||||
const runs = await state.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, f.companyId), sql`${heartbeatRuns.status} = 'running'`));
|
||||
expect(runs.map(run => run.id)).toContain(live.runId);
|
||||
}, 1_800_000);
|
||||
});
|
||||
@@ -1,6 +1,13 @@
|
||||
import { sql } from "drizzle-orm";
|
||||
import { issues } from "@paperclipai/db";
|
||||
|
||||
/** Tasks whose own rows changed since `since`, for the backstop between full
|
||||
* passes. An assignee agent's row counts only for a change that can make its
|
||||
* tasks need recovery (a pause, resume, hold or configuration change). Run
|
||||
* lifecycle writes are left out: a run's finish records a heartbeat, and a run's
|
||||
* start flips the agent to running, which makes none of its other tasks
|
||||
* stranded. Every run start would otherwise re-check all of the agent's tasks
|
||||
* for two backstop windows. */
|
||||
export function changedTaskCondition(since: Date) {
|
||||
const at = sql`${since.toISOString()}::timestamptz`;
|
||||
return sql`(${issues.updatedAt} >= ${at}
|
||||
@@ -15,5 +22,6 @@ export function changedTaskCondition(since: Date) {
|
||||
or exists (select 1 from issue_comments c where c.company_id = ${issues.companyId}
|
||||
and c.issue_id = ${issues.id} and c.created_at >= ${at})
|
||||
or exists (select 1 from agents ag where ag.id = ${issues.assigneeAgentId} and ag.updated_at >= ${at}
|
||||
and ag.status <> 'running'
|
||||
and (ag.last_heartbeat_at is null or ag.last_heartbeat_at < ag.updated_at)))`;
|
||||
}
|
||||
Reference in new issue
Block a user