From da8cfda42e499e0c3a5206ef44b7d67407022585 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 15 Nov 2026 16:49:44 -0600 Subject: [PATCH] perf(recovery): stop counting a run start as a change to every task of its agent (#21831) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues. The `stranded-assignments` queue checks assigned tasks that have no live run. One check costs about 22 database statements. > - #21729 made the 5-minute backstop re-check only tasks whose own rows changed. A change to the assignee agent's row counts as a change to every task of that agent. #21765 left out the agent's heartbeat writes. > - Execution start in `heartbeat.ts` writes `agents.status = 'running'` and `updated_at`, but not `last_heartbeat_at`. So each run start still makes every task of that agent "changed". The backstop window overlaps the previous one by 5 minutes, so each run start re-checks those tasks in two windows. > - On an almost idle hosted instance with one live run and a new run every few minutes, this gives bursts of about 1,450 checks for about 855 settled tasks, every few minutes. > - This pull request stops counting an agent row whose current status is `running` as a change to its tasks. > - The benefit is no re-checks of settled tasks beside a live run: 540 backstop checks in 30 simulated minutes drop to 6 (only the live task itself), and statements drop from 15,280 to 3,984. ## Linked Issues or Issue Description Refs #21729, #21765, #21795 No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The changed-tasks backstop of the `stranded-assignments` and `assigned-work-invariant` recovery queues (`server/src/services/recovery/changed-tasks.ts`). **Subsystem affected** Server: issue recovery background work. **Current behavior** On staging at near idle (one live native run, about six new runs in 30 minutes), the stranded-check statement bundle runs in bursts of about 10 checks per second for about 2.5 minutes, then stops, then starts again about 4 minutes later. The tasks do not change. The cause: every run start writes the agent row without a heartbeat time. The backstop then loads every open task of that agent as "changed", in two consecutive windows. **Proposed behavior** An agent row counts as a change to its tasks only when the agent is not `running`. A pause, a resume, a hold or a configuration change of an idle agent still counts. A run's own task still gets its run hints and an immediate targeted check. **Reason and benefit** Recovery work becomes proportional to activity, not to the number of tasks held by an agent that just started a run. ## What Changed - `server/src/services/recovery/changed-tasks.ts`: the assignee-agent clause of `changedTaskCondition` now requires `ag.status <> 'running'`. A comment explains why run lifecycle writes are left out. - `server/src/__tests__/issue-recovery-live-run-agent-census.test.ts` (new): a staging-shaped census on the embedded test database with the real recovery workers and a fake clock. It seeds 800 settled in_review tasks with a succeeded latest run (3 workers, about 266 tasks each), 30 in_progress and 30 todo tasks without live runs, and one live native run. The run's agent gets the execution-start write, and the run publishes a run hint every 10 seconds for 30 simulated minutes. It also has three correctness guards: a task of the live agent whose own row changes without a hint is checked within the backstop bound; a resumed agent's task is checked within the backstop bound when the resume sends no hint; and the hourly full pass still checks every settled task while the run stays live. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: a fast test that a run-start write is not a change and a pause during the run is a change. ## Verification Red on `origin/master` (fix reverted), green with the fix: | 30 simulated minutes beside one live run | before | after | | --- | --- | --- | | stranded checks, backstop wake | 540 | 6 | | stranded checks, hint wake (the live task's own run hints) | 178 | 174 | | checks of settled tasks | 534 | 0 | | database statements | 15,280 | 3,984 | | work signals beyond the run's own hints | 0 | 0 | - `npx vitest run src/__tests__/issue-recovery-live-run-agent-census.test.ts` (4 tests): the census fails on master with `expected 534 to be less than or equal to 50` and passes with the fix. The three guards pass. - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts -t "run start"`: fails on master, passes with the fix. - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts src/__tests__/issue-recovery-idle-backstop.test.ts src/__tests__/issue-recovery-activity-hints.test.ts src/__tests__/issue-recovery-signal-fanout.test.ts`: 31 passed, 1 skipped. - `tsc --noEmit -p server/tsconfig.json`: clean. ## Risks - Low risk. The change narrows one clause of the backstop query. No schema change. - If an agent is resumed and then starts a run in the same window, the resume no longer counts through the agent row. A resume already sends its own `pause_lifted` hint (`agent-lifecycle-driver.ts`), and the hourly full pass still covers every task. Before this change, a run finish already hid a resume the same way, because the finish writes a heartbeat time. - A configuration change of an agent while it runs is caught by its next idle write or by the hourly full pass, not by the 5-minute backstop. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../issue-recovery-deadlines.test.ts | 15 ++ ...sue-recovery-live-run-agent-census.test.ts | 169 ++++++++++++++++++ server/src/services/recovery/changed-tasks.ts | 8 + 3 files changed, 192 insertions(+) create mode 100644 server/src/__tests__/issue-recovery-live-run-agent-census.test.ts diff --git a/server/src/__tests__/issue-recovery-deadlines.test.ts b/server/src/__tests__/issue-recovery-deadlines.test.ts index b2344c7c4fc..f11f3f541f5 100644 --- a/server/src/__tests__/issue-recovery-deadlines.test.ts +++ b/server/src/__tests__/issue-recovery-deadlines.test.ts @@ -240,6 +240,21 @@ describeIssueRecovery("issue recovery production producers and deadlines", () => await state.db.update(agents).set({ status: "idle", updatedAt: new Date(heartbeatAt.getTime() + 1000) }).where(eq(agents.id, f.agentId)); expect(await changed()).toEqual(tasks.map(row => row.id).sort()); }); + it("counts an agent's run start as no change to its other tasks, and a pause during the run as a change", async () => { + const f = await fixture(); + const old = new Date(Date.now() - 60 * 60_000); + await state.db.update(agents).set({ lastHeartbeatAt: old, updatedAt: old }).where(eq(agents.id, f.agentId)); + const tasks = await state.db.insert(issues).values(Array.from({ length: 5 }, (_, index) => ({ companyId: f.companyId, + title: `Settled ${index}`, status: "in_review", assigneeAgentId: f.agentId, createdAt: old, updatedAt: old }))).returning({ id: issues.id }); + const since = new Date(Date.now() - 5 * 60_000); + const changed = async () => (await state.db.select({ id: issues.id }).from(issues) + .where(and(eq(issues.companyId, f.companyId), changedTaskCondition(since)))).map(row => row.id).sort(); + // The write of execution start in heartbeat.ts: a status flip with no heartbeat time. + await state.db.update(agents).set({ status: "running", updatedAt: new Date() }).where(eq(agents.id, f.agentId)); + expect(await changed()).toEqual([]); + await state.db.update(agents).set({ status: "paused", updatedAt: new Date() }).where(eq(agents.id, f.agentId)); + expect(await changed()).toEqual(tasks.map(row => row.id).sort()); + }); it("resolves a run without a task to no tasks and a missing task to the whole company", async () => { const f = await fixture(); const [taskless] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running", diff --git a/server/src/__tests__/issue-recovery-live-run-agent-census.test.ts b/server/src/__tests__/issue-recovery-live-run-agent-census.test.ts new file mode 100644 index 00000000000..94032f7f2e8 --- /dev/null +++ b/server/src/__tests__/issue-recovery-live-run-agent-census.test.ts @@ -0,0 +1,169 @@ +import { setImmediate as immediate } from "node:timers/promises"; +import { and, eq, sql } from "drizzle-orm"; +import { expect, it, vi } from "vitest"; +import { agents, heartbeatRuns, issues } from "@paperclipai/db"; +import { workSignalDebug } from "@paperclipai/db/work-signal-debug"; +import { ISSUE_RECOVERY_BACKSTOP_MS, ISSUE_RECOVERY_FULL_BACKSTOP_MS, notifyIssueGraphRecovery } + from "../services/issue-recovery-notifications.js"; +import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js"; +import { seedRecoveryPopulation } from "./helpers/recovery-population.js"; + +vi.hoisted(() => { process.env.PAPERCLIP_DEBUG_WORK_SIGNALS = "stranded-assignments:1,assigned-work-invariant:1"; }); + +const MINUTE = 60_000; +const TOPICS = new Set(["stranded-assignments", "assigned-work-invariant"]); +type Recorded = { at: number; topic: string; name: string; fields: Record }; + +/** + * Staging shape: about 860 settled assigned tasks, most of them held by the + * same few agents, and one agent whose native run stays live for the whole + * window, publishing a run hint every 10 seconds. Nothing about the settled + * tasks changes, so the recovery queues must not re-check them. + */ +describeIssueRecovery("stranded-task checks beside one long live run (census)", () => { + const state = issueRecoveryFixture({ meter: true }); + + async function advance(ms: number, onTick?: () => Promise) { + const until = Date.now() + ms; + let nextTick = Date.now(); + while (Date.now() < until) { + if (onTick && Date.now() >= nextTick) { await onTick(); nextTick = Date.now() + 10_000; } + await vi.advanceTimersByTimeAsync(Math.min(1000, until - Date.now())); + await Promise.allSettled([...state.passes]); await immediate(); + } + } + + function record() { + const events: Recorded[] = []; + vi.spyOn(workSignalDebug, "event").mockImplementation((topic, name, fields = {}) => { + if (TOPICS.has(topic)) events.push({ at: Date.now(), topic, name, fields }); + }); + vi.spyOn(workSignalDebug, "signal").mockImplementation(topic => { events.push({ at: Date.now(), topic, name: "signal", fields: {} }); }); + return events; + } + + const checksOf = (events: Recorded[], topic: string) => events.filter(event => event.topic === topic && event.name === "check"); + function summarize(events: Recorded[]) { + const byWake = (topic: string) => { + const counts: Record = {}; + for (const event of checksOf(events, topic)) counts[String(event.fields.wake)] = (counts[String(event.fields.wake)] ?? 0) + 1; + return counts; + }; + return { stranded: checksOf(events, "stranded-assignments").length, assignedWork: checksOf(events, "assigned-work-invariant").length, + strandedByWake: byWake("stranded-assignments"), assignedWorkByWake: byWake("assigned-work-invariant") }; + } + + async function seed() { + const f = await state.fixture(); + // 12 agents leave 3 workers, so each holds about 266 of the settled reviews. + const population = await seedRecoveryPopulation(state.db, f.companyId, { agents: 12, inReview: 800, inProgress: 30, todo: 30 }); + const liveAgentId = population.agentIds[7]!; + const [liveTask] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Live native task", status: "in_progress", + assigneeAgentId: liveAgentId }).returning({ id: issues.id }); + // A paused agent with one waiting task, both settled long before the window. + const old = new Date(Date.now() - 2 * 60 * MINUTE); + const [held] = await state.db.insert(agents).values({ companyId: f.companyId, name: "Held worker", role: "engineer", status: "paused", + lifecycleHolds: ["manual"], adapterType: "process", reportsTo: population.agentIds[1], createdAt: old, updatedAt: old, + adapterConfig: { command: process.execPath, args: ["-e", "0"], cwd: process.cwd() }, + runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: true } } }).returning({ id: agents.id }); + const [waiting] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Waits for its agent", status: "todo", + assigneeAgentId: held!.id, createdAt: old, updatedAt: old }).returning({ id: issues.id }); + return { ...f, population, liveAgentId, liveTaskId: liveTask!.id, heldAgentId: held!.id, waitingTaskId: waiting!.id }; + } + + async function boot() { + const f = await seed(); + const events = record(); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); + await state.start({}); + await advance(10 * MINUTE); + events.length = 0; + return { ...f, events }; + } + + /** What execution start writes (heartbeat.ts): the run, then the agent's + * status flip to running without a heartbeat time. */ + async function startLiveRun(f: Awaited>) { + const [run] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.liveAgentId, status: "running", + runtimeMode: "native", startedAt: new Date(), contextSnapshot: { issueId: f.liveTaskId }, nativeIssueId: f.liveTaskId }) + .returning({ id: heartbeatRuns.id }); + await state.db.update(issues).set({ executionRunId: run!.id }).where(eq(issues.id, f.liveTaskId)); + await state.db.update(agents).set({ status: "running", updatedAt: new Date() }).where(eq(agents.id, f.liveAgentId)); + const runId = run!.id; + return { + runId, + /** A run event: the run row moves and its writers publish a run hint. */ + tick: () => state.db.transaction(async tx => { + await tx.update(heartbeatRuns).set({ updatedAt: new Date() }).where(eq(heartbeatRuns.id, runId)); + await notifyIssueGraphRecovery(tx, { companyId: f.companyId, runId }); + }), + }; + } + + it("does not re-check the live agent's settled tasks while its run stays live", async () => { + const f = await boot(); + const live = await startLiveRun(f); + let ticks = 0; + const before = state.meter.read(); + await advance(30 * MINUTE, () => { ticks += 1; return live.tick(); }); + const after = state.meter.read(); + // Each run event publishes one signal to each of the 7 recovery queues; anything above that is the sweep's own. + const signals = f.events.filter(event => event.name === "signal").length; + const result = { ...summarize(f.events), statements: after.statements - before.statements, ticks, signals, + signalsBeyondRunEvents: signals - ticks * 7 }; + console.info(`live-run census (30 min): ${JSON.stringify(result)}`); + const settled = new Set([...f.population.inReview, ...f.population.inProgress, ...f.population.todo, f.waitingTaskId]); + const settledChecks = f.events.filter(event => event.name === "check" && settled.has(String(event.fields.issueId))); + // Before the fix: 534 settled checks and 540 backstop checks. After: 0 and 6 (the live task itself). + expect(settledChecks.length).toBe(0); + expect(result.strandedByWake.backstop ?? 0).toBeLessThanOrEqual(10); + // The live task's own run hints: about one check per run event. + expect(result.strandedByWake.hint ?? 0).toBeLessThanOrEqual(ticks + 10); + expect(result.assignedWork).toBeLessThanOrEqual(10); + expect(result.signalsBeyondRunEvents).toBe(0); + }, 1_800_000); + + it("still checks a task of the live agent within the backstop bound when its own rows change without a hint", async () => { + const f = await boot(); + const live = await startLiveRun(f); + await advance(6 * MINUTE, live.tick); + f.events.length = 0; + const target = f.population.inReview.find((_, index) => index % 3 === 0)!; + const [current] = await state.db.select({ assignee: issues.assigneeAgentId }).from(issues).where(eq(issues.id, target)); + expect(current!.assignee).toBe(f.liveAgentId); + const changedAt = Date.now(); + // A direct write that sends no hint (for example an edit by an older server). + await state.db.update(issues).set({ status: "todo", updatedAt: new Date() }).where(eq(issues.id, target)); + await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, live.tick); + const checked = f.events.find(event => event.topic === "stranded-assignments" && event.name === "check" && event.fields.issueId === target); + expect(checked).toBeDefined(); + expect(checked!.at - changedAt).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE); + }, 1_800_000); + + it("still checks a resumed agent's task within the backstop bound when the resume sends no hint", async () => { + const f = await boot(); + const live = await startLiveRun(f); + await advance(6 * MINUTE, live.tick); + f.events.length = 0; + const waiting = f.waitingTaskId; + const changedAt = Date.now(); + await state.db.update(agents).set({ status: "idle", lifecycleHolds: [], updatedAt: new Date() }) + .where(eq(agents.id, f.heldAgentId)); + await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, live.tick); + const checked = f.events.find(event => event.topic === "stranded-assignments" && event.name === "check" && event.fields.issueId === waiting); + expect(checked).toBeDefined(); + expect(checked!.at - changedAt).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE); + }, 1_800_000); + + it("still re-checks every settled task in the hourly full pass", async () => { + const f = await boot(); + const live = await startLiveRun(f); + await advance(ISSUE_RECOVERY_FULL_BACKSTOP_MS + 5 * MINUTE, live.tick); + const checked = new Set(checksOf(f.events, "stranded-assignments").map(event => String(event.fields.issueId))); + const missing = f.population.inReview.filter(id => !checked.has(id)); + expect(missing).toEqual([]); + const runs = await state.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.companyId, f.companyId), sql`${heartbeatRuns.status} = 'running'`)); + expect(runs.map(run => run.id)).toContain(live.runId); + }, 1_800_000); +}); diff --git a/server/src/services/recovery/changed-tasks.ts b/server/src/services/recovery/changed-tasks.ts index f28c55bbce8..aecbf30a0e6 100644 --- a/server/src/services/recovery/changed-tasks.ts +++ b/server/src/services/recovery/changed-tasks.ts @@ -1,6 +1,13 @@ import { sql } from "drizzle-orm"; import { issues } from "@paperclipai/db"; +/** Tasks whose own rows changed since `since`, for the backstop between full + * passes. An assignee agent's row counts only for a change that can make its + * tasks need recovery (a pause, resume, hold or configuration change). Run + * lifecycle writes are left out: a run's finish records a heartbeat, and a run's + * start flips the agent to running, which makes none of its other tasks + * stranded. Every run start would otherwise re-check all of the agent's tasks + * for two backstop windows. */ export function changedTaskCondition(since: Date) { const at = sql`${since.toISOString()}::timestamptz`; return sql`(${issues.updatedAt} >= ${at} @@ -15,5 +22,6 @@ export function changedTaskCondition(since: Date) { or exists (select 1 from issue_comments c where c.company_id = ${issues.companyId} and c.issue_id = ${issues.id} and c.created_at >= ${at}) or exists (select 1 from agents ag where ag.id = ${issues.assigneeAgentId} and ag.updated_at >= ${at} + and ag.status <> 'running' and (ag.last_heartbeat_at is null or ag.last_heartbeat_at < ag.updated_at)))`; }