perf(recovery): stop counting a run start as a change to every task of its agent (#21831)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues. The `stranded-assignments`
queue checks assigned tasks that have no live run. One check costs about
22 database statements.
> - #21729 made the 5-minute backstop re-check only tasks whose own rows
changed. A change to the assignee agent's row counts as a change to
every task of that agent. #21765 left out the agent's heartbeat writes.
> - Execution start in `heartbeat.ts` writes `agents.status = 'running'`
and `updated_at`, but not `last_heartbeat_at`. So each run start still
makes every task of that agent "changed". The backstop window overlaps
the previous one by 5 minutes, so each run start re-checks those tasks
in two windows.
> - On an almost idle hosted instance with one live run and a new run
every few minutes, this gives bursts of about 1,450 checks for about 855
settled tasks, every few minutes.
> - This pull request stops counting an agent row whose current status
is `running` as a change to its tasks.
> - The benefit is no re-checks of settled tasks beside a live run: 540
backstop checks in 30 simulated minutes drop to 6 (only the live task
itself), and statements drop from 15,280 to 3,984.

## Linked Issues or Issue Description

Refs #21729, #21765, #21795

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The changed-tasks backstop of the `stranded-assignments` and
`assigned-work-invariant` recovery queues
(`server/src/services/recovery/changed-tasks.ts`).

**Subsystem affected**

Server: issue recovery background work.

**Current behavior**

On staging at near idle (one live native run, about six new runs in 30
minutes), the stranded-check statement bundle runs in bursts of about 10
checks per second for about 2.5 minutes, then stops, then starts again
about 4 minutes later. The tasks do not change. The cause: every run
start writes the agent row without a heartbeat time. The backstop then
loads every open task of that agent as "changed", in two consecutive
windows.

**Proposed behavior**

An agent row counts as a change to its tasks only when the agent is not
`running`. A pause, a resume, a hold or a configuration change of an
idle agent still counts. A run's own task still gets its run hints and
an immediate targeted check.

**Reason and benefit**

Recovery work becomes proportional to activity, not to the number of
tasks held by an agent that just started a run.

## What Changed

- `server/src/services/recovery/changed-tasks.ts`: the assignee-agent
clause of `changedTaskCondition` now requires `ag.status <> 'running'`.
A comment explains why run lifecycle writes are left out.
- `server/src/__tests__/issue-recovery-live-run-agent-census.test.ts`
(new): a staging-shaped census on the embedded test database with the
real recovery workers and a fake clock. It seeds 800 settled in_review
tasks with a succeeded latest run (3 workers, about 266 tasks each), 30
in_progress and 30 todo tasks without live runs, and one live native
run. The run's agent gets the execution-start write, and the run
publishes a run hint every 10 seconds for 30 simulated minutes. It also
has three correctness guards: a task of the live agent whose own row
changes without a hint is checked within the backstop bound; a resumed
agent's task is checked within the backstop bound when the resume sends
no hint; and the hourly full pass still checks every settled task while
the run stays live.
- `server/src/__tests__/issue-recovery-deadlines.test.ts`: a fast test
that a run-start write is not a change and a pause during the run is a
change.

## Verification

Red on `origin/master` (fix reverted), green with the fix:

| 30 simulated minutes beside one live run | before | after |
| --- | --- | --- |
| stranded checks, backstop wake | 540 | 6 |
| stranded checks, hint wake (the live task's own run hints) | 178 | 174
|
| checks of settled tasks | 534 | 0 |
| database statements | 15,280 | 3,984 |
| work signals beyond the run's own hints | 0 | 0 |

- `npx vitest run
src/__tests__/issue-recovery-live-run-agent-census.test.ts` (4 tests):
the census fails on master with `expected 534 to be less than or equal
to 50` and passes with the fix. The three guards pass.
- `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts -t "run
start"`: fails on master, passes with the fix.
- `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts
src/__tests__/issue-recovery-idle-backstop.test.ts
src/__tests__/issue-recovery-activity-hints.test.ts
src/__tests__/issue-recovery-signal-fanout.test.ts`: 31 passed, 1
skipped.
- `tsc --noEmit -p server/tsconfig.json`: clean.

## Risks

- Low risk. The change narrows one clause of the backstop query. No
schema change.
- If an agent is resumed and then starts a run in the same window, the
resume no longer counts through the agent row. A resume already sends
its own `pause_lifted` hint (`agent-lifecycle-driver.ts`), and the
hourly full pass still covers every task. Before this change, a run
finish already hid a resume the same way, because the finish writes a
heartbeat time.
- A configuration change of an agent while it runs is caught by its next
idle write or by the hourly full pass, not by the 5-minute backstop.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-15 16:49:44 -06:00
1 parent 8299f7c389
commit da8cfda42e
3 files changed
+192

No files matched your search

@@ -240,6 +240,21 @@ describeIssueRecovery("issue recovery production producers and deadlines", () =>
await state.db.update(agents).set({ status: "idle", updatedAt: new Date(heartbeatAt.getTime() + 1000) }).where(eq(agents.id, f.agentId));
expect(await changed()).toEqual(tasks.map(row => row.id).sort());
});
it("counts an agent's run start as no change to its other tasks, and a pause during the run as a change", async () => {
const f = await fixture();
const old = new Date(Date.now() - 60 * 60_000);
await state.db.update(agents).set({ lastHeartbeatAt: old, updatedAt: old }).where(eq(agents.id, f.agentId));
const tasks = await state.db.insert(issues).values(Array.from({ length: 5 }, (_, index) => ({ companyId: f.companyId,
title: `Settled ${index}`, status: "in_review", assigneeAgentId: f.agentId, createdAt: old, updatedAt: old }))).returning({ id: issues.id });
const since = new Date(Date.now() - 5 * 60_000);
const changed = async () => (await state.db.select({ id: issues.id }).from(issues)
.where(and(eq(issues.companyId, f.companyId), changedTaskCondition(since)))).map(row => row.id).sort();
// The write of execution start in heartbeat.ts: a status flip with no heartbeat time.
await state.db.update(agents).set({ status: "running", updatedAt: new Date() }).where(eq(agents.id, f.agentId));
expect(await changed()).toEqual([]);
await state.db.update(agents).set({ status: "paused", updatedAt: new Date() }).where(eq(agents.id, f.agentId));
expect(await changed()).toEqual(tasks.map(row => row.id).sort());
});
it("resolves a run without a task to no tasks and a missing task to the whole company", async () => {
const f = await fixture();
const [taskless] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running",
@@ -0,0 +1,169 @@
import { setImmediate as immediate } from "node:timers/promises";
import { and, eq, sql } from "drizzle-orm";
import { expect, it, vi } from "vitest";
import { agents, heartbeatRuns, issues } from "@paperclipai/db";
import { workSignalDebug } from "@paperclipai/db/work-signal-debug";
import { ISSUE_RECOVERY_BACKSTOP_MS, ISSUE_RECOVERY_FULL_BACKSTOP_MS, notifyIssueGraphRecovery }
from "../services/issue-recovery-notifications.js";
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
import { seedRecoveryPopulation } from "./helpers/recovery-population.js";
vi.hoisted(() => { process.env.PAPERCLIP_DEBUG_WORK_SIGNALS = "stranded-assignments:1,assigned-work-invariant:1"; });
const MINUTE = 60_000;
const TOPICS = new Set(["stranded-assignments", "assigned-work-invariant"]);
type Recorded = { at: number; topic: string; name: string; fields: Record<string, unknown> };
/**
* Staging shape: about 860 settled assigned tasks, most of them held by the
* same few agents, and one agent whose native run stays live for the whole
* window, publishing a run hint every 10 seconds. Nothing about the settled
* tasks changes, so the recovery queues must not re-check them.
*/
describeIssueRecovery("stranded-task checks beside one long live run (census)", () => {
const state = issueRecoveryFixture({ meter: true });
async function advance(ms: number, onTick?: () => Promise<unknown>) {
const until = Date.now() + ms;
let nextTick = Date.now();
while (Date.now() < until) {
if (onTick && Date.now() >= nextTick) { await onTick(); nextTick = Date.now() + 10_000; }
await vi.advanceTimersByTimeAsync(Math.min(1000, until - Date.now()));
await Promise.allSettled([...state.passes]); await immediate();
}
}
function record() {
const events: Recorded[] = [];
vi.spyOn(workSignalDebug, "event").mockImplementation((topic, name, fields = {}) => {
if (TOPICS.has(topic)) events.push({ at: Date.now(), topic, name, fields });
});
vi.spyOn(workSignalDebug, "signal").mockImplementation(topic => { events.push({ at: Date.now(), topic, name: "signal", fields: {} }); });
return events;
}
const checksOf = (events: Recorded[], topic: string) => events.filter(event => event.topic === topic && event.name === "check");
function summarize(events: Recorded[]) {
const byWake = (topic: string) => {
const counts: Record<string, number> = {};
for (const event of checksOf(events, topic)) counts[String(event.fields.wake)] = (counts[String(event.fields.wake)] ?? 0) + 1;
return counts;
};
return { stranded: checksOf(events, "stranded-assignments").length, assignedWork: checksOf(events, "assigned-work-invariant").length,
strandedByWake: byWake("stranded-assignments"), assignedWorkByWake: byWake("assigned-work-invariant") };
}
async function seed() {
const f = await state.fixture();
// 12 agents leave 3 workers, so each holds about 266 of the settled reviews.
const population = await seedRecoveryPopulation(state.db, f.companyId, { agents: 12, inReview: 800, inProgress: 30, todo: 30 });
const liveAgentId = population.agentIds[7]!;
const [liveTask] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Live native task", status: "in_progress",
assigneeAgentId: liveAgentId }).returning({ id: issues.id });
// A paused agent with one waiting task, both settled long before the window.
const old = new Date(Date.now() - 2 * 60 * MINUTE);
const [held] = await state.db.insert(agents).values({ companyId: f.companyId, name: "Held worker", role: "engineer", status: "paused",
lifecycleHolds: ["manual"], adapterType: "process", reportsTo: population.agentIds[1], createdAt: old, updatedAt: old,
adapterConfig: { command: process.execPath, args: ["-e", "0"], cwd: process.cwd() },
runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: true } } }).returning({ id: agents.id });
const [waiting] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Waits for its agent", status: "todo",
assigneeAgentId: held!.id, createdAt: old, updatedAt: old }).returning({ id: issues.id });
return { ...f, population, liveAgentId, liveTaskId: liveTask!.id, heldAgentId: held!.id, waitingTaskId: waiting!.id };
}
async function boot() {
const f = await seed();
const events = record();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
await state.start({});
await advance(10 * MINUTE);
events.length = 0;
return { ...f, events };
}
/** What execution start writes (heartbeat.ts): the run, then the agent's
* status flip to running without a heartbeat time. */
async function startLiveRun(f: Awaited<ReturnType<typeof boot>>) {
const [run] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.liveAgentId, status: "running",
runtimeMode: "native", startedAt: new Date(), contextSnapshot: { issueId: f.liveTaskId }, nativeIssueId: f.liveTaskId })
.returning({ id: heartbeatRuns.id });
await state.db.update(issues).set({ executionRunId: run!.id }).where(eq(issues.id, f.liveTaskId));
await state.db.update(agents).set({ status: "running", updatedAt: new Date() }).where(eq(agents.id, f.liveAgentId));
const runId = run!.id;
return {
runId,
/** A run event: the run row moves and its writers publish a run hint. */
tick: () => state.db.transaction(async tx => {
await tx.update(heartbeatRuns).set({ updatedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
await notifyIssueGraphRecovery(tx, { companyId: f.companyId, runId });
}),
};
}
it("does not re-check the live agent's settled tasks while its run stays live", async () => {
const f = await boot();
const live = await startLiveRun(f);
let ticks = 0;
const before = state.meter.read();
await advance(30 * MINUTE, () => { ticks += 1; return live.tick(); });
const after = state.meter.read();
// Each run event publishes one signal to each of the 7 recovery queues; anything above that is the sweep's own.
const signals = f.events.filter(event => event.name === "signal").length;
const result = { ...summarize(f.events), statements: after.statements - before.statements, ticks, signals,
signalsBeyondRunEvents: signals - ticks * 7 };
console.info(`live-run census (30 min): ${JSON.stringify(result)}`);
const settled = new Set([...f.population.inReview, ...f.population.inProgress, ...f.population.todo, f.waitingTaskId]);
const settledChecks = f.events.filter(event => event.name === "check" && settled.has(String(event.fields.issueId)));
// Before the fix: 534 settled checks and 540 backstop checks. After: 0 and 6 (the live task itself).
expect(settledChecks.length).toBe(0);
expect(result.strandedByWake.backstop ?? 0).toBeLessThanOrEqual(10);
// The live task's own run hints: about one check per run event.
expect(result.strandedByWake.hint ?? 0).toBeLessThanOrEqual(ticks + 10);
expect(result.assignedWork).toBeLessThanOrEqual(10);
expect(result.signalsBeyondRunEvents).toBe(0);
}, 1_800_000);
it("still checks a task of the live agent within the backstop bound when its own rows change without a hint", async () => {
const f = await boot();
const live = await startLiveRun(f);
await advance(6 * MINUTE, live.tick);
f.events.length = 0;
const target = f.population.inReview.find((_, index) => index % 3 === 0)!;
const [current] = await state.db.select({ assignee: issues.assigneeAgentId }).from(issues).where(eq(issues.id, target));
expect(current!.assignee).toBe(f.liveAgentId);
const changedAt = Date.now();
// A direct write that sends no hint (for example an edit by an older server).
await state.db.update(issues).set({ status: "todo", updatedAt: new Date() }).where(eq(issues.id, target));
await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, live.tick);
const checked = f.events.find(event => event.topic === "stranded-assignments" && event.name === "check" && event.fields.issueId === target);
expect(checked).toBeDefined();
expect(checked!.at - changedAt).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE);
}, 1_800_000);
it("still checks a resumed agent's task within the backstop bound when the resume sends no hint", async () => {
const f = await boot();
const live = await startLiveRun(f);
await advance(6 * MINUTE, live.tick);
f.events.length = 0;
const waiting = f.waitingTaskId;
const changedAt = Date.now();
await state.db.update(agents).set({ status: "idle", lifecycleHolds: [], updatedAt: new Date() })
.where(eq(agents.id, f.heldAgentId));
await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, live.tick);
const checked = f.events.find(event => event.topic === "stranded-assignments" && event.name === "check" && event.fields.issueId === waiting);
expect(checked).toBeDefined();
expect(checked!.at - changedAt).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE);
}, 1_800_000);
it("still re-checks every settled task in the hourly full pass", async () => {
const f = await boot();
const live = await startLiveRun(f);
await advance(ISSUE_RECOVERY_FULL_BACKSTOP_MS + 5 * MINUTE, live.tick);
const checked = new Set(checksOf(f.events, "stranded-assignments").map(event => String(event.fields.issueId)));
const missing = f.population.inReview.filter(id => !checked.has(id));
expect(missing).toEqual([]);
const runs = await state.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, f.companyId), sql`${heartbeatRuns.status} = 'running'`));
expect(runs.map(run => run.id)).toContain(live.runId);
}, 1_800_000);
});
@@ -1,6 +1,13 @@
import { sql } from "drizzle-orm";
import { issues } from "@paperclipai/db";
/** Tasks whose own rows changed since `since`, for the backstop between full
* passes. An assignee agent's row counts only for a change that can make its
* tasks need recovery (a pause, resume, hold or configuration change). Run
* lifecycle writes are left out: a run's finish records a heartbeat, and a run's
* start flips the agent to running, which makes none of its other tasks
* stranded. Every run start would otherwise re-check all of the agent's tasks
* for two backstop windows. */
export function changedTaskCondition(since: Date) {
const at = sql`${since.toISOString()}::timestamptz`;
return sql`(${issues.updatedAt} >= ${at}
@@ -15,5 +22,6 @@ export function changedTaskCondition(since: Date) {
or exists (select 1 from issue_comments c where c.company_id = ${issues.companyId}
and c.issue_id = ${issues.id} and c.created_at >= ${at})
or exists (select 1 from agents ag where ag.id = ${issues.assigneeAgentId} and ag.updated_at >= ${at}
and ag.status <> 'running'
and (ag.last_heartbeat_at is null or ag.last_heartbeat_at < ag.updated_at)))`;
}