From 83541e89ad3b90740ffc61a32c396dddb5834526 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 15 Nov 2026 05:50:59 -0600 Subject: [PATCH] perf(recovery): re-check only changed tasks between hourly full backstop passes (#21729) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues on one Node main thread. `stranded-assignments` and `assigned-work-invariant` find assigned tasks that have no live run and restart them. > - #21552 scoped run hints to the run's task graph and kept a company "backstop" pass to catch a missed hint. Any graph hint arms the backstop, and the backstop runs at most every 5 minutes after the last full pass. > - The backstop reloads and re-checks every open assigned task of the company. With any trickle of activity it runs every 5 minutes. On the hosted staging instance that is about 850 tasks every 5 minutes, about 2.8 checks per second at idle, close to the measured 3.2 per second. That is work proportional to history, not to activity. > - This pull request makes the 5-minute backstop re-check only tasks whose own rows changed since the previous look, with one set-based query. A full company pass runs at most once an hour, still armed only by activity. > - The benefit is no re-checks between hourly passes when nothing changed: idle statements drop from 59 per second to 5.5 per second in a staging-shaped fixture. A task whose own state changed without a hint is still recovered within 5 minutes. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The missed-hint backstop of the `stranded-assignments` and `assigned-work-invariant` recovery queues (`server/src/services/recovery/work.ts`). **Subsystem affected** Server: issue recovery background work. **Current behavior** Each graph hint arms a `{companyId, backstop: true}` scope. When it opens (5 minutes after the last full pass), the queue loads every open assigned task of the company and checks each one again: about 22 statements per task for the stranded check. With a hint at least every 5 minutes, this repeats every 5 minutes, whatever changed. **Proposed behavior** Between full passes, the backstop loads only the tasks whose own rows changed since the previous look (minus a 5-minute overlap for late commits): the task row, its runs, its recovery actions, its wake requests, its thread interactions, its comments, or its assignee agent. A full company pass runs when the last one is an hour old. A real change still sends its own hint and gets an immediate targeted check. **Reason and benefit** On staging at idle, the stranded check runs about 3.2 times per second with zero runs. A fixture with staging's population shape attributes all of it to the backstop. ## What Changed - `server/src/services/recovery/work.ts`: - `fullPasses()` keeps the time of the last full pass and of the last changed-tasks pass per company. The backstop gate is 5 minutes after the later of the two. - A backstop scan decides on its first page: a full load when the last full pass is at least `ISSUE_RECOVERY_FULL_BACKSTOP_MS` (1 hour) old, otherwise a changed-tasks load since the previous look minus `ISSUE_RECOVERY_BACKSTOP_MS`. All pages of one scan use the same decision. - Each scan records its start time on its first page. A completed scan advances the saved full or changed time only to that start, so a change behind the cursor of a slow scan falls in the next window. - The stranded queue adds `changedTaskCondition(since)` to its load query. The assigned-work queue passes `changedSince` to `listSuspects`. Both return `keepUnseen: true` for a changed-tasks load. - A new `fullBackstopMs` option. When a caller sets only `backstopMs` (tests), every backstop stays a full pass, as before. - With `PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments`, each `check` event now names why the task was loaded: `deadline`, `reload` (unscoped reconstruction), `backstop`, `company` (company-wide hint) or `hint` (task or graph hint). - `server/src/services/recovery/changed-tasks.ts` (new): the set-based "changed since" condition. Each part uses an existing index: `issues (company_id, updated_at)`, runs by `(company_id, context issueId)`, wakes by `(company_id, payload issueId)`, interactions and comments by `(company_id, issue_id)`. - `server/src/services/scoped-deadline-work.ts`: a load result may set `keepUnseen`. Then a completed scan does not drop the pending entries it did not load, so their saved deadlines stay. - `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects` accepts `changedSince`. - `server/src/services/issue-recovery-notifications.ts`: `ISSUE_RECOVERY_FULL_BACKSTOP_MS = 60 * 60_000`. - Tests: - `server/src/__tests__/issue-recovery-idle-backstop.test.ts` (new): the staging-shaped fixture, see Verification. - The staging-shaped population helper and the fixture's statement meter come from #21724, now merged into this branch. - `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`: the check event now names its load reason (`hint` or, when the backstop loaded the task last, `backstop`) instead of `load`. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: the backlog helper writes its rows with the test's fake clock, as production writes use the server clock. Without it, rows stamped with the database clock look older than the fake-clock window. ## Verification - Fixture (`issue-recovery-idle-backstop.test.ts`, embedded PostgreSQL, the real recovery, dispatch and retry-cancellation workers, fake clock, statement meter, `PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`): one company, 50 agents, 780 in-review tasks with succeeded native runs and no execution state, 40 in-progress and 30 to-do tasks of a paused agent, 12 tasks with an active agent-owned recovery action, 3 tasks of a terminated agent. After boot, one unrelated graph hint per minute stands in for idle activity. - Wake reasons on master over 30 idle minutes with that trickle: 5,100 stranded checks, all `backstop`; 0 from hints, 0 from company hints, 0 from deadlines. That is 2.83 checks per second and 106,632 statements (59.2 per second). With no hint at all after boot, master still made one backstop pass (850 checks) in 30 minutes, armed by boot activity. - This branch, same fixture: | | master | this branch | | --- | --- | --- | | Stranded checks, first 30 idle minutes | 5,100 | 0 | | Statements, first 30 idle minutes | 106,632 | 1,044 | | Stranded checks, one idle hour (one full pass) | about 10,200 | 850 | | Statements per second over that hour | 59.2 | 5.5 | - Missed hint: a task inserted with no hint is checked and gets a run within 5 minutes plus one pass (`ISSUE_RECOVERY_BACKSTOP_MS + 2 min` asserted). - A change outside the task's own rows (its agent resumed by a raw update that sets no `updated_at`, with no hint) is not seen by the 5-minute pass. It is checked and gets a run after more than 10 minutes and within 62 minutes, through the hourly full pass. - Prompt pickup with the backstop closed: a finished run (`notifyNativeStatusReassessment`), a user comment, a hold release (`pause_lifted`, which a budget raise and an agent resume send) and an agent resume each get a check of the affected task within 60 seconds. The company-wide events re-check the whole company at 25 tasks per pass, as before. - Two regression tests in `issue-recovery-deadlines.test.ts`, red on 379dda9e26 and green on 802a76c153: - "keeps a saved deadline of a task that a changed-tasks backstop pass does not load": a task with a 16-minute saved deadline is checked at that deadline after two changed-tasks passes that did not load it. - "starts the next changed-tasks window from the start of a slow backstop scan": each check advances the clock by 20 s, so the scan spans more than 10 minutes. A task changed behind the cursor early in the scan is read by the next changed-tasks pass. - #21552 backstop tests in `issue-recovery-deadlines.test.ts` pass, including "finds a stranded task outside the affected set within the backstop bound", which now runs with the production defaults. - Regression, embedded PostgreSQL: issue-recovery-idle-backstop, issue-recovery-deadlines, scoped-deadline-work, assigned-work-recheck-idle, live-recovery-work (98 tests; "resolves a run's task graph through index lookups" failed once under 3 parallel workers and passes alone; it does not touch this change); live-recovery-work, assigned-work-recheck-idle, assigned-work-invariant, recovery-deadline-acceptance, issue-recovery-recorded-decision-idle, continuation-delivery-unwakeable-agent-idle. - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - **The missed-hint bound changes for one class of change. The owner must see this.** A change that touches the task's own rows (task, runs, recovery actions, wakes, interactions, comments, or the assignee agent's row) and sends no hint is still recovered within 5 minutes plus one pass. A change elsewhere that sends no hint (company or budget rows, tree holds, an execution-state participant agent, or a write that does not set `updated_at`) is now recovered within 60 minutes plus one pass, where it was 5 minutes. Every such writer that I found sends a company or graph hint, which still triggers an immediate check. - The changed-tasks window trusts `updated_at` stamps. A transaction that commits more than 5 minutes after its stamp can be missed until the next full pass. - The fixture cannot run the full set of workers that `index.ts` registers. It runs the recovery, dispatch and retry-cancellation workers, and stands in for other activity with one graph hint per minute. - Not changed here: a company-wide hint (`{companyId}` with no task) still reloads and re-checks every task, at most every 30 seconds. About 20 writers publish one, for example `documents.ts`, `task-watchdogs.ts`, `issue-tree-control.ts`, `issues.ts`, `heartbeat/queue.ts` and `run-dispatch-notifications.ts` without an issue. In the fixture these came only from boot escalations, not at idle. - At boot, a graph hint that arrives before the first full reconstruction completes still opens one full backstop pass. - master (with #21724) is merged into this branch. The changed-tasks backstop and #21724's `beforeChecks` pass with its shared review-wait read work together: the focused tests of both pull requests pass on the merged tree (107 tests). On the merged tree the hour that holds the full pass reads 7.7 MB instead of 15.0 MB, because of #21724's lean reads. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local PostgreSQL test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../issue-recovery-deadlines.test.ts | 66 ++++++- .../issue-recovery-idle-backstop.test.ts | 166 ++++++++++++++++++ ...ue-recovery-work-debug-attribution.test.ts | 2 +- .../services/issue-recovery-notifications.ts | 1 + .../services/recovery/assigned-work-sweep.ts | 5 +- server/src/services/recovery/changed-tasks.ts | 18 ++ server/src/services/recovery/work.ts | 56 ++++-- server/src/services/scoped-deadline-work.ts | 7 +- 8 files changed, 298 insertions(+), 23 deletions(-) create mode 100644 server/src/__tests__/issue-recovery-idle-backstop.test.ts create mode 100644 server/src/services/recovery/changed-tasks.ts diff --git a/server/src/__tests__/issue-recovery-deadlines.test.ts b/server/src/__tests__/issue-recovery-deadlines.test.ts index f87ab9e9564..a63b1e3a983 100644 --- a/server/src/__tests__/issue-recovery-deadlines.test.ts +++ b/server/src/__tests__/issue-recovery-deadlines.test.ts @@ -1,7 +1,7 @@ import { setImmediate as immediate } from "node:timers/promises"; import { idleWorkSnapshot } from "../services/task-admission.js"; import { randomUUID } from "node:crypto"; -import { and, eq, sql } from "drizzle-orm"; +import { and, eq, inArray, sql } from "drizzle-orm"; import { expect, it, vi } from "vitest"; import { agents, budgetPolicies, heartbeatRuns, issueRecoveryActions, issueRelations, issues, issueWatchdogs } from "@paperclipai/db"; import { heartbeatService } from "../services/heartbeat.js"; @@ -113,7 +113,7 @@ describeIssueRecovery("issue recovery production producers and deadlines", () => const [run] = await state.db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", finishedAt: new Date(), contextSnapshot: { issueId: task.id } }).returning({ id: heartbeatRuns.id }); const backlog = async (count: number) => (await state.db.insert(issues).values(Array.from({ length: count }, (_, index) => ({ - companyId, title: `Backlog ${index}`, status: "todo", assigneeAgentId: other.id, + companyId, title: `Backlog ${index}`, status: "todo", assigneeAgentId: other.id, createdAt: new Date(), updatedAt: new Date(), }))).returning({ id: issues.id })).map(row => row.id); return { runId: run.id, taskId: task.id, siblingId: sibling.id, backlog, affected: [parent.id, task.id, dependent.id].sort() }; } @@ -223,6 +223,68 @@ describeIssueRecovery("issue recovery production producers and deadlines", () => expect(checkedAt.get(later)! - eventAt).toBeLessThanOrEqual(5_000); await quiet(); }); + async function advanceWithoutWaiting(ms: number) { + const until = Date.now() + ms; + while (Date.now() < until) { await vi.advanceTimersByTimeAsync(Math.min(1000, until - Date.now())); await immediate(); } + } + it("keeps a saved deadline of a task that a changed-tasks backstop pass does not load", async () => { + const f = await fixture(); + const graph = await runGraph(f.companyId, f.agentId); + const [target] = await graph.backlog(20); + const checkedAt: number[] = []; + await recordedRecovery({ stranded: async scope => { + if (scope.issueId !== target) return; + checkedAt.push(Date.now()); + scope.onDeferredDeadline(Date.now() + 16 * 60_000); + } }); + await settleWork({ backstopPending: true }); + for (let round = 0; round < 2; round++) { + await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000); + await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId })); + await settleWork({ backstopPending: true }); + } + const savedFor = checkedAt.at(-1)! + 16 * 60_000; + checkedAt.length = 0; + await vi.advanceTimersByTimeAsync(savedFor - Date.now() + 2_000); + await settleWork({ backstopPending: true }); + expect(checkedAt.length).toBeGreaterThan(0); + expect(checkedAt[0]! - savedFor).toBeLessThanOrEqual(5_000); + }); + it("starts the next changed-tasks window from the start of a slow backstop scan", async () => { + const f = await fixture(); + const graph = await runGraph(f.companyId, f.agentId); + const backlog = (await graph.backlog(250)).sort(); + const behind = backlog[0]!; + let slow = false, changedBehind = false; + const checkedAt = new Map(); + await recordedRecovery({ stranded: async scope => { + checkedAt.set(scope.issueId, [...checkedAt.get(scope.issueId) ?? [], Date.now()]); + if (!slow) return; + vi.setSystemTime(Date.now() + 20_000); + if (scope.issueId === behind && !changedBehind) { + changedBehind = true; + await state.db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, behind)); + } + } }, { fullBackstopMs: 24 * 60 * 60_000 }); + await settleWork({ backstopPending: true }); + await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000); + await state.db.update(issues).set({ updatedAt: new Date() }).where(inArray(issues.id, backlog)); + await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000); + slow = true; + checkedAt.clear(); + const scanStartedAt = Date.now(); + await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId })); + await settleWork({ backstopPending: true }); + slow = false; + expect(changedBehind).toBe(true); + expect(backlog.every(id => checkedAt.has(id))).toBe(true); + expect(Date.now() - scanStartedAt).toBeGreaterThan(2 * ISSUE_RECOVERY_BACKSTOP_MS); + checkedAt.clear(); + await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000); + await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId })); + await settleWork({ backstopPending: true }); + expect(checkedAt.has(behind)).toBe(true); + }, 300_000); it("keeps the company pass for hold releases that can admit any task", async () => { const f = await fixture(); const graph = await runGraph(f.companyId, f.agentId); diff --git a/server/src/__tests__/issue-recovery-idle-backstop.test.ts b/server/src/__tests__/issue-recovery-idle-backstop.test.ts new file mode 100644 index 00000000000..baf8fd03798 --- /dev/null +++ b/server/src/__tests__/issue-recovery-idle-backstop.test.ts @@ -0,0 +1,166 @@ +import { setImmediate as immediate } from "node:timers/promises"; +import { and, eq, sql } from "drizzle-orm"; +import { expect, it, vi } from "vitest"; +import { agents, heartbeatRuns, issues } from "@paperclipai/db"; +import { workSignalDebug } from "@paperclipai/db/work-signal-debug"; +import { ISSUE_RECOVERY_BACKSTOP_MS, ISSUE_RECOVERY_FULL_BACKSTOP_MS, notifyIssueGraphRecovery } + from "../services/issue-recovery-notifications.js"; +import { issueService } from "../services/issues.js"; +import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js"; +import { promoteDeferredWorkIntents } from "../services/work-intent-outbox.js"; +import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js"; +import { seedRecoveryPopulation } from "./helpers/recovery-population.js"; + +vi.hoisted(() => { process.env.PAPERCLIP_DEBUG_WORK_SIGNALS = "stranded-assignments:1"; }); + +const MINUTE = 60_000; +type Recorded = { at: number; name: string; fields: Record }; + +describeIssueRecovery("stranded-task checks at idle for a staging-shaped company", () => { + const state = issueRecoveryFixture({ meter: true }); + + async function advance(ms: number) { + const until = Date.now() + ms; + while (Date.now() < until) { + await vi.advanceTimersByTimeAsync(Math.min(1000, until - Date.now())); + await Promise.allSettled([...state.passes]); await immediate(); + } + } + + function record() { + const events: Recorded[] = []; + vi.spyOn(workSignalDebug, "event").mockImplementation((topic, name, fields = {}) => { + if (topic === "stranded-assignments") events.push({ at: Date.now(), name, fields }); + }); + vi.spyOn(workSignalDebug, "signal").mockImplementation(() => {}); + return events; + } + + function summarize(events: Recorded[]) { + const checks: Record = {}, loads: Record = {}; + for (const event of events) { + if (event.name === "check") checks[String(event.fields.wake)] = (checks[String(event.fields.wake)] ?? 0) + 1; + if (event.name === "load") { + const scope = String(event.fields.scope); + const kind = scope === "*" ? "reload" : scope.endsWith(",true]") ? "backstop" : /^\["[^"]+",null,null,null,null\]$/.test(scope) + ? "company" : "hint"; + loads[kind] = (loads[kind] ?? 0) + 1; + } + } + return { checks, loads, totalChecks: Object.values(checks).reduce((sum, count) => sum + count, 0) }; + } + + const checkedAt = (events: Recorded[], issueId: string) => + events.find(event => event.name === "check" && event.fields.issueId === issueId)?.at ?? null; + + async function seed() { + const f = await state.fixture(); + const population = await seedRecoveryPopulation(state.db, f.companyId, { agents: 50, inReview: 780, inProgress: 40, todo: 30, + recoveryActions: 12, terminated: 3 }); + const [side] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Finished side task", status: "done" }) + .returning({ id: issues.id }); + const activity = () => state.db.transaction(tx => notifyIssueGraphRecovery(tx, { companyId: f.companyId, issueId: side!.id })); + return { ...f, population, activity }; + } + + async function boot() { + const f = await seed(); + const events = record(); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); + await state.start({}); + await advance(10 * MINUTE); + return { ...f, events, boot: summarize(events.splice(0)) }; + } + + async function heldTask(f: Awaited>) { + const old = new Date(Date.now() - 2 * 60 * MINUTE); + const [held] = await state.db.insert(agents).values({ companyId: f.companyId, name: "Held worker", role: "engineer", status: "paused", + lifecycleHolds: ["manual"], adapterType: "process", reportsTo: f.population.agentIds[1], createdAt: old, updatedAt: old, + adapterConfig: { command: process.execPath, args: ["-e", "0"], cwd: process.cwd() }, + runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: true } } }).returning({ id: agents.id }); + const [waiting] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Waits for its agent", status: "todo", + assigneeAgentId: held!.id, createdAt: old, updatedAt: old }).returning({ id: issues.id }); + return { held: held!, waiting: waiting! }; + } + + async function trickle(f: Awaited>, minutes: number) { + const before = state.meter.read(); + for (let minute = 0; minute < minutes; minute++) { await f.activity(); await advance(MINUTE); } + const after = state.meter.read(); + const raw = f.events.splice(0); + return { ...summarize(raw), raw, statements: after.statements - before.statements, + bytes: after.bytesReceived - before.bytesReceived }; + } + + it("checks only changed tasks between hourly full passes while unrelated activity continues", async () => { + const f = await boot(); + const between = await trickle(f, 30); + const hour = await trickle(f, 30); + const { raw: _first, ...firstHalfHour } = between, { raw: _second, ...secondHalfHour } = hour; + console.info(JSON.stringify({ boot: f.boot, firstHalfHour, secondHalfHour, + perSecond: { checks: (between.totalChecks + hour.totalChecks) / 3600, statements: (between.statements + hour.statements) / 3600 } })); + expect(f.boot.totalChecks).toBeGreaterThanOrEqual(f.population.inReview.length); + expect(between.totalChecks).toBe(0); + expect(between.statements).toBeLessThan(2_000); + expect(hour.checks.backstop ?? 0).toBeLessThanOrEqual(f.population.inReview.length + 100); + }, 1_800_000); + + it("recovers a task whose hint was never sent within the backstop bound", async () => { + const f = await boot(); + const worker = f.population.agentIds[7]!; + const now = new Date(); + const [missed] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Created without a hint", status: "todo", + assigneeAgentId: worker, createdAt: now, updatedAt: now }).returning({ id: issues.id }); + const window = await trickle(f, Math.ceil((ISSUE_RECOVERY_BACKSTOP_MS + MINUTE) / MINUTE) + 1); + const checked = checkedAt(window.raw, missed!.id); + expect(checked).not.toBeNull(); + expect(checked! - now.getTime()).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE); + const runs = await state.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${missed!.id}`)); + expect(runs.length).toBeGreaterThan(0); + }, 1_800_000); + + it("re-checks a task whose only change is outside its own rows within the hourly full pass", async () => { + const f = await boot(); + const { held, waiting } = await heldTask(f); + await state.db.execute(sql`update agents set status = 'idle', lifecycle_holds = '[]'::jsonb where id = ${held.id}`); + const changedAt = Date.now(); + const window = await trickle(f, Math.ceil((ISSUE_RECOVERY_FULL_BACKSTOP_MS + 2 * MINUTE) / MINUTE)); + const checks = window.raw.filter(event => event.name === "check" && event.fields.issueId === waiting.id); + expect(checks.length).toBeGreaterThan(0); + expect(checks[0]!.at - changedAt).toBeGreaterThan(2 * ISSUE_RECOVERY_BACKSTOP_MS); + expect(checks[0]!.at - changedAt).toBeLessThanOrEqual(ISSUE_RECOVERY_FULL_BACKSTOP_MS + 2 * MINUTE); + const runs = await state.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.companyId, f.companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${waiting.id}`)); + expect(runs.length).toBeGreaterThan(0); + }, 1_800_000); + + it("picks up a finished run, a user message, a raised budget and a resumed agent within a minute", async () => { + const f = await boot(); + await trickle(f, 6); + const [target, messaged] = f.population.inReview; + const within = async (issueId: string, act: () => Promise) => { + f.events.length = 0; + const at = Date.now(); + await act(); + await advance(60_000); + const checked = checkedAt(f.events, issueId); + expect(checked, issueId).not.toBeNull(); + expect(checked! - at).toBeLessThanOrEqual(60_000); + }; + const [run] = await state.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .where(sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${target!}`).limit(1); + await within(target!, () => state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: run!.id }))); + await advance(31_000); + await within(messaged!, () => issueService(state.db).addComment(messaged!, "Please look again", { userId: "recovery-user" })); + await advance(31_000); + await within(target!, () => state.db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "pause_lifted" }))); + await advance(31_000); + const { held, waiting } = await heldTask(f); + await advance(31_000); + await within(waiting.id, async () => { + await state.db.update(agents).set({ status: "idle", lifecycleHolds: [], updatedAt: new Date() }).where(eq(agents.id, held.id)); + await state.db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "pause_lifted" })); + }); + }, 1_800_000); +}); diff --git a/server/src/__tests__/issue-recovery-work-debug-attribution.test.ts b/server/src/__tests__/issue-recovery-work-debug-attribution.test.ts index 8b0282ac0e5..eceba1544ee 100644 --- a/server/src/__tests__/issue-recovery-work-debug-attribution.test.ts +++ b/server/src/__tests__/issue-recovery-work-debug-attribution.test.ts @@ -39,7 +39,7 @@ describeIssueRecovery("issue recovery work signal attribution", () => { expect(stranded).toContainEqual(["stranded-assignments", "load", { scope: JSON.stringify([f.companyId, null, run.id, null, null]) }]); expect(stranded).toContainEqual(["stranded-assignments", "check", - expect.objectContaining({ issueId: task.id, companyId: f.companyId, wake: "load" })]); + expect.objectContaining({ issueId: task.id, companyId: f.companyId, wake: expect.stringMatching(/^(hint|backstop)$/) })]); } finally { enabled.mockRestore(); event.mockRestore(); signal.mockRestore(); } }); diff --git a/server/src/services/issue-recovery-notifications.ts b/server/src/services/issue-recovery-notifications.ts index 08751b36220..5c5038d5425 100644 --- a/server/src/services/issue-recovery-notifications.ts +++ b/server/src/services/issue-recovery-notifications.ts @@ -10,6 +10,7 @@ export type IssueRecoveryScope = { companyId: string; issueId?: string; graphRun export const ISSUE_RECOVERY_MAX_ISSUE_IDS = 100; export const ISSUE_RECOVERY_BACKSTOP_MS = 5 * 60_000; +export const ISSUE_RECOVERY_FULL_BACKSTOP_MS = 60 * 60_000; const UUID_PATTERN = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"; const COMPANY_QUEUES: DeliveryQueue[] = [DELIVERY_QUEUES.orphanBlockers, DELIVERY_QUEUES.activeRecoveryActions, DELIVERY_QUEUES.taskWatchdogs]; diff --git a/server/src/services/recovery/assigned-work-sweep.ts b/server/src/services/recovery/assigned-work-sweep.ts index 3315703383e..1983df05181 100644 --- a/server/src/services/recovery/assigned-work-sweep.ts +++ b/server/src/services/recovery/assigned-work-sweep.ts @@ -20,6 +20,7 @@ import { import { logger } from "../../middleware/logger.js"; import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js"; import { evaluateAgentInvokability } from "../agent-invokability.js"; +import { changedTaskCondition } from "./changed-tasks.js"; import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js"; import { unadmittedChatWakeupCondition } from "../durable-chat-wakeup.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; @@ -511,7 +512,8 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { order by lr.created_at desc limit 1), false)`; } - async function listSuspects(scope: { companyId?: string; issueId?: string; issueIds?: string[]; createdAfter?: Date | null }, + async function listSuspects(scope: { companyId?: string; issueId?: string; issueIds?: string[]; createdAfter?: Date | null; + changedSince?: Date }, cursor: string | null, limit: number) { const at = now(); const liveRun = sql`exists (select 1 from heartbeat_runs r where r.company_id = ${issues.companyId} @@ -531,6 +533,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { scope.issueIds ? inArray(issues.id, scope.issueIds) : undefined, cursor ? sql`${issues.id} > ${cursor}` : undefined, scope.createdAfter ? gte(issues.createdAt, scope.createdAfter) : undefined, + scope.changedSince ? changedTaskCondition(scope.changedSince) : undefined, inArray(issues.status, [...OPEN_STATUSES]), sql`${issues.assigneeAgentId} is not null`, isNull(issues.assigneeUserId), diff --git a/server/src/services/recovery/changed-tasks.ts b/server/src/services/recovery/changed-tasks.ts new file mode 100644 index 00000000000..2a0832131aa --- /dev/null +++ b/server/src/services/recovery/changed-tasks.ts @@ -0,0 +1,18 @@ +import { sql } from "drizzle-orm"; +import { issues } from "@paperclipai/db"; + +export function changedTaskCondition(since: Date) { + const at = sql`${since.toISOString()}::timestamptz`; + return sql`(${issues.updatedAt} >= ${at} + or exists (select 1 from heartbeat_runs r where r.company_id = ${issues.companyId} + and (r.context_snapshot->>'issueId') = ${issues.id}::text and r.updated_at >= ${at}) + or exists (select 1 from issue_recovery_actions a where a.company_id = ${issues.companyId} + and a.source_issue_id = ${issues.id} and a.updated_at >= ${at}) + or exists (select 1 from agent_wakeup_requests w where w.company_id = ${issues.companyId} + and (w.payload->>'issueId') = ${issues.id}::text and w.updated_at >= ${at}) + or exists (select 1 from issue_thread_interactions t where t.company_id = ${issues.companyId} + and t.issue_id = ${issues.id} and t.updated_at >= ${at}) + or exists (select 1 from issue_comments c where c.company_id = ${issues.companyId} + and c.issue_id = ${issues.id} and c.created_at >= ${at}) + or exists (select 1 from agents ag where ag.id = ${issues.assigneeAgentId} and ag.updated_at >= ${at}))`; +} diff --git a/server/src/services/recovery/work.ts b/server/src/services/recovery/work.ts index 6a8115737ed..9926f1c9fb8 100644 --- a/server/src/services/recovery/work.ts +++ b/server/src/services/recovery/work.ts @@ -3,14 +3,15 @@ import { and, asc, eq, gt, inArray, isNotNull, isNull, or, sql, type SQL } from import { issueRecoveryActions, issueRelations, issues, type Db } from "@paperclipai/db"; import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; import { DELIVERY_QUEUES, type DeliveryQueue } from "../delivery-work-notifications.js"; -import { decodeIssueRecoveryHint, ISSUE_RECOVERY_BACKSTOP_MS, issueRecoveryBackstopHint, issueRecoveryScopeMatches, resolveIssueGraph, type IssueRecoveryScope } +import { decodeIssueRecoveryHint, ISSUE_RECOVERY_BACKSTOP_MS, ISSUE_RECOVERY_FULL_BACKSTOP_MS, issueRecoveryBackstopHint, issueRecoveryScopeMatches, resolveIssueGraph, type IssueRecoveryScope } from "../issue-recovery-notifications.js"; import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; import { workSignalDebug } from "@paperclipai/db/work-signal-debug"; import { recoveryActionSweepCondition } from "./blocker-stall-wakeups.js"; +import { changedTaskCondition } from "./changed-tasks.js"; import { createRecoveryPassReads, type RecoveryPassReads } from "./pass-reads.js"; -type Entry = { id: string; companyId: string; dueAt: number; fromDeadline?: true }; +type Entry = { id: string; companyId: string; dueAt: number; fromDeadline?: true; origin?: "reload" | "backstop" | "company" | "hint" }; type Scope = { companyId: string; issueId: string; onDeferredDeadline: (at: number, detail?: { unchanged: boolean }) => void; pass?: RecoveryPassReads }; const PAGE_SIZE = 100; @@ -35,22 +36,39 @@ export function registerIssueRecoveryWork( schedulingSuppressed(): Promise; /** The assigned-work invariant (assigned-work-sweep.ts). */ assignedWork?: { - listSuspects(scope: { companyId?: string; issueId?: string; issueIds?: string[] }, cursor: string | null, limit: number): + listSuspects(scope: { companyId?: string; issueId?: string; issueIds?: string[]; changedSince?: Date }, cursor: string | null, limit: number): Promise>; checkIssue(scope: Scope): Promise; }; }, canRun: () => boolean, - options: { backstopMs?: number } = {}, + options: { backstopMs?: number; fullBackstopMs?: number } = {}, ) { + const backstopMs = options.backstopMs ?? ISSUE_RECOVERY_BACKSTOP_MS; + const fullBackstopMs = options.fullBackstopMs ?? options.backstopMs ?? ISSUE_RECOVERY_FULL_BACKSTOP_MS; function fullPasses() { - const completedAt = new Map(); + const completedAt = new Map(), changedAt = new Map(); + const scans = new Map(); + const lastFull = (companyId: string) => Math.max(completedAt.get("*") ?? 0, completedAt.get(companyId) ?? 0); + const lastLook = (companyId: string) => Math.max(lastFull(companyId), changedAt.get(companyId) ?? 0); return { - backstop: (scope: IssueRecoveryScope | undefined) => scope?.backstop - ? Math.max(completedAt.get("*") ?? 0, completedAt.get(scope.companyId) ?? 0) + (options.backstopMs ?? ISSUE_RECOVERY_BACKSTOP_MS) - : null, - loaded(scope: IssueRecoveryScope | undefined, complete: boolean) { - if (complete && !scope?.issueId && (!isGraph(scope) || scope?.wholeCompany)) completedAt.set(scope?.companyId ?? "*", Date.now()); + backstop: (scope: IssueRecoveryScope | undefined) => scope?.backstop ? lastLook(scope.companyId) + backstopMs : null, + changedSince(scope: IssueRecoveryScope | undefined, key: string, firstPage: boolean): Date | null { + if (firstPage || !scans.has(key)) scans.set(key, { startedAt: Date.now(), + since: !scope?.backstop || Date.now() - lastFull(scope.companyId) >= fullBackstopMs ? null + : new Date(lastLook(scope.companyId) - ISSUE_RECOVERY_BACKSTOP_MS) }); + return scans.get(key)!.since; + }, + loaded(scope: IssueRecoveryScope | undefined, key: string, complete: boolean) { + if (!complete) return; + const scan = scans.get(key); + scans.delete(key); + const startedAt = scan?.startedAt ?? Date.now(); + if (scan?.since && scope) changedAt.set(scope.companyId, Math.max(changedAt.get(scope.companyId) ?? 0, startedAt)); + else if (!scope?.issueId && (!isGraph(scope) || scope?.wholeCompany)) { + const companyKey = scope?.companyId ?? "*"; + completedAt.set(companyKey, Math.max(completedAt.get(companyKey) ?? 0, startedAt)); + } }, }; } @@ -98,8 +116,10 @@ export function registerIssueRecoveryWork( if (!enabled) { cursors.delete(key); return []; } if (await toCompany(scope, worker)) return { rows: null, complete: true }; const cursor = cursors.get(key); + const changedSince = passes.changedSince(scope, key, !cursor); const rows = await db.select({ id: issues.id, companyId: issues.companyId }).from(issues).where(and( scope ? eq(issues.companyId, scope.companyId) : undefined, + changedSince ? changedTaskCondition(changedSince) : undefined, scope?.issueId ? eq(issues.id, scope.issueId) : undefined, scope?.issueIds ? inArray(issues.id, scope.issueIds) : undefined, cursor ? gt(issues.id, cursor) : undefined, @@ -111,13 +131,16 @@ export function registerIssueRecoveryWork( )).orderBy(asc(issues.id)).limit(PAGE_SIZE); const complete = rows.length < PAGE_SIZE; if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id); - passes.loaded(scope, complete); - return { rows: rows.map(row => ({ ...row, dueAt: Date.now() })), complete }; + passes.loaded(scope, key, complete); + const keepUnseen = changedSince ? { keepUnseen: true as const } : {}; + const origin = !workSignalDebug.enabled ? undefined : !scope ? "reload" as const : scope.backstop ? "backstop" as const + : scope.issueId || isGraph(scope) ? "hint" as const : "company" as const; + return { rows: rows.map(row => ({ ...row, dueAt: Date.now(), ...(origin ? { origin } : {}) })), complete, ...keepUnseen }; }, check: async entry => { if (!enabled) return null; if (workSignalDebug.enabled) workSignalDebug.event(queue, "check", { issueId: entry.id, companyId: entry.companyId, - wake: entry.fromDeadline ? "deadline" : "load", dueAt: entry.dueAt }); + wake: entry.fromDeadline ? "deadline" : entry.origin ?? "load", dueAt: entry.dueAt }); let next: number | null = null; await check({ companyId: entry.companyId, issueId: entry.id, pass, onDeferredDeadline: at => { next = next === null ? at : Math.min(next, at); } }); @@ -149,13 +172,14 @@ export function registerIssueRecoveryWork( const key = scopeKey(scope); if (!enabled) { cursors.delete(key); return []; } if (await toCompany(scope, worker)) return { rows: null, complete: true }; + const changedSince = passes.changedSince(scope, key, !cursors.has(key)); const rows = await sweep.listSuspects(scope ? { companyId: scope.companyId, issueId: scope.issueId, - issueIds: scope.issueIds } : {}, cursors.get(key) ?? null, PAGE_SIZE); + issueIds: scope.issueIds, ...(changedSince ? { changedSince } : {}) } : {}, cursors.get(key) ?? null, PAGE_SIZE); const complete = rows.length < PAGE_SIZE; if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id); - passes.loaded(scope, complete); + passes.loaded(scope, key, complete); return { rows: rows.map(row => ({ id: row.id, companyId: row.companyId, updatedAt: row.updatedAt.getTime(), dueAt: Date.now(), - ...(scope?.backstop ? { fromBackstop: true as const } : {}) })), complete }; + ...(scope?.backstop ? { fromBackstop: true as const } : {}) })), complete, ...(changedSince ? { keepUnseen: true as const } : {}) }; }, check: async entry => { if (!enabled) return null; diff --git a/server/src/services/scoped-deadline-work.ts b/server/src/services/scoped-deadline-work.ts index 232b816e484..647f75d9c57 100644 --- a/server/src/services/scoped-deadline-work.ts +++ b/server/src/services/scoped-deadline-work.ts @@ -37,7 +37,7 @@ export function registerScopedDeadlineWork( * the scopes of its own key (and unkeyed scopes), not against every scope. */ scopeKey?: { entry(entry: Entry): string; scope(scope: Scope): string | undefined }; - load(scope?: Scope, signal?: AbortSignal): Promise; + load(scope?: Scope, signal?: AbortSignal): Promise; // Owners may retain pending queue priority while refreshing eligibility. mergeEntry?(pending: Entry, loaded: Entry): Entry; priority?(entry: Entry): number; @@ -165,7 +165,8 @@ export function registerScopedDeadlineWork( const scan = scans.get(hint) ?? { version: versions.get(hint) ?? 0, seen: new Set() }; scans.set(hint, scan); const loaded = await options.load(scope, signal); - const result = Array.isArray(loaded) ? { rows: loaded, complete: true } : loaded; + const result: { rows: Entry[] | null; complete: boolean; keepUnseen?: boolean } = Array.isArray(loaded) + ? { rows: loaded, complete: true } : loaded; const progressScopes = [...scans].map(([other, progress]) => ({ progress, scope: decode(other) })); for (const entry of result.rows ?? []) { const pending = entries.get(entry.id); @@ -174,7 +175,7 @@ export function registerScopedDeadlineWork( for (const { progress, scope: other } of progressScopes) if (options.matches(entry, other)) progress.seen.add(entry.id); } if (!result.complete) { dirty.add(hint); continue; } - if (result.rows !== null) for (const entry of entriesIn(scope)) if (options.matches(entry, scope) && !scan.seen.has(entry.id)) { + if (result.rows !== null && !result.keepUnseen) for (const entry of entriesIn(scope)) if (options.matches(entry, scope) && !scan.seen.has(entry.id)) { entries.delete(entry.id); failures.delete(entry.id); } scans.delete(hint);