From 54cde3353b11fbe3d2979b3972d37bf7f9378729 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 19:42:25 -0600 Subject: [PATCH] perf(recovery): back off assigned-work re-checks that find the same refused state (#21691) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The assigned-work sweep checks every open task assigned to an agent (`todo`, `in_progress`, `blocked`). When a task has nothing running, nothing pending and no one else's move, it claims one automatic continuation and asks the wake queue to start it. > - When the wake queue refuses that continuation, the claim is recorded as "not started" and the task is checked again 90 seconds later. The per-task spacing (2, 4, 8, then 15 minutes) lets it claim again. The refusal is not counted toward the hourly cap, so this never ends. > - Each cycle reads all of the task's facts, writes an activity entry and saves a skipped wake row. For 10 such tasks this is about 5,000 statements and 50 claims per hour, forever, on an idle server. > - This pull request stores a fingerprint of the task's facts with each refused continuation. A later check that finds the same facts waits, with reads only, on a backoff that doubles to a 6-hour ceiling. > - The benefit is that a task that cannot progress costs almost nothing while idle, and any real change on the task is still picked up within seconds. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** `act` in `server/src/services/recovery/assigned-work-sweep.ts` repairs a `no_pending_path` or `retry_offered` task with `deps.enqueueWakeup`. When admission refuses (for example the assignee has `wakeOnDemand: false`), `act` returns `enqueue_refused` with `retryAt` 90 seconds later. The next check claims again as soon as `continuationAllowance` allows: 2, 4, 8, then 15 minutes. Refused claims do not count toward `CONTINUATION_CAP`, so the task is re-claimed forever. Each claim writes an `issue.assigned_work_continued` activity entry ("could not start yet. It will try again shortly."), and each wake attempt saves a skipped wake row. Census on a staging-shaped fixture (real PostgreSQL, real report): 60 tasks in review, 10 running, 5 assigned to a paused agent and 10 whose continuation is refused. Only the 10 refused tasks are violations (`retry_offered`). Tasks in review are not in the sweep's scope at all, and a paused assignee is an "agent paused" wait. So on a hosted instance where most open tasks are in review, only the tasks whose continuation is refused pay this cost. One claim check costs 102 statements. A backed-off re-check costs 34 statements and writes nothing. **Expected behavior** A re-check that finds the same state as the last refused continuation stores nothing and backs off. A real change on the task (a run finishes, a status change, an assignment, a budget or pause change, a new wake) re-checks it promptly. **Steps to reproduce** 1. Create 10 tasks in `in_progress`, assigned to an agent with `runtimeConfig.heartbeat = { enabled: false, wakeOnDemand: false }`, each with a failed run. 2. Start the issue-state recovery work (it registers the assigned-work sweep). 3. Leave the instance idle. The sweep claims and is refused about every 2 to 15 minutes per task, forever. **Paperclip version or commit** master at 3eabdd6d10. ## What Changed - `server/src/services/recovery/assigned-work-sweep.ts`: - `act` computes a fingerprint of the task's facts and the violation (`stateKey`, without the task's `updatedAt`, which a refused claim on a blocked task rewrites). Before it claims, it reads the task's latest continuation entries (`refusalsInState`, one indexed read). When the newest entries are refused continuations with the same fingerprint, it waits until the last refusal plus 2 minutes, doubled per refusal, up to 6 hours. That check returns the new outcome `waiting_unchanged` and writes nothing. - A refused continuation (the ordinary path and the stalled-run replacement) stores `stateKey` and `refusedAt` in its activity entry. From the second refusal in the same state, the entry says "Paperclip still could not start this task. It will keep trying, less often each time, and right away when something on the task changes." - The agent fact now carries the agent's `updatedAt`, so a change to the agent's configuration changes the fingerprint. - `server/src/services/recovery/work.ts`: a task that waits on `waiting_unchanged` is marked on its queue entry. When an event reloads the task, the entry becomes due at once instead of keeping the later deadline. A backstop reload keeps the deadline. - `server/src/services/recovery/assigned-work-invariant.ts`: the new activity text, and the optional `updatedAt` on the agent fact. - `server/src/__tests__/assigned-work-invariant.test.ts`: the "too soon" check after a refusal now reports `waiting_unchanged` instead of `deferred_rate_limit`. - `server/src/__tests__/assigned-work-recheck-idle.test.ts`: three new tests on real PostgreSQL with the real coordinator, the real issue-state recovery work and a fake clock. What the user sees: the first refused continuation still says "could not start yet. It will try again shortly." Later refusals in the same state say Paperclip keeps trying less often. The task stays assigned, and any Retry the task shows is unchanged. About `retry_offered`: the request was to treat it as "a person was offered Retry, so nothing re-checks until they act". The code does not support that reading. `retry_offered` is defined as "the visible state offers a Retry that only the system can act on", and the sweep continues it so that the person does not need to press Retry. A rule that stops after one refusal would turn every transient refusal into a task that needs a person. So this pull request gives `retry_offered` the same backoff as `no_pending_path`. A person's Retry or message is still an event that re-checks the task at once. ## Verification - `pnpm exec vitest run src/__tests__/assigned-work-recheck-idle.test.ts src/__tests__/assigned-work-invariant.test.ts` (in `server/`), 35 tests pass. - `backs off while every check finds the same refused state`: 10 refused tasks, measured on the fake clock. | Window | master claims | master statements | this branch claims | this branch statements | | --- | --- | --- | --- | --- | | first idle hour | 50 | 5,060 | 40 | 5,078 | | idle hours 1 to 6 | 1,460 | 147,752 | 21 | 2,866 | The master numbers after the first hour are inflated by the test: the cap and spacing read activity `created_at`, which comes from the database clock, not the fake clock, so after one fake hour master spaces claims by only 2 minutes. At the production spacing (15 minutes, about 6 claims per task per hour), master would do about 300 claims and 25,000 statements in the same 5 hours. The assertion (at most 30 claims and fewer than 5,000 statements in hours 1 to 6) fails on master and passes on this branch. - `re-checks a backed-off task within two minutes when one of its runs finishes`: after 3 idle hours, a new failed run of one task is notified. Within 2 minutes (the 90-second settling grace) the sweep claims that task again. Without the `work.ts` change, the event keeps the backed-off deadline and this test fails. - `leaves review work alone and re-checks a refused task with reads only while it waits`: the census above, plus one claim check (102 statements, writes an activity entry) and one backed-off re-check (34 statements, no insert, update or delete). - Neighbouring suites pass locally: `assigned-work-sweep-round-trips`, `assigned-work-stalled-queued-run`, `failed-continuation-continues-again`, `automatic-run-brake-recovery-passes`, `issue-recovery-recorded-decision-idle`, `recovery-wake-herd`, `issue-recovery-signal-fanout`, `saved-work-recovery-overdue`. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - A refusal whose cause clears without any change to the task's facts or a reloading event (for example a scheduling suppression that lifts) is retried on the backoff, at most 6 hours later, instead of within 15 minutes. Changes to the task, its runs, its wakes, the agent row, budget or company pause change the fingerprint and claim at once. - The fingerprint covers the full fact set. A fact that changes on every check would turn the backoff off for that task and keep the master behavior. The idle test shows that the refused fixture keeps a stable fingerprint. - An event that reloads a backed-off task now re-checks it at once. That check is read-only while the state is the same (34 statements). - No schema or API change. The new fields live in the existing activity `details` JSON. - Separate finding, not changed here: a task whose pending continuation counts as its pending path is not re-checked when that continuation's bound ends unless something reloads it. With continuation delivery running this is rare, but it is a missed re-check rather than a loop. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local PostgreSQL test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #21687, #21621, #21674, #21675, #21676, #21617. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../__tests__/assigned-work-invariant.test.ts | 2 +- .../assigned-work-recheck-idle.test.ts | 109 ++++++++++++++++++ .../recovery/assigned-work-invariant.ts | 4 +- .../services/recovery/assigned-work-sweep.ts | 51 ++++++-- server/src/services/recovery/work.ts | 17 +-- 5 files changed, 166 insertions(+), 17 deletions(-) create mode 100644 server/src/__tests__/assigned-work-recheck-idle.test.ts diff --git a/server/src/__tests__/assigned-work-invariant.test.ts b/server/src/__tests__/assigned-work-invariant.test.ts index 4261c4b1598..a94b211d0f6 100644 --- a/server/src/__tests__/assigned-work-invariant.test.ts +++ b/server/src/__tests__/assigned-work-invariant.test.ts @@ -446,7 +446,7 @@ describePostgres("assigned-work invariant sweep", () => { expect(deadlines).toHaveLength(1); // Too soon: spaced, not repaired and not reported as fine. clock = new Date(clock.getTime() + 60_000); - expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "deferred_rate_limit", verdict: { kind: "violation" } }); + expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "waiting_unchanged", verdict: { kind: "violation" } }); // After the spacing it is admitted; the queue later declines the claim // (for example behind the old run's pending cleanup) and the wake is skipped. admit = true; diff --git a/server/src/__tests__/assigned-work-recheck-idle.test.ts b/server/src/__tests__/assigned-work-recheck-idle.test.ts new file mode 100644 index 00000000000..90ccd0b4314 --- /dev/null +++ b/server/src/__tests__/assigned-work-recheck-idle.test.ts @@ -0,0 +1,109 @@ +import { setImmediate as immediate } from "node:timers/promises"; +import { and, desc, eq } from "drizzle-orm"; +import { expect, it, vi } from "vitest"; +import { activityLog, agents, heartbeatRuns, issues } from "@paperclipai/db"; +import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js"; +import { ASSIGNED_WORK_CONTINUED_ACTION } from "../services/recovery/assigned-work-sweep.js"; +import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js"; + +const MINUTE = 60_000; +const HOUR = 60 * MINUTE; + +describeIssueRecovery("assigned-work re-checks of tasks whose continuation is refused", () => { + const state = issueRecoveryFixture(); + const TASKS = 10; + + async function seed() { + const f = await state.fixture(); + const db = state.db; + await db.update(agents).set({ runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: false } } }) + .where(eq(agents.id, f.agentId)); + const tasks = await db.insert(issues).values(Array.from({ length: TASKS }, (_, index) => ({ + companyId: f.companyId, title: `Refused task ${index}`, status: "in_progress", assigneeAgentId: f.agentId, + }))).returning({ id: issues.id }); + await db.insert(heartbeatRuns).values(tasks.map(task => ({ companyId: f.companyId, agentId: f.agentId, + status: "failed", errorCode: "adapter_failed", finishedAt: new Date(Date.now() - HOUR), contextSnapshot: { issueId: task.id } }))); + return { ...f, tasks }; + } + + async function advance(ms: number) { + const until = Date.now() + ms; + while (Date.now() < until) { + await vi.advanceTimersByTimeAsync(Math.min(500, until - Date.now())); + await Promise.allSettled([...state.passes]); await immediate(); + } + } + + async function countStatements(work: () => Promise) { + const statements: string[] = [], prior = state.db.$client.options.debug; + state.db.$client.options.debug = (_connection, query) => { statements.push(query); }; + try { await work(); } finally { state.db.$client.options.debug = prior; } + return statements; + } + + const attempts = async (companyId: string, issueId?: string) => (await state.db.select().from(activityLog).where(and( + eq(activityLog.companyId, companyId), eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION), + issueId ? eq(activityLog.entityId, issueId) : undefined))).length; + + async function boot(f: Awaited>) { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); + await state.start({}); + await advance(5 * MINUTE); + expect(await attempts(f.companyId)).toBeGreaterThanOrEqual(TASKS); + const report = await state.heartbeat.reportAssignedWorkInvariant(f.companyId); + expect(report.violations).toEqual({ retry_offered: TASKS }); + } + + it("backs off while every check finds the same refused state", async () => { + const f = await seed(); + await boot(f); + await advance(HOUR); + const before = await attempts(f.companyId); + const idle = await countStatements(() => advance(5 * HOUR)); + expect(await attempts(f.companyId) - before).toBeLessThanOrEqual(3 * TASKS); + expect(idle.length).toBeLessThan(TASKS * 500); + const [latest] = await state.db.select().from(activityLog).where(and(eq(activityLog.entityId, f.tasks[0]!.id), + eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION))).orderBy(desc(activityLog.createdAt)).limit(1); + expect(latest?.details).toMatchObject({ outcome: "not_started", stateKey: expect.any(String), + message: expect.stringContaining("still could not start") }); + }, 600_000); + + it("re-checks a backed-off task within two minutes when one of its runs finishes", async () => { + const f = await seed(); + await boot(f); + await advance(3 * HOUR); + const target = f.tasks[0]!.id; + const before = await attempts(f.companyId, target); + await state.db.transaction(async tx => { + const [run] = await tx.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "failed", + errorCode: "adapter_failed", finishedAt: new Date(), contextSnapshot: { issueId: target } }).returning(); + await notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: run!.id }); + }); + await advance(2 * MINUTE); + expect(await attempts(f.companyId, target)).toBeGreaterThan(before); + }, 600_000); + + it("leaves review work alone and re-checks a refused task with reads only while it waits", async () => { + const f = await seed(); + const db = state.db; + const [paused] = await db.insert(agents).values({ companyId: f.companyId, name: "Paused reviewer", role: "engineer", status: "paused", + adapterType: "process", adapterConfig: {} }).returning(); + await db.insert(issues).values(Array.from({ length: 60 }, (_, index) => ({ companyId: f.companyId, title: `Review ${index}`, + status: "in_review", assigneeAgentId: f.agentId, executionState: { status: "pending" } }))); + const running = await db.insert(issues).values(Array.from({ length: 10 }, (_, index) => ({ companyId: f.companyId, + title: `Running ${index}`, status: "in_progress", assigneeAgentId: f.agentId }))).returning({ id: issues.id }); + await db.insert(heartbeatRuns).values(running.map(task => ({ companyId: f.companyId, agentId: f.agentId, status: "running", + startedAt: new Date(), contextSnapshot: { issueId: task.id } }))); + await db.insert(issues).values(Array.from({ length: 5 }, (_, index) => ({ companyId: f.companyId, title: `Paused ${index}`, + status: "todo", assigneeAgentId: paused!.id, createdAt: new Date(Date.now() - HOUR) }))); + const report = await state.heartbeat.reportAssignedWorkInvariant(f.companyId); + expect(report).toMatchObject({ openTasks: 25, violations: { retry_offered: TASKS }, + ok: { live_run: 10, "waiting:agent_paused": 5 } }); + const target = f.tasks[0]!.id; + const claim = await countStatements(() => state.heartbeat.checkAssignedWorkInvariant({ companyId: f.companyId, issueId: target })); + const recheck = await countStatements(() => state.heartbeat.checkAssignedWorkInvariant({ companyId: f.companyId, issueId: target })); + expect(claim.some(query => /^insert into "activity_log"/i.test(query))).toBe(true); + expect(recheck.filter(query => /^(insert|update|delete)/i.test(query.trim()))).toEqual([]); + expect(recheck.length).toBeLessThan(claim.length); + }, 600_000); +}); diff --git a/server/src/services/recovery/assigned-work-invariant.ts b/server/src/services/recovery/assigned-work-invariant.ts index 317f69c5516..9bca61b2f82 100644 --- a/server/src/services/recovery/assigned-work-invariant.ts +++ b/server/src/services/recovery/assigned-work-invariant.ts @@ -177,7 +177,7 @@ export type AssignedWorkFacts = { /** A saved-work recovery the recovery queue still owes this task: the * task's saved files must be recovered before any run can start. */ savedWorkRecovery?: SavedWorkRecoveryFact | null; - agent: { status: string; invokable: boolean } | null; + agent: { status: string; invokable: boolean; updatedAt?: Date } | null; companyPaused: boolean; pauseHold: boolean; budgetStop: boolean; @@ -385,6 +385,8 @@ export const CONTINUATION_REVIEW_MESSAGE = "Returned to review automatically after an interruption. Nothing needed you, so the review picks up where it stopped."; export const CONTINUATION_REFUSED_MESSAGE = "Paperclip tried to continue this task automatically, but it could not start yet. It will try again shortly."; +export const CONTINUATION_REFUSED_AGAIN_MESSAGE = + "Paperclip still could not start this task. It will keep trying, less often each time, and right away when something on the task changes."; /** Plain-language activity text for an admitted continuation. No codes, no ids. */ export function continuationActivityMessage(reason: ViolationReason): string { diff --git a/server/src/services/recovery/assigned-work-sweep.ts b/server/src/services/recovery/assigned-work-sweep.ts index 9317316ab80..c3ca5f80ba1 100644 --- a/server/src/services/recovery/assigned-work-sweep.ts +++ b/server/src/services/recovery/assigned-work-sweep.ts @@ -1,3 +1,4 @@ +import { createHash } from "node:crypto"; import { and, desc, eq, gte, inArray, isNull, or, sql, type SQL, type SQLWrapper } from "drizzle-orm"; import { activityLog, @@ -47,6 +48,8 @@ import { SAVED_WORK_RECOVERY_START_BOUND_MS, savedWorkRecoveryDueSince, CONTINUATION_REFUSED_MESSAGE, + CONTINUATION_REFUSED_AGAIN_MESSAGE, + CONTINUATION_BASE_BACKOFF_MS, CONTINUATION_REVIEW_MESSAGE, CONTINUATION_CAP_WINDOW_MS, evaluateAssignedWork, @@ -69,6 +72,8 @@ export const STRANDED_TRANSIENT_CONTINUED_OUTCOME = "stranded_transient_continue export const ASSIGNED_WORK_REPAIRS_PER_MINUTE = 10; /** Upper bound on tasks one report reads. */ export const ASSIGNED_WORK_REPORT_LIMIT = 2_000; +export const UNCHANGED_REFUSAL_MAX_WAIT_MS = 6 * 60 * 60_000; +const UNCHANGED_REFUSAL_CHAIN = 9; const OPEN_STATUSES = ["todo", "in_progress", "blocked"] as const; const PENDING_WAKE_STATUSES = ["queued", "deferred_issue_execution", "claimed"]; @@ -108,7 +113,7 @@ export type AssignedWorkCheck = { verdict: AssignedWorkVerdict; outcome: "ok" | "settling" | "repaired" | "settled_lost_run" | "deferred_rate_limit" | "deferred_cap" | "deferred_reattach_phase" | "stale_reported" | "left_to_hold_owner" | "left_to_native_recovery" | "changed" | "enqueue_refused" | "waiting_on_recovery" | "started_saved_work_recovery" | "saved_work_recovery_busy" - | "replaced_stalled_run" | "left_person_work" | "left_stalled_replacement"; + | "replaced_stalled_run" | "left_person_work" | "left_stalled_replacement" | "waiting_unchanged"; retryAt: Date | null; }; @@ -458,7 +463,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { resumeAt: PROVIDER_LIMIT_CAUSES.includes(action.cause) && interruption?.nextRetryAt ? new Date(interruption.nextRetryAt) : null, continuedBySweep: strandedHoldContinuesAutomatically(action, strandedRuns.get(String(action.evidence?.latestRunId ?? "")) ?? null, issue.assigneeAgentId, at) } : null, - agent: agent ? { status: agent.status, invokable: agent.invokable } : null, + agent: agent ? { status: agent.status, invokable: agent.invokable, updatedAt: agent.updatedAt } : null, companyPaused: company?.status === "paused", pauseHold, budgetStop: companyBudget || agent?.pauseReason === "budget" || agent?.pauseReason === "budget_exceeded", @@ -582,14 +587,15 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { } /** Evaluate one task, and repair it when it violates the invariant. */ - async function checkIssue(scope: { companyId: string; issueId: string; onDeferredDeadline?: (at: number) => void }): Promise { + async function checkIssue(scope: { companyId: string; issueId: string; + onDeferredDeadline?: (at: number, detail?: { unchanged: boolean }) => void }): Promise { const [issue] = await loadIssues(scope.companyId, scope.issueId, 1); if (!issue) return null; const facts = (await loadFacts(scope.companyId, [issue])).get(issue.id)!; const at = now(); const verdict = evaluateAssignedWork(facts, at); const check = await act(issue, facts, verdict, at); - if (check.retryAt) scope.onDeferredDeadline?.(check.retryAt.getTime()); + if (check.retryAt) scope.onDeferredDeadline?.(check.retryAt.getTime(), { unchanged: check.outcome === "waiting_unchanged" }); passLog.record(check); return check; } @@ -639,6 +645,14 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { retryAt: new Date(at.getTime() + CONTINUATION_CAP_WINDOW_MS) }; } } + const key = stateKey(facts, verdict); + const refusals = await refusalsInState(issue, key, at); + if (refusals.count && refusals.lastAt !== null) { + const waitUntil = refusals.lastAt + Math.min(UNCHANGED_REFUSAL_MAX_WAIT_MS, CONTINUATION_BASE_BACKOFF_MS * 2 ** (refusals.count - 1)); + if (waitUntil > at.getTime()) return { verdict, outcome: "waiting_unchanged", retryAt: new Date(waitUntil) }; + } + const refusal = { stateKey: key, refusedAt: at.toISOString(), + ...(refusals.count ? { message: CONTINUATION_REFUSED_AGAIN_MESSAGE } : {}) }; // A small process-wide budget keeps a post-deploy backlog from starting // hundreds of runs at once. Deferred tasks come back on their deadline. const budgetDeferral = repairBudgetDeferral(at); @@ -650,7 +664,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { if (claimed.kind !== "claimed") return { verdict, outcome: claimed.kind === "capped" ? "deferred_cap" : claimed.kind === "changed" ? "changed" : "deferred_rate_limit", retryAt: claimed.kind === "changed" ? null : claimed.retryAt }; repairTimes.push(at.getTime()); - if (claimed.reason === "queued_run_stalled") return replaceStalledRun(issue, claimed, claimed.verdict.runId!); + if (claimed.reason === "queued_run_stalled") return replaceStalledRun(issue, claimed, claimed.verdict.runId!, refusal); // A block that interrupted a review goes back to review; the review's own // recovery wakes its participant. No agent work starts from here. if (claimed.restoredTo === "in_review") { @@ -694,10 +708,31 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { if (!run) await release(); if (!run && claimed.restoredTo) await revertRestoredBlock(issue, claimed); await settleContinuationActivity(claimed, claimed.reason, run ? "queued" : "not_started", - fresh?.kind === "person_owned" ? { waitingFor: fresh.reason } : {}); + { ...(fresh?.kind === "person_owned" ? { waitingFor: fresh.reason } : {}), ...(run ? {} : refusal) }); return { verdict: claimed.verdict, outcome: run ? "repaired" : "enqueue_refused", retryAt: claimed.nextCheckAt }; } + function stateKey(facts: AssignedWorkFacts, verdict: Extract) { + const { updatedAt: _updatedAt, ...issue } = facts.issue; + return createHash("sha256").update(JSON.stringify({ reason: verdict.reason, ...facts, issue })).digest("hex").slice(0, 32); + } + + async function refusalsInState(issue: IssueRow, key: string, at: Date) { + const rows = await db.select({ details: activityLog.details }).from(activityLog).where(and( + eq(activityLog.companyId, issue.companyId), eq(activityLog.entityType, "issue"), eq(activityLog.entityId, issue.id), + gte(activityLog.createdAt, new Date(at.getTime() - UNCHANGED_REFUSAL_CHAIN * UNCHANGED_REFUSAL_MAX_WAIT_MS)), + eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION))).orderBy(desc(activityLog.createdAt)).limit(UNCHANGED_REFUSAL_CHAIN); + let count = 0, lastAt: number | null = null; + for (const row of rows) { + const details = json(row.details); + if (details.outcome !== "not_started" || details.stateKey !== key) break; + count += 1; + const refusedAt = Date.parse(String(details.refusedAt)); + if (lastAt === null && Number.isFinite(refusedAt)) lastAt = refusedAt; + } + return { count, lastAt }; + } + function repairBudgetDeferral(at: Date): Date | null { const minuteAgo = at.getTime() - 60_000; while (repairTimes.length && repairTimes[0] <= minuteAgo) repairTimes.shift(); @@ -715,7 +750,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { "queued run its agent keeps passing over was left in place"); } - async function replaceStalledRun(issue: IssueRow, claimed: Claimed, runId: string): Promise { + async function replaceStalledRun(issue: IssueRow, claimed: Claimed, runId: string, refusal: Record): Promise { const slot = await reserveAutomaticRunCreation(db, issue.companyId, { issueId: issue.id, sourceRunId: runId }); if (slot.heldUntil) { await settleContinuationActivity(claimed, claimed.reason, "not_started", @@ -743,7 +778,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { return null; }); if (!run) await slot.release(); - await settleContinuationActivity(claimed, claimed.reason, run ? "queued" : "not_started"); + await settleContinuationActivity(claimed, claimed.reason, run ? "queued" : "not_started", run ? {} : refusal); return { verdict: claimed.verdict, outcome: run ? "replaced_stalled_run" : "enqueue_refused", retryAt: claimed.nextCheckAt }; } diff --git a/server/src/services/recovery/work.ts b/server/src/services/recovery/work.ts index 6fae89d60e0..fc7b39b628b 100644 --- a/server/src/services/recovery/work.ts +++ b/server/src/services/recovery/work.ts @@ -10,7 +10,7 @@ import { workSignalDebug } from "@paperclipai/db/work-signal-debug"; import { recoveryActionSweepCondition } from "./blocker-stall-wakeups.js"; type Entry = { id: string; companyId: string; dueAt: number; fromDeadline?: true }; -type Scope = { companyId: string; issueId: string; onDeferredDeadline: (at: number) => void }; +type Scope = { companyId: string; issueId: string; onDeferredDeadline: (at: number, detail?: { unchanged: boolean }) => void }; const PAGE_SIZE = 100; export const resolvedDependencyCandidate = sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId} @@ -126,7 +126,7 @@ export function registerIssueRecoveryWork( * company dirty; checks schedule their own grace and back-off deadlines. * Newest-updated tasks are checked first, a small batch per pass. */ function registerAssignedWork(sweep: NonNullable) { - type Suspect = Entry & { updatedAt: number }; + type Suspect = Entry & { updatedAt: number; unchanged?: true; fromBackstop?: true }; const cursors = new Map(); const passes = fullPasses(); let enabled = false; @@ -139,7 +139,8 @@ export function registerIssueRecoveryWork( decodeScope: decodeIssueRecoveryHint, matches: issueRecoveryScopeMatches, priority: entry => entry.updatedAt, - mergeEntry: (pending, loaded) => ({ ...loaded, dueAt: Math.max(pending.dueAt, loaded.dueAt) }), + mergeEntry: (pending, loaded) => pending.unchanged && !loaded.fromBackstop ? loaded + : { ...loaded, dueAt: Math.max(pending.dueAt, loaded.dueAt), ...(pending.unchanged ? { unchanged: true as const } : {}) }, load: async scope => { const key = scopeKey(scope); if (!enabled) { cursors.delete(key); return []; } @@ -149,14 +150,16 @@ export function registerIssueRecoveryWork( const complete = rows.length < PAGE_SIZE; if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id); passes.loaded(scope, complete); - return { rows: rows.map(row => ({ id: row.id, companyId: row.companyId, updatedAt: row.updatedAt.getTime(), dueAt: Date.now() })), complete }; + return { rows: rows.map(row => ({ id: row.id, companyId: row.companyId, updatedAt: row.updatedAt.getTime(), dueAt: Date.now(), + ...(scope?.backstop ? { fromBackstop: true as const } : {}) })), complete }; }, check: async entry => { if (!enabled) return null; - let next: number | null = null; + let next: number | null = null, unchanged = false; await sweep.checkIssue({ companyId: entry.companyId, issueId: entry.id, - onDeferredDeadline: at => { next = next === null ? at : Math.min(next, at); } }); - return next === null ? null : { ...entry, dueAt: next }; + onDeferredDeadline: (at, detail) => { next = next === null ? at : Math.min(next, at); unchanged ||= detail?.unchanged === true; } }); + const { unchanged: _previous, fromBackstop: _source, ...rest } = entry; + return next === null ? null : { ...rest, dueAt: next, ...(unchanged ? { unchanged: true as const } : {}) }; }, }, canRun); return worker;