perf(recovery): back off assigned-work re-checks that find the same refused state (#21691)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The assigned-work sweep checks every open task assigned to an agent
(`todo`, `in_progress`, `blocked`). When a task has nothing running,
nothing pending and no one else's move, it claims one automatic
continuation and asks the wake queue to start it.
> - When the wake queue refuses that continuation, the claim is recorded
as "not started" and the task is checked again 90 seconds later. The
per-task spacing (2, 4, 8, then 15 minutes) lets it claim again. The
refusal is not counted toward the hourly cap, so this never ends.
> - Each cycle reads all of the task's facts, writes an activity entry
and saves a skipped wake row. For 10 such tasks this is about 5,000
statements and 50 claims per hour, forever, on an idle server.
> - This pull request stores a fingerprint of the task's facts with each
refused continuation. A later check that finds the same facts waits,
with reads only, on a backoff that doubles to a 6-hour ceiling.
> - The benefit is that a task that cannot progress costs almost nothing
while idle, and any real change on the task is still picked up within
seconds.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

`act` in `server/src/services/recovery/assigned-work-sweep.ts` repairs a
`no_pending_path` or `retry_offered` task with `deps.enqueueWakeup`.
When admission refuses (for example the assignee has `wakeOnDemand:
false`), `act` returns `enqueue_refused` with `retryAt` 90 seconds
later. The next check claims again as soon as `continuationAllowance`
allows: 2, 4, 8, then 15 minutes. Refused claims do not count toward
`CONTINUATION_CAP`, so the task is re-claimed forever. Each claim writes
an `issue.assigned_work_continued` activity entry ("could not start yet.
It will try again shortly."), and each wake attempt saves a skipped wake
row.

Census on a staging-shaped fixture (real PostgreSQL, real report): 60
tasks in review, 10 running, 5 assigned to a paused agent and 10 whose
continuation is refused. Only the 10 refused tasks are violations
(`retry_offered`). Tasks in review are not in the sweep's scope at all,
and a paused assignee is an "agent paused" wait. So on a hosted instance
where most open tasks are in review, only the tasks whose continuation
is refused pay this cost. One claim check costs 102 statements. A
backed-off re-check costs 34 statements and writes nothing.

**Expected behavior**

A re-check that finds the same state as the last refused continuation
stores nothing and backs off. A real change on the task (a run finishes,
a status change, an assignment, a budget or pause change, a new wake)
re-checks it promptly.

**Steps to reproduce**

1. Create 10 tasks in `in_progress`, assigned to an agent with
`runtimeConfig.heartbeat = { enabled: false, wakeOnDemand: false }`,
each with a failed run.
2. Start the issue-state recovery work (it registers the assigned-work
sweep).
3. Leave the instance idle. The sweep claims and is refused about every
2 to 15 minutes per task, forever.

**Paperclip version or commit**

master at 3eabdd6d10.

## What Changed

- `server/src/services/recovery/assigned-work-sweep.ts`:
- `act` computes a fingerprint of the task's facts and the violation
(`stateKey`, without the task's `updatedAt`, which a refused claim on a
blocked task rewrites). Before it claims, it reads the task's latest
continuation entries (`refusalsInState`, one indexed read). When the
newest entries are refused continuations with the same fingerprint, it
waits until the last refusal plus 2 minutes, doubled per refusal, up to
6 hours. That check returns the new outcome `waiting_unchanged` and
writes nothing.
- A refused continuation (the ordinary path and the stalled-run
replacement) stores `stateKey` and `refusedAt` in its activity entry.
From the second refusal in the same state, the entry says "Paperclip
still could not start this task. It will keep trying, less often each
time, and right away when something on the task changes."
- The agent fact now carries the agent's `updatedAt`, so a change to the
agent's configuration changes the fingerprint.
- `server/src/services/recovery/work.ts`: a task that waits on
`waiting_unchanged` is marked on its queue entry. When an event reloads
the task, the entry becomes due at once instead of keeping the later
deadline. A backstop reload keeps the deadline.
- `server/src/services/recovery/assigned-work-invariant.ts`: the new
activity text, and the optional `updatedAt` on the agent fact.
- `server/src/__tests__/assigned-work-invariant.test.ts`: the "too soon"
check after a refusal now reports `waiting_unchanged` instead of
`deferred_rate_limit`.
- `server/src/__tests__/assigned-work-recheck-idle.test.ts`: three new
tests on real PostgreSQL with the real coordinator, the real issue-state
recovery work and a fake clock.

What the user sees: the first refused continuation still says "could not
start yet. It will try again shortly." Later refusals in the same state
say Paperclip keeps trying less often. The task stays assigned, and any
Retry the task shows is unchanged.

About `retry_offered`: the request was to treat it as "a person was
offered Retry, so nothing re-checks until they act". The code does not
support that reading. `retry_offered` is defined as "the visible state
offers a Retry that only the system can act on", and the sweep continues
it so that the person does not need to press Retry. A rule that stops
after one refusal would turn every transient refusal into a task that
needs a person. So this pull request gives `retry_offered` the same
backoff as `no_pending_path`. A person's Retry or message is still an
event that re-checks the task at once.

## Verification

- `pnpm exec vitest run src/__tests__/assigned-work-recheck-idle.test.ts
src/__tests__/assigned-work-invariant.test.ts` (in `server/`), 35 tests
pass.
- `backs off while every check finds the same refused state`: 10 refused
tasks, measured on the fake clock.

| Window | master claims | master statements | this branch claims | this
branch statements |
    | --- | --- | --- | --- | --- |
    | first idle hour | 50 | 5,060 | 40 | 5,078 |
    | idle hours 1 to 6 | 1,460 | 147,752 | 21 | 2,866 |

The master numbers after the first hour are inflated by the test: the
cap and spacing read activity `created_at`, which comes from the
database clock, not the fake clock, so after one fake hour master spaces
claims by only 2 minutes. At the production spacing (15 minutes, about 6
claims per task per hour), master would do about 300 claims and 25,000
statements in the same 5 hours. The assertion (at most 30 claims and
fewer than 5,000 statements in hours 1 to 6) fails on master and passes
on this branch.
- `re-checks a backed-off task within two minutes when one of its runs
finishes`: after 3 idle hours, a new failed run of one task is notified.
Within 2 minutes (the 90-second settling grace) the sweep claims that
task again. Without the `work.ts` change, the event keeps the backed-off
deadline and this test fails.
- `leaves review work alone and re-checks a refused task with reads only
while it waits`: the census above, plus one claim check (102 statements,
writes an activity entry) and one backed-off re-check (34 statements, no
insert, update or delete).
- Neighbouring suites pass locally: `assigned-work-sweep-round-trips`,
`assigned-work-stalled-queued-run`,
`failed-continuation-continues-again`,
`automatic-run-brake-recovery-passes`,
`issue-recovery-recorded-decision-idle`, `recovery-wake-herd`,
`issue-recovery-signal-fanout`, `saved-work-recovery-overdue`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- A refusal whose cause clears without any change to the task's facts or
a reloading event (for example a scheduling suppression that lifts) is
retried on the backoff, at most 6 hours later, instead of within 15
minutes. Changes to the task, its runs, its wakes, the agent row, budget
or company pause change the fingerprint and claim at once.
- The fingerprint covers the full fact set. A fact that changes on every
check would turn the backoff off for that task and keep the master
behavior. The idle test shows that the refused fixture keeps a stable
fingerprint.
- An event that reloads a backed-off task now re-checks it at once. That
check is read-only while the state is the same (34 statements).
- No schema or API change. The new fields live in the existing activity
`details` JSON.
- Separate finding, not changed here: a task whose pending continuation
counts as its pending path is not re-checked when that continuation's
bound ends unless something reloads it. With continuation delivery
running this is rare, but it is a missed re-check rather than a loop.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21687, #21621, #21674, #21675, #21676, #21617.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 19:42:25 -06:00
1 parent f4e025a1cf
commit 54cde3353b
5 files changed
+166 -17

No files matched your search

@@ -446,7 +446,7 @@ describePostgres("assigned-work invariant sweep", () => {
expect(deadlines).toHaveLength(1);
// Too soon: spaced, not repaired and not reported as fine.
clock = new Date(clock.getTime() + 60_000);
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "deferred_rate_limit", verdict: { kind: "violation" } });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "waiting_unchanged", verdict: { kind: "violation" } });
// After the spacing it is admitted; the queue later declines the claim
// (for example behind the old run's pending cleanup) and the wake is skipped.
admit = true;
@@ -0,0 +1,109 @@
import { setImmediate as immediate } from "node:timers/promises";
import { and, desc, eq } from "drizzle-orm";
import { expect, it, vi } from "vitest";
import { activityLog, agents, heartbeatRuns, issues } from "@paperclipai/db";
import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js";
import { ASSIGNED_WORK_CONTINUED_ACTION } from "../services/recovery/assigned-work-sweep.js";
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
const MINUTE = 60_000;
const HOUR = 60 * MINUTE;
describeIssueRecovery("assigned-work re-checks of tasks whose continuation is refused", () => {
const state = issueRecoveryFixture();
const TASKS = 10;
async function seed() {
const f = await state.fixture();
const db = state.db;
await db.update(agents).set({ runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: false } } })
.where(eq(agents.id, f.agentId));
const tasks = await db.insert(issues).values(Array.from({ length: TASKS }, (_, index) => ({
companyId: f.companyId, title: `Refused task ${index}`, status: "in_progress", assigneeAgentId: f.agentId,
}))).returning({ id: issues.id });
await db.insert(heartbeatRuns).values(tasks.map(task => ({ companyId: f.companyId, agentId: f.agentId,
status: "failed", errorCode: "adapter_failed", finishedAt: new Date(Date.now() - HOUR), contextSnapshot: { issueId: task.id } })));
return { ...f, tasks };
}
async function advance(ms: number) {
const until = Date.now() + ms;
while (Date.now() < until) {
await vi.advanceTimersByTimeAsync(Math.min(500, until - Date.now()));
await Promise.allSettled([...state.passes]); await immediate();
}
}
async function countStatements(work: () => Promise<unknown>) {
const statements: string[] = [], prior = state.db.$client.options.debug;
state.db.$client.options.debug = (_connection, query) => { statements.push(query); };
try { await work(); } finally { state.db.$client.options.debug = prior; }
return statements;
}
const attempts = async (companyId: string, issueId?: string) => (await state.db.select().from(activityLog).where(and(
eq(activityLog.companyId, companyId), eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION),
issueId ? eq(activityLog.entityId, issueId) : undefined))).length;
async function boot(f: Awaited<ReturnType<typeof seed>>) {
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
await state.start({});
await advance(5 * MINUTE);
expect(await attempts(f.companyId)).toBeGreaterThanOrEqual(TASKS);
const report = await state.heartbeat.reportAssignedWorkInvariant(f.companyId);
expect(report.violations).toEqual({ retry_offered: TASKS });
}
it("backs off while every check finds the same refused state", async () => {
const f = await seed();
await boot(f);
await advance(HOUR);
const before = await attempts(f.companyId);
const idle = await countStatements(() => advance(5 * HOUR));
expect(await attempts(f.companyId) - before).toBeLessThanOrEqual(3 * TASKS);
expect(idle.length).toBeLessThan(TASKS * 500);
const [latest] = await state.db.select().from(activityLog).where(and(eq(activityLog.entityId, f.tasks[0]!.id),
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION))).orderBy(desc(activityLog.createdAt)).limit(1);
expect(latest?.details).toMatchObject({ outcome: "not_started", stateKey: expect.any(String),
message: expect.stringContaining("still could not start") });
}, 600_000);
it("re-checks a backed-off task within two minutes when one of its runs finishes", async () => {
const f = await seed();
await boot(f);
await advance(3 * HOUR);
const target = f.tasks[0]!.id;
const before = await attempts(f.companyId, target);
await state.db.transaction(async tx => {
const [run] = await tx.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "failed",
errorCode: "adapter_failed", finishedAt: new Date(), contextSnapshot: { issueId: target } }).returning();
await notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: run!.id });
});
await advance(2 * MINUTE);
expect(await attempts(f.companyId, target)).toBeGreaterThan(before);
}, 600_000);
it("leaves review work alone and re-checks a refused task with reads only while it waits", async () => {
const f = await seed();
const db = state.db;
const [paused] = await db.insert(agents).values({ companyId: f.companyId, name: "Paused reviewer", role: "engineer", status: "paused",
adapterType: "process", adapterConfig: {} }).returning();
await db.insert(issues).values(Array.from({ length: 60 }, (_, index) => ({ companyId: f.companyId, title: `Review ${index}`,
status: "in_review", assigneeAgentId: f.agentId, executionState: { status: "pending" } })));
const running = await db.insert(issues).values(Array.from({ length: 10 }, (_, index) => ({ companyId: f.companyId,
title: `Running ${index}`, status: "in_progress", assigneeAgentId: f.agentId }))).returning({ id: issues.id });
await db.insert(heartbeatRuns).values(running.map(task => ({ companyId: f.companyId, agentId: f.agentId, status: "running",
startedAt: new Date(), contextSnapshot: { issueId: task.id } })));
await db.insert(issues).values(Array.from({ length: 5 }, (_, index) => ({ companyId: f.companyId, title: `Paused ${index}`,
status: "todo", assigneeAgentId: paused!.id, createdAt: new Date(Date.now() - HOUR) })));
const report = await state.heartbeat.reportAssignedWorkInvariant(f.companyId);
expect(report).toMatchObject({ openTasks: 25, violations: { retry_offered: TASKS },
ok: { live_run: 10, "waiting:agent_paused": 5 } });
const target = f.tasks[0]!.id;
const claim = await countStatements(() => state.heartbeat.checkAssignedWorkInvariant({ companyId: f.companyId, issueId: target }));
const recheck = await countStatements(() => state.heartbeat.checkAssignedWorkInvariant({ companyId: f.companyId, issueId: target }));
expect(claim.some(query => /^insert into "activity_log"/i.test(query))).toBe(true);
expect(recheck.filter(query => /^(insert|update|delete)/i.test(query.trim()))).toEqual([]);
expect(recheck.length).toBeLessThan(claim.length);
}, 600_000);
});
@@ -177,7 +177,7 @@ export type AssignedWorkFacts = {
/** A saved-work recovery the recovery queue still owes this task: the
* task's saved files must be recovered before any run can start. */
savedWorkRecovery?: SavedWorkRecoveryFact | null;
agent: { status: string; invokable: boolean } | null;
agent: { status: string; invokable: boolean; updatedAt?: Date } | null;
companyPaused: boolean;
pauseHold: boolean;
budgetStop: boolean;
@@ -385,6 +385,8 @@ export const CONTINUATION_REVIEW_MESSAGE =
"Returned to review automatically after an interruption. Nothing needed you, so the review picks up where it stopped.";
export const CONTINUATION_REFUSED_MESSAGE =
"Paperclip tried to continue this task automatically, but it could not start yet. It will try again shortly.";
export const CONTINUATION_REFUSED_AGAIN_MESSAGE =
"Paperclip still could not start this task. It will keep trying, less often each time, and right away when something on the task changes.";
/** Plain-language activity text for an admitted continuation. No codes, no ids. */
export function continuationActivityMessage(reason: ViolationReason): string {
@@ -1,3 +1,4 @@
import { createHash } from "node:crypto";
import { and, desc, eq, gte, inArray, isNull, or, sql, type SQL, type SQLWrapper } from "drizzle-orm";
import {
activityLog,
@@ -47,6 +48,8 @@ import {
SAVED_WORK_RECOVERY_START_BOUND_MS,
savedWorkRecoveryDueSince,
CONTINUATION_REFUSED_MESSAGE,
CONTINUATION_REFUSED_AGAIN_MESSAGE,
CONTINUATION_BASE_BACKOFF_MS,
CONTINUATION_REVIEW_MESSAGE,
CONTINUATION_CAP_WINDOW_MS,
evaluateAssignedWork,
@@ -69,6 +72,8 @@ export const STRANDED_TRANSIENT_CONTINUED_OUTCOME = "stranded_transient_continue
export const ASSIGNED_WORK_REPAIRS_PER_MINUTE = 10;
/** Upper bound on tasks one report reads. */
export const ASSIGNED_WORK_REPORT_LIMIT = 2_000;
export const UNCHANGED_REFUSAL_MAX_WAIT_MS = 6 * 60 * 60_000;
const UNCHANGED_REFUSAL_CHAIN = 9;
const OPEN_STATUSES = ["todo", "in_progress", "blocked"] as const;
const PENDING_WAKE_STATUSES = ["queued", "deferred_issue_execution", "claimed"];
@@ -108,7 +113,7 @@ export type AssignedWorkCheck = {
verdict: AssignedWorkVerdict;
outcome: "ok" | "settling" | "repaired" | "settled_lost_run" | "deferred_rate_limit" | "deferred_cap" | "deferred_reattach_phase" | "stale_reported" | "left_to_hold_owner" | "left_to_native_recovery" | "changed" | "enqueue_refused"
| "waiting_on_recovery" | "started_saved_work_recovery" | "saved_work_recovery_busy"
| "replaced_stalled_run" | "left_person_work" | "left_stalled_replacement";
| "replaced_stalled_run" | "left_person_work" | "left_stalled_replacement" | "waiting_unchanged";
retryAt: Date | null;
};
@@ -458,7 +463,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
resumeAt: PROVIDER_LIMIT_CAUSES.includes(action.cause) && interruption?.nextRetryAt ? new Date(interruption.nextRetryAt) : null,
continuedBySweep: strandedHoldContinuesAutomatically(action,
strandedRuns.get(String(action.evidence?.latestRunId ?? "")) ?? null, issue.assigneeAgentId, at) } : null,
agent: agent ? { status: agent.status, invokable: agent.invokable } : null,
agent: agent ? { status: agent.status, invokable: agent.invokable, updatedAt: agent.updatedAt } : null,
companyPaused: company?.status === "paused",
pauseHold,
budgetStop: companyBudget || agent?.pauseReason === "budget" || agent?.pauseReason === "budget_exceeded",
@@ -582,14 +587,15 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
}
/** Evaluate one task, and repair it when it violates the invariant. */
async function checkIssue(scope: { companyId: string; issueId: string; onDeferredDeadline?: (at: number) => void }): Promise<AssignedWorkCheck | null> {
async function checkIssue(scope: { companyId: string; issueId: string;
onDeferredDeadline?: (at: number, detail?: { unchanged: boolean }) => void }): Promise<AssignedWorkCheck | null> {
const [issue] = await loadIssues(scope.companyId, scope.issueId, 1);
if (!issue) return null;
const facts = (await loadFacts(scope.companyId, [issue])).get(issue.id)!;
const at = now();
const verdict = evaluateAssignedWork(facts, at);
const check = await act(issue, facts, verdict, at);
if (check.retryAt) scope.onDeferredDeadline?.(check.retryAt.getTime());
if (check.retryAt) scope.onDeferredDeadline?.(check.retryAt.getTime(), { unchanged: check.outcome === "waiting_unchanged" });
passLog.record(check);
return check;
}
@@ -639,6 +645,14 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
retryAt: new Date(at.getTime() + CONTINUATION_CAP_WINDOW_MS) };
}
}
const key = stateKey(facts, verdict);
const refusals = await refusalsInState(issue, key, at);
if (refusals.count && refusals.lastAt !== null) {
const waitUntil = refusals.lastAt + Math.min(UNCHANGED_REFUSAL_MAX_WAIT_MS, CONTINUATION_BASE_BACKOFF_MS * 2 ** (refusals.count - 1));
if (waitUntil > at.getTime()) return { verdict, outcome: "waiting_unchanged", retryAt: new Date(waitUntil) };
}
const refusal = { stateKey: key, refusedAt: at.toISOString(),
...(refusals.count ? { message: CONTINUATION_REFUSED_AGAIN_MESSAGE } : {}) };
// A small process-wide budget keeps a post-deploy backlog from starting
// hundreds of runs at once. Deferred tasks come back on their deadline.
const budgetDeferral = repairBudgetDeferral(at);
@@ -650,7 +664,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
if (claimed.kind !== "claimed") return { verdict, outcome: claimed.kind === "capped" ? "deferred_cap" : claimed.kind === "changed" ? "changed" : "deferred_rate_limit",
retryAt: claimed.kind === "changed" ? null : claimed.retryAt };
repairTimes.push(at.getTime());
if (claimed.reason === "queued_run_stalled") return replaceStalledRun(issue, claimed, claimed.verdict.runId!);
if (claimed.reason === "queued_run_stalled") return replaceStalledRun(issue, claimed, claimed.verdict.runId!, refusal);
// A block that interrupted a review goes back to review; the review's own
// recovery wakes its participant. No agent work starts from here.
if (claimed.restoredTo === "in_review") {
@@ -694,10 +708,31 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
if (!run) await release();
if (!run && claimed.restoredTo) await revertRestoredBlock(issue, claimed);
await settleContinuationActivity(claimed, claimed.reason, run ? "queued" : "not_started",
fresh?.kind === "person_owned" ? { waitingFor: fresh.reason } : {});
{ ...(fresh?.kind === "person_owned" ? { waitingFor: fresh.reason } : {}), ...(run ? {} : refusal) });
return { verdict: claimed.verdict, outcome: run ? "repaired" : "enqueue_refused", retryAt: claimed.nextCheckAt };
}
function stateKey(facts: AssignedWorkFacts, verdict: Extract<AssignedWorkVerdict, { kind: "violation" }>) {
const { updatedAt: _updatedAt, ...issue } = facts.issue;
return createHash("sha256").update(JSON.stringify({ reason: verdict.reason, ...facts, issue })).digest("hex").slice(0, 32);
}
async function refusalsInState(issue: IssueRow, key: string, at: Date) {
const rows = await db.select({ details: activityLog.details }).from(activityLog).where(and(
eq(activityLog.companyId, issue.companyId), eq(activityLog.entityType, "issue"), eq(activityLog.entityId, issue.id),
gte(activityLog.createdAt, new Date(at.getTime() - UNCHANGED_REFUSAL_CHAIN * UNCHANGED_REFUSAL_MAX_WAIT_MS)),
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION))).orderBy(desc(activityLog.createdAt)).limit(UNCHANGED_REFUSAL_CHAIN);
let count = 0, lastAt: number | null = null;
for (const row of rows) {
const details = json(row.details);
if (details.outcome !== "not_started" || details.stateKey !== key) break;
count += 1;
const refusedAt = Date.parse(String(details.refusedAt));
if (lastAt === null && Number.isFinite(refusedAt)) lastAt = refusedAt;
}
return { count, lastAt };
}
function repairBudgetDeferral(at: Date): Date | null {
const minuteAgo = at.getTime() - 60_000;
while (repairTimes.length && repairTimes[0] <= minuteAgo) repairTimes.shift();
@@ -715,7 +750,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
"queued run its agent keeps passing over was left in place");
}
async function replaceStalledRun(issue: IssueRow, claimed: Claimed, runId: string): Promise<AssignedWorkCheck> {
async function replaceStalledRun(issue: IssueRow, claimed: Claimed, runId: string, refusal: Record<string, string>): Promise<AssignedWorkCheck> {
const slot = await reserveAutomaticRunCreation(db, issue.companyId, { issueId: issue.id, sourceRunId: runId });
if (slot.heldUntil) {
await settleContinuationActivity(claimed, claimed.reason, "not_started",
@@ -743,7 +778,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
return null;
});
if (!run) await slot.release();
await settleContinuationActivity(claimed, claimed.reason, run ? "queued" : "not_started");
await settleContinuationActivity(claimed, claimed.reason, run ? "queued" : "not_started", run ? {} : refusal);
return { verdict: claimed.verdict, outcome: run ? "replaced_stalled_run" : "enqueue_refused", retryAt: claimed.nextCheckAt };
}
+10 -7
View File
@@ -10,7 +10,7 @@ import { workSignalDebug } from "@paperclipai/db/work-signal-debug";
import { recoveryActionSweepCondition } from "./blocker-stall-wakeups.js";
type Entry = { id: string; companyId: string; dueAt: number; fromDeadline?: true };
type Scope = { companyId: string; issueId: string; onDeferredDeadline: (at: number) => void };
type Scope = { companyId: string; issueId: string; onDeferredDeadline: (at: number, detail?: { unchanged: boolean }) => void };
const PAGE_SIZE = 100;
export const resolvedDependencyCandidate = sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId}
@@ -126,7 +126,7 @@ export function registerIssueRecoveryWork(
* company dirty; checks schedule their own grace and back-off deadlines.
* Newest-updated tasks are checked first, a small batch per pass. */
function registerAssignedWork(sweep: NonNullable<typeof owner.assignedWork>) {
type Suspect = Entry & { updatedAt: number };
type Suspect = Entry & { updatedAt: number; unchanged?: true; fromBackstop?: true };
const cursors = new Map<string, string>();
const passes = fullPasses();
let enabled = false;
@@ -139,7 +139,8 @@ export function registerIssueRecoveryWork(
decodeScope: decodeIssueRecoveryHint,
matches: issueRecoveryScopeMatches,
priority: entry => entry.updatedAt,
mergeEntry: (pending, loaded) => ({ ...loaded, dueAt: Math.max(pending.dueAt, loaded.dueAt) }),
mergeEntry: (pending, loaded) => pending.unchanged && !loaded.fromBackstop ? loaded
: { ...loaded, dueAt: Math.max(pending.dueAt, loaded.dueAt), ...(pending.unchanged ? { unchanged: true as const } : {}) },
load: async scope => {
const key = scopeKey(scope);
if (!enabled) { cursors.delete(key); return []; }
@@ -149,14 +150,16 @@ export function registerIssueRecoveryWork(
const complete = rows.length < PAGE_SIZE;
if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id);
passes.loaded(scope, complete);
return { rows: rows.map(row => ({ id: row.id, companyId: row.companyId, updatedAt: row.updatedAt.getTime(), dueAt: Date.now() })), complete };
return { rows: rows.map(row => ({ id: row.id, companyId: row.companyId, updatedAt: row.updatedAt.getTime(), dueAt: Date.now(),
...(scope?.backstop ? { fromBackstop: true as const } : {}) })), complete };
},
check: async entry => {
if (!enabled) return null;
let next: number | null = null;
let next: number | null = null, unchanged = false;
await sweep.checkIssue({ companyId: entry.companyId, issueId: entry.id,
onDeferredDeadline: at => { next = next === null ? at : Math.min(next, at); } });
return next === null ? null : { ...entry, dueAt: next };
onDeferredDeadline: (at, detail) => { next = next === null ? at : Math.min(next, at); unchanged ||= detail?.unchanged === true; } });
const { unchanged: _previous, fromBackstop: _source, ...rest } = entry;
return next === null ? null : { ...rest, dueAt: next, ...(unchanged ? { unchanged: true as const } : {}) };
},
}, canRun);
return worker;